Bias in Performance Ratings: Clarifying the Role of Positive Versus Negative Escalation

This study replicated and refined research on the effects of escalation of commitment in performance ratings. We utilized experimental manipulations in a laboratory setting to determine whether positive escalation or negative escalation (or both) could be responsible for the effect. In one session, participants (N = 210) were assigned to the perspective of the potential employee's supervisor and chose 1 of 2 candidates for a sales position. In a second session, participants rated the performance of (a) the individual they selected; (b) the individual they rejected; or (c) a third individual, whose preselection information they never viewed. Results replicated previous findings, such that ratings were biased upward when participants rated the performance of the salesperson they had originally selected. Results were not biased downward when individuals rated the performance of the salesperson they had rejected, however. Thus, our results suggest that positive escalation, and not negative escalation, was the cause of the bias.     

Managing Salesperson Motivation in a Territory Realignment

Although typically undertaken to improve aggregate sales force performance, the realignment of sales territories can be quite de-motivational to individual salespersons. Findings of two studies reported here demonstrate that salesperson motivation and performance can be positively influenced by the managerial actions taken during the implementation phase of a territory realignment. Actions based on expectancy theory were found to be more effective for salespeople whose territory realignment increased sales potential, and justice-based actions were more effective for salespersons experiencing a reduction in territory potential.     

A self-fulfilling prophecy in a competitive psychomotor task

Fifty-six pairs of male college students competed in games of electronic table tennis. The participants were led to expect a bias in the game equipment, such that one player would have an advantage and the other a disadvantage. All participants recorded levels of aspiration both before and after being informed of the alleged bias. Upon completion of the game, the subjects made ratings of causal attribution to five factors: ability, task difficulty, effort, luck, and equipment bias. The results showed that the students with the putative disadvantage set significantly lower aspiration levels and performed at a significantly lower level than those students with the supposed advantage. Thus the putative bias became a real bias, although a psychological rather than a physical one. It is suggested that the source of the bias and the performance differences was a self-fulfilling prophecy. In contrast to the results of earlier studies, there were no meaningful differences in ratings of causal attribution.     

A Meta-Analytic Comparison of Managerial Ratings and Self-Evaluations

The purpose of this study was to examine the equivalence of self-reports and managerial ratings of salesperson job performance. A meta-analysis showed that the two measures exhibited low convergent validity (mean corrected correlation = 0.19), which indicated that they are not interchangeable. The predictive validity of self-reports and managerial ratings was compared. Managerial ratings had a corrected mean correlation of 0.44 with objective performance, whereas the corrected mean correlation between self-reports and objective measures was 0.34. Further meta-analytic investigations showed that the divergence between self-reports and managerial ratings was attributable to the performance effect. A discussion of the findings and avenues for further research are also provided.     

The Relation of Job Knowledge,Job Performance,and Supervisory Ratings as a Function of Ratee Race

Racial bias in performance ratings may be inferred when ratings hold differ- ent meanings for different racial subgroups. Operationally, this would be indi- cated by differences (by ratee race) in the correlation between performance ratings and objective indices of performance. In this study, the effects of rate race on the relations between supervisory ratings and more objective criteria of job knowledge and work performance were examined by aggregating corre- lations across 25 studies. The results indicated that supervisory ratings were more highly related to work-performance measures-and to a lesser extent to job-knowledge measures-for Black than for White ratees. Tivo theories were proposed that could account for such differences.     

Self-serving bias effects on job analysis ratings

The purpose of this study was to investigate whether worker-oriented job analysis importance ratings were influenced by subject matter experts' (SME) standing (as measured by self-rated performance) on a competency. This type of relationship (whereby SMEs indicate that the traits they have are important for successful job performance) is an example of the self-serving bias (which is widely described in the social cognition literature and rarely described in the industrial/organizational psychology literature). An archival dataset covering 57 clerical and technical occupations with 26,682 participants was used. Support was found for the relationship between self-rated performance and importance ratings. Significant relationships (typically in the .30s) were observed for all 31 competencies that were studied. Controls were taken to account for common method bias and differences in the competencies required for each of the 57 occupations. Past research has demonstrated the effects of the self-serving bias on personality-based job analysis ratings. This study was the first to extend these findings to traditional job analysis, which covers other competencies in addition to personality. In addition, this study is the first to use operational field data instead of laboratory data.     

The Influence of Indirect Knowledge of Previous Performance on Ratings of Present Performance: The Effects of Job Familiarity and Rater Training

In a 2 × 2 × 2 crossed factorial design, trained or untrained subjects viewed a videotape and evaluated performance on either a familiar (college lecturer) or unfamiliar (salesperson) job. Prior to viewing the videotape, some subjects reviewed positive information about the ratee's prior performance, whereas other subjects did not review any prior performance information. To determine whether assimilation or contrast effects occurred, we compared ratings provided by subjects who reviewed positive information about prior performance with ratings provided by subjects who did not review any prior performance information. A three-way interaction was obtained. Ratings of performance on the familiar job by untrained or trained subjects revealed only a small assimilation effect. However, when rating performance on the unfamiliar job, a large assimilation effect was observed among untrained subjects, whereas a large contrast effect was observed among trained subjects. The results indicate that rater error training may reverse, rather than reduce or eliminate, rating errors that arise from knowledge of a ratee's prior performance. Implications for further understanding and reducing assimilation and contrast effects are discussed.     

Selling and Sales Management in Action: Attitudes and Applications of Quotas by Sales Executives and Sales Managers

A survey of 186 sales managers and executives revealed that in setting sales quotas, factors such as the sales territory and the support provided by the manager were considered more important than the past performance of the sales representative. As would be expected, in an inspection of general opinions about sales quotas, the managers demonstrated a preference for linking the sales quota to the compensation of the salesperson.     

Stuck like glue: the formation and consequences of brand attachments among salespeople

Although managers and researchers put utmost importance on uncovering factors that affect effort and performance of sales representatives, they have largely overlooked a key factor. This research aims to remedy this neglect by shedding light on salesperson brand attachment. A qualitative study first examines the prevalence of attachment and uncovers several potential antecedents. Two follow-up empirical studies show that identification and psychological ownership are significant predictors of salesperson brand attachment. Informed by this research, managers can encourage brand attachment, with the end goal of increasing salesperson effort and ultimately performance, by increasing feelings of identification and psychological ownership among salespeople. This impact is particularly notable among brands with low market share, which are often a key concern of managers. The findings provide novel insights into how attachment affects sales representatives, as well as potential methods to cultivate such an attachment.     

A Meta-Analysis of the Relationship Between Rater Liking and Performance Ratings

This meta-analysis reviewed the magnitude and moderators of the relationship between rater liking and performance ratings. The results revealed substantial overlap between rater liking and performance ratings (ρ = .77). Although this relationship is often interpreted as indicative of bias, we review studies that indicate that to some extent the relationship between liking and performance ratings potentially reflects “true” differences in ratee performance. Moderator analyses indicated that the relationship between liking and performance ratings was weaker for ratings of organizational citizenship behaviors, ratings made by peer raters, ratings in nonsales jobs, and ratings made for development; however, the relationship was strong across moderator levels, underscoring the robustness of this relationship. Implications for the interpretation of performance ratings are discussed.     

CULTURAL RELATIVITY IN ACTION: A COMPARISON OF SELF-RATINGS MADE BY CHINESE AND U.S. WORKERS

Using a sample of 982 leader-subordinate dyads drawn from nine different organizations in the Republic of China (i.e., Taiwan), self-ratings of performance were compared with supervisory ratings of subordinate performance. Results indicated that Chinese employees rated their job performance less favorably than did their supervisors (i.e., they exhibited a modesty bias). This modesty bias occurred relatively uniformly across gender, various educational levels, and age groups. These results are contrary to the typically reported U.S. finding that self-ratings of performance are more lenient than are supervisory ratings. A further comparison of the means of supervisory and self-ratings between this study and previous U.S. research revealed that the modesty bias appeared to be produced by the lower self-ratings made by Chinese workers as compared to their U.S. counterparts. The findings suggest that culture plays a critical role in shaping workers' perceptions of their own work performance. Results of this study are discussed in terms of their implications for future research and practice in international human resource management.     

What's Context Got to Do with It? Comparative Difficulty of Test Questions Influences Metacognition and Corrected Scores for Formula‐scored Exams

Summary: On formula‐scored exams students receive points and penalties for correct and incorrect answers, respectively, but they can avoid the penalty by withholding incorrect answers. However, test‐takers have difficulty strategically regulating their accuracy and often set an overly conservative metacognitive response bias (e.g., Higham, 2007). The current experiments extended these findings by exploring whether the comparative difficulty of surrounding test questions (i.e., easy vs. hard)—a factor unrelated to the knowledge being tested—impacts metacognitive response bias for medium‐difficulty test questions. Comparative difficulty had no significant influence on participants' ability to choose correct answers for medium questions, but it did affect willingness to report answers and confidence ratings. This difference carried over to corrected scores (scores after penalties are applied) when comparative difficulty was manipulated within‐subjects: Scores were higher in the hard condition. Results are discussed in terms of implications for interpreting formula‐scored tests and underlying mechanisms of performance.Copyright © 2017 John Wiley & Sons, Ltd.     

Examining the reliability and predictive validity of performance assessments by soccer coaches and scouts: The influence of structured collection and mechanical combination of information

Soccer coaches and scouts typically assess in-game soccer performance to predict players’ future performance. However, there is hardly any research on the reliability and predictive validity of coaches’ and scouts’ performance assessments, or on strategies they can use to optimize their predictions. In the current study, we examined whether robust principles from psychological research on selection – namely structured information collection and mechanical combination of predictor information through a decision-rule – improve soccer coaches’ and scouts’ performance assessments. A total of n = 96 soccer coaches and scouts participated in an elaborate within-subjects experiment. Participants watched soccer players’ performance on video, rated their performance in both a structured and unstructured manner, and combined their ratings in a holistic and mechanical way. We examined the inter-rater reliability of the ratings and assessed the predictive validity by relating the ratings to players’ future market values. Contrary to our expectations, we did not find that ratings based on structured assessment paired with mechanical combination of the ratings showed higher inter-rater reliability and predictive validity. In contrast, unstructured-holistic ratings yielded the highest reliability and predictive validity, although differences were marginal. Overall, reliability was poor and predictive validities small-to-moderate, regardless of the approach used to rate players’ performance. The findings provide insights into the difficulty of predicting future performance in soccer.     

Understanding and Removing the Effects of Performance Cues on Behavioral Ratings

Previous research has demonstrated that performance information (e.g., prior supervisor ratings) can bias behavioral ratings. However, research has not fully explored the effects of performance cues on raters' memory. In addition, no studies have attempted to eliminate this performance cue effect. This study addressed these deficiencies by collecting both free recall and recognition memory measures while testing an unstructured free recall intervention. Results indicate that performance cues do affect the recall of performance relevant behaviors from memory. Contrary to expectations, free recall did not prove to be an effective intervention. Implications of these findings for future attempts to remove the performance cue effect are discussed.     

A Hierarchical Rater Model for Longitudinal Data

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM.     

Reducing the Effects of Gender Stereotypes on Performance Evaluations

The purpose of this research was to extend previous work on gender bias in performance evaluation. Specifically, we examined whether a structured free recall intervention could decrease the influence of traditional gender-stereotypes on the performance evaluations of women. Two hundred and forty-seven college students provided performance ratings for vignettes that described the performance of male or female college professors. Results indicated that without the intervention, raters who have traditional stereotypes evaluated women less accurately and more negatively. Conversely, the structured free recall intervention successfully eliminated these effects. The usefulness of the structured free recall intervention as a tool for decreasing the influence of gender stereotypes on performance ratings is discussed.     

What is acceptable for women may not be for men: The effect of family conflicts with work on job‐performance ratings

We conducted a laboratory study examining the effect of a family conflict with work on performance appraisal ratings given to men and women. Overall, the experience of a family conflict was associated with lower performance ratings, and ratee sex moderated this relationship. Men who experienced a family conflict received lower overall performance ratings and lower reward recommendations than men who did not, whereas ratings of women were unaffected by the experience of a family conflict. The sex bias was not evident when performance was evaluated on the more specific dimension of planning. Neither rater gender nor work‐family role attitudes moderated the sex bias. Implications for future research and practice are discussed.     

Sex Bias at Work: The Effects of Attentional and Memory Demands on Performance Ratings of Men and Women1

The present study examined the impact of attentional and memory demands on work performance ratings accorded men and women in traditionally male jobs. Of interest was whether sex discrimination would abate in the face of individuating and job-relevant work behavior even when the demands likely to be faced in actual work settings were taken into account. Two hundred and two subjects read a vignette depicting the work behavior of a male or female police officer and then rated the individual's work performance. The attentional demands imposed on subjects while reading the vignette and the amount of time elapsed prior to issuing the performance ratings were systematically varied. As predicted, men were evaluated more favorably than women when raters were faced with an additional task requiring attention and time pressures were made salient. Only when subjects were able to carefully allocate all of their attentional resources did sex bias in work performance ratings abate. Memory demands had no effects on work performance ratings. Gender-related work characterizations paralleled the performance ratings, providing support for the idea that sex stereotypes mediate discrimination in performance appraisal judgments. The theoretical and practical implications of these findings, as well as suggestions for future research, are discussed.     

Performance Trends and Salesperson Evaluations: The Moderating Roles of Evaluation Task,Managerial Risk Propensity,and Firm Strategic Orientation

Understanding managers’ evaluations of salesperson performance is important for both theoretical and pragmatic reasons. We examine several factors that moderate the influence of performance trends on salesperson evaluations provided by practicing sales managers. Preliminary data from a pilot study indicate that performance trends exert a stronger influence on salesperson evaluations that are prospective (e.g., making a hiring decision) rather than retrospective (e.g., providing an annual evaluation). Study 1’s findings reveal an additional contingency by demonstrating that these trend effects were particularly pronounced for risk-taking sales managers providing prospective evaluations. Finally, evidence from Study 2 documents the moderating influence of firm strategic orientation by showing that trend effects were most prominent for managers making forward-looking evaluations under the aegis of a prospector-based strategic orientation. The latter two studies also offer mediational evidence indicating that trends influence performance evaluations by affecting inferences about a salesperson’s future productivity.