Sex Effects in Performance--Related Judgments

This study examined the relationships among objective performance, performance ratings, and pay allocations made by male and female managers. Because rater sex has been found to affect performance ratings and pay allocations, but little research has explored the causes of these sex differences, this study explored the effects of contextual factors that may moderate differences between the performance ratings and pay allocations of male and female raters. Two hundred twenty-nine managers assigned performance ratings and merit pay allocations to performance profiles that varied in levels of objective performance. It was found that: (a) sex effects on performance ratings were moderated by contextual variables, and b) sex and objective performance did not influence merit pay allocations independent of their separate effects on performance ratings.     

RATER SOURCE EFFECTS ARE ALIVE AND WELL AFTER ALL

Recent research has questioned the importance of rater perspective effects on multisource performance ratings (MSPRs). Although making a valuable contribution, we hypothesize that this research has obscured evidence for systematic rater source effects as a result of misspecified models of the structure of multisource performance ratings and inappropriate analytic methods. Accordingly, this study provides a reexamination of the impact of rater source on multisource performance ratings by presenting a set of confirmatory factor analyses of two large samples of multisource performance rating data in which source effects are modeled in the form of second-order factors. Hierarchical confirmatory factor analysis of both samples revealed that the structure of multisource performance ratings can be characterized by general performance, dimensional performance, idiosyncratic rater, and source factors, and that source factors explain (much) more variance in multisource performance ratings whereas general performance explains (much) less variance than was previously believed. These results reinforce the value of collecting performance data from raters occupying different organizational levels and have important implications for research and practice.     

Differences in Inter‐Rater Reliability and Accuracy for a Treatment Adherence Scale

Inter‐rater reliability and accuracy are measures of rater performance. Inter‐rater reliability is frequently used as a substitute for accuracy despite conceptual differences and literature suggesting important differences between them. The aims of this study were to compare inter‐rater reliability and accuracy among a group of raters, using a treatment adherence scale, and to assess for factors affecting the reliability of these ratings. Paired undergraduate raters assessed therapist behavior by viewing videotapes of 4 therapists' cognitive behavioral therapy sessions. Ratings were compared with expert‐generated criterion ratings and between raters using intraclass correlation (2,1). Inter‐rater reliability was marginally higher than accuracy (p = 0.09). The specific therapist significantly affected inter‐rater reliability and accuracy. The frequency and intensity of the therapists' ratable behaviors of criterion ratings correlated only with rater accuracy. Consensus ratings were more accurate than individual ratings, but composite ratings were not more accurate than consensus ratings. In conclusion, accuracy cannot be assumed to exceed inter‐rater reliability or vice versa, and both are influenced by multiple factors. In this study, the subject of the ratings (i.e. the therapist and the intensity and frequency of rated behaviors) was shown to influence inter‐rater reliability and accuracy. The additional resources needed for a composite rating, a rating based on the average score of paired raters, may be justified by improved accuracy over individual ratings. The additional time required to arrive at a consensus rating, a rating generated following discussion between 2 raters, may not be warranted. Further research is needed to determine whether these findings hold true with other raters and treatment adherence scales.     

A Social Relations Analysis of Team Performance Ratings

The current study used a Social Relations Model to analyze self and peer ratings to explore the dynamics of team member perceptions and performance ratings. The results from 29 organizational teams who completed performance ratings of themselves and team members indicated that the most rating variance was attributed to the relationship component, followed by the ratee component, followed by the rater component. Among other findings, the results indicated that self‐ratings were related to how one rates, and is rated by, others; that there were high levels of reciprocity between peers for dimensions that were interpersonal in nature; and that raters tended to evaluate others within, but not necessarily across, dimensions similarly.     

Competencies,Personality Traits,and Organizational Rewards of Middle Managers: A Motive-Based Approach

Previous literature suggests that performance ratings are saturated with rater-related idiosyncratic variance. Given that modern psychometric theories relegate this source of variance to measurement error, it has not been the subject of much previous research. Of importance, identifying and estimating the variance components underlying idiosyncratic rater variance will inform our understanding of the nature of this variance. In a sample of managerial performance ratings we report on components of variance and find that the idiosyncratic rater variance component is about one third rater main effects variance, one third Rater × Ratee interaction effects variance, and one third upper-bound Rater × Ratee × Dimension interaction effects variance. Further, results indicate that variance components are moderated by the acquaintanceship time between the rater and the ratee.     

Liking, Arousal Potential, and the Emotions Expressed by Music

Previous research indicates that the specific emotions expressed by stimuli may be closely associated with their pleasing and arousing qualities, and this parallels psychomusicological research on the relationship between these two stimulus qualities. In light of this, the present research contends that the emotions expressed by musical stimuli are associated with their pleasing and arousing qualities. Sixty subjects rated 32 musical excerpts on 11-point scales representing the expression of eight specific emotions. Statistical analyses showed that these emotion ratings were predictable on the basis of 60 additional subjects' ratings of each excerpt in terms of 'liking' and 'arousal potential'. This indicates that ratings of liking and arousal potential are essentially similar to ratings of the specific emotions expressed by musical stimuli. These results are discussed in terms of the relationship between liking and arousal potential, and the implications that this may have for research on affective processes.     

Race of ratee and anonymity of rater: a study comparing students with practitioners as performance appraisers

123 students and 123 nonstudent supervisors viewed videotapes which displayed four supposed subordinate supervisors, two African Americans and two Caucasians, who individually described their respective performances during the past year. After being told either that the supposed subordinates would or that they would not have access to the performance rating, the subjects rated the performance of those subordinate supervisors. While anonymity of rater and race of rater had no evaluative effect on the performance ratings given by the nonstudent subjects, the student subjects gave higher ratings when they believed that their ratings would be made public. Also, the nonstudent subjects' ratings differed as a function of whether they worked closely with others of another race and as a function of the frequency with which they actually discussed performance evaluations with their own subordinates.     

What is acceptable for women may not be for men: The effect of family conflicts with work on job‐performance ratings

We conducted a laboratory study examining the effect of a family conflict with work on performance appraisal ratings given to men and women. Overall, the experience of a family conflict was associated with lower performance ratings, and ratee sex moderated this relationship. Men who experienced a family conflict received lower overall performance ratings and lower reward recommendations than men who did not, whereas ratings of women were unaffected by the experience of a family conflict. The sex bias was not evident when performance was evaluated on the more specific dimension of planning. Neither rater gender nor work‐family role attitudes moderated the sex bias. Implications for future research and practice are discussed.     

ORGANIZATION AND RATER DIFFERENCES IN PERFORMANCE APPRAISALS

This study examines the effects of organizational differences and rater differences on performance appraisals. Self, peer, and supervisory ratings of performance for nurses in four hospitals and self, student, peer, and supervisory ratings for resident advisors in seven university dormitory complexes were used in this study. The analyses indicate that both organization and rater differences have significant, independent effects on performance ratings. The findings suggest that organizational differences may restrict the generality of the findings of performance appraisal studies across organizational settings. They also may have a negative impact on the usefulness of any particular performance appraisal form in different settings, and on the ability of managers to accurately interpret and compare performance ratings for individuals in different organizational subunits.     

A FIELD STUDY OF THE EFFECTS OF RATING PURPOSE ON THE QUALITY OF MULTISOURCE RATINGS

Using a field sample of peers and subordinates, the current study employed generalizability theory to estimate sources of systematic variability associated with both developmental and administrative ratings (variance due to items, raters, etc.) and then used these values to estimate the dependability (i.e., reliability) of the performance ratings under various conditions. Results indicated that the combined rater and rater-by-ratee interaction effect and the residual effect were substantially larger than the person effect (i.e., object of measurement) for both rater sources across both purpose conditions. For subordinates, the person effect accounted for a significantly greater percentage of total variance in developmental ratings than in administrative ratings; however, no differences were observed for peer ratings as a function of rating purpose. These results suggest that subordinate ratings are of significantly better quality when made for developmental than for administrative purposes, but the same is not true for peer ratings.     

Moderating Effects of Rater Personality on the Relation Between Candidate Self‐Monitoring and Selection Interview Ratings

The present study examined the moderating effect of rater personality – extroversion and sensitivity to others – on the relations between selection interview ratings and measures of candidate self‐monitoring (SM) and social anxiety (SA). In a real‐life military selection procedure setting in which 445 candidates and 93 raters participated, rater extroversion moderated the relation between candidate SM and selection interview ratings so that this relation was negative for raters low on extroversion and positive for raters high on extroversion. Rater extroversion was also found to moderate the negative relation between candidate SA and selection interview ratings. No support was found for the moderating effect of rater sensitivity to others. An explanation of the moderating effect of rater extroversion based on the assumption that extroversion is negatively related to critical interpersonal sensitivity was suggested.     

The impact of rater agreeableness and rating context on the evaluation of poor performance

We tested the effects of rater agreeableness on the rating of others’ poor performance in performance appraisal (PA). We also examined the interactions between rater agreeableness and two aspects of the rating context: ratee self‐ratings and the prospect of future collaboration with the ratee. Participants (n= 230) were allocated to one of six experimental groups (a 3 × 2 between‐groups design) or a control group (n= 20). Participants received accurate, low‐deviated, or high‐deviated self‐ratings from the ratee. Half were notified they would collaborate with the ratee in a future task. High rater agreeableness, positive deviations in self‐rating, and the prospect of future collaboration were all independent predictors of higher PA ratings. The interactions between rater agreeableness and rating context were very small. We argue that conflict avoidance is an important motivation in the PA process.     

The mere exposure effect and recognition memory

This investigation explores the relationship between liking ratings and recognition performance for obscure classical and Russian music melodies. Past studies have explored if awareness of stimulus presentation affects the mere exposure effect (MEE) (Bornstein, 1989; Bornstein & D'Agostino, 1992). We investigate if the type of awareness (i.e., "remembering" an actual occurrence of hearing the melody vs. "knowing" that one is familiar with the melody in a more general, less contextualised sense; Tulving, 1985) affects the MEE. In Experiment 1, we administered the liking test and the recognition test within the same test block and found that liking ratings for "remembered" melodies were on average higher than those for melodies that were merely "known". Although the recognition data replicate the findings of Gardiner, Kaminska, Dixon, and Java (1996) whereby remember and know responses to a given stimulus react differently to repetition from one to three trials, liking data did not vary with the degree of exposure. In addition, our adoption of the recognition category "guess" resulted in a pattern of results that is not only different from previous studies but illustrates the importance of judged, instead of true, old/new status in determining liking. The basic findings of Experiment 1 were replicated in Experiment 2 in which the liking test was conducted prior to the recognition test. Implications of these findings for theories of MEE are discussed.     

Impact and Causes of Rater Severity/Leniency in Appraisals without Postevaluation Communication Between Raters and Ratees

In performance appraisals, some assessors are substantially more lenient than others. Research on this effect in appraisals involving communication and interaction between raters and ratees after the performance evaluation has taken place indicates that it may be at least partly caused by individual differences in assessor personality. However, little is known about the impact or causes of rater severity versus leniency in situations in which there is little or no contact between raters and ratees after the performance evaluation. In Study 1 (N = 174) the strength of the severity–leniency effect in this ‘no‐contact’ context is estimated and found to be similar to that reported for ‘with‐contact’ appraisals. No evidence of an association between assessor personality and assessor severity (vs. leniency) is found in the ‘no‐contact’ context. In Study 2 (N = 54) there is no evidence of an association between the fluid cognitive ability of assessors and the severity of their ratings in a no‐contact context. It is concluded that the severity versus leniency effect probably has a considerable impact on performance ratings in ‘no‐contact’ appraisal settings, but that neither rater personality nor rater cognitive ability appear to play a significant role in this.     

RATER-RATEE RACE EFFECTS IN DEVELOPMENTAL PERFORMANCE RATINGS OF MANAGERS

The effects of rater and ratee race on performance ratings of managers were examined. Ratings were obtained from peers, subordinates and bosses as part of a multirater, developmental feedback program for managers. Two data sets were created for purposes of this study. The between-subjects data set consisted of ratings from over 20,000 bosses, over 50,000 peers, and over 40,000 subordinates. The repeated measures data set was substantially smaller because it included only those Black and White managers who were rated by both a Black and White rater from each of the three perspectives. Results for rater race indicated that Black raters from all perspectives (peers, subordinates, and bosses) assigned more favorable ratings to ratees of their own race. Results for White raters differed according to the particular rating source. White bosses assigned more favorable ratings to ratees of their own race, but White subordinates did not. White peers assigned more favorable ratings to Whites in the repeated measures analysis, but not in the between-subjects analysis. Results for ratee race indicated that both White and Black managers received higher ratings from Black raters than from White raters, and the effect was more pronounced for ratings assigned to Black managers.     

Physical Attractiveness Bias in Employee Termination

We examined the impact of attractiveness on employment termination decisions. After viewing a file that contained a poor performance review and a badge with a photograph of an extremely attractive, moderately attractive, or unattractive employee, 178 participants were asked if they would terminate the employee, rated how much they liked the employee, and made a judgment of attribution of her poor performance. Participants were willing to terminate the unattractive woman more frequently than the moderate or extremely attractive women. The participants also liked the unattractive woman less than the others. When asked to rate causes of poor performance, no differences were found based directly on attractiveness, but attributions mediated the relationship between liking ratings and termination decisions.     

Leniency effect as a function of rating format,purpose for appraisal,and rater individual differences

The effect of rating format and non-performance variables on rating leniency were studied in two law enforcement organizations. One of these variables, trust in the appraisal process, was defined as the extent to which a rater believes that fair and accurate appraisal will be made in the organization. A measure of trust in appraisal accounted for a significant proportion of variance in performance ratings. The purpose of appraisal (i.e., feedback or promotion) also accounted for rating variance. A mixed-standard rating format showed less susceptibility to the non-performance variables on the extent of leniency. Discussion centers on the usefulness of rater and organizational variables in performance appraisal research.     

Effects of a favor and liking on compliance

A laboratory experiment was conducted to examine the effects of a favor and of liking on compliance with a request for assistance from a confederate. Liking for the confederate was manipulated, and male subjects then received a soft drink from the confederate, from the experimenter, or received no favor. Compliance with the confederate's request to purchase some raffle tickets was measured, as was liking for the confederate. The results showed that the favor increased liking for the confederate and compliance with his request, but the effect of manipulated liking was weak. Detailed ratings of the confederate as well as correlational data suggested that the relationship between favors and compliance is mediated, not by liking for the favor-doer, but by normative pressure to reciprocate.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.