RATER-RATEE RACE EFFECTS IN DEVELOPMENTAL PERFORMANCE RATINGS OF MANAGERS

The effects of rater and ratee race on performance ratings of managers were examined. Ratings were obtained from peers, subordinates and bosses as part of a multirater, developmental feedback program for managers. Two data sets were created for purposes of this study. The between-subjects data set consisted of ratings from over 20,000 bosses, over 50,000 peers, and over 40,000 subordinates. The repeated measures data set was substantially smaller because it included only those Black and White managers who were rated by both a Black and White rater from each of the three perspectives. Results for rater race indicated that Black raters from all perspectives (peers, subordinates, and bosses) assigned more favorable ratings to ratees of their own race. Results for White raters differed according to the particular rating source. White bosses assigned more favorable ratings to ratees of their own race, but White subordinates did not. White peers assigned more favorable ratings to Whites in the repeated measures analysis, but not in the between-subjects analysis. Results for ratee race indicated that both White and Black managers received higher ratings from Black raters than from White raters, and the effect was more pronounced for ratings assigned to Black managers.     

Frame-of-reference training and cognitive categorization: an empirical investigation of rater memory issues.

We considered the effects of frame-of-reference (FOR) training on raters' ability to correctly classify ratee performance as well as their ability to recognize previously observed behaviors. The purpose was to examine the cognitive changes associated with FOR training to better understand why such training generally improves rating accuracy. We trained college students (N = 94) using either FOR or control procedures, had them observe three managers on videotape, and rate the managers on three performance dimensions. Results supported our hypotheses that, compared with control training, FOR training led to better rating accuracy and better classification accuracy. Also consistent with predictions, FOR training resulted in lower decision criteria (i.e., higher bias) and lower behavioral accuracy on a recognition memory task involving impression-consistent behaviors. The implications of these results are discussed, particularly in terms of the ability of FOR-trained raters to provide accurate performance feedback to ratees.     

The Effects of Ratee Prototypicality on Rater Observation and Accuracy1

Social cognition theory asserts that perceivers (raters) assign stimulus persons (ratees) to social categories. These categories help the raters encode, store, and recall information. In a longitudinal design that represented a performance appraisal situation, this study examined the effects of information about a ratee's category membership on the amount of information that raters collected about the ratee prior to rating. One hundred fourteen subjects participated in three separate experimental sessions which spanned a 3-week time period. Among other tasks, subjects were required to rate a subordinate who was described in a manner which made it either difficult or easy to assign the subordinate to a social category. It was predicted and found that raters of ratees who were easily categoriezed spent less time observing the ratees' performance than raters of ratees who were less easily classified. Furthermore, results indicated that it was the effect of rater categorization on observation time that was critical to rating accuracy.     

Cultural Values and Performance Appraisal: Assessing the Effects of Rater Self-Construal on Performance Ratings

Much of the prior research investigating the influence of cultural values on performance ratings has focused either on conducting cross-national comparisons among raters or using cultural level individualism/collectivism scales to measure the effects of cultural values on performance ratings. Recent research has shown that there is considerable within country variation in cultural values, i.e. people in one country can be more individualistic or collectivistic in nature. Taking the latter perspective, the present study used Markus and Kitayama's (1991) conceptualization of independent and interdependent self-construals as measures of individual variations in cultural values to investigate within culture variations in performance ratings. Results suggest that rater self-construal has a significant influence on overall performance evaluations; specifically, raters with a highly interdependent self-construal tend to show a preference for interdependent ratees, whereas raters high on independent self-construal do not show a preference for specific type of ratees when making overall performance evaluations. Although rater self-construal significantly influenced overall performance evaluations, no such effects were observed for specific dimension ratings. Implications of these results for performance appraisal research and practice are discussed.     

A LONGITUDINAL EXAMINATION OF RATER AND RATEE EFFECTS IN PERFORMANCE RATINGS

The consistency and loci of leniency, halo, and range restriction effects in performance ratings were investigated in a longitudinal study. Ratings were provided by approximately 90 supervisors in a metropolitan police department, who rated approximately 350 police-rank subordinates on five occasions over a three and one-half year period. Rating effects were computed separately as rater-and ratee-based statistics, and intercorrelated among the five rating periods. The nature of the data set made it possible to hold either raters or ratees constant for each analysis, thus permitting inferences regarding the sources of reliable variance in effects as due to raters or ratees. It was concluded that reliable variance in mean ratings is partly attributable to ratees, but mainly introduced by raters. Reliable halo variance is attributable to raters, and range restriction is a product of stable group performance variability within intact ratee groups. Implications of these results for future rating process research are discussed.     

Effects of rater conscientiousness on evaluations of task and contextual performance of older and younger co-workers

The possibility for age discrimination and stereotypes to affect performance evaluations is rising. Although careful evaluations might be expected from conscientious raters, little is known about whether they might show more or less bias towards certain age groups. Therefore, in our study using a time-lagged design, we investigated the effects of rater conscientiousness on the performance evaluations of younger and older actual co-worker (N = 242). We found that raters who were more conscientious provided higher ratings for older workers than for younger workers on task performance and organizational citizenship behaviours. Specifically, we tested the model of mediated moderation, in which the relation between rater conscientiousness and ratee age predicts ratee-perceived conscientiousness, which in turn predicts performance ratings. The model was significant for older ratees, but not for younger ratees. We discuss our results in terms of the “similar to me” effects and implications for organizational practices.     

Impact and Causes of Rater Severity/Leniency in Appraisals without Postevaluation Communication Between Raters and Ratees

In performance appraisals, some assessors are substantially more lenient than others. Research on this effect in appraisals involving communication and interaction between raters and ratees after the performance evaluation has taken place indicates that it may be at least partly caused by individual differences in assessor personality. However, little is known about the impact or causes of rater severity versus leniency in situations in which there is little or no contact between raters and ratees after the performance evaluation. In Study 1 (N = 174) the strength of the severity–leniency effect in this ‘no‐contact’ context is estimated and found to be similar to that reported for ‘with‐contact’ appraisals. No evidence of an association between assessor personality and assessor severity (vs. leniency) is found in the ‘no‐contact’ context. In Study 2 (N = 54) there is no evidence of an association between the fluid cognitive ability of assessors and the severity of their ratings in a no‐contact context. It is concluded that the severity versus leniency effect probably has a considerable impact on performance ratings in ‘no‐contact’ appraisal settings, but that neither rater personality nor rater cognitive ability appear to play a significant role in this.     

A Cognitive Evaluation of Frame-of-Reference Rater Training: Content and Process Issues

A multivariate, cognitive evaluation model was used to examine the previously noted effects of frame-of-reference (FOR) training on rating accuracy. The tested model emphasized two content-related issues associated with FOR training: (a) the content of the training itself and the extent that raters agreed with it, which provided an index of the amount of overlap between the theory of performance taught in training and a rater's implicit theory of performance following training; and (b) the content of raters' performance impressions of ratees based on a comprehensive and integrative model of the cognitive representations of persons— Associated Systems Theory (Carlston, 1992, 1994). Undergraduates (N= 172) were trained with FOR or control procedures using two different performance theories, observed and rated videotaped manager performance on three performance dimensions, and engaged in written free-recall of target performance vignettes. A structural model was tested incorporating a latent differential rating accuracy construct. FOR training was associated with higher demonstrated levels of agreement and with changes in ratee representation that were more abstract and more target-referent (i.e., less idiosyncratic) than control training. Agreement was found to be associated with enhanced rating accuracy and more self-referent impressions; however, target-referent impressions were shown to be related to better rating accuracy. Discussion focuses on the interrelated nature of content and process issues in performance appraisal. Using the AST framework, it is also proposed that longstanding manager/subordinate dyads could be expected to benefit the most from FOR training.     

The role of the supervisor in career and organizational commitment

Supervisors are increasingly required to support employees in their learning and career development. This study investigated relationships between supervisor support activities and their subordinates' career and organizational commitment. Findings suggested that employees who saw their supervisors as giving them trust and the authority to do the job were more committed to their organization, as were those who perceived their supervisors to engage in feedback and goal-setting. This suggested the potential organizational benefits of performance feedback from managers and the importance of supervisors developing the confidence and skills to delegate effectively and have trust in their subordinates. No relationships were found between supervisor support and career identity, career resilience or career planning. The relationship between perceptions of overall supervisor support and organizational commitment was moderated by employee gender. The findings suggested a positive linear relationship between support and commitment for women, but the association between support and commitment was less straightforward in the male sample.     

Putting Raters in Ratees’ Shoes: Perspective Taking and Assessment of Creative Products

This study reported 2 experiments that studied the effect of perspective taking on assessment of creative products by using human raters. Forty responses of 2 alternative uses tasks (AUTs) and 15 alien stories generated by 6th-grade students were used as assessment materials. Undergraduate students as the novice raters assessed the products under 3 experimental conditions: assessing without any information of the ratees, assessing only with age information of the ratees, and assessing with age information and taking the perspective of the ratees. Results of Experiment 1 showed significant differences in creativity ratings between group 1 and 2. But no significant difference was found between group 2 and 3. In Experiment 2, raters in group 1 used objective perception and raters in group 3 were asked to take the perspective for more time. Raters in group 3 assigned higher ratings than the other 2 groups but no difference was found between group 1 and 2. Overall, the results showed the effect of perspective taking on assessment of creative products. Possible mechanisms of the effect and implications for creativity assessment and creativity teaching practice were discussed.     

Additional Evidence of Attitudinal Influences in Performance Appraisal

The attitudes of twenty-nine managers toward performance appraisal and toward their organization were used to predict differences in mean ratings and measures of discrimination among ratees and performance dimensions. Substantial multiple correlations were obtained for all three dependent variables (between 32 and 46% of the variance in each measure was explained), and several individual correlations were statistically significant. Raters who showed low levels of confidence in the appraisal system, high levels of discomfort, or high levels of instrumental commitment were more likely to provide ratings that were unusually high or that did not discriminate well among ratees and/or dimensions. Raters who showed higher levels of attitudinal commitment or who perceived more risks associated with distorting ratings tended to provide lower ratings and to discriminate among ratees and/or dimensions.     

Performance Appraisal Process Research in the 1980s: What Has It Contributed to Appraisals in Use?

In the early 1980s, Landy and Farr (1980) and Feldman (1981) redirected performance appraisal research from issues related to the development of psychometrically sound rating scales to those involving the cognitive processes of raters. Since that time, several reviews have attempted to translate principles from social cognition and cognitive psychology to the specific conditions of formal appraisal systems in work-oriented organizations. In addition, a number of empirical studies have been conducted on this topic. This article reviews empirical research during the 1980s that focused on performance appraisal processes, particularly the research that has focused upon rating accuracy. The review is structured around a three-stage process model of gathering, storing, and retrieving information about social stimuli for the purposes of rating performance. Factors affecting this process are clustered into four categories: appraisal settings, ratees, raters, and the nature of the scales used for the appraisal. Once reviewed, the research is evaluated in terms of its contributions to improving the quality of appraisal systems as they are used in organizations.     

AN EMPIRICAL INVESTIGATION OF PERCEPTIONS OF RATEE BEHAVIOR FREQUENCY AND RATEE BEHAVIOR CHANGE USING BEHAVIORAL EXPECTATION SCALES (BES)

Behavioral Expectation Scales (BES) research has concentrated upon psychometric properties, while evidence for potential operational advantages has been largely anecdotal. This empirical study explored the operational value of BES as a technique for identifying divergent rater-ratee perceptions of ratee behavior frequency and for facilitating subsequent ratee behavior change. Results indicated that ratees perceived desired behaviors as occurring more often and undesired behaviors occurring less often than did raters. Behavior change was demonstrated both after development and after implementation of BES. Such operational advantages of BES are argued as a crucial factor in the choice of an organization's performance appraisal format.     

The Effect of Feedback Packaging on Ratee Reactions1

In performance‐feedback situations, reactions to the rater have been examined rarely. A clearer understanding of what causes negative reactions toward raters might be used by them to better control feedback outcomes without having to distort the feedback message. In Study I, ratee reactions to the packaging of feedback messages were examined in a laboratory experiment. A legitimizing statement included in the feedback message resulted in more positive reactions to the rater than when no such statement was presented. In addition, the use of less personal feedback language resulted in more positive reactions to the rater than when more personal language was used. Neither legitimization statement nor type of language significantly impacted reactions to the feedback message or perceptions of performance, indicating that they did not distort the feedback message. In Study 2, a laboratory observation, the use of more personal language by the rater was related negatively to ratee confidence in rater judgment and rater likability. More research on feedback packaging, with the goal of training raters in how best to convey the feedback message, is needed.     

THE EFFECTS OF INTERRATER AND SELF-OTHER AGREEMENT ON PERFORMANCE IMPROVEMENT FOLLOWING UPWARD FEEDBACK

This study investigated (a) whether performance improvement following upward feedback is related to self-other rating discrepancies, (b) how self-ratings change after feedback, and (c) whether agreement among raters influences performance improvement. Self-ratings and subordinate ratings were collected from 1,888 managers at 2 points in time 1 year apart. Using polynomial regression equations, we determined that managers who overrated themselves relative to how others rated them tended to improve their performance from 1 year to the next, and underraters tended to decline. This is consistent with what would be predicted by self-consistency theory (Korman, 1976). Self-ratings tended to decrease for overraters and increase for underraters, but this effect was not constant throughout the range of self-ratings. Agreement among subordinate raters was negatively related to performance improvement, but this is likely a reflection of differential regression to the mean.     

Assessment of Differential Rater Functioning in Latent Classes with New Mixture Facets Models

Multifaceted data are very common in the human sciences. For example, test takers' responses to essay items are marked by raters. If multifaceted data are analyzed with standard facets models, it is assumed there is no interaction between facets. In reality, an interaction between facets can occur, referred to as differential facet functioning. A special case of differential facet functioning is the interaction between ratees and raters, referred to as differential rater functioning (DRF). In existing DRF studies, the group membership of ratees is known, such as gender or ethnicity. However, DRF may occur when the group membership is unknown (latent) and thus has to be estimated from data. To solve this problem, in this study, we developed a new mixture facets model to assess DRF when the group membership is latent and we provided two empirical examples to demonstrate its applications. A series of simulations were also conducted to evaluate the performance of the new model in the DRF assessment in the Bayesian framework. Results supported the use of the mixture facets model because all parameters were recovered fairly well, and the more data there were, the better the parameter recovery.     

The role of appraisal purpose: Effects of purpose on information acquisition and utilization

Past research on the role of appraisal purpose in the appraisal decision-making process has concentrated on the motivational role of purpose. Research has found that raters are less willing to give poor ratings when appraisals are to be used for some purposes rather than others. The present paper describes two experiments which explore how appraisal purpose might affect rater cognitive activities as well. The first experiment investigated how appraisal purpose and outcomes affect how raters differentially utilize information to make appraisal decisions. Few differences were found. The second experiment investigated how raters differentially search for performance information to make appraisal decisions for different purposes and outcomes. Raters were found to search for more comparative information when they had to select one of several ratees for some treatment. The results also indicated a discrepancy between how information is collected and how it is used. Implications for defining the role of purpose in the appraisal process, as well as for recent process approaches to performance appraisal, are discussed.     

多源评价的特点与内在机制

多源评价在国外企业中的运用已日益成熟, 但在我国还停留在探索与发展阶段。基于已有的研究发现, 围绕评价过程、评价源及被评价者三方面对多源评价的特点及内在机制进行了探讨与分析。从评价过程看, 其评价目的具有多重性, 评价形式注重匿名性, 且评价结果的合理应用非常重要; 从评价源看, 不同评价源间的评价一致性较低, 且易造成晕轮效应和宽大效应; 从被评价者来看, 个体对多源评价结果的反应, 受到个性特征、反馈信号及自我-他人评价间差距等因素影响。研究也发现, 多源评价所带来的绩效改进结果具有不稳定性。基于此, 如何提高多源评价过程的有效性与准确性, 改善评价者对评价结果的反应, 以及如何对多源评价结果进行有效汇总等是未来值得研究的重要内容。     

SELF-APPRAISAL BASED UPON SUPERVISORY FEEDBACK

Self-appraisals of job performance have historically proven to be weakly related to appraisals conducted by supervisory personnel. Two studies sought to facilitate rater agreement by invoking supervisory performance feedback as a frame of reference for subordinate raters. In the first study, a sample of 401 branch managers from a large lending institution performed a conventional self-appraisal and an appraisal containing instructions referencing supervisory feedback (Feedback Based Self-Appraisal). Feedback Based Self-Appraisals exhibited significantly smaller leniency error, greater total rating variance, and more agreement with superior ratings than conventional self-appraisals. FBSAs produced small improvements over conventional self-appraisals in predicting objective criteria of managerial job performance. Study 2 examined FBSA-supervisory appraisal agreement for two samples of military organization personnel. The relative degree of feedback available in a work environment was found to moderate FBSA-superior agreement.