多源评价的特点与内在机制

多源评价在国外企业中的运用已日益成熟, 但在我国还停留在探索与发展阶段。基于已有的研究发现, 围绕评价过程、评价源及被评价者三方面对多源评价的特点及内在机制进行了探讨与分析。从评价过程看, 其评价目的具有多重性, 评价形式注重匿名性, 且评价结果的合理应用非常重要; 从评价源看, 不同评价源间的评价一致性较低, 且易造成晕轮效应和宽大效应; 从被评价者来看, 个体对多源评价结果的反应, 受到个性特征、反馈信号及自我-他人评价间差距等因素影响。研究也发现, 多源评价所带来的绩效改进结果具有不稳定性。基于此, 如何提高多源评价过程的有效性与准确性, 改善评价者对评价结果的反应, 以及如何对多源评价结果进行有效汇总等是未来值得研究的重要内容。     

Multisource Feedback and Leaders' Goal Performance: Moderating effects of rating purpose,rater perspective,and performance dimension

The main purposes of this study are to examine whether multisource feedback ratings predict leaders' organizational goal performance, and whether the relationships are consistent across the two rating purposes (developmental, administrative), two leadership dimensions behaviors (Consideration, Initiating Structure), and three rating perspectives (supervisor, self, and ‘other’ raters, i.e., peers and subordinates). Leaders (n=396) in a large organization in the transportation industry participated in two multisource feedback programs, the first for developmental purposes and the second 8 months later for administrative purposes. Approximately 1 month later, they were rated by their supervisor on their effectiveness in attaining five organizational performance goals (financial, safety, customer satisfaction, employee satisfaction, diversity). Results revealed that both developmental ratings and administrative ratings uniquely predicted leaders' goal performance. However, both leadership dimension and rater perspective moderated these relationships. Leadership behaviors associated with Consideration were stronger predictors of goal performance for supervisor ratings, whereas behaviors associated with Initiating Structure were stronger predictors of goal performance for self and other ratings.     

PSYCHOMETRIC PROPERTIES OF SELF-APPRAISALS OF JOB PERFORMANCE

This paper reviews literature of psychometric properties of self-appraisals of work performance. It summarizes evidence of leniency, variability, halo, bias, and construct validity. Comparisons with appraisals by supervisors, peers, and subordinates suggest that self-appraisals tend to show more leniency, less variability, and less discriminant validity. Different factor structures have been found among self, supervisor and peer-ratings. On the other hand, self-appraisals showed less halo. Self-appraisals were significantly correlated with other sources in some studies and failed to correlate in many others. Existing data do not allow any conclusion whether the quality of self-appraisals is a function of scale format, amount or rater training, type of judgment, or purpose of appraisal. The effects of the observed psychometric qualities of self-appraisals on various applications are discussed. Problems may exist when they are used for administrative decision making, diagnosis of training needs, applied criterion measurement, measurement of constructs in basic research, or for selection purposes.     

Do Cross‐Cultural Values Affect Multisource Feedback Dynamics? The Case of High Power Distance and Collectivism in Two Latin American Countries

In this investigation we report results of a study designed to test the effect of cross‐cultural values on multisource feedback (MSF). With the participation of a sample of 501 managers from Venezuela and Colombia, two collectivistic and high power distance countries, results of the study indicate that cultural values distort the evaluations involved in MSF systems. Specifically, unlike reports of studies conducted in individualistic and low power distance environments, we found that (a) peers are the least discrepant source of information, (b) subordinates tend to provide the highest evaluations across feedback sources, and (c) there is an excessive emphasis on people‐oriented behaviors. Results are discussed in light of the conflict between our findings and those in existing literature. The practical implications of MSF in multi‐cultural workplaces are addressed.     

RATER-RATEE RACE EFFECTS IN DEVELOPMENTAL PERFORMANCE RATINGS OF MANAGERS

The effects of rater and ratee race on performance ratings of managers were examined. Ratings were obtained from peers, subordinates and bosses as part of a multirater, developmental feedback program for managers. Two data sets were created for purposes of this study. The between-subjects data set consisted of ratings from over 20,000 bosses, over 50,000 peers, and over 40,000 subordinates. The repeated measures data set was substantially smaller because it included only those Black and White managers who were rated by both a Black and White rater from each of the three perspectives. Results for rater race indicated that Black raters from all perspectives (peers, subordinates, and bosses) assigned more favorable ratings to ratees of their own race. Results for White raters differed according to the particular rating source. White bosses assigned more favorable ratings to ratees of their own race, but White subordinates did not. White peers assigned more favorable ratings to Whites in the repeated measures analysis, but not in the between-subjects analysis. Results for ratee race indicated that both White and Black managers received higher ratings from Black raters than from White raters, and the effect was more pronounced for ratings assigned to Black managers.     

Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice.     

Rater bias in the EASI temperament scales: a twin study

Under trait theory, ratings may be modeled as a function of the temperament of the child and the bias of the rater. Two linear structural equation models are described, one for mutual self- and partner ratings, and one for multiple ratings of related individuals. Application of the first model to EASI temperament data collected from spouses rating each other shows moderate agreement between raters and little rating bias. Spouse pairs agree moderately when rating their twin children, but there is significantly rater bias, with greater bias for monozygotic than for dizygotic twins. MLE's of heritability are approximately .5 for all temperament scales with no common environmental variance. Results are discussed with reference to trait validity, the person-situation debate, halo effects, and stereotyping. Questionnaire development using ratings on family members permits increased rater agreement and reduced rater bias.     

Leniency effect as a function of rating format,purpose for appraisal,and rater individual differences

The effect of rating format and non-performance variables on rating leniency were studied in two law enforcement organizations. One of these variables, trust in the appraisal process, was defined as the extent to which a rater believes that fair and accurate appraisal will be made in the organization. A measure of trust in appraisal accounted for a significant proportion of variance in performance ratings. The purpose of appraisal (i.e., feedback or promotion) also accounted for rating variance. A mixed-standard rating format showed less susceptibility to the non-performance variables on the extent of leniency. Discussion centers on the usefulness of rater and organizational variables in performance appraisal research.     

Effects of multisource feedback and a feedback facilitator on the influence behavior of managers toward subordinates

The authors compared a feedback workshop with both a no-feedback control group and a comparison group of managers who received a feedback report but no feedback workshop. The multisource feedback was based on ratings of a manager's influence behavior by subordinates, peers, and bosses. Managers in the feedback workshop increased their use of some core influence tactics with subordinates, whereas there was no change in behavior for the control group or for the comparison group. The feedback was perceived to be more useful by managers who received it in a workshop with a facilitator than by managers who received only a printed feedback report.     

Rater bias in psychological research: when is it a problem and what can we do about it?

Rater bias is a substantial source of error in psychological research. Bias distorts observed effect sizes beyond the expected level of attenuation due to intrarater error, and the impact of bias is not accurately estimated using conventional methods of correction for attenuation. Using a model based on multivariate generalizability theory, this article illustrates how bias affects research results. The model identifies 4 types of bias that may affect findings in research using observer ratings, including the biases traditionally termed leniency and halo errors. The impact of bias depends on which of 4 classes of rating design is used, and formulas are derived for correcting observed effect sizes for attenuation (due to bias variance) and inflation (due to bias covariance) in each of these classes. The rater bias model suggests procedures for researchers seeking to minimize adverse impact of bias on study findings.     

Rating ability in performance judgments: The joint influence of implicit theories and intelligence

The purpose of this study was to approach the issue of rating ability by examining the influence of rater implicit theories and rater intelligence on rating outcomes. Using the inferential accuracy model (Jackson, 1972), raters were identified as either possessing a normative or idiosyncratic implicit theory of the occupation of college instructor. In a laboratory setting, 50 normative and 50 idiosyncratic raters judged the videotaped performance of either a good or poor lecturer. Results showed that (a) intelligence was positively related to rating accuracy and to possessing a normative implicit theory, (b) rater type moderated the relationship between intelligence and rating accuracy, and (c) controlling for intelligence, normative raters committed stronger halo effects than idiosyncratic raters. These results were discussed in relation to furthering the understanding of rating ability.     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

The Relationship of Personality test Scores to School Marks and Intelligence Quotients

Summary

After Kipnis et al., it was hypothesized that superiors who believe they employ strong means of influence attribute the behavior of their subordinates to that influence, devalue the contributions of their subordinates, and increase the social distance between themselves and their subordinates. Also, the effects of superior power from the viewpoint of the subordinate were assessed. With the use of different measures of power, attribution, and performance from Kipnis et al. and a sample of 77 female and male American nurses, it was found that superiors who perceive themselves to be relatively powerful report a greater social distance from their subordinates than less powerful superiors. Kipnis et al.'s other findings generally were not replicated. Examined subordinate reactions were shown to be minimal. The limitations of the study and the need to further explore the role of superior power in work settings are discussed.     

A LONGITUDINAL EXAMINATION OF RATER AND RATEE EFFECTS IN PERFORMANCE RATINGS

The consistency and loci of leniency, halo, and range restriction effects in performance ratings were investigated in a longitudinal study. Ratings were provided by approximately 90 supervisors in a metropolitan police department, who rated approximately 350 police-rank subordinates on five occasions over a three and one-half year period. Rating effects were computed separately as rater-and ratee-based statistics, and intercorrelated among the five rating periods. The nature of the data set made it possible to hold either raters or ratees constant for each analysis, thus permitting inferences regarding the sources of reliable variance in effects as due to raters or ratees. It was concluded that reliable variance in mean ratings is partly attributable to ratees, but mainly introduced by raters. Reliable halo variance is attributable to raters, and range restriction is a product of stable group performance variability within intact ratee groups. Implications of these results for future rating process research are discussed.     

Power and Personality Perception in Real‐Life Hierarchical Relationships

The present study examined how Big Five personality ratings of the same target individuals differ as a function of the power relation between the target and the judge. Our targets were 37 employees with leadership duties from two large organizations. The targets' subordinates (N = 352), peers (N = 186), and superiors (N = 62) constituted our groups of judges. The targets and judges also provided self‐reports of personality. Subordinate judges showed higher consensus but not higher self‐other agreement than peer or superior judges. Furthermore, the targets were judged as more extraverted, more emotionally stable, less agreeable, and less open to experience by their subordinates than by their superiors. The results suggest that (i) observer consensus, but not self‐other agreement or assumed similarity varies as a function of real‐life power; (ii) the effects of power on mean trait scores are mostly congruent with the previously observed effects of power on behaviour and on stereotypes. Copyright © 2012 John Wiley & Sons, Ltd.     

A CRITICAL ANALYSIS OF STUDIES COMPARING BEHAVIORALLY ANCHORED RATING SCALES (BARS) AND OTHER RATING FORMATS1

All published studies comparing behaviorally anchored rating scales (BARS) with other rating formats were reviewed. Results indicate that BARS and other formats appear to differ relatively little (if at all) with regard to psychometric characteristics (leniency, halo, interrater agreement, ratee discriminability, validity and “accuracy”). Other variables, such as degree of rater participation in scale construction and rater preferences for different rating scale formats, also exhibited relatively little differences between BARS and other formats. A close examination of the studies, however, reveals numerous methodological problems confounding interpretations of results, making conclusions about the superiority or inferiority of BARS quite tenuous. Suggestions are made for more appropriate research directions and questions which seem more appropriate for rating scale investigations.     

RATER SOURCE EFFECTS ARE ALIVE AND WELL AFTER ALL

Recent research has questioned the importance of rater perspective effects on multisource performance ratings (MSPRs). Although making a valuable contribution, we hypothesize that this research has obscured evidence for systematic rater source effects as a result of misspecified models of the structure of multisource performance ratings and inappropriate analytic methods. Accordingly, this study provides a reexamination of the impact of rater source on multisource performance ratings by presenting a set of confirmatory factor analyses of two large samples of multisource performance rating data in which source effects are modeled in the form of second-order factors. Hierarchical confirmatory factor analysis of both samples revealed that the structure of multisource performance ratings can be characterized by general performance, dimensional performance, idiosyncratic rater, and source factors, and that source factors explain (much) more variance in multisource performance ratings whereas general performance explains (much) less variance than was previously believed. These results reinforce the value of collecting performance data from raters occupying different organizational levels and have important implications for research and practice.     

TRAIT, RATER AND LEVEL EFFECTS IN 360-DEGREE PERFORMANCE RATINGS

Method and trait effects in multitrait-multirater (MTMR) data were examined in a sample of 2,350 managers who participated in a developmental feedback program. Managers rated their own performance and were also rated by two subordinates, two peers, and two bosses. The primary purpose of the study was to determine whether method effects are associated with the level of the rater (boss, peer, subordinate, self) or with each individual rater, or both. Previous research which has tacitly assumed that method effects are associated with the level of the rater has included only one rater from each level; consequently, method effects due to the rater's level may have been confounded with those due to the individual rater. Based on confirmatory factor analysis, the present results revealed that of the five models tested, the best fit was the 10-factor model which hypothesized 7 method factors (one for each individual rater) and 3 trait factors. These results suggest that method variance in MTMR data is more strongly associated with individual raters than with the rater's level. Implications for research and practice pertaining to multirater feedback programs are discussed.     

Understanding the latent structure of job performance ratings

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n = 2,350 and n = 2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%.