SELF-OTHER AGREEMENT: DOES IT REALLY MATTER?

A current controversy in the self-other rating and 360-degree feedback literature is the extent to which self-other agreement (and lack of agreement) has an impact on individual and organizational outcomes. Using a large sample and a multi-source data set, the current study addressed some methodological limitations of prior research. Results from polynomial regression analyses demonstrated that both self- and other ratings are related to performance outcomes. This procedure revealed the underlying three-dimensional relationship between self-ratings, other ratings, and effectiveness. Findings indicate that the relationship between self-ratings, other ratings and outcomes are somewhat more complex than previous conceptualizations in this area. Simultaneous consideration of both self- and other ratings in terms of the direction and magnitude of self- and other ratings is important for explaining effectiveness outcomes.     

PSYCHOMETRIC PROPERTIES OF SUBORDINATE RATINGS OF MANAGERIAL PERFORMANCE

The purpose of the present study was to compare the psychometric properties of subordinate ratings of managerial performance to those of supervisor and self ratings. Subordinate ratings (n = 365), self-ratings (n = 80) and supervisor ratings ( n = 80) were obtained on eight performance dimensions for middle level managers in a multinational corporation. Multitrait-multimethod analysis of variance indicated significant effects for convergent validity, halo error, and discriminant validity. Comparisons between rating sources indicated that subordinate ratings are more similar to supervisor ratings than to self ratings in terms of convergent validity and leniency effect. A comparison of the present results to other studies indicated less convergent validity, more discriminant validity, and less halo error than found by others.     

A COMPARISON OF VALIDATION CRITERIA: OBJECTIVE VERSUS SUBJECTIVE PERFORMANCE MEASURES AND SELF- VERSUS SUPERVISOR RATINGS

This study compared four criteria–two objective (production quantity and production quality) and two subjective (supervisor and self-ratings)–for their predictability in a criterion-related validity study. Results from this sample of 212 maintenance, mechanic, and field service workers replicated previous meta-analytic results with clerical workers (Nathan & Alexander, 1988); supervisor ratings and objective productivity indices provided similar and significant validity coefficients with a unit-weighted composite of five cognitive ability tests. The objective quality index and employee self-ratings resulted in near zero correlations with the same predictor battery. Additional productivity and quality objective criterion data were available for 2 years since the original validation study; no change in validity was found.     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

Self‐Supervisor Agreement: The Influence of Feedback Seeking on the Relationship Between Self and Supervisor Ratings of Performance1

Self‐assessment research has continued to search for those factors that increase self‐other rating agreement. The current field study investigated the feedback‐seeking strategies (i. e., monitoring and inquiry) used by 125 employees to obtain performance information, as well as the relationship between feedback‐seeking strategy use and self‐supervisor performance‐rating agreement. Results indicate that the frequency of monitoring reported by employees significantly moderated the relationship between self and supervisor ratings of performance. Individuals who reported higher levels of feedback seeking through monitoring were more likely to have self‐assessments that were congruent with their supervisors' ratings of performance.     

Construct Validity of Multi‐Source Performance Ratings: An Examination of the Relationship of Self‐, Supervisor‐, and Peer‐Ratings with Cognitive and Personality Measures

Although more and more organizations prefer using multi‐source performance ratings or 360° feedback over traditional performance appraisals, researchers have been rather skeptical regarding the reliability and validity of such ratings. The present study examined the validity of self‐, supervisor‐, and peer‐ratings of 195 employees in a Dutch public organization, using scores on an In‐Basket exercise, an intelligence test, and a personality questionnaire as external criterion measures. Interrater agreement ranged from .28 to .38. Variance in the ratings was explained by both method and content factors. Support for the external construct validity was rather weak. Supervisor‐ratings were not found to be superior to self‐ and peer‐ratings in predicting the scores on the external measures.     

THE EFFECTS OF PERCEIVED SYSTEM KNOWLEDGE ON THE AGREEMENT BETWEEN SELF-RATINGS AND SUPERVISOR RATINGS

The viability of self-assessment in the performance appraisal process has been the focus of recent work. Although some contend that self-appraisals can be beneficial, the consistent finding of lenient and inaccurate self-ratings has led others to discount and doubt their usefulness. Recent research (Farh & Dobbins, 1989a,b) has investigated the influence of a variety of factors in improving the accuracy of self-ratings. The present study was conducted to examine the effect of perceived system knowledge (PSK) on the discrepancy between self- and supervisor ratings. Seventy-three employees of two banking institutions participated by completing a questionnaire on their perceived knowledge of the performance appraisal system. The results of moderated regression analyses suggest that the level of knowledge did in fact influence the agreement among subordinates and supervisors. Self-ratings were more congruent with supervisors'ratings when the subordinates reported high levels of perceived system knowledge. Implications of these findings for organizations and areas for future research are discussed.     

Unanswered questions in assessment center research

This study compared the predictive validity of two types of overall assessment center ratings (clinical and mechanical) and two types of performance ratings (supervisor and subordinate). Data were gathered on approximately 50 law enforcement agency managers over a four-year time span. Results indicated that clinical predictions were not significantly better than mechanical ones and that prior performance ratings were better predictors of future performance ratings than were assessment centers. Further, dimension ratings within the assessment center were found to be unrelated to ratings of the same dimensions on the job four years later, supporting the notion of subtle criterion contamination (Klimoski & Strickland, 1977) and raising questions about the central assumptions underlying the assessment center approach.     

Reactions to Appraisal Discrepancies: Performance Ratings and Attributions

The traditional top-down approach to performance appraisal has been criticized as not keeping pace with the move toward more participative organizations of the last decade. This has led many organizations to adopt newer performance appraisal systems that use multiple feedback sources (e.g., 360-degree feedback systems). However, research indicates that multiple feedback sources often disagree on performance ratings and attributions. The current paper reports the results of one laboratory and one field study that show the importance of performance and attributional rating discrepancies on the appraisal process.     

A MULTIPLE LEVEL INVESTIGATION OF PERSONNEL RATINGS

A multiple levels of analysis approach was used to assess whether job performance and skill importance ratings were based on individual differences, incumbent/supervisor dyadic agreement, group membership, or functional department. Data were collected as part of a managerial training needs analysis conducted at a large gold and silver mining organization. Participants included 140 incumbent managers and their immediate supervisors, who were embedded in 140 incumbent/supervisor dyads, 39 managerial work groups, and eight departments of the organization. Results from Within and Between Analysis (WABA) suggested that (a) incumbent ratings of both performance and skill importance were characterized by individual differences, (b) supervisory ratings of both performance and skill importance were primarily characterized by between-groups (supervisors) differences, (c) matched incumbent-supervisory performance ratings displayed individual differences, and (d) matched incumbent-supervisory importance ratings displayed dyadic agreement (i.e., between-dyads differences). Practical implications of these results for interpreting these and other types of personnel-related data are discussed.     

MODESTY BIAS IN SELF-RATINGS OF PERFORMANCE: A TEST OF THE CULTURAL RELATIVITY HYPOTHESIS

Although self-ratings of performance are usually higher than ratings obtained from supervisors, Farh, Dobbins, and Cheng (1991) found that Taiwanese workers exhibited modesty bias (i.e., self-ratings that were lower than supervisory ratings). They explained their findings in terms of broad cultural differences between Taiwanese and Western workers. To test this cultural relativity hypothesis, we replicated their study using data from several organizations in mainland China. As is typically found in Western research, Chinese workers showed leniency in self-ratings (i.e., self-ratings that were higher than supervisor or peer ratings), which suggests that broad cultural factors do not explain the modesty bias reported by Farh et al.     

The influence of group discussion on performance judgments: rating accuracy, contrast effects, and halo

The authors investigated the effect of group discussion, such as may occur formally in panel interview scenarios, assessment centers, or 360-degree feedback situations, on judgments of performance. Research on group polarization suggests that the effect of group discussion combined with raters' preexisting impressions of ratees or interviewees should result in an extremitization of impressions. Thus, the authors hypothesized that group discussion would (a) make ratings less accurate, (b) polarize impressions that were already good or poor as reflected by greater contrast effects, and (c) increase positive halo. Results indicated that group discussion resulted in less accurate ratings and greater contrast effects. Additional analyses suggested that group discussion increased positive halo. The authors discuss implications for research on group or panel judgments.     

Loving yourself abundantly: relationship of the narcissistic personality to self- and other perceptions of workplace deviance, leadership, and task and contextual performance

The authors report results from 2 studies assessing the extent to which narcissism is related to self- and other ratings of leadership, workplace deviance, and task and contextual performance. Study 1 results revealed that narcissism was related to enhanced self-ratings of leadership, even when controlling for the Big Five traits. Study 2 results also revealed that narcissism was related to enhanced leadership self-perceptions; indeed, whereas narcissism was significantly positively correlated with self-ratings of leadership, it was significantly negatively related to other ratings of leadership. Study 2 also revealed that narcissism was related to more favorable self-ratings of workplace deviance and contextual performance compared to other (supervisor) ratings. Finally, as hypothesized, narcissism was more strongly negatively related to contextual performance than to task performance.     

RATINGS OF MANAGERIAL CHARACTERISTICS: EVALUATION DIFFICULTY, CO-WORKER AGREEMENT, AND SELF-AWARENESS

This research examined variables contributing to self-awareness and coworker agreement on 30 managerial characteristics. The first study collected data from 233 management students on the difficulty of rating different characteristics. It also developed reliable measures of self-protection mechanisms that may negatively influence self-awareness. Four self-protection factors were identified: Denial, Giving Up, Self-Promotion, and Fear of Failure. The second study collected self-, supervisor, peer, and subordinate ratings of the managerial characteristics to derive multiple measures of self-awareness. There was a total of 283 co-worker ratings for 36 target middle managers. The results showed that within-sample co-worker disagreement measures from Study 2 were higher for items that were perceived by the sample in Study 1 as more difficult to rate. Self-ratings were more highly related to the average of co-worker ratings than to the ratings provided by any one co-worker alone. Co-workers' ratings of the target's use of self-protection mechanisms were negatively associated with self-awareness measures. Ideas are discussed for other correlates of self-awareness and ways to enhance the value of feedback processes.     

Multisource Feedback and Leaders' Goal Performance: Moderating effects of rating purpose,rater perspective,and performance dimension

The main purposes of this study are to examine whether multisource feedback ratings predict leaders' organizational goal performance, and whether the relationships are consistent across the two rating purposes (developmental, administrative), two leadership dimensions behaviors (Consideration, Initiating Structure), and three rating perspectives (supervisor, self, and ‘other’ raters, i.e., peers and subordinates). Leaders (n=396) in a large organization in the transportation industry participated in two multisource feedback programs, the first for developmental purposes and the second 8 months later for administrative purposes. Approximately 1 month later, they were rated by their supervisor on their effectiveness in attaining five organizational performance goals (financial, safety, customer satisfaction, employee satisfaction, diversity). Results revealed that both developmental ratings and administrative ratings uniquely predicted leaders' goal performance. However, both leadership dimension and rater perspective moderated these relationships. Leadership behaviors associated with Consideration were stronger predictors of goal performance for supervisor ratings, whereas behaviors associated with Initiating Structure were stronger predictors of goal performance for self and other ratings.     

Internationalizing 360-Degree Feedback: Are Subordinate Ratings Comparable?

ABSTRACT: This study addressed whether 360-degree feedback ratings made by subordinates are equivalent across national cultures (Great Britain, Hong Kong, Japan, U.S.). Employees ratings of their leaders were collected via a companys existing 360-degree feedback survey. A two-factor baseline model was developed using exploratory factor analysis and tested across cultures utilizing confirmatory factor analysis techniques. Results revealed differences in factor form, suggesting that the constructs underlying the survey and their relationship to the survey items differed across cultures. These results emphasize the need for multinational companies to use caution when transporting human resources practices, such as 360-degree feedback, to international locations.The author would like to express appreciation to Alice Stuhlmacher for her valuable comments on earlier versions of this article. This research was conducted as part of the authors doctoral dissertation. Portions of this article were presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada, April 2002.     

THE RELATIONSHIP BETWEEN SUPERVISORY- AND SELF-APPRAISALS OF EXECUTIVE PERFORMANCE

There has been increased use of self-appraisal programs for evaluating the performance of executive personnel. Because objective measures of performance are rarely available or adequate for this level of personnel, ratings are often relied upon. The purpose of this study was to investigate the relationship between supervisory-perceptions and incumbent self-perceptions of the performance of executive personnel. The rationale for this study was that the relationship between supervisory-ratings and self-ratings has important implications for the usefulness of self-appraisal programs. If self-evaluations are to be effective in enlisting a person's cooperation and participation, it seems essential that his perceptions agree with those of his superior. This study was designed to identify those areas of executive performance where there might be significant amounts of disagreement. Knowledge of these problem areas would then be useful in conducting self-appraisal programs. Supervisory-ratings and self-ratings on 27 performance variables were obtained for the top executives of a large manufacturing corporation. A measure of promotability was used as the criterion and analysis of variance and correlation analyses were carried out to attain the objectives. The major conclusions of the study were: (1) Executives tended to rate themselves higher than they were rated by their supervisors, (2) Disagreement in the relationship of the two sets of ratings was found in several areas, (3) Executives who tended to overrate themselves were found to be the ones who were considered least promotable on the basis of a criterion measure of success in the organization.     

Positivity bias in employees' self-ratings of performance relative to supervisor ratings: The roles of performance type,performance-approach goal orientation,and perceived influence

As hypothesized, data from two field studies among employees and their supervisors showed that employees are more likely to positively bias their self-ratings relative to supervisor ratings when creative performance rather than inrole performance is being evaluated. In addition, employees' performance-approach goal orientation and perceived influence were found to enhance positivity bias in self-ratings of creative performance. Moreover, performance-approach goal orientation and perceived influence appeared to combine and interact in their effects, such that the impact of perceived influence on positivity bias in self-ratings was larger when performance-approach goal orientation was weaker rather than stronger.     

A META-ANALYSIS OF SELF-SUPERVISOR, SELF-PEER, AND PEER-SUPERVISOR RATINGS

Reviews of self–supervisor, self–peer, and peer–supervisor ratings have generally concluded that there is at best a modest correlation between different rating sources. Nevertheless, there has been much inconsistency across studies. Accordingly, a meta-analysis was conducted. The results indicated a relatively high correlation between peer and supervisor ratings (ρ= .62) but only a moderate correlation between self-supervisor (ρ= .35) and self-peer ratings (ρ= .36). While rating format (dimensional versus global) and rating scale (trait versus behavioral) had little impact as moderators, job type (managerial/professional versus blue-collar/service) did seem to moderate self-peer and self-supervisor ratings.     

ASSESSMENT CENTERS AND SUBORDINATE APPRAISALS OF MANAGERS: A SEVEN-YEAR EXAMINATION OF PREDICTIVE VALIDITY

Assessment center ratings of 52 state law enforcement agency managers predicted both a contaminated promotions criterion and two uncontaminated ratings criteria, but they did significantly better in predicting the former. Criteria data were gathered two, four, and seven years after the assessment center, and a modest upward trend in validity coefficients was observed. Ratings by subordinates demonstrated promise as a near term predictor, outperforming the assessment center on uncontaminated criteria in the two- and four-year time frames. Last, regression analyses using Year 7 criteria revealed unique predictive power for both ratings by subordinates and assessment centers above and beyond that provided by supervisor ratings.
An earlier version of this paper was presented at the Academy of Management meeting, Anaheim, California, August 7-10, 1988.