A COMPARISON OF VALIDATION CRITERIA: OBJECTIVE VERSUS SUBJECTIVE PERFORMANCE MEASURES AND SELF- VERSUS SUPERVISOR RATINGS

This study compared four criteria–two objective (production quantity and production quality) and two subjective (supervisor and self-ratings)–for their predictability in a criterion-related validity study. Results from this sample of 212 maintenance, mechanic, and field service workers replicated previous meta-analytic results with clerical workers (Nathan & Alexander, 1988); supervisor ratings and objective productivity indices provided similar and significant validity coefficients with a unit-weighted composite of five cognitive ability tests. The objective quality index and employee self-ratings resulted in near zero correlations with the same predictor battery. Additional productivity and quality objective criterion data were available for 2 years since the original validation study; no change in validity was found.     

PSYCHOMETRIC PROPERTIES OF SUBORDINATE RATINGS OF MANAGERIAL PERFORMANCE

The purpose of the present study was to compare the psychometric properties of subordinate ratings of managerial performance to those of supervisor and self ratings. Subordinate ratings (n = 365), self-ratings (n = 80) and supervisor ratings ( n = 80) were obtained on eight performance dimensions for middle level managers in a multinational corporation. Multitrait-multimethod analysis of variance indicated significant effects for convergent validity, halo error, and discriminant validity. Comparisons between rating sources indicated that subordinate ratings are more similar to supervisor ratings than to self ratings in terms of convergent validity and leniency effect. A comparison of the present results to other studies indicated less convergent validity, more discriminant validity, and less halo error than found by others.     

RATER-RATEE RACE EFFECTS IN DEVELOPMENTAL PERFORMANCE RATINGS OF MANAGERS

The effects of rater and ratee race on performance ratings of managers were examined. Ratings were obtained from peers, subordinates and bosses as part of a multirater, developmental feedback program for managers. Two data sets were created for purposes of this study. The between-subjects data set consisted of ratings from over 20,000 bosses, over 50,000 peers, and over 40,000 subordinates. The repeated measures data set was substantially smaller because it included only those Black and White managers who were rated by both a Black and White rater from each of the three perspectives. Results for rater race indicated that Black raters from all perspectives (peers, subordinates, and bosses) assigned more favorable ratings to ratees of their own race. Results for White raters differed according to the particular rating source. White bosses assigned more favorable ratings to ratees of their own race, but White subordinates did not. White peers assigned more favorable ratings to Whites in the repeated measures analysis, but not in the between-subjects analysis. Results for ratee race indicated that both White and Black managers received higher ratings from Black raters than from White raters, and the effect was more pronounced for ratings assigned to Black managers.     

TRAIT, RATER AND LEVEL EFFECTS IN 360-DEGREE PERFORMANCE RATINGS

Method and trait effects in multitrait-multirater (MTMR) data were examined in a sample of 2,350 managers who participated in a developmental feedback program. Managers rated their own performance and were also rated by two subordinates, two peers, and two bosses. The primary purpose of the study was to determine whether method effects are associated with the level of the rater (boss, peer, subordinate, self) or with each individual rater, or both. Previous research which has tacitly assumed that method effects are associated with the level of the rater has included only one rater from each level; consequently, method effects due to the rater's level may have been confounded with those due to the individual rater. Based on confirmatory factor analysis, the present results revealed that of the five models tested, the best fit was the 10-factor model which hypothesized 7 method factors (one for each individual rater) and 3 trait factors. These results suggest that method variance in MTMR data is more strongly associated with individual raters than with the rater's level. Implications for research and practice pertaining to multirater feedback programs are discussed.     

SELF- VERSUS OTHERS' RATINGS AS PREDICTORS OF ASSESSMENT CENTER RATINGS: VALIDATION EVIDENCE FOR 360-DEGREE FEEDBACK PROGRAMS

Although 360-degree feedback programs are rapidly increasing in popularity, few studies have examined how well ratings from these programs predict an independent criterion. This study had 2 main aims: First, to examine the validity of ratings from a 360-degree feedback program using assessment center ratings as an independent criterion and to determine which source (i.e., self, supervisor, peers, or subordinates) provided the most valid predictor of the criterion measure of competency. Second, to better understand the relationship between self-observer discrepancies and an independent criterion. The average of supervisor, peer, and subordinate ratings predicted performance on the assessment center, as did the supervisor ratings alone. The self-ratings were negatively and nonlinearly related to performance with some of those who gave themselves the highest ratings having the lowest performance on the assessment center. Supervisor ratings successfully discriminated between overestimators but were not as successful at discriminating underestimators, suggesting that more modest feedback recipients might be underrated by their supervisors. Peers overestimated performance for poor performers. Explanations of the results and the implications for the use of self-ratings in evaluations, the design of feedback reports, and the use of 360-degree feedback programs for involving and empowering staff are discussed.     

THE SOUNDNESS OF SUPERVISORY RATINGS AS PREDICTORS OF MANAGERIAL SUCCESS

A model of career development is presented. Using this model, three types of validity, empirical, content, and construct, are considered for supervisors' rating of non-managers as a predictor of success as managers. As denned in the Uniform Guidelines and the professional literature, all three are inapplicable to this prediction situation. Characteristics of a sound rating system are suggested.     

EFFECTS OF STANDARD EXTREMITY ON MIXED STANDARD SCALE PERFORMANCE RATINGS1

It was suggested that the extremity of the scale values associated with standards used to represent effective and ineffective performance in Mixed Standard Scales may affect the nature of performance ratings derived from MSS responses and decisions based on MSS ratings. When the extremity of standards was experimentally manipulated, it was found that standard extremity affects both the level of performance ratings and the proportion of logically inconsistent response patterns observed. In addition, standard extremity appears to affect the rankings on performance of ratees. The implications of these observations for the development of Mixed Standard Scales were discussed.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.     

PLACING PEER RATINGS IN CONTEXT: SYSTEMATIC INFLUENCES BEYOND RATEE PERFORMANCE

Performance evaluation research indicates that variance in ratings may be attributable to systematic sources beyond the actual performance of the ratee. However, the majority of prior work compares ratings across sources and uses ratings from a single rating event. Using confirmatory factor analysis and multivariate latent growth modeling (MLGM), we specifically examine peer ratings from 740 participants on 5 performance dimensions across 3 distinct performance situations for systematic sources of variance beyond ratee performance. Results demonstrate that both ratee performance and the performance context have systematic effects, with contextual effects varying by how "strong" or "weak" the situation is for a given performance dimension. Furthermore, MLGM results suggest that the influence of performance dynamism is only meaningfully interpreted when contextual effects are modeled.     

WHEN PERFORMANCE IMPROVEMENT IS THE GOAL: A NEW SET OF CRITERIA FOR CRITERIA

Weitz (1961), aware of the lure of tradition and expedience, urged the identification of evaluation standards for dependent variables, which he referred to as criteria for criteria. In this article, five criteria are proposed using the mnemonic SURF & C: the directness of sampling (S); the responsiveness of the target (i.e., making sure the dependent variable is under (U) the worker's control); the reliability (R) of observers; the frequent (F) assessment of the target during the intervention period; and the critical (C) nature of the target. Together the criteria provide guidelines for what and how targets should be assessed. Their necessity is illustrated in two year-long experiments designed to improve the preventive maintenance of heavy equipment in the U.S. Marine Corps. Although the criteria are limited to evaluating dependent variables in field experiments, they are recommended as the foundation for successful performance efforts in any applied setting.     

THE INFLUENCE OF SUBORDINATE AGE ON PERFORMANCE RATINGS AND CAUSAL ATTRIBUTIONS

Three issues were examined in this study relative to the role of subordinate age in performance evaluations. First, it was found that supervisors rated older subordinates lower than younger subordinates doing the same job. Second, differences between self-ratings of performance and supervisory ratings seemed to be a function of the subordinate's age. Third, a significant age × performance interaction was detected on ability attributions.     

A LONGITUDINAL EXAMINATION OF RATER AND RATEE EFFECTS IN PERFORMANCE RATINGS

The consistency and loci of leniency, halo, and range restriction effects in performance ratings were investigated in a longitudinal study. Ratings were provided by approximately 90 supervisors in a metropolitan police department, who rated approximately 350 police-rank subordinates on five occasions over a three and one-half year period. Rating effects were computed separately as rater-and ratee-based statistics, and intercorrelated among the five rating periods. The nature of the data set made it possible to hold either raters or ratees constant for each analysis, thus permitting inferences regarding the sources of reliable variance in effects as due to raters or ratees. It was concluded that reliable variance in mean ratings is partly attributable to ratees, but mainly introduced by raters. Reliable halo variance is attributable to raters, and range restriction is a product of stable group performance variability within intact ratee groups. Implications of these results for future rating process research are discussed.     

ADJUSTING SALES RESULTS MEASURES FOR USE AS CRITERIA

The use of objective criterion measures in validation research raises the issue of criterion contamination. Several methods of treating sales measures in the empirical keying of a biodata instrument are compared. The relative contaminating effects of local economic conditions and company factors are evaluated. This study is an example of how practitioners can use their knowledge about the selection context to develop acceptable criterion norming strategies.     

A FIELD STUDY OF PERFORMANCE APPRAISAL PURPOSE: RESEARCH- VERSUS ADMINISTRATIVE- BASED RATINGS

Many researchers have discussed the theoretical and practical importance of rating purpose. Nevertheless, the body of empirical studies, the majority of which were conducted in a laboratory setting, focus on leniency. There has been little research on other effects of rating purpose. The present study examines 223 ratees in a field setting for whom there were both administrative-based performance appraisal ratings (which were actually used for personnel decisions) and research-based performance appraisal ratings (obtained for a validation study). Two of the hypotheses were supported; administrative ratings were more lenient than research-based ratings. The administrative-based ratings demonstrated a statistically significant relationship with ratee seniority, while the research-based ratings did not. There was mixed support for a third hypothesis: Research ratings were significantly correlated with a predictor, while the administrative ratings were not. The difference between the validity coefficients, however, was not significant. Contrary to the hypothesis, the rank order between administrative-based and research-based ratings was relatively high ( r = 33).     

THE RELATIONSHIP BETWEEN SUPERVISORY RATINGS AND RESULTS-ORIENTED MEASURES OF PERFORMANCE: A META-ANALYSIS

A meta-analysis was conducted with 23 studies to assess the relationship between supervisory ratings and results-oriented measures of performance. As hypothesized, the corrected mean correlation was higher when a relative (versus absolute) rating format was used and when a composite (versus overall) rating method was used. These differences did not, however, account for all of the remaining variance around the relationship between ratings and results. Suggestions are offered for the direction of future research and practice.     

EXPLORING RELATIONS BETWEEN TYPICAL AND MAXIMUM PERFORMANCE RATINGS AND THE FIVE FACTOR MODEL OF PERSONALITY

The study tests the distinction between typical and maximum criteria with ratings of transformational leadership performance, and examines whether the criterion-related validities of the five factor model differ for the two types of criteria. Using an East Asian military sample ( n = 1,259) where multiple ratings of typical and maximum performance were obtained from different sources, we used structural equation modeling to test the typical/maximum performance distinction. Results found that typical and maximum performance are different latent constructs and that this distinction is present even after considering rating method factors (i.e., rater source, time). The importance of this distinction is shown by the fact that validities for the personality constructs were not equally predictive of both criteria: Openness was most predictive of maximum performance, Neuroticism was most predictive of typical performance, and Extroversion was predictive of both. By distinguishing typical from maximum performance constructs, relationships between personality and transformational leadership were found to be stronger than previous research suggested.     

COMBINING BIODATA TEST AND INTERVIEW INFORMATION: PREDICTING DECISIONS AND PERFORMANCE CRITERIA

An important question in real world selection situations is: How do interviewers integrate employment test scores and interview information in making selection decisions? The present study expands on previous research by examining how valid biodata test scores and interview information interact in the prediction of actual interviewer selection decisions and actual job performance criteria for hired candidates. The sample for the study was 577 actual candidates for the position of life insurance sales agent evaluated by insurance agency managers from a single company. The results indicated the presence of interaction effects in predicting both the interviewers' decision and 12-month survival. The interview information predicted the decision and survival best for candidates with low passing scores on the biodata test and poorest for candidates with high passing scores. The results clarify the conditions under which interview information is most predictive of job performance criteria.     

DYNAMIC CRITERIA REVISITED: A LONGITUDINAL STUDY OF PERFORMANCE STABILITY AND PREDICTIVE VALIDITY

The concept of dynamic criteria has been the subject of a recent debate regarding both the definition and prevalence of the phenomenon (Austin, Humphreys, & Hulin, 1989; Barrett & Alexander, 1989; Barrett, Caldwell, & Alexander, 1985). The present paper questions the adequacy of the conceptual framework underlying the debate and provides data supporting a refined concept of dynamic criteria. The incidence and possible causes of change in relative performance were investigated using weekly performance data from 509 sewing machine operators. Analyses were conducted to determine the degree of performance consistency, potential moderators of consistency, and the stability of predictor-criteria relationships using multiple predictors and criteria. Results revealed a steady decline in performance stability coefficients as the interval between measures increased. This decay was evident regardless of employees' prior job experience, cognitive ability, or psychomotor ability. Analyses of predictive validity coefficients revealed temporal changes in validity for both objective and subjective criteria, but not in the expected direction. The validity of cognitive ability increased, the validity of psychomotor ability was stable, and that of prior job experience decreased over time. Implications for theory and research are discussed.