SELF- VERSUS OTHERS' RATINGS AS PREDICTORS OF ASSESSMENT CENTER RATINGS: VALIDATION EVIDENCE FOR 360-DEGREE FEEDBACK PROGRAMS

Although 360-degree feedback programs are rapidly increasing in popularity, few studies have examined how well ratings from these programs predict an independent criterion. This study had 2 main aims: First, to examine the validity of ratings from a 360-degree feedback program using assessment center ratings as an independent criterion and to determine which source (i.e., self, supervisor, peers, or subordinates) provided the most valid predictor of the criterion measure of competency. Second, to better understand the relationship between self-observer discrepancies and an independent criterion. The average of supervisor, peer, and subordinate ratings predicted performance on the assessment center, as did the supervisor ratings alone. The self-ratings were negatively and nonlinearly related to performance with some of those who gave themselves the highest ratings having the lowest performance on the assessment center. Supervisor ratings successfully discriminated between overestimators but were not as successful at discriminating underestimators, suggesting that more modest feedback recipients might be underrated by their supervisors. Peers overestimated performance for poor performers. Explanations of the results and the implications for the use of self-ratings in evaluations, the design of feedback reports, and the use of 360-degree feedback programs for involving and empowering staff are discussed.     

INFLUENCE OF ASSESSMENT CENTER METHODS ON ASSESSORS'RATINGS

Recently a number of authors have argued persuasively that performance ratings are influenced to a large extent by the way the rater selects, organizes, stores, and recalls information. Although the influence of the rater's cognitive processes on the obtained ratings has been considered in the job-performance evaluation literature, this issue has not been considered in the assessment center literature. The purpose of the present study was to examine how assessment center methods affect the way assessors organize and process assessment center information and affect the ratings they make. Independent groups of assessors underwent one of two methods for evaluating candidates in an assessment center. Data indicated differences in the convergent and discriminant validities and differences in the factor structures of the ratings for the two methods. The pattern of results suggested that the two methods for evaluating assessment center candidates affected the way the assessors organized the assessment center information and affected the obtained ratings. Future research should consider assessment center methods and assessors as sources of variation in assessment center ratings.     

RATER-RATEE RACE EFFECTS IN DEVELOPMENTAL PERFORMANCE RATINGS OF MANAGERS

The effects of rater and ratee race on performance ratings of managers were examined. Ratings were obtained from peers, subordinates and bosses as part of a multirater, developmental feedback program for managers. Two data sets were created for purposes of this study. The between-subjects data set consisted of ratings from over 20,000 bosses, over 50,000 peers, and over 40,000 subordinates. The repeated measures data set was substantially smaller because it included only those Black and White managers who were rated by both a Black and White rater from each of the three perspectives. Results for rater race indicated that Black raters from all perspectives (peers, subordinates, and bosses) assigned more favorable ratings to ratees of their own race. Results for White raters differed according to the particular rating source. White bosses assigned more favorable ratings to ratees of their own race, but White subordinates did not. White peers assigned more favorable ratings to Whites in the repeated measures analysis, but not in the between-subjects analysis. Results for ratee race indicated that both White and Black managers received higher ratings from Black raters than from White raters, and the effect was more pronounced for ratings assigned to Black managers.     

A FIELD STUDY OF THE EFFECTS OF RATING PURPOSE ON THE QUALITY OF MULTISOURCE RATINGS

Using a field sample of peers and subordinates, the current study employed generalizability theory to estimate sources of systematic variability associated with both developmental and administrative ratings (variance due to items, raters, etc.) and then used these values to estimate the dependability (i.e., reliability) of the performance ratings under various conditions. Results indicated that the combined rater and rater-by-ratee interaction effect and the residual effect were substantially larger than the person effect (i.e., object of measurement) for both rater sources across both purpose conditions. For subordinates, the person effect accounted for a significantly greater percentage of total variance in developmental ratings than in administrative ratings; however, no differences were observed for peer ratings as a function of rating purpose. These results suggest that subordinate ratings are of significantly better quality when made for developmental than for administrative purposes, but the same is not true for peer ratings.     

IMPLICATIONS OF TRAIT-ACTIVATION THEORY FOR EVALUATING THE CONSTRUCT VALIDITY OF ASSESSMENT CENTER RATINGS

Assessment centers have often been criticized for lacking evidence supporting the construct validity of dimension ratings. This study examines whether the poor convergence of assessment center ratings is a result of correlating ratings from exercises that differ in the extent that behavior relevant to personality traits can be observed. Using data from a promotional assessment center for law enforcement officers ( n = 79), the convergence of assessment center ratings was evaluated within the context of the five factor model by comparing the average within-dimension correlation of ratings from exercises that allowed for more opportunity to observe trait-relevant behavior to the average of those involving exercises where there was less opportunity. For each personality trait, ratings from exercises judged by experts to be high in trait-activation potential displayed stronger convergence (mean r = .30) than did ratings from exercises that were low in activation potential for that trait (mean r = .15). Implications for evaluating the construct validity of assessment centers are discussed along with future directions for classifying exercises based on situational similarity.     

CONSTRUCT VALIDITY OF TWO CATEGORIES OF ASSESSMENT CENTER DIMENSION RATINGS

The construct validity of assessment center final dimension ratings was examined within a nomological network of cognitive and personality measures. Four hundred forty-one employees of a large mid-western petroleum company were assessed on 11 dimensions in two broad categories and completed four tests. Results showed that several cognitive ability measures related more strongly to performance-style dimension ratings than to interpersonal-style dimension ratings, providing evidence for convergent and discriminant validity. Correlation analysis and factor analysis support the two a priori interpersonal- and performance-style categories. The results suggest that final dimension ratings possess construct validity and that assessors can differentiate between two broad categories of assessment dimensions.     

AN EXAMINATION OF THE EFFECTS OF AN UPWARD FEEDBACK PROGRAM OVER TIME

Despite the growing popularity of upward feedback programs, there has been very little research on the effects of upward feedback on changes in managerial performance over time. We examined the effects of implementing an upward feedback program in which subordinates' ratings were collected for 238 first-through fifth-level managers in a large corporation at two points in time about 6 months apart. Research questions drawn from goal-setting, control, and social cognition theories guided our investigation of the effects of (a) receiving feedback, (b) completing self-ratings, (c) managers' initial level of performance, and (d) subordinate's familiarity with the manager on changes over time in the level of ratings and reported opportunity to observe managerial behaviors. Ratings were provided on a 33-item behavioral observation scale that focused on areas such as the manager's commitment to quality, communications, support of subordinates (encouraging participation, coaching), and fairness. Based on subordinates' ratings, managers whose initial level of performance was moderate or low improved over the 6-month period, and this improvement could not be attributed solely to regression to the mean. Also, both managers and their subordinates became more likely over time to indicate that the managers had an opportunity to demonstrate behaviors measured by the upward feedback instrument.     

Should assessment center dimension ratings be mechanically-derived?

This study examined the relationship between consensusderived and mechanically-derived assessment center ratings using a sample of 110 police candidates. It was determined that the two sets of dimension ratings were highly correlated (medianr=.83). Further, the mechanically-derived and the consensus-derived assessment ratings both correlated with the overall assessment ratings (OAR) at .71. It was estimated that the organization in this study could save $6137.92 a year, and $36.89 per assessee by eliminating the team meeting (consensus decision-making).     

Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

Personality Correlates of Assessment Center Consensus Competency Ratings: Evidence from Russia

Controversy has revolved around whether assessment center ratings have construct validity to measure intended dimensions of managerial performance. In contrast to much recent research on the internal structure of assessment center ratings, the present studies investigated the relationship of final competency ratings derived by consensus discussion with external questionnaire measures of personality characteristics. Expanding on previous studies showing correlations of dimension scores in relation to individual trait measures, this study investigated the relationship of complex competencies with both single personality traits and with composites of personality traits. Evidence from two samples of managers in Russia shows that final competency ratings are related to predicted composites of personality factors more consistently than to single factors. Taken together, these findings provide evidence that assessment center ratings derived by consensus discussion show construct validity in relationship with predicted composites of personality characteristics.     

AN EXAMINATION OF THE EFFECTS OF USING BEHAVIOR CHECKLISTS ON THE CONSTRUCT VALIDITY OF ASSESSMENT CENTER DIMENSIONS

Although research has established the criterion-related validity of assessment centers for selection purposes, the construct validity of dimension ratings has not been demonstrated. A quasi-experimental design was used to investigate the influence of retranslated behavior checklists on the construct validity of dimension ratings for two assessment center exercises. Assessor use of behavior checklists increased the average convergent (i.e., same dimension across exercise) validity from .24 to .43 while decreasing the average discriminant (i.e., different dimension within exercise) validity (.47 to .41). Behavior checklist sums were moderately correlated with corresponding dimension ratings and demonstrated a comparable level of construct validity. It is suggested that using behavior checklists may improve dimension construct validity by reducing the cognitive demands placed on raters.     

COORIENTATION ON COMMUNICATION RULES IN MANAGERIAL DYADS

This field study compared the effects of three dimensions of supervisor-subordinate coorientation regarding two communication rules (initiation and termination) on satisfaction with supervision and performance evaluations. For supervisors accuracy and perceived agreement on rules had positive effects on performance ratings of subordinates, even after controlling for frequency of contact. For subordinates, only perceived agreement on rules had an impact on satisfaction with supervision. For both supervisors and subordinates agreement on communication rules was not a significant predictor of evaluations.     

The Influence of Candidate Social Effectiveness on Assessment Center Performance Ratings: A field study

The present research examined the influence of constructs representing social effectiveness on assessment center (AC) ratings in two samples. We expected different effects of self‐monitoring (SM) on different dimension ratings, a positive effect of the ability to identify criteria (ATIC) on the overall AC rating and a moderating effect of the ATIC on the relationship between SM and the dimension rating. Forty‐six (Study 1) and 115 (Study 2) applicants participated in ACs in field settings. Across both studies, SM had a negative effect on the integrity rating. No relationship was identified between SM and social sensitivity or problem solving ratings. In Study 1, the ATIC had a positive effect on the overall AC rating. No support was identified for a moderating effect of the ATIC on the relationship between SM and the social sensitivity rating.     

THE EFFECTS OF FEEDBACK ACCOUNTABILITY ON UPWARD APPRAISAL RATINGS

Do subordinates feel and respond differently to upward appraisal procedures depending on whether they are accountable or anonymous? Accountability requires subordinates to identify themselves on the upward appraisal questionnaires they complete. In an experimental field study, 38 managers and their subordinates from an insurance company were randomly assigned to use one of the appraisal procedures. As hypothesized, managers who received feedback from specific individuals in the accountability procedure viewed the upward appraisal process more positively than did managers in the anonymity procedure. However, subordinates felt more comfortable giving anonymous responses. As a result, subordinates who used the accountability procedure rated their managers significantly higher than the subordinates who responded anonymously. Therefore, the results of this study suggest that an upward appraisal procedure in which individuals are accountable for their responses may produce inflated ratings of managers' performance. Further research should compare the accountability and anonymous upward appraisal procedures to determine which procedure produces more accurate ratings.     

THE ROLE OF COGNITIVE ABILITY IN THE SUBGROUP DIFFERENCES AND INCREMENTAL VALIDITY OF ASSESSMENT CENTER EXERCISES

This study investigates the degree to which subgroup (Black-White) mean differences on various assessment center exercises (e.g., in-basket, role play) may be a function of the type of exercise employed; and furthermore, begins to explore why these different types of exercises result in subgroup differences. The sample consisted of 633 participants who completed a managerial assessment center that evaluated them on 14 ability dimensions across 7 different types of assessment exercises. In addition, each participant completed a cognitive ability measure. The results suggest that subgroup differences varied by type of assessment exercise; and furthermore that the subgroup difference appeared to be a function of the cognitive component of the exercise. Lastly, preliminary support is found that the validity of some of the assessment center exercises in predicting supervisor ratings of job performance is based, in part, on their cognitive component; however, evidence of incremental validity does exist.     

Unanswered questions in assessment center research

This study compared the predictive validity of two types of overall assessment center ratings (clinical and mechanical) and two types of performance ratings (supervisor and subordinate). Data were gathered on approximately 50 law enforcement agency managers over a four-year time span. Results indicated that clinical predictions were not significantly better than mechanical ones and that prior performance ratings were better predictors of future performance ratings than were assessment centers. Further, dimension ratings within the assessment center were found to be unrelated to ratings of the same dimensions on the job four years later, supporting the notion of subtle criterion contamination (Klimoski & Strickland, 1977) and raising questions about the central assumptions underlying the assessment center approach.     

THE INFLUENCE OF SUBORDINATE AGE ON PERFORMANCE RATINGS AND CAUSAL ATTRIBUTIONS

Three issues were examined in this study relative to the role of subordinate age in performance evaluations. First, it was found that supervisors rated older subordinates lower than younger subordinates doing the same job. Second, differences between self-ratings of performance and supervisory ratings seemed to be a function of the subordinate's age. Third, a significant age × performance interaction was detected on ability attributions.     

A META-ANALYSIS OF THE CRITERION-RELATED VALIDITY OF ASSESSMENT CENTER DIMENSIONS

We used meta-analytic procedures to investigate the criterion-related validity of assessment center dimension ratings. By focusing on dimension-level information, we were able to assess the extent to which specific constructs account for the criterion-related validity of assessment centers. From a total of 34 articles that reported dimension-level validities, we collapsed 168 assessment center dimension labels into an overriding set of 6 dimensions: (a) consideration/awareness of others, (b) communication, (c) drive, (d) influencing others, (e) organizing and planning, and (f) problem solving. Based on this set of 6 dimensions, we extracted 258 independent data points. Results showed a range of estimated true criterion-related validities from .25 to .39. A regression-based composite consisting of 4 out of the 6 dimensions accounted for the criterion-related validity of assessment center ratings and explained more variance in performance (20%) than Gaugler, Rosenthal, Thornton, and Bentson (1987) were able to explain using the overall assessment center rating (14%).     

THE ROLE OF UPWARD INFLUENCE TACTICS IN HUMAN RESOURCE DECISIONS

A model was examined which proposes that employee influence tactics impact human resource decisions by affecting managers' perceptions of the subordinate's interpersonal skills, manager liking of subordinates, and managers' perceptions of similarity to subordinates. The human resource decisions investigated were performance ratings, promotability assessments, and salary. Data were collected from 247 subordinates and their managers and from company records. Structural equation modeling results indicated that influence tactics were related to managers' perceptions of subordinates' interpersonal skills, liking, and perceptions of similarity to subordinates. Specifically, subordinates' use of reasoning, assertiveness, and favor rendering were positively related to managers' perceptions whereas bargaining and self-promotion were negatively related to these perceptions. Finally, there was no support for direct relationships between influence tactics and HR decisions.