A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed.     

Dimension and exercise variance in assessment center scores: a large-scale evaluation of multitrait-multimethod studies.

This study addresses 3 questions regarding assessment center construct validity: (a) Are assessment center ratings best thought of as reflecting dimension constructs (dimension model), exercises (exercise model), or a combination? (b) To what extent do dimensions or exercises account for variance? (c) Which design characteristics increase dimension variance? To this end, a large set of multitrait-multimethod studies (N = 34) were analyzed, showing that assessment center ratings were best represented (i.e., in terms of fit and admissible solutions) by a model with correlated dimensions and exercises specified as correlated uniquenesses. In this model, dimension variance equals exercise variance. Significantly more dimension variance was found when fewer dimensions were used and when assessors were psychologists. Use of behavioral checklists, a lower dimension-exercise ratio, and similar exercises also increased dimension variance.     

Three method factors explaining the low correlations between assessment center dimension ratings and scores on personality inventories

In general, correlations between assessment centre (AC) ratings and personality inventories are low. In this paper, we examine three method factors that may be responsible for these low correlations: differences in (i) rating source (other versus self), (ii) rating domain (general versus specific), and (iii) rating format (multi‐ versus single item). This study tests whether these three factors diminish correlations between AC exercise ratings and external indicators of similar dimensions. Ratings of personality and performance were combined in an analytical framework following a 2 × 2 × 2 (source, domain, format) completely crossed, within subjects design. Results showed partial support for the influence of each of the three method factors. Implications for future research are discussed. Copyright © 2004 John Wiley & Sons, Ltd.     

Further evidence for the validity of assessment center dimensions: a meta-analysis of the incremental criterion-related validity of dimension ratings

This study investigates the incremental variance in job performance explained by assessment center (AC) dimensions over and above personality and cognitive ability. The authors extend previous research by using meta-analysis to examine the relationships between AC dimensions, personality, cognitive ability, and job performance. The results indicate that the 7 summary AC dimensions postulated by W. Arthur, Jr., E. A. Day, T. L. McNelly, & P. S. Edens (2003) are distinguishable from popular individual difference constructs and explain a sizeable proportion of variance in job performance beyond cognitive ability and personality. (PsycINFO Database Record (c) 2008 APA, all rights reserved).     

A further examination of the constructs underlying assessment center ratings

Previous factor analytic work (Sackett and Dreher, 1982) has found exercise factors, rather than the expected dimension factors, underlying ratings made in managerial assessment centers. The present paper responds to criticisms that inappropriate analyses were conducted and that unrepresentative assessment centers were studied. Exercise factors were found to predominate in two new sets of assessment data which we obtained and in three studies published by other authors, thus indicating that the results were not idiosyncratic to the three centers studied by Sackett and Dreher. Analyses using confirmatory, rather than exploratory, factor analysis also support the dominance of exercise factors. Although dimension factors were found in three of four data sets studied, variables loading on a dimension factor consistently loaded more highly on an exercise factor as well.     

Ethnic and gender subgroup differences in assessment center ratings: a meta-analysis

Assessment centers are widely believed to have relatively small standardized subgroup differences (d). However, no meta-analytic review to date has examined ds for assessment centers. The authors conducted a meta-analysis of available data and found an overall Black-White d of 0.52, an overall Hispanic-White d of 0.28, and an overall male-female d of -0.19. Consistent with our expectations, results suggest that Black-White ds in assessment center data may be larger than was previously thought. Hispanic-White comparisons were smaller than were Black-White comparisons. Females, on average, scored higher than did males in assessment centers. As such, assessment centers may be associated with more adverse impact against Blacks than is portrayed in the literature, but the predictor may have less adverse impact and be more "diversity friendly" for Hispanics and females.     

Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

Moderators of sex- and race-based subgroup differences in assessment center ratings: A meta-analysis

This meta-analysis tested a series of moderators of sex- and race-based subgroup differences using assessment center (AC) field data. We found that sex-based subgroup differences favoring female assessees were smaller among studies that reported: combining AC scores with other tests to compute overall assessment ratings, lower mean correlations between rating dimensions, using more than one assessor to rate assessees in exercises, and providing assessor training. In contrast, we found larger sex-based subgroup differences favoring female assessees among studies that reported: lower proportions of females in assessee pools, conducting a job analysis to design the AC, and using multiple observations of AC dimensions across exercises. We also observed a polynomial effect showing that subgroup differences most strongly favored female assessees in jobs with the highest and lowest rates of female incumbents. We found race-based subgroup differences favoring White assessees were smaller on less cognitively loaded rating dimensions and for jobs with lower rates of Black incumbents. Studies reporting greater overall methodological rigor also showed smaller subgroup differences favoring White assessees. Regarding specific rigor features, studies reporting use of highly qualified assessors and integrating dimension ratings from separate exercises into overall dimension scores showed significantly lower differences favoring White assessees.     

Adjective ratings and dimension analyses of auditory rhythm patterns

Gabrielsson, A. Adjective ratings and dimension analyses of auditory rhythm patterns. Scand. J. psychol., 1973, 14, 244–260.-Rhythm experience for three classes of auditory rhythm stimuli, including music, was investigated by a variant of semantic differential technique in six experiments. The rating scales were chosen by means of questionnaires to musicians. The subjects rated the rhythms in a number of the scales and the ratings were subjected to factor analysis. The factors were interpreted as "uniformity-variation" or "simplicity-complexity", "accentuated first beat", "meter", "basic pattern", "rapidity", "movement characters" (floating-stuttering, dancing-walking and others), "vital-dull", "excited-calm", and "rigid-flexible".     

A test of the generalizability of a recently suggested conceptual model for assessment center ratings

The present study replicated and extended research concerning a recently suggested conceptual model of the underlying factors of dimension ratings in assessment centers (ACs) proposed by Hoffman, Melchers, Blair, Kleinmann, and Ladd that includes broad dimension factors, exercise factors, and a general performance factor. We evaluated the criterion-related validity of these different components and expanded their nomological network. Results showed that all components (i.e., broad dimensions, exercises, general performance) were significant predictors of training performance. Furthermore, broad dimensions showed incremental validity beyond exercises and general performance. Finally, relationships between the AC factors and individual difference constructs (e.g., Big Five, core self-evaluations, positive and negative affectivity) supported the construct-related validity of broad dimensions and provided further insights in the nature of the different AC components.     

Rose colored webcam: Discrepancies in personality estimates and interview performance ratings

Companies increasingly use computer-controlled interviews as a less expensive and more efficient way to screen job applicants. Despite these advantages, this interview format may prevent evaluators from accurately judging an applicant’s personality traits, which, in turn, may influence hiring decisions. Two traits in particular, agreeableness and conscientiousness, have been found to predict performance in many occupational settings. In the current research, participants randomly were assigned to either a face-to-face (FTF) or computer-controlled (CC) mock job interview. Interviewees were rated by external observers as higher in conscientiousness and agreeableness when the interview was CC rather than FTF. In addition, observers rated interview performance more positively than did the interviewees themselves – particularly when the interview was CC. Finally, the discrepancy between self and observer judgments of the interviewees’ personality (in terms of agreeableness and conscientiousness) mediated the relation between interview format and the discrepancy between self and observer ratings of interview performance. These findings suggest that CC interviews have the potential to yield overly positive evaluations of interviewees, thereby biasing personality judgments and estimations of ultimate job performance.     

The effect of dimension content on observation and ratings of job performance

Recent theory and research in the performance appraisal area (e.g., Denisi, Cafferty, & Meglino, 1984; Feldman, 1986; Ilgen & Feldman, 1983; Williams, Denisi, & Blencoe, 1985) have suggested that providing information regarding the performance dimension to be rated will cause raters to select appropriate observational schemata and, as a result, produce higher quality ratings. A study was conducted to determine if giving raters dimension-relevant information prior to performance observation would affect their attention processes and rating quality. Prior to watching a videotape of an instructor giving a lecture, 156 subjects were given either: 1) correct information, 2) incorrect information, or 3) no information regarding dimensions of performance they would subsequently be asked to rate. The results indicated that giving prior information regarding dimension content affected subjects' attention processes. Further, raters receiving no information and those receiving misinformation prior to performance observation produced less accurate ratings compared to expert raters. Ratings produced by subjects receiving correct information did not differ significantly from experts' ratings. These results are discussed in terms of both their practical and theoretical implications.     

Diagnostic importance of combined parent and teacher ratings on the Revised Behavior Problem Checklist

The Revised Behavior Problem Checklist (RBPC) was completed by parents and teachers for 105 child psychiatry outpatient boys aged 6 to 12 years. In addition, DSM-III diagnoses were determined independent of checklist results, and the children were divided into major diagnostic groups of externalizing, internalizing, and mixed disorders. Combined parent and teacher ratings proved more effective than separate ratings in distinguishing the externalizing and mixed groups from the internalizing group on the RBPC externalizing factors. The overall classification of individual boys with discriminant function analysis increased from 72% to 83% when the ratings of both parents, rather than one parent, were combined with teacher ratings. Enhancing diagnostic accuracy of type of psychiatric disorder by using combined parent and teacher behavior checklist ratings is discussed.     

The variance of d' estimates obtained in yes-no and two-interval forced choice procedures

In studies of detection and discrimination, data are often obtained in the form of a 2 x 2 matrix and then converted to an estimate of d' based on the assumptions that the underlying decision distributions are Gaussian and equal in variance. The statistical properties of the estimate of d', d' are well understood for data obtained using the yes-no procedure, but less effort has been devoted to the more commonly used two-interval forced choice (2IFC) procedure. The variance associated with d' is a function of true d' in both procedures, but for small values of true d' the variance of d' obtained using the 2IFC procedure is predicted to be less than the variance of d' obtained using yes-no; for large values of true d', the variance of d' obtained using the 2IFC procedure is predicted to be greater than the variance of d' from yes-no. These results follow from standard assumptions about the relationship between the two procedures. The present paper reviews the statistical properties of d' obtained using the two standard procedures and compares estimates of the variance of d' as a function of true d' with the variance observed in values of d' obtained with a 2IFC procedure.