评价中心的结构效度研究

评价中心虽然具备很高的预测效度,但其结构效度指标却不太理想,如研究普遍发现其汇聚效度和区分效度较低。影响评价中心结构效度的因素众多,如评分维度因素（数量和类型）、评价者因素（培训方式和人员类型）、测评方法因素（情景导向特征、特质激活潜力、测评活动形式）以及系统的观察与评价程序。该文从上述因素出发,综述了评价中心结构效度的相关研究,总结了提高评价中心结构效度的措施,并指出了未来的研究方向     

Trying to understand the different pieces of the construct validity puzzle of assessment centers: an examination of assessor and assessee effects

This study examined the effects of assessor-related factors (i.e., type of assessor) and assessee-related factors (i.e., type of assessee profile) on the construct validity of assessment center ratings. In particular, 3 types of assessors (26 industrial/organizational [I/O] psychologists, 20 managers, and 27 students), rated assessee performances that varied according to cross-exercise consistency (i.e., relatively inconsistent vs. relatively consistent) and dimension differentiation (relatively undifferentiated vs. relatively differentiated). Construct validity evidence was established for only one assessee profile and only in the I/O psychologist and managerial samples. More generally, these results indicate that 3 factors (poor design, assessor unreliability, and especially cross-situational inconsistent assessee performances) may explain why construct validity evidence is often not established in operational assessment centers.     

An investigation into assessment centre validity,fairness, and selection drivers

This study examined the construct‐related validity of an assessment centre (AC) developed by a national distribution company for the selection and development of lower‐grade managers. In five locations throughout Britain, 487 individuals were observed on nine dimensions, each of which was measured through six distinct exercises. Multitrait‐multimethod analyses conducted to investigate the convergent and discriminant validity of the AC revealed strong exercise (“method”) effects. This finding was corroborated by an exploratory factor analysis showing that AC ratings clustered into factors according to exercises, rather than according to performance dimensions. A series of MANOVAs and chi‐squared tests demonstrated that neither the exercise ratings nor the selection decision were biased by sex, ethnicity, or training location, and a logistic regression determined which exercises had most impact on the final decision.     

EXERCISES AND DIMENSIONS ARE THE CURRENCY OF ASSESSMENT CENTERS

A novel assessment center (AC) structure that models broad dimension factors, exercise factors, and a general performance factor is proposed and supported in 4 independent samples of AC ratings. Consistent with prior research, the variance attributable to dimension and exercise factors varied widely across ACs. To investigate the construct validity of these empirically supported components of AC ratings, the nomological network of broad dimensions, exercises, and general performance was examined. Results supported the criterion‐related validity of broad dimensions and exercises as predictors of effectiveness and success criteria as well as the incremental validity of broad dimensions beyond exercises and general performance. Finally, the relationships between individual differences and AC factors supported the construct validity of broad dimension factors and provide initial insight as to the meaning of exercise specific variance and general AC performance.     

Impact of Common Rater Variance on Construct Validity of Assessment Center Dimension Judgments

In an assessment center (AC), assessors generally rate an applicant's performance on multiple dimensions in just 1 exercise. This rating procedure introduces common rater variance within exercises but not between exercises. This article hypothesizes that this phenomenon is partly responsible for the consistently reported result that the AC lacks construct validity. Therefore, in this article, the rater effect is standardized on discriminant and convergent validity via a multitrait-multimethod design in which each matrix cell is based on ratings of different assessors. Two independent studies (N = 200, N = 52) showed that, within exercises, correlations decrease when common rater variance is excluded both across exercises (by having assessors rate only 1 exercise) and within exercises (by having assessors rate only 1 dimension per exercise). Implications are discussed in the context of the recent discussion around the appropriateness of the within-exercise versus the within-dimension evaluation method.     

AN EXAMINATION OF IMPRESSION MANAGEMENT USE AND EFFECTIVENESS ACROSS ASSESSMENT CENTER EXERCISES: THE ROLE OF COMPETENCY DEMANDS

We report 2 studies that examine how promotional candidates use verbal and nonverbal impression management (IM) tactics across several structured assessment center exercises that differ in the competency demands they place on candidates. Based on the competency-demand hypothesis ( Shoda, Mischel, & Wright, 1993a, 1993b ), it was predicted that IM use would occur most frequently and have the strongest effects on assessor evaluations in exercises that place greater demands on candidates' interpersonal skills than in exercises that depend primarily on technical skills. In both studies, IM tactics were generally used more frequently and there was more variability in IM use for those exercises requiring candidates to display interpersonal competencies (i.e., the role-plays and mock presentation) relative to the exercise that did not (i.e., the tactical exercise). The relationship between IM use and assessor evaluations was also influenced by the competencies assessed by the exercises, and IM use related to both interpersonal and noninterpersonal ratings of performance.     

AN EXAMINATION OF THE EFFECTS OF USING BEHAVIOR CHECKLISTS ON THE CONSTRUCT VALIDITY OF ASSESSMENT CENTER DIMENSIONS

Although research has established the criterion-related validity of assessment centers for selection purposes, the construct validity of dimension ratings has not been demonstrated. A quasi-experimental design was used to investigate the influence of retranslated behavior checklists on the construct validity of dimension ratings for two assessment center exercises. Assessor use of behavior checklists increased the average convergent (i.e., same dimension across exercise) validity from .24 to .43 while decreasing the average discriminant (i.e., different dimension within exercise) validity (.47 to .41). Behavior checklist sums were moderately correlated with corresponding dimension ratings and demonstrated a comparable level of construct validity. It is suggested that using behavior checklists may improve dimension construct validity by reducing the cognitive demands placed on raters.     

Dimension and exercise variance in assessment center scores: a large-scale evaluation of multitrait-multimethod studies.

This study addresses 3 questions regarding assessment center construct validity: (a) Are assessment center ratings best thought of as reflecting dimension constructs (dimension model), exercises (exercise model), or a combination? (b) To what extent do dimensions or exercises account for variance? (c) Which design characteristics increase dimension variance? To this end, a large set of multitrait-multimethod studies (N = 34) were analyzed, showing that assessment center ratings were best represented (i.e., in terms of fit and admissible solutions) by a model with correlated dimensions and exercises specified as correlated uniquenesses. In this model, dimension variance equals exercise variance. Significantly more dimension variance was found when fewer dimensions were used and when assessors were psychologists. Use of behavioral checklists, a lower dimension-exercise ratio, and similar exercises also increased dimension variance.     

Preventing adolescent smoking: what have we learned about treatment construct validity?

In recent years, numerous programs introduced to prevent adolescent smoking have demonstrated some success. This paper reviews the treatment construct validity of such programs; that is, we seek to determine how and why programs reduce adolescent smoking. The review leads to the conclusion that little is presently known about the construct validity of successful programs, a problem that results primarily from the neglect of process assessment and analyses. The advantages and disadvantages of several future research approaches are discussed, including: (a) utilization of process measures within large scale treatment/no-treatment designs, (b) small-scale studies to test the effects of prevention components on process measures (e.g., attitudes, intentions to smoke), and (c) combinations of these two approaches.     

The Construct Validity of Assessment Centre Performance Dimensions

Evidence regarding the construct validity of assessment centre performance dimensions is reviewed. The evidence strongly suggests that variance in ratings tends to reflect exercises more than individual performance dimensions, thus calling into question the construct validity and utility of these dimensions. A number of biases in the assessment centre process, as well as more general rating biases are noted that may be responsible for these pervasive exercise effects. Suggestions are made for enhancing the construct validity of performance dimensions.     

Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity

This study compares the effects of data-driven assessor training with schema-driven assessor training and control training. The sample consisted of 229 industrial and organizational psychology students and 161 managers who were randomly assigned to 1 of these training strategies. Participants observed and rated candidates in an assessment center exercise. The data-driven and schema-driven assessor training approaches outperformed the control training on all 3 dependent variables. The schema-driven assessor training resulted in the largest values of interrater reliability, dimension differentiation, and accuracy. Managers provided significantly more accurate ratings than students but distinguished less between the dimensions. Practical implications regarding the design of assessor trainings and the composition of assessor teams are proposed.     

Assessor Constructs in Use as the Missing Component in Validation of Assessment Center Dimensions: A critique and directions for research

The basis for assessment center dimension ratings has been examined through the lens of various multitrait–multimethod approaches, with some researchers concluding that dimension ratings are not representative of meaningful constructs. This presents a serious challenge for those who would generalize predictor constructs, particularly those that are broadly articulated in management assessment center dimensions. This paper addresses the problems with applying such analyses to assessment center dimension ratings without first articulating expected construct relationships. Ackerman and Humphreys' analysis of constructs will be used to better frame assessment center dimensions and to articulate possible assessor constructs in use. Theories of performance held by the managers, who serve both as subject matter experts when dimensional constructs are initially defined, and as the assessors who impose these dimensions on assessees' behavior in assessment center ratings, are suggested as important bases for further construct validation. Approaches for generalization of assessor constructs in use to the managerial work domain will be discussed.     

Extending the Nomological Network of Assessment Center Construct Validity: Prediction of Cross-Situationally Consistent and Specific Aspects of Assessment Center Performance

The purpose of this study was to expand the nomological validity of assessment centers (ACs) by investigating predictors of cross-situationally consistent versus specific aspects of AC performance. Consistent with hypotheses, (a) Big Five personality factors predicted AC performance as it related to a cross-situationally consistent general performance factor but not as it related to exercise (i.e., situationally specific) factors, and (b) job knowledge predicted performance as it related to both the general performance factor and exercise-specific factors. Results are interpreted as they relate to the growing literature on AC construct validity.     

Methodological Issues in the Direct Observation of Parent–Child Interaction: Do Observational Findings Reflect the Natural Behavior of Participants?

This review examines evidence for the utility and validity of direct observational techniques for answering particular research and clinical questions. Observational techniques often involve recording behavior in settings that are relatively unnatural for families. However, it is argued that construct validity of observational methods depends partly on whether the findings are representative of participants' typical everyday behavior. Evidence is reviewed concerning whether observational findings are affected by the presence of the observer, and by two factors which have been neglected in the literature, namely the type of task imposed by the observer (e.g., directing parent and child to play rather than observing spontaneous interaction) and the location of the observations (e.g., clinic or laboratory rather than home). The review suggests that the presence of an observer does not necessarily distort the nature of interactions. However, the small number of studies in this area suggest that interactions in structured or artificial settings are not necessarily representative of those normally taking place at home.     

A Different Look at Assessment Centers: Views of Assessment Center Users

This study aims to shed light on possible problems of assessment center users and designers when developing and implementing assessment centers. Semi-structured interviews with a representative sample of assessment center users in Flanders revealed that, besides a large variability in assessment center practice, practitioners experience problems with dimension selection and definition, exercise design, line/staff managers as assessors, distinguishing between observation and evaluation, and with the content of assessor training programs. Solutions for these problems are suggested.     

Personality Correlates of Assessment Center Consensus Competency Ratings: Evidence from Russia

Controversy has revolved around whether assessment center ratings have construct validity to measure intended dimensions of managerial performance. In contrast to much recent research on the internal structure of assessment center ratings, the present studies investigated the relationship of final competency ratings derived by consensus discussion with external questionnaire measures of personality characteristics. Expanding on previous studies showing correlations of dimension scores in relation to individual trait measures, this study investigated the relationship of complex competencies with both single personality traits and with composites of personality traits. Evidence from two samples of managers in Russia shows that final competency ratings are related to predicted composites of personality factors more consistently than to single factors. Taken together, these findings provide evidence that assessment center ratings derived by consensus discussion show construct validity in relationship with predicted composites of personality characteristics.     

Same‐gender and same‐race bias in assessment center ratings: A rating error approach to understanding subgroup differences

This study investigated leniency and similar‐to‐me bias as mechanisms underlying demographic subgroup differences among assessees in assessors’ initial dimension ratings from three assessment center (AC) simulation exercises used as part of high‐stakes promotional testing. It examined whether even small individual‐level effects can accumulate (i.e., “trickle‐up”) to produce larger subgroup‐level differences. Individual‐level analyses were conducted using cross‐classified multilevel modeling and conducted separately for each exercise. Results demonstrated weak evidence of leniency toward White assessees and similar‐to‐me bias among non‐White assessee–assessor pairs. Similar leniency was found toward female assessees, but no statistically significant effects were found for assessee or assessor gender or assessee–assessor gender similarity. Using traditional d effect size estimates, weak individual level assessee effects translated into small but consistent subgroup differences favoring White and female assessees. Generally small but less consistent subgroup differences indicated that non‐White and male assessors gave higher ratings. Moreover, analyses of overall promotion decisions indicate the absence of adverse impact. Findings from this AC provide some support for the “trickle‐up” effect, but the effect on subgroup differences is trivial. The results counter recent reviews of AC studies suggesting larger than previously assumed subgroup differences. Consequently, the findings demonstrate the importance of following established best practices when developing and implementing the AC method for selection purposes to minimize subgroup differences.     

EXAMINING SELECTION UTILITY WHERE COMPETING PREDICTORS DIFFER IN ADVERSE IMPACT

This paper discusses the roles of validity, cut score choice, and adverse impact on selection system utility using data from two concurrent validation studies. We contrast an assessment center and published aptitude test on several metrics, including validity, testing costs, adverse impact, and utility. The assessment center produced slightly lower validity than the aptitude test while costing roughly 10 times as much per candidate. In spite of these advantages for the aptitude test, the assessment center produced so much less adverse impact its operational utility would be higher given cut scores likely to be chosen in this organization. Potential concerns with applying net utility models to this type of situation are discussed in comparison to gross utility models.     

THE ROLE OF COGNITIVE ABILITY IN THE SUBGROUP DIFFERENCES AND INCREMENTAL VALIDITY OF ASSESSMENT CENTER EXERCISES

This study investigates the degree to which subgroup (Black-White) mean differences on various assessment center exercises (e.g., in-basket, role play) may be a function of the type of exercise employed; and furthermore, begins to explore why these different types of exercises result in subgroup differences. The sample consisted of 633 participants who completed a managerial assessment center that evaluated them on 14 ability dimensions across 7 different types of assessment exercises. In addition, each participant completed a cognitive ability measure. The results suggest that subgroup differences varied by type of assessment exercise; and furthermore that the subgroup difference appeared to be a function of the cognitive component of the exercise. Lastly, preliminary support is found that the validity of some of the assessment center exercises in predicting supervisor ratings of job performance is based, in part, on their cognitive component; however, evidence of incremental validity does exist.     

Constructs and developmental assessment centers: Further troubling empirical findings

Research indicates that assessment center (AC) ratings typically demonstrate poor construct validity; that is, they do not measure the intended dimensions of managerial performance (e.g., Sackett & Harris, 1988). The purpose of this study was to investigate the construct validity of dimension ratings from a developmental assessment center (N=102), using multitrait-multimethod analysis and factor analysis. The relationships between AC ratings, job performance ratings, and personality measures also were investigated. Results indicate that the AC ratings failed to demonstrate construct validity. The ratings did not show the expected relationships with the job performance and personality measures. Additionally, the factors underlying these ratings were found to be the AC exercises, rather than the managerial dimensions as expected. Potentially, this lack of construct validity of the dimension ratings is a serious problem for a developmental assessment center. There is little evidence that the managerial weaknesses identified by the AC are the dimensions that actually need to be improved on the job. Methods are discussed for improving the construct validity of AC ratings, for example, by decreasing the cognitive demands on the assessors.This study is based on a dissertation submitted to North Carolina State University. Portions of this paper were presented at the meeting of the Society for Industrial and Organizational Psychology in Montreal, Quebec, May, 1992. I am grateful to Paul Thayer, Bert Westbrook, James W. Cunningham, and Patrick Hauenstein for their contributions to this research. I also thank several anonymous reviewers for their comments on this article.