AN EXAMINATION OF THE EFFECTS OF USING BEHAVIOR CHECKLISTS ON THE CONSTRUCT VALIDITY OF ASSESSMENT CENTER DIMENSIONS

Although research has established the criterion-related validity of assessment centers for selection purposes, the construct validity of dimension ratings has not been demonstrated. A quasi-experimental design was used to investigate the influence of retranslated behavior checklists on the construct validity of dimension ratings for two assessment center exercises. Assessor use of behavior checklists increased the average convergent (i.e., same dimension across exercise) validity from .24 to .43 while decreasing the average discriminant (i.e., different dimension within exercise) validity (.47 to .41). Behavior checklist sums were moderately correlated with corresponding dimension ratings and demonstrated a comparable level of construct validity. It is suggested that using behavior checklists may improve dimension construct validity by reducing the cognitive demands placed on raters.     

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.     

CONSTRUCT VALIDITY OF TWO CATEGORIES OF ASSESSMENT CENTER DIMENSION RATINGS

The construct validity of assessment center final dimension ratings was examined within a nomological network of cognitive and personality measures. Four hundred forty-one employees of a large mid-western petroleum company were assessed on 11 dimensions in two broad categories and completed four tests. Results showed that several cognitive ability measures related more strongly to performance-style dimension ratings than to interpersonal-style dimension ratings, providing evidence for convergent and discriminant validity. Correlation analysis and factor analysis support the two a priori interpersonal- and performance-style categories. The results suggest that final dimension ratings possess construct validity and that assessors can differentiate between two broad categories of assessment dimensions.     

Dimension and exercise variance in assessment center scores: a large-scale evaluation of multitrait-multimethod studies.

This study addresses 3 questions regarding assessment center construct validity: (a) Are assessment center ratings best thought of as reflecting dimension constructs (dimension model), exercises (exercise model), or a combination? (b) To what extent do dimensions or exercises account for variance? (c) Which design characteristics increase dimension variance? To this end, a large set of multitrait-multimethod studies (N = 34) were analyzed, showing that assessment center ratings were best represented (i.e., in terms of fit and admissible solutions) by a model with correlated dimensions and exercises specified as correlated uniquenesses. In this model, dimension variance equals exercise variance. Significantly more dimension variance was found when fewer dimensions were used and when assessors were psychologists. Use of behavioral checklists, a lower dimension-exercise ratio, and similar exercises also increased dimension variance.     

评价中心的构想效度和结构模型

采用多质多法和验证性因素分析的方法,对以无领导小组讨论、文件筐和人格测验构成的一个评价中心的构想效度和结构模型进行了研究。通过对136名被试在四个测评维度上的施测,其结果表明,在评价中心中会聚效度低于区分效度,影响评价中心测评结果的主要因素是测评方法而不是测评维度,从而得到了一个以测评方法为潜变量的评价中心结构模型。从该结构模型来看,评价中心之所以起作用是由于其多个测评方法(情景)的结果。表明测评情景对于构建评价中心有着至关重要的作用。     

EXERCISES AND DIMENSIONS ARE THE CURRENCY OF ASSESSMENT CENTERS

A novel assessment center (AC) structure that models broad dimension factors, exercise factors, and a general performance factor is proposed and supported in 4 independent samples of AC ratings. Consistent with prior research, the variance attributable to dimension and exercise factors varied widely across ACs. To investigate the construct validity of these empirically supported components of AC ratings, the nomological network of broad dimensions, exercises, and general performance was examined. Results supported the criterion‐related validity of broad dimensions and exercises as predictors of effectiveness and success criteria as well as the incremental validity of broad dimensions beyond exercises and general performance. Finally, the relationships between individual differences and AC factors supported the construct validity of broad dimension factors and provide initial insight as to the meaning of exercise specific variance and general AC performance.     

IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

A test of the generalizability of a recently suggested conceptual model for assessment center ratings

The present study replicated and extended research concerning a recently suggested conceptual model of the underlying factors of dimension ratings in assessment centers (ACs) proposed by Hoffman, Melchers, Blair, Kleinmann, and Ladd that includes broad dimension factors, exercise factors, and a general performance factor. We evaluated the criterion-related validity of these different components and expanded their nomological network. Results showed that all components (i.e., broad dimensions, exercises, general performance) were significant predictors of training performance. Furthermore, broad dimensions showed incremental validity beyond exercises and general performance. Finally, relationships between the AC factors and individual difference constructs (e.g., Big Five, core self-evaluations, positive and negative affectivity) supported the construct-related validity of broad dimensions and provided further insights in the nature of the different AC components.     

Constructs and developmental assessment centers: Further troubling empirical findings

Research indicates that assessment center (AC) ratings typically demonstrate poor construct validity; that is, they do not measure the intended dimensions of managerial performance (e.g., Sackett & Harris, 1988). The purpose of this study was to investigate the construct validity of dimension ratings from a developmental assessment center (N=102), using multitrait-multimethod analysis and factor analysis. The relationships between AC ratings, job performance ratings, and personality measures also were investigated. Results indicate that the AC ratings failed to demonstrate construct validity. The ratings did not show the expected relationships with the job performance and personality measures. Additionally, the factors underlying these ratings were found to be the AC exercises, rather than the managerial dimensions as expected. Potentially, this lack of construct validity of the dimension ratings is a serious problem for a developmental assessment center. There is little evidence that the managerial weaknesses identified by the AC are the dimensions that actually need to be improved on the job. Methods are discussed for improving the construct validity of AC ratings, for example, by decreasing the cognitive demands on the assessors.This study is based on a dissertation submitted to North Carolina State University. Portions of this paper were presented at the meeting of the Society for Industrial and Organizational Psychology in Montreal, Quebec, May, 1992. I am grateful to Paul Thayer, Bert Westbrook, James W. Cunningham, and Patrick Hauenstein for their contributions to this research. I also thank several anonymous reviewers for their comments on this article.     

中国人人格量表的信度与效度

在初步确定中国人人格量表（Qingnian Zhongguo Personality Scale, QZPS）的基础上,通过对5000多被试的施测,确定了QZPS的项目构成及信度和效度。研究中对QZPS的项目重新进行探索性因素分析,确认了中国人人格的七因素结构,根据对每一个大因素的因素分析和项目鉴别度分析,确定了QZPS各个因素的项目构成和内部一致性系数,共180个项目。通过自-他评定的相关、自评与标尺评定的相关、以及对特殊群体分数分布的分析验证了QZPS的效度。研究结果确认QZPS有良好的信度和效度,可以应用于人格的理论和应用领域。     

A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed.     

评价中心的构想效度谜题:测量维度还是活动?

评价中心是一种高保真度的情境模拟,它被设计用来在多种与工作相关的活动中测量多项维度.30年来的大量研究发现,评价中心具备良好的内容效度和效标关联效度,但构想效度却始终不理想,评价中心评分反映的总是由活动而非预先设想的维度带来的效应.这一评价中心“构想效度谜题”吸引了大量研究关注,并逐步形成了维度中心取向、活动中心取向及交互作用取向三种主要观点,分别主张控制各种误差因素以改善维度测量、放弃维度而转向活动或任务以及关注维度与活动的共同作用.未来研究应在传统的维度中心取向之外给予活动中心取向足够重视,并重点发展交互作用取向.     

AN EVALUATION OF THE VALIDITY OF THE PERSONALITY RESEARCH FORM WITH FILIPINO UNIVERSITY STUDENTS

The cross-cultural validity of a North American personality inventory, namely, the Personality Research Form (Jackson 1984) was examined using 394 university students in the Philippines who were able to speak and read English. Scale validities, with self and peer ratings as criteria, were generally significant but modest. Moderate scale and peer rating reliabilities probably contributed to these results. Elevated scores on a PRF scale designed to detect careless responding suggested failure to understand instructions or insufficient motivation may also account for the findings. Interestingly, recalculating validities for subsamples comprising ‘dependable’ and ‘undependable’ subjects yielded no substantial differences in overall validity. Implications for cross-cultural personality assessment are discussed.     

Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

USING A META-ANALYTIC PERSPECTIVE TO ENHANCE JOB COMPONENT VALIDATION

This paper develops synthetic validity estimates based on a meta-analytic-weighted least squares (WLS) approach to job component validity (JCV), using position analysis questionnaire (PAQ) estimates of job characteristics, and the Data, People, & Things ratings from the  Dictionary of Occupational Titles  as indices of job complexity. For the general aptitude test battery database of 40,487 employees, nine validity coefficients were estimated for 192 positions. The predicted validities from the WLS approach had lower estimated variability than would be obtained from either the classic JCV approach or local criterion-related validity studies. Data, People, & Things summary ratings did not consistently moderate validity coefficients, whereas the PAQ data did moderate validity coefficients. In sum, these results suggest that synthetic validity procedures should incorporate a WLS regression approach. Moreover, researchers should consider a comprehensive set of job characteristics when considering job complexity rather than a single aggregated index.     

Predicting management career success from assessment center data: A longitudinal study

The authors investigated trends over a 13-year period in the predictive validity of assessment center data for objective career advancement for a sample of 456 academic graduates. Using year of entry and tenure as controls, findings from research into managerial effectiveness and development were confirmed with respect to the importance of interpersonal effectiveness, firmness, and ambition for long-term career success. Within the long-tenured group the validity of the overall assessment rating (OAR) corrected for initial differences in starting salaries and for restriction in range, was 0.35. Substantial long-term validities were obtained for ratings from the assessment center exercises. Computation of validities after three to five years of tenure of the same cohort in addition revealed some dynamic validity patterns. For instance, the OAR was important for early and late career success only for the group that was still present after 12 years, and its validity increased over time.     

A new frame for frame-of-reference training: enhancing the construct validity of assessment centers

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (a) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (b) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (c) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice.     

The Effects of Social Desirability and Faking on Personality and Integrity Assessment for Personnel Selection

A review of the extant literature and new empirical research suggests that social desirability is not much of a concern in personality and integrity testing for personnel selection. In particular, based on meta-analytically derived evidence, it appears that social desirability influences do not destroy the convergent and discriminant validity of the Big Five dimensions of personality (Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness). We also present new empirical evidence regarding gender and age differences in socially desirable re- sponding. Although social desirability predicts a number of important work variables such as job satisfaction, organizational commitment, and supervisor ratings of training success, social desirability does not seem to be a predictor of overall job performance and is only very weakly related to specific dimensions of job performance such as technical proficiency (r = -.07) and personal discipline ( r = .05). Large sample investigations of the moderating influences of social desirability in actual work settings indicate that social desirability does not moderate the criterion-related validities of personality variables or integrity tests. The criterion-related validity of integrity tests for overall job performance with applicant samples in predictive studies is .41. Controlling for social desirability in integrity or personality test scores leaves the operational validities intact, thereby suggesting that social desirability functions neither as a mediator nor as a suppressor variable in personality-performance.     

IMPLICATIONS OF TRAIT-ACTIVATION THEORY FOR EVALUATING THE CONSTRUCT VALIDITY OF ASSESSMENT CENTER RATINGS

Assessment centers have often been criticized for lacking evidence supporting the construct validity of dimension ratings. This study examines whether the poor convergence of assessment center ratings is a result of correlating ratings from exercises that differ in the extent that behavior relevant to personality traits can be observed. Using data from a promotional assessment center for law enforcement officers ( n = 79), the convergence of assessment center ratings was evaluated within the context of the five factor model by comparing the average within-dimension correlation of ratings from exercises that allowed for more opportunity to observe trait-relevant behavior to the average of those involving exercises where there was less opportunity. For each personality trait, ratings from exercises judged by experts to be high in trait-activation potential displayed stronger convergence (mean r = .30) than did ratings from exercises that were low in activation potential for that trait (mean r = .15). Implications for evaluating the construct validity of assessment centers are discussed along with future directions for classifying exercises based on situational similarity.     

Personality Correlates of Assessment Center Consensus Competency Ratings: Evidence from Russia

Controversy has revolved around whether assessment center ratings have construct validity to measure intended dimensions of managerial performance. In contrast to much recent research on the internal structure of assessment center ratings, the present studies investigated the relationship of final competency ratings derived by consensus discussion with external questionnaire measures of personality characteristics. Expanding on previous studies showing correlations of dimension scores in relation to individual trait measures, this study investigated the relationship of complex competencies with both single personality traits and with composites of personality traits. Evidence from two samples of managers in Russia shows that final competency ratings are related to predicted composites of personality factors more consistently than to single factors. Taken together, these findings provide evidence that assessment center ratings derived by consensus discussion show construct validity in relationship with predicted composites of personality characteristics.