不同类型的测评维度对评价中心结构效度的影响研究

本研究将测评维度分为行为能力和心理特质两组,考察对评价中心结构效度的不同影响。A公司216名部门副经理级员工接受了本次管理素质评价中心测评,采用公文筐、角色扮演和无领导小组讨论三种方法,测查行为能力和心理特质两组维度,各自有三个维度被一种以上的方法测量。多质多法和验证性因素分析的结果表明,评价中心以行为能力比以心理特质为测评维度结构效度好;以行为能力为测评维度时,会聚效度和区分效度都较好。     

评价中心的结构效度研究

评价中心虽然具备很高的预测效度,但其结构效度指标却不太理想,如研究普遍发现其汇聚效度和区分效度较低。影响评价中心结构效度的因素众多,如评分维度因素（数量和类型）、评价者因素（培训方式和人员类型）、测评方法因素（情景导向特征、特质激活潜力、测评活动形式）以及系统的观察与评价程序。该文从上述因素出发,综述了评价中心结构效度的相关研究,总结了提高评价中心结构效度的措施,并指出了未来的研究方向     

评价中心结构效度的多元概化理论分析研究

评价中心技术是现代人事测评的一种主要形式,主要通过多种模拟任务来测评高级管理人才。然而,少有研究能证明它具有理想的结构效度。该研究运用多质多法分析了某金融企业一次真实的评价中心测评项目,发现测评的结构效度并不理想。接着,通过多元概化理论的分析,探讨了结构效度不理想的原因,并指出了评价中心任务与所测维度之间的关系,以及对各维度的测量信度。最后,根据分析结果,探讨了优化评价中心结构效度和提升整体测量信度的途径。     

人事测评中的概化理论应用

概化理论是现代心理与教育测量理论之一,可应用在各种人事测评中,如表现性评价、多源评估、心理测验、结构化面试、水平测试、工作分析、评价中心等.与经典测量理论相比,概化理论应用于人事测评,表现出较强的优势,能够同时考察多种因素、确定多个维度权重等,其应用对象主要包括两大类,即企业和机构.概化理论应用于人事测评,存在应用领域、样本数据、测评效度和微观分析等问题.     

评价中心测评的评分误差分析研究

本研究应用概化理论对评价中心中的评分误差控制问题进行了系统的分析探讨,得出了以下研究结论：(1)总体上,在评价中心(以LGD为例)测评中,评价员对研究设计的测评维度的评分基本反映了被试的真实能力水平,测量误差较小;(2)评价员对合作能力、应变能力和总体印象等维度的评价标准的理解具有较高的一致性,但对决策能力等维度的评价标准的理解存在一定的偏差;(3)在评价中心(以LGD为例)测评中,采用4名评价员能达到预期的测量目标,符合测评应用的经济性、有效性原则。     

无领导小组讨论的多侧面Rasch模型应用*

采用项目反应理论(IRT)的多侧面Rasch模型(MFRM),分析评价中心技术中无领导小组讨论(LGD)的测评结果,探讨被试能力水平、评委评分宽严度、评分内部一致性、维度难度和评定等级等问题,进而讨论各种偏差。通过 MFRM 分析人事测评结果,可深入了解被试能力的真实差异、甑别维度难度、探查测评误差源,从而完善测评试题编制、评估或诊断评委合格性、提高测评维度与测评目的匹配性,为拓展项目反应理论在人事测评中的应用提供独特视角。     

评价中心的评分维度和评分效果

对近年来国内外关于评价中心的研究进行了比较系统的介绍。首先,文章讨论了评分维度的数目对于评分结果的影响,以及评价中心中的4个元维度;其次,介绍了评价中心中衡量评分效果的指标,并讨论了评分者培训的分类及其对评分效果的影响;第三,虽然评价中心具有良好的效标关联效度,但对于其结构效度的研究至今尚无统一结论。最后,文章对评价中心未来的研究趋势进行了探讨。     

评价中心法在评价中心管理潜能中的应用及其结构效度检验

本文介绍了评价中心在我国企业评价干部管理潜能中的应用,并以公文处理法在四通集团公司管理人员测评中的应用为基础,运用因素分析法对其结构效度进行了检验。研究结论对评价中心的可行性作了肯定,并提出了值得进一步探讨的问题。     

应对量表(COPE)测评维度结构研究

该文旨在对应对量表(COPE)的测评维度结构进行进一步的鉴别分析和验证研究。研究一对736名大学生的应对量表中文修订本(C-COPE)测评数据进行探索性二阶因素分析;研究二根据已有研究关于应对量表测评维度组构模式的不同结论,以及研究一的结果,提出十个假设模型,采用验证性因素分析测试这些模型与另一大学生样本(N：465)测评数据的拟合度。研究结果支持C-COPE八因子斜交模型。该量表如何进一步修订也在文中予以讨论。     

评价中心的构想效度谜题:测量维度还是活动?

评价中心是一种高保真度的情境模拟,它被设计用来在多种与工作相关的活动中测量多项维度.30年来的大量研究发现,评价中心具备良好的内容效度和效标关联效度,但构想效度却始终不理想,评价中心评分反映的总是由活动而非预先设想的维度带来的效应.这一评价中心“构想效度谜题”吸引了大量研究关注,并逐步形成了维度中心取向、活动中心取向及交互作用取向三种主要观点,分别主张控制各种误差因素以改善维度测量、放弃维度而转向活动或任务以及关注维度与活动的共同作用.未来研究应在传统的维度中心取向之外给予活动中心取向足够重视,并重点发展交互作用取向.     

EXERCISES AND DIMENSIONS ARE THE CURRENCY OF ASSESSMENT CENTERS

A novel assessment center (AC) structure that models broad dimension factors, exercise factors, and a general performance factor is proposed and supported in 4 independent samples of AC ratings. Consistent with prior research, the variance attributable to dimension and exercise factors varied widely across ACs. To investigate the construct validity of these empirically supported components of AC ratings, the nomological network of broad dimensions, exercises, and general performance was examined. Results supported the criterion‐related validity of broad dimensions and exercises as predictors of effectiveness and success criteria as well as the incremental validity of broad dimensions beyond exercises and general performance. Finally, the relationships between individual differences and AC factors supported the construct validity of broad dimension factors and provide initial insight as to the meaning of exercise specific variance and general AC performance.     

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.     

An investigation into assessment centre validity,fairness, and selection drivers

This study examined the construct‐related validity of an assessment centre (AC) developed by a national distribution company for the selection and development of lower‐grade managers. In five locations throughout Britain, 487 individuals were observed on nine dimensions, each of which was measured through six distinct exercises. Multitrait‐multimethod analyses conducted to investigate the convergent and discriminant validity of the AC revealed strong exercise (“method”) effects. This finding was corroborated by an exploratory factor analysis showing that AC ratings clustered into factors according to exercises, rather than according to performance dimensions. A series of MANOVAs and chi‐squared tests demonstrated that neither the exercise ratings nor the selection decision were biased by sex, ethnicity, or training location, and a logistic regression determined which exercises had most impact on the final decision.     

Why Assessment Centers Do Not Work the Way They Are Supposed To

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing "construct validity problem" for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task- or role-based ACs and away from traditional dimension-based ACs.     

Are assessment center behaviors' meanings consistent across exercises? A measurement invariance approach

To examine the appropriateness of a Multi‐Trait–Multi‐Method framework for testing construct validity of Assessment Centers (ACs) and get practical implications for the improved AC design, degree to which the AC dimension‐related performance behaviors consistently manifest across multiple AC rating situations was investigated. The present study used a large sample (N = 5,006) to apply a measurement invariance analysis. AC rating situations generally produced consistent factor loadings for items on AC dimensions, item residuals, dimension factor variances, and covariance between dimensions. The AC rating situation of interview tended to produce higher ratings and less item residuals. These findings support the consistency in constructs assessed across different AC rating situations, while some exercises may be better for teasing apart particular dimensions than others.     

Constructs and developmental assessment centers: Further troubling empirical findings

Research indicates that assessment center (AC) ratings typically demonstrate poor construct validity; that is, they do not measure the intended dimensions of managerial performance (e.g., Sackett & Harris, 1988). The purpose of this study was to investigate the construct validity of dimension ratings from a developmental assessment center (N=102), using multitrait-multimethod analysis and factor analysis. The relationships between AC ratings, job performance ratings, and personality measures also were investigated. Results indicate that the AC ratings failed to demonstrate construct validity. The ratings did not show the expected relationships with the job performance and personality measures. Additionally, the factors underlying these ratings were found to be the AC exercises, rather than the managerial dimensions as expected. Potentially, this lack of construct validity of the dimension ratings is a serious problem for a developmental assessment center. There is little evidence that the managerial weaknesses identified by the AC are the dimensions that actually need to be improved on the job. Methods are discussed for improving the construct validity of AC ratings, for example, by decreasing the cognitive demands on the assessors.This study is based on a dissertation submitted to North Carolina State University. Portions of this paper were presented at the meeting of the Society for Industrial and Organizational Psychology in Montreal, Quebec, May, 1992. I am grateful to Paul Thayer, Bert Westbrook, James W. Cunningham, and Patrick Hauenstein for their contributions to this research. I also thank several anonymous reviewers for their comments on this article.     

Impact of Common Rater Variance on Construct Validity of Assessment Center Dimension Judgments

In an assessment center (AC), assessors generally rate an applicant's performance on multiple dimensions in just 1 exercise. This rating procedure introduces common rater variance within exercises but not between exercises. This article hypothesizes that this phenomenon is partly responsible for the consistently reported result that the AC lacks construct validity. Therefore, in this article, the rater effect is standardized on discriminant and convergent validity via a multitrait-multimethod design in which each matrix cell is based on ratings of different assessors. Two independent studies (N = 200, N = 52) showed that, within exercises, correlations decrease when common rater variance is excluded both across exercises (by having assessors rate only 1 exercise) and within exercises (by having assessors rate only 1 dimension per exercise). Implications are discussed in the context of the recent discussion around the appropriateness of the within-exercise versus the within-dimension evaluation method.     

Moderators of sex- and race-based subgroup differences in assessment center ratings: A meta-analysis

This meta-analysis tested a series of moderators of sex- and race-based subgroup differences using assessment center (AC) field data. We found that sex-based subgroup differences favoring female assessees were smaller among studies that reported: combining AC scores with other tests to compute overall assessment ratings, lower mean correlations between rating dimensions, using more than one assessor to rate assessees in exercises, and providing assessor training. In contrast, we found larger sex-based subgroup differences favoring female assessees among studies that reported: lower proportions of females in assessee pools, conducting a job analysis to design the AC, and using multiple observations of AC dimensions across exercises. We also observed a polynomial effect showing that subgroup differences most strongly favored female assessees in jobs with the highest and lowest rates of female incumbents. We found race-based subgroup differences favoring White assessees were smaller on less cognitively loaded rating dimensions and for jobs with lower rates of Black incumbents. Studies reporting greater overall methodological rigor also showed smaller subgroup differences favoring White assessees. Regarding specific rigor features, studies reporting use of highly qualified assessors and integrating dimension ratings from separate exercises into overall dimension scores showed significantly lower differences favoring White assessees.