评价中心的结构效度研究

评价中心虽然具备很高的预测效度,但其结构效度指标却不太理想,如研究普遍发现其汇聚效度和区分效度较低。影响评价中心结构效度的因素众多,如评分维度因素（数量和类型）、评价者因素（培训方式和人员类型）、测评方法因素（情景导向特征、特质激活潜力、测评活动形式）以及系统的观察与评价程序。该文从上述因素出发,综述了评价中心结构效度的相关研究,总结了提高评价中心结构效度的措施,并指出了未来的研究方向     

Impact of Common Rater Variance on Construct Validity of Assessment Center Dimension Judgments

In an assessment center (AC), assessors generally rate an applicant's performance on multiple dimensions in just 1 exercise. This rating procedure introduces common rater variance within exercises but not between exercises. This article hypothesizes that this phenomenon is partly responsible for the consistently reported result that the AC lacks construct validity. Therefore, in this article, the rater effect is standardized on discriminant and convergent validity via a multitrait-multimethod design in which each matrix cell is based on ratings of different assessors. Two independent studies (N = 200, N = 52) showed that, within exercises, correlations decrease when common rater variance is excluded both across exercises (by having assessors rate only 1 exercise) and within exercises (by having assessors rate only 1 dimension per exercise). Implications are discussed in the context of the recent discussion around the appropriateness of the within-exercise versus the within-dimension evaluation method.     

Extending the Nomological Network of Assessment Center Construct Validity: Prediction of Cross-Situationally Consistent and Specific Aspects of Assessment Center Performance

The purpose of this study was to expand the nomological validity of assessment centers (ACs) by investigating predictors of cross-situationally consistent versus specific aspects of AC performance. Consistent with hypotheses, (a) Big Five personality factors predicted AC performance as it related to a cross-situationally consistent general performance factor but not as it related to exercise (i.e., situationally specific) factors, and (b) job knowledge predicted performance as it related to both the general performance factor and exercise-specific factors. Results are interpreted as they relate to the growing literature on AC construct validity.     

Situational Tests in Student Selection: An Examination of Predictive Validity,Adverse Impact,and Construct Validity

The Flemish Admission Exam ‘Medical and Dental Studies’ is comprised of four cognitive ability tests and four situational tests, namely two work samples (i.e., a lecture and a medical text) and two video‐based situational judgement tests (i.e., a physician–patient interaction and a medical expert discussion). On the basis of the Admission Exam scores of 941 candidates (359 men, 582 women) this study shows that situational tests significantly can predict better than cognitive ability tests, with lecture and text emerging as significant predictors. When situational tests are combined with cognitive ability tests, there are no mean gender differences. Situational tests also enable us to measure a broader range of constructs. For example, in this study, the personality factor Openness is related to better situational test performance. Overall, this study demonstrates that situational tests may be a useful complement to traditional student selection procedures.     

Assessment Center Graduate Selection: Decision Processes,Validity, and Evaluation by Candidates

In this contribution, results are presented of research on the predictive validity of a procedure for the career-oriented selection of graduates in The Netherlands. After selection on account of applicant letter, the selection procedure consists of four selective steps: interview with a ‘recruitment officer’, mental test, interview with line manager, and an assessment center comprising two situational exercises. Performance indicators investigated are present salary, average annual changes in function, appraisal of performance, appraisal of management potential, and average annual salary progression. We report on the relationships between the predictors, between the criteria, and between predictors and criteria. The assessment center, together with other steps from the selection procedure, appear to be valid for a number of performance indicators. The correlation between the overall assessment center rating and present salary, corrected for restriction in range, equals 0.55. Stepwise regression indicates that elements from all procedure steps contribute to the prediction of the criterion of average annual salary progression. The disturbing influence of possible ‘crown prince’ -effects on these results, are discussed. Also, the decision process taking place during the end meeting of the assessors is investigated. Finally, the evaluation of the procedure, both by accepted and by rejected candidates, is discussed. These and other results are compared to what is generally known about assessment centers.     

Transparency: A Necessary Requirement for the Construct Validity of Assessment Centres

La validité de construction des centres d'évaluation et la performance perçue des candidats sont-elles sensibles au fait que ceux-ci aient ou non connaissance des dimensions jugées pertinentes? 119 étudiants de 1° cycle ont participéà une session de centre d'évaluation dans des conditions de transparence et de non transparence. II est apparu que la validité de construction était supérieure quand les participants connaissaient les dimensions recherchées et le comportement attendu de leur part. En outre, les notations des sujets de la condition "transparence" qui admettaient avoir soumis leur conduite aux dimensions attendues bénéficiaient d'une validité discriminante et convergente supérieure à celle des notations des sujets qui affirmaient n'avoir tenu aucun compte des dites dimensions. L'impact de la transparence des dimensions recherchées sur la validité de construction des centres d'évaluation est discuté.
The present study tested whether the construct validity of assessment centres and the perceived performance of candidates were influenced by whether or not the requirement dimensions were made transparent for participants. A total of 119 college students participated in an assessment centre under either transparent or nontransparent conditions. Results showed that construct validity was higher when participants knew the requirement dimensions and what behaviour was required of them. In addition, the ratings of subjects in the transparency condition who reported that they had oriented their behaviour towards the requirement dimensions had higher convergent and discriminant validity than the ratings of subjects who reported that they disregarded the requirement dimensions. The consequences of the transparency of requirement dimensions for the construct validity of assessment centres are discussed.     

不同类型的测评维度对评价中心结构效度的影响研究

本研究将测评维度分为行为能力和心理特质两组,考察对评价中心结构效度的不同影响。A公司216名部门副经理级员工接受了本次管理素质评价中心测评,采用公文筐、角色扮演和无领导小组讨论三种方法,测查行为能力和心理特质两组维度,各自有三个维度被一种以上的方法测量。多质多法和验证性因素分析的结果表明,评价中心以行为能力比以心理特质为测评维度结构效度好;以行为能力为测评维度时,会聚效度和区分效度都较好。     

Negative Psychological Effects of Selection Methods: Construct Formulation and an Empirical Investigation into an Assessment Center

This paper defines and formulates the construct of negative psychological effects (NPEs) of selection and assessment methods upon applicants. The results of an empirical study into NPEs in an operational assessment center are also reported. Applicants (n=107) completed measures at three timepoints: Time 1 (just before the AC), Time 2 (immediately after the AC but before outcome decisions were known), and Time 3 (6 months after the AC). Both accepted and rejected candidates completed all three measures, which included self‐esteem, mental health, positive and negative affect, and career exploration behavior. No evidence of NPEs was found for rejected candidates despite significant between‐group differences on feedback reactions level items. Interestingly, well‐being and positive affect declined slightly for successful candidates at subsequent measurement compared against baseline Time 1 norms. Reactions level outcomes did, however, differ significantly between accepted and rejected candidates, with the latter rating feedback dimensions far less favorably. These results are discussed in relation to future research into the psychological impact of selection procedures upon candidates and the need for applied psychologists and HR practitioners to demonstrate that assessment methods do not exhibit NPEs upon applicants.     

The Construct Validity of Assessment Centre Performance Dimensions

Evidence regarding the construct validity of assessment centre performance dimensions is reviewed. The evidence strongly suggests that variance in ratings tends to reflect exercises more than individual performance dimensions, thus calling into question the construct validity and utility of these dimensions. A number of biases in the assessment centre process, as well as more general rating biases are noted that may be responsible for these pervasive exercise effects. Suggestions are made for enhancing the construct validity of performance dimensions.     

Nonverbal Memory Assessment with Designs: Construct Validity and Clinical Utility

The evaluation of memory is often a principal objective in psychological testing, especially for older adults. Differentiation between subtypes of memory functioning is critical in making differential diagnoses and in predicting everyday functioning. In this paper, 23 tests using memory for designs and developed for clinical or research applications are reviewed. Overall, reliabilities are low, normative samples varied in size, and findings regarding the construct validity of design memory testing are mixed. Tests with higher reliability, large numbers of designs, and recognition formats may be more specific to nonverbal memory. A series of studies that systematically establishes construct validity for design memory tests will enhance clinical utility.     

Factors which Improve the Construct Validity of Assessment Centers: A Review

This article reviews 21 studies which manipulated specific variables to determine their impact on the construct validity of assessment centers. This review shows that the studies regarding the impact of different observation, evaluation, and integration procedures yielded mixed results. Conversely, dimension factors (number, conceptual distinctiveness, and transparency), assessor factors (type of assessor and type of assessor training), and exercise factors (exercise form and use of role-players) were found to moderate construct validity. On the basis of the review, practical recommendations are derived to maximize the probability that practitioners design and administer an assessment center with construct validity. Finally, new perspectives for future research are identified.     

Assessor Constructs in Use as the Missing Component in Validation of Assessment Center Dimensions: A critique and directions for research

The basis for assessment center dimension ratings has been examined through the lens of various multitrait–multimethod approaches, with some researchers concluding that dimension ratings are not representative of meaningful constructs. This presents a serious challenge for those who would generalize predictor constructs, particularly those that are broadly articulated in management assessment center dimensions. This paper addresses the problems with applying such analyses to assessment center dimension ratings without first articulating expected construct relationships. Ackerman and Humphreys' analysis of constructs will be used to better frame assessment center dimensions and to articulate possible assessor constructs in use. Theories of performance held by the managers, who serve both as subject matter experts when dimensional constructs are initially defined, and as the assessors who impose these dimensions on assessees' behavior in assessment center ratings, are suggested as important bases for further construct validation. Approaches for generalization of assessor constructs in use to the managerial work domain will be discussed.     

Type D Personality and Chronic Pain: Construct and Concurrent Validity of the DS14

A body of research has demonstrated that the Type D personality is a risk factor among cardiac patients. Previous studies validating the Type D Scale (DS14) across other clinical groups have not included chronic pain patients in their samples. The purpose of this study was to investigate the construct and concurrent validity of the DS14 using the MMPI-2. The DS14 and its two component subscales demonstrated strong internal consistency among chronic pain patients. The two subscales of the DS14 were found to be related to similar clinical scales on the MMPI-2, and significant differences were found in the MMPI-2 profiles of individuals with and without the Type D personality. Considerations for clinical practice and research are discussed.     

Effects of Disaggregation on Managerial Assessment Center Validity

Assessment center validity data from a large, international petrochemical company were disaggregated by managerial level and business stream. As with previous research, the correlations were higher for the current job grade criterion than for performance ratings. Overall, disaggregating the data did not have the effect of increasing the assessment center—performance rating validities. However, when the validation data were disaggregated by business stream, there were differences in the pattern of the relationships between assessment factors and performance.     

Construct and Criterion Validity of the Comprehensive Trail Making Test in Children and Adolescents with Traumatic Brain Injury

The current study examined the construct and criterion validity of the Comprehensive Trail Making Test (CTMT) when used to evaluate children and adolescents with traumatic brain injury (TBI). Participants included 100 children and adolescents, 50 who had sustained TBI and 50 normal comparisons (NC). Analyses indicated that the CTMT factor scores were significantly correlated with tests of perceptual organizational ability, processing speed, and motor function and provided support for its construct validity. Additionally, correlations between the various CTMT scores suggested that a different pattern of associations was present in the TBI group compared to the NC group. Finally, the TBI group performed significantly worse (p < .001) on all of the CTMT scores, including each of the five CTMT trails as well as the factor and composite index scores. Results support the construct and criterion validity of the CTMT when used to assess children and adolescents with TBI.     

The Construct Validity of Assertion: Contributions of Four Assessment Procedures and Norman's Personality Factors

ABSTRACT The construct validity of assertion was analyzed in three independent correlational studies The convergence of self-reported, laboratory roleplayed, self-observed, and peer-rated measures was examined, as well as their relationship to Norman's peer-rated five personality factors of extraversion, agreeableness, conscientiousness, emotional stability, and culture Twenty-three and 29 undergraduate students participated in Studies 1 and 2, respectively In general, results showed poor convergence among assertiveness measures even when increasing the representativeness and reliability of the role-played laboratory and self-observed measures, as in Study 2 In Study 3, the role-playing test was abandoned and 45 subjects served as peers for one another The results of Study 3 supported the prediction that assertiveness measures showed at least moderate convergence, in addition there were significant relationships between assertiveness and extraversion and between peer-rated assertiveness and agreeableness, conscientiousness, and culture The relative contribution of each assessment procedure to the construct validity of assertion is discussed     

Cognitive Tests Used in Selection Can Have Content Validity as Well as Criterion Validity: A broader research review and implications for practice

Many industrial/organizational (I/O) psychologists, both academics and practitioners, believe that the content validity model is not appropriate for cognitive ability measures used in personnel selection. They believe that cognitive tests can have criterion validity and construct validity but not content validity. Based on a review of the broader differential psychology research literature on cognitive skills, aptitudes, and abilities, this article demonstrates that with the proper content validity procedures, cognitive ability measures, including, ultimately, some de facto measures of general cognitive ability, can have content validity in addition to criterion and construct validity. Finally, the article considers, critiques, and refutes the specific arguments contending that content validity is inappropriate for use with cognitive skills and abilities. These research facts have implications for I/O practice, professional standards, and legal defensibility of selection programs.     

Social Status: Construct and External Validity

Abstract

The present study examined the construct and external validity of social status based on data covering various aspects of the construct, and collected from adolescent samples in Sweden, Australia, and the United States. Using correlation and factor analytic techniques it was found that (a) the various social status variables were, in general, only moderately related within the different countries, (b) the relations among variables were not congruent between countries, (c) two social status factors were revealed in all countries, separating educational-occupational status from social-economic status, and (d) only the educational-occupational factor showed high congruence between all of the countries.     

Validity of the Suicide Assessment Checklist in an Emergency Crisis Center

Validity of the Suicide Assessment Checklist (SAC; J. R. Rogers, 1990) was investigated in a sample of 1,969 admissions to a psychiatric emergency crisis center. Internal consistency reliability for rated items of the SAC was .87. Supporting construct‐related validity, total score differences were found in the expected directions as a function of referral reason. Convergent validity was based on observed correlations between selected SAC items and conceptually similar items on the Beck Depression Inventory (A. T. Beck, 1970). Supporting criterion‐related validity, total score differences in the expected directions were found as a function of disposition setting. Recommendations are made for the use of the SAC in the practice of counseling.