Same‐gender and same‐race bias in assessment center ratings: A rating error approach to understanding subgroup differences

This study investigated leniency and similar‐to‐me bias as mechanisms underlying demographic subgroup differences among assessees in assessors’ initial dimension ratings from three assessment center (AC) simulation exercises used as part of high‐stakes promotional testing. It examined whether even small individual‐level effects can accumulate (i.e., “trickle‐up”) to produce larger subgroup‐level differences. Individual‐level analyses were conducted using cross‐classified multilevel modeling and conducted separately for each exercise. Results demonstrated weak evidence of leniency toward White assessees and similar‐to‐me bias among non‐White assessee–assessor pairs. Similar leniency was found toward female assessees, but no statistically significant effects were found for assessee or assessor gender or assessee–assessor gender similarity. Using traditional d effect size estimates, weak individual level assessee effects translated into small but consistent subgroup differences favoring White and female assessees. Generally small but less consistent subgroup differences indicated that non‐White and male assessors gave higher ratings. Moreover, analyses of overall promotion decisions indicate the absence of adverse impact. Findings from this AC provide some support for the “trickle‐up” effect, but the effect on subgroup differences is trivial. The results counter recent reviews of AC studies suggesting larger than previously assumed subgroup differences. Consequently, the findings demonstrate the importance of following established best practices when developing and implementing the AC method for selection purposes to minimize subgroup differences.     

评价中心的结构效度研究

评价中心虽然具备很高的预测效度,但其结构效度指标却不太理想,如研究普遍发现其汇聚效度和区分效度较低。影响评价中心结构效度的因素众多,如评分维度因素（数量和类型）、评价者因素（培训方式和人员类型）、测评方法因素（情景导向特征、特质激活潜力、测评活动形式）以及系统的观察与评价程序。该文从上述因素出发,综述了评价中心结构效度的相关研究,总结了提高评价中心结构效度的措施,并指出了未来的研究方向     

Factors which Improve the Construct Validity of Assessment Centers: A Review

This article reviews 21 studies which manipulated specific variables to determine their impact on the construct validity of assessment centers. This review shows that the studies regarding the impact of different observation, evaluation, and integration procedures yielded mixed results. Conversely, dimension factors (number, conceptual distinctiveness, and transparency), assessor factors (type of assessor and type of assessor training), and exercise factors (exercise form and use of role-players) were found to moderate construct validity. On the basis of the review, practical recommendations are derived to maximize the probability that practitioners design and administer an assessment center with construct validity. Finally, new perspectives for future research are identified.     

AN EXAMINATION OF THE EFFECTS OF USING BEHAVIOR CHECKLISTS ON THE CONSTRUCT VALIDITY OF ASSESSMENT CENTER DIMENSIONS

Although research has established the criterion-related validity of assessment centers for selection purposes, the construct validity of dimension ratings has not been demonstrated. A quasi-experimental design was used to investigate the influence of retranslated behavior checklists on the construct validity of dimension ratings for two assessment center exercises. Assessor use of behavior checklists increased the average convergent (i.e., same dimension across exercise) validity from .24 to .43 while decreasing the average discriminant (i.e., different dimension within exercise) validity (.47 to .41). Behavior checklist sums were moderately correlated with corresponding dimension ratings and demonstrated a comparable level of construct validity. It is suggested that using behavior checklists may improve dimension construct validity by reducing the cognitive demands placed on raters.     

Personality Assessment in the Diagnostic Manuals: On Mindfulness,Multiple Methods,and Test Score Discontinuities

Recent controversies have illuminated the strengths and limitations of different frameworks for conceptualizing personality pathology (e.g., trait perspectives, categorical models), and stimulated debate regarding how best to diagnose personality disorders (PDs) in the Diagnostic and Statistical Manual of Mental Disorders (5th ed.), and in other diagnostic systems (i.e., the International Classification of Diseases, the Psychodynamic Diagnostic Manual). In this article I argue that regardless of how PDs are conceptualized and which diagnostic system is employed, multimethod assessment must play a central role in PD diagnosis. By complementing self-reports with evidence from other domains (e.g., performance-based tests), a broader range of psychological processes are engaged in the patient, and the impact of self-perception and self-presentation biases can be better understood. By providing the assessor with evidence drawn from multiple modalities, some of which provide converging patterns and some of which yield divergent results, a multimethod assessment compels the assessor to engage this evidence more deeply. The mindful processing that ensues can help minimize the deleterious impact of naturally occurring information processing bias and distortion on the part of the clinician (e.g., heuristics, attribution errors), bringing greater clarity to the synthesis and integration of assessment data.     

Assessee and Assessor Experiences of Significant Events in Psychological Assessment Feedback

Although a growing body of scholarly work explores the unique utility and therapeutic uses of psychological assessment, less work has focused specifically on feedback, with few studies that have explored empirically the underlying processes that may describe the role of feedback in a successful assessment. The purpose of this project was to add to the discourse on this topic by engaging in an empirical study exploring assessee and assessor experiences of significant events in psychological assessment feedback. The methodology is qualitative and modeled after significant events research in the study of psychotherapy process. I analyze the accounts of 6 assessment clients and 6 assessment clinicians regarding key events in their experience of feedback.     

Construct validity of the Rosenzweig Picture-Frustration Study

Projective and semiprojective techniques are both tools and tests. A schema for assessing the validity of projective devices is proposed: (1) Primary, consisting of (a) construct validity i.e., the soundness of the techniques vis-a-vis its underlying concepts; and (b) criterion-related validity, i.e., the concordance between tool results and specified external criteria. (2) Secondary, viz., pragmatic validity, i.e., the usefulness of the instrument in applied fields. Discussion then focuses on the construct validity of the conceptually-based, semiprojective Rosenzweig P-F Study. specifically the evidence on (a) the self-consistency of its conceptual dimensions; (b) the sequential. social development of the Types and Directions of Aggression; (c) levels of behavior elicited; (d) sensitivity to scorable changes after induced frustration: and (e) physiological correlates oft he aggression categories. In each of these areas evidence exists in support of the validity of the P-F Study.     

Assessee and assessor experiences of significant events in psychological assessment feedback

Although a growing body of scholarly work explores the unique utility and therapeutic uses of psychological assessment, less work has focused specifically on feedback, with few studies that have explored empirically the underlying processes that may describe the role of feedback in a successful assessment. The purpose of this project was to add to the discourse on this topic by engaging in an empirical study exploring assessee and assessor experiences of significant events in psychological assessment feedback. The methodology is qualitative and modeled after significant events research in the study of psychotherapy process. I analyze the accounts of 6 assessment clients and 6 assessment clinicians regarding key events in their experience of feedback.     

Extending the Nomological Network of Assessment Center Construct Validity: Prediction of Cross-Situationally Consistent and Specific Aspects of Assessment Center Performance

The purpose of this study was to expand the nomological validity of assessment centers (ACs) by investigating predictors of cross-situationally consistent versus specific aspects of AC performance. Consistent with hypotheses, (a) Big Five personality factors predicted AC performance as it related to a cross-situationally consistent general performance factor but not as it related to exercise (i.e., situationally specific) factors, and (b) job knowledge predicted performance as it related to both the general performance factor and exercise-specific factors. Results are interpreted as they relate to the growing literature on AC construct validity.     

Assessment Centers: Current Practices in the United States

Purpose  

The goals of this investigation were to review current AC practices in the United States by evaluating whether they follow the Guidelines and Ethical Considerations for Assessment Center Operations (International Task Force on Assessment Centers, 2000). We both expanded upon and compared our results to a prior benchmarking study (Spychalski et al. in Personnel Psychol, 50:71–90, 1997), and investigated practices regarding job analysis, AC development, dimensions (i.e., job requirements), exercises, assessor characteristics and training, behavior recording, data integration, organizational policy, assessee rights, AC evaluation and AC technology. Data were collected via an online survey completed by individuals from human resource departments of organizations (N = 54) across the U.S; organizations to whom the survey was sent were selected by sampling Fortune 500 organizations based on economic sector.     

Constructs and developmental assessment centers: Further troubling empirical findings

Research indicates that assessment center (AC) ratings typically demonstrate poor construct validity; that is, they do not measure the intended dimensions of managerial performance (e.g., Sackett & Harris, 1988). The purpose of this study was to investigate the construct validity of dimension ratings from a developmental assessment center (N=102), using multitrait-multimethod analysis and factor analysis. The relationships between AC ratings, job performance ratings, and personality measures also were investigated. Results indicate that the AC ratings failed to demonstrate construct validity. The ratings did not show the expected relationships with the job performance and personality measures. Additionally, the factors underlying these ratings were found to be the AC exercises, rather than the managerial dimensions as expected. Potentially, this lack of construct validity of the dimension ratings is a serious problem for a developmental assessment center. There is little evidence that the managerial weaknesses identified by the AC are the dimensions that actually need to be improved on the job. Methods are discussed for improving the construct validity of AC ratings, for example, by decreasing the cognitive demands on the assessors.This study is based on a dissertation submitted to North Carolina State University. Portions of this paper were presented at the meeting of the Society for Industrial and Organizational Psychology in Montreal, Quebec, May, 1992. I am grateful to Paul Thayer, Bert Westbrook, James W. Cunningham, and Patrick Hauenstein for their contributions to this research. I also thank several anonymous reviewers for their comments on this article.     

A new frame for frame-of-reference training: enhancing the construct validity of assessment centers

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (a) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (b) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (c) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice.     

Nonlinear personality-performance relationships and the spurious moderating effects of traitedness

This article evaluated the validity of two traditional traitedness measures (i.e., standard deviation traitedness measures and global, single-item traitedness measures) and three new traitedness measures (i.e., multi-item, ranking, and facet traitedness measures). Overall, there was poor construct validity for both the traditional and new measures of traitedness. Furthermore, traitedness was a spurious moderator of the relationship between openness to experience and academic performance. Finally, evidence of nonlinear personality-academic performance relationships was discovered.     

Validity Concerns in the Measurement of Women’s and Men’s Report of Intimate Partner Violence

The measurement of intimate partner violence (IPV) has proven to be more complex than originally anticipated and content and construct validity need to be greatly improved for IPV assessment. For measurement of IPV in the United States, these validity issues range from providing the most accurate wording for the content domain to controversies as to when violent actions are counted (e.g., self-defense) or whether to include mild aggression (e.g., psychological conflict tactics) that may be more normative and not harmful. The three major forms of IPV (i.e., physical, sexual, and psychological abuse) have distinct validity issues and may require different modalities for assessment. Gender needs to be considered when establishing construct validity due to differences in the meaning of aggression, impacts of abuse, and even patterns of violence for women and men. External threats to validity include potential bias of self-report and motivations when reporting on a partner, discrepancies in couples’ reports, the influence of response styles, and design issues affecting reporting. Traditional methods used to establish validity for IPV scales are reviewed and critiqued. Recommendations for enhancing validity in IPV assessment are provided.     

Applicant Selection Expectations: Validating a multidimensional measure in the military

Research has suggested the importance of applicants' expectations of forthcoming selection procedures in predicting how applicants react to selection procedures. Validated measures of selection expectations are still scarce, however. This study reports on the validation of the Applicant Expectation Survey (AES), intended to measure applicants' expectations of forthcoming selection procedures. The AES was validated using three military applicant samples and showed sound psychometric properties (i.e., reliability, measurement invariance, discriminant validity) for a five‐factorial oblique structure consisting of 26 items. The five factors (i.e., Warmth/respect, Chance to demonstrate potential, Difficulty of faking, Unbiased assessment, Feedback) were positively related to several organizational outcome measures and to applicants' perceptions of the selection procedure, providing evidence for the predictive validity of the AES.     

DISENTANGLING THE MEANING OF MULTISOURCE PERFORMANCE RATING SOURCE AND DIMENSION FACTORS

We extend multisource performance rating (MSPR) construct validity research by examining the pattern of relationships between factor analytically derived MSPR rating source and performance dimension factors and externally measured constructs (e.g., assessment center dimensions, personality constructs, and intelligence). The pattern of relationships among MSPR dimensions and external constructs provides modest construct validity evidence for the MSPR dimensions. In addition, MSPR source factors were differentially correlated with externally measured constructs, suggesting that MSPR source effects represent substantively meaningful source specific variance, as opposed to bias. These findings are discussed in the context of managerial skill diagnosis and the efficacy of collecting performance data from multiple sources.     

Child sex moderates the association between negative parenting and childhood conduct problems

Although multiple dimensions of negative parenting behavior are associated with childhood conduct problems (CP), there is relatively little research on whether the association is equally robust in boys and girls. To improve the specificity of current models of negative parenting and offspring CP, we explored the potential moderating role of child sex in a sample of 179 5- to 10-year-old ethnically diverse boys and girls with and without attention-deficit/hyperactivity disorder (ADHD) who were assessed using multiple methods (i.e., rating scales, semistructured interviews) and informants (i.e., parents, teachers). Controlling for children's age, race-ethnicity, and ADHD diagnostic status (i.e., ADHD vs. non-ADHD), inconsistent discipline was positively associated with offspring aggression and rule-breaking behavior, whereas harsh punishment was positively associated with aggression, rule-breaking behavior, and oppositional defiant disorder symptoms. Furthermore, child sex significantly moderated the association of inconsistent discipline and aggression and rule-breaking behavior, such that inconsistent discipline was positively associated with CP for boys, but not for girls. Given the centrality of negative parenting to theories of and efficacious interventions for aggression and CP, we discuss these findings within a developmental psychopathology framework and consider their implications for intervention.     

On the myth and the reality of the temporal validity degradation of general mental ability test scores

Claims of changes in the validity coefficients associated with general mental ability (GMA) tests due to the passage of time (i.e., temporal validity degradation) have been the focus of an on-going debate in applied psychology. To evaluate whether and, if so, under what conditions this degradation may occur, we integrate evidence from multiple sub-disciplines of psychology. The temporal stability of construct validity is considered in light of the evidence regarding the differential stability of g and the invariance of measurement properties of GMA tests over the adult life-span. The temporal stability of criterion-related validity is considered in light of evidence from long-term predictive validity studies in educational and occupational realms. The evidence gained from this broad-ranging review suggests that temporal degradation of the construct- and criterion-related validity of ability test scores may not be as ubiquitous as some have previously concluded. Rather, it appears that both construct and criterion-related validity coefficients are reasonably robust over time and that any apparent degradation of criterion-related validity coefficients has more to do with changes in the determinants of task performance and changes in the nature of the criterion domain rather temporal degradation per se (i.e., the age of the test scores). A key exception to the conclusion that temporal validity degradation is more myth than reality concerns decision validity. Although the evidence is sparse, it is likely that the utility of a given GMA test score for making diagnostic decisions about an individual deteriorates over time. Importantly, we also note several areas in need of additional and more rigorous research before strong conclusions can be supported.     

A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed.