Ethnic and gender subgroup differences in assessment center ratings: a meta-analysis

Assessment centers are widely believed to have relatively small standardized subgroup differences (d). However, no meta-analytic review to date has examined ds for assessment centers. The authors conducted a meta-analysis of available data and found an overall Black-White d of 0.52, an overall Hispanic-White d of 0.28, and an overall male-female d of -0.19. Consistent with our expectations, results suggest that Black-White ds in assessment center data may be larger than was previously thought. Hispanic-White comparisons were smaller than were Black-White comparisons. Females, on average, scored higher than did males in assessment centers. As such, assessment centers may be associated with more adverse impact against Blacks than is portrayed in the literature, but the predictor may have less adverse impact and be more "diversity friendly" for Hispanics and females.     

The criterion-related validity of integrity tests: an updated meta-analysis

Integrity tests have become a prominent predictor within the selection literature over the past few decades. However, some researchers have expressed concerns about the criterion-related validity evidence for such tests because of a perceived lack of methodological rigor within this literature, as well as a heavy reliance on unpublished data from test publishers. In response to these concerns, we meta-analyzed 104 studies (representing 134 independent samples), which were authored by a similar proportion of test publishers and non-publishers, whose conduct was consistent with professional standards for test validation, and whose results were relevant to the validity of integrity-specific scales for predicting individual work behavior. Overall mean observed validity estimates and validity estimates corrected for unreliability in the criterion (respectively) were .12 and .15 for job performance, .13 and .16 for training performance, .26 and .32 for counterproductive work behavior, and .07 and .09 for turnover. Although data on restriction of range were sparse, illustrative corrections for indirect range restriction did increase validities slightly (e.g., from .15 to .18 for job performance). Several variables appeared to moderate relations between integrity tests and the criteria. For example, corrected validities for job performance criteria were larger when based on studies authored by integrity test publishers (.27) than when based on studies from non-publishers (.12). In addition, corrected validities for counterproductive work behavior criteria were larger when based on self-reports (.42) than when based on other-reports (.11) or employee records (.15).     

A meta-analysis of the criterion-related validity of prehire work experience

Organizations frequently screen or select job applicants based on their work experience. Despite this, surprisingly little is known about the criterion-related validity of prehire experience, which reflects the amount, duration, or type of experience workers have acquired before they enter a new organization. To address this critical gap in the literature, we used meta-analysis to synthesize data from 81 independent samples that reported relations between prehire experience and performance or turnover. Results revealed overall corrected correlations of .06 for job performance (k = 44, n = 11,785), .11 for training performance (k = 21, n = 8,176), and .00 for turnover (k = 32, n = 11,676). Measures that capture prehire experience with tasks, jobs, or occupations relevant to workers’ current position also are only weakly related to the outcomes (e.g.,  = .07 for job performance). Two exceptions to our main findings are that (a) prehire experience is somewhat more predictive of job performance when workers first start a new job and (b) measures of task-level experience predict training performance, although these results are based on small subsets of primary studies. Overall, the present findings suggest that the types of prehire experience measures organizations currently use to screen job applicants generally are poor predictors of future performance and turnover. We therefore caution organizations from selecting employees based on such measures unless more positive evidence emerges.     

Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings

The authors reanalyzed assessment center (AC) multitrait-multimethod (MTMM) matrices containing correlations among postexercise dimension ratings (PEDRs) reported by F. Lievens and J. M. Conway (2001). Unlike F. Lievens and J. M. Conway, who used a correlated dimension-correlated uniqueness model, we used a different set of confirmatory-factor-analysis-based models (1-dimension-correlated Exercise and 1-dimension-correlated uniqueness models) to estimate dimension and exercise variance components in AC PEDRs. Results of reanalyses suggest that, consistent with previous narrative reviews, exercise variance components dominate over dimension variance components after all. Implications for AC construct validity and possible redirections of research on the validity of ACs are discussed.     

Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

Moderators of sex- and race-based subgroup differences in assessment center ratings: A meta-analysis

This meta-analysis tested a series of moderators of sex- and race-based subgroup differences using assessment center (AC) field data. We found that sex-based subgroup differences favoring female assessees were smaller among studies that reported: combining AC scores with other tests to compute overall assessment ratings, lower mean correlations between rating dimensions, using more than one assessor to rate assessees in exercises, and providing assessor training. In contrast, we found larger sex-based subgroup differences favoring female assessees among studies that reported: lower proportions of females in assessee pools, conducting a job analysis to design the AC, and using multiple observations of AC dimensions across exercises. We also observed a polynomial effect showing that subgroup differences most strongly favored female assessees in jobs with the highest and lowest rates of female incumbents. We found race-based subgroup differences favoring White assessees were smaller on less cognitively loaded rating dimensions and for jobs with lower rates of Black incumbents. Studies reporting greater overall methodological rigor also showed smaller subgroup differences favoring White assessees. Regarding specific rigor features, studies reporting use of highly qualified assessors and integrating dimension ratings from separate exercises into overall dimension scores showed significantly lower differences favoring White assessees.     

Further validation of the IDAS: evidence of convergent, discriminant, criterion, and incremental validity

The authors explicated the validity of the Inventory of Depression and Anxiety Symptoms (IDAS; D. Watson et al., 2007) in 2 samples (306 college students and 605 psychiatric patients). The IDAS scales showed strong convergent validity in relation to parallel interview-based scores on the Clinician Rating version of the IDAS; the mean convergent correlations were .51 and .62 in the student and patient samples, respectively. With the exception of the Well-Being Scale, the scales also consistently demonstrated significant discriminant validity. Furthermore, the scales displayed substantial criterion validity in relation to Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994) mood and anxiety disorder diagnoses in the patient sample. The authors identified particularly clear and strong associations between (a) major depression and the IDAS General Depression, Dysphoria and Well-Being scales, (b) panic disorder and IDAS Panic, (c) posttraumatic stress disorder and IDAS Traumatic Intrusions, and (d) social phobia and IDAS Social Anxiety. Finally, in logistic regression analyses, the IDAS scales showed significant incremental validity in predicting several DSM-IV diagnoses when compared against the Beck Depression Inventory-II (A. T. Beck, R. A. Steer, & G. K. Brown, 1996) and the Beck Anxiety Inventory (A. T. Beck & R. A. Steer, 1990).     

A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed.     

Further evidence for the validity of the Taylor Aggression Paradigm

The purpose of this study was to evaluate the validity of a modified version of the Taylor Aggression Paradigm (TAP) as a measure of direct physical aggression. Hypotheses were generated from recent theory pertinent to the categorization and measurement of aggressive behavior as well as widely supported effects of alcohol intoxication and gender on aggression. Participants were 328 (163 men and 165 women) healthy social drinkers between 21 and 35 years of age who completed self-report personality inventories designed to assess one's propensity toward direct physical aggression, verbal aggression, trait anger, and hostility. Following the consumption of either an alcohol or a placebo beverage, participants were tested on the TAP, in which mild electric shocks were received from, and administered to, a fictitious opponent during a competitive task. Direct physical aggression was operationalized as the shock intensities (i.e., first trial shock intensity, mean shock intensity, proportion of highest shock) administered to the fictitious opponent. Although all self-report measures were significantly associated with the three TAP indices, the associations involving physical aggression were strongest. In addition, self-report measures of physical aggression consistently predicted higher levels of aggression on the TAP indices in men, compared with women, and in intoxicated, relative to sober, participants. Taken as a whole, this pattern of findings provides further evidence for the validity of the TAP as a measure of direct physical aggression for men and women.     

Three method factors explaining the low correlations between assessment center dimension ratings and scores on personality inventories

In general, correlations between assessment centre (AC) ratings and personality inventories are low. In this paper, we examine three method factors that may be responsible for these low correlations: differences in (i) rating source (other versus self), (ii) rating domain (general versus specific), and (iii) rating format (multi‐ versus single item). This study tests whether these three factors diminish correlations between AC exercise ratings and external indicators of similar dimensions. Ratings of personality and performance were combined in an analytical framework following a 2 × 2 × 2 (source, domain, format) completely crossed, within subjects design. Results showed partial support for the influence of each of the three method factors. Implications for future research are discussed. Copyright © 2004 John Wiley & Sons, Ltd.     

On the usefulness of measures of normal personality for clinical assessment: evidence of the incremental validity of the Revised NEO Personality Inventory

As a means of examining the incremental validity of a normal personality measure in the prediction of selected Axis I and II diagnoses, 1,342 inpatient substance abusers completed the Revised NEO Personality Inventory (NEO-PI-R) and the Minnesota Multiphasic Personality Inventory--2 (MMPI-2) and were assessed with structured clinical interviews to determine diagnostic status. Results demonstrated that scores from the NEO-PI-R (a) were substantially related to the majority of diagnoses, accounting for between 8% and 26% of the variance in the diagnostic criteria; (b) explained an additional 3% to 8% of the variability beyond 28 selected MMPI-2 scale scores; (c) increased diagnostic classification an additional 7% to 23% beyond MMPI-2 scale scores; and (d) were significantly more useful when examined at the facet trait level than at the domain trait level. Implications for incorporating measures of normal personality into clinical assessment batteries are discussed.     

Responsibility to continue thinking and worrying: evidence of incremental validity

This two-part study examined the role of the responsibility to continue thinking, beliefs that one needs prolonged thinking about stressful problems, in the prediction of excessive worry. This construct is considered to reflect high levels of motivation to continue inflexible thinking and the use of rigid stop rules. In Study 1, 122 students completed questionnaires. A regression analysis revealed that responsibility accounted for a unique variance beyond negative meta-cognitive beliefs about worry. One hundred and fifty students participated in Study 2, where worry was regressed on emotional instability (Neuroticism), responsibility, and other worry-related cognitive variables (intolerance of uncertainty, positive/negative meta-cognitive beliefs, poor problem-solving orientation, and cognitive avoidance). Again, responsibility was a significant predictor, after controlling for emotional instability (Neuroticism) and other worry-related cognitions. These results indicate the incremental validity of the responsibility to continue thinking.     

Further evidence for construct validity of the Interpersonal Relationship Inventory: Japanese version

The Japanese versions of the Interpersonal Relationship Inventory, the Satisfaction with Life Scale, and the Self-esteem Scale were administered to 153 Japanese college students. Weak to modest significant correlations between scores on the inventory and measure of life satisfaction and self-esteem provided further incremental support for construct validity of the inventory.     

The use of person-organization fit in employment decision making: an assessment of its criterion-related validity

Because measures of person-organization (P-O) fit are accountable to the same psychometric and legal standards used for other employment tests when they are used for personnel decision making, the authors assessed the criterion-related validity of P-O fit as a predictor of job performance and turnover. Meta-analyses resulted in estimated true criterion-related validities of .15 (k = 36, N = 5,377) for P-O fit as a predictor of job performance and .24 (k = 8, N = 2,476) as a predictor of turnover, compared with a stronger effect of .31 (k = 109, N = 108,328) for the more commonly studied relation between P-O fit and work attitudes. In contrast to the relations between P-O fit and work attitudes, the lower 95% credibility values for the job performance and turnover relations included zero. In addition, P-O fit's relations with job performance and turnover were partially mediated by work attitudes. Potential concerns pertaining to the use of P-O fit in employment decision making are discussed in light of these results.     

A test of the generalizability of a recently suggested conceptual model for assessment center ratings

The present study replicated and extended research concerning a recently suggested conceptual model of the underlying factors of dimension ratings in assessment centers (ACs) proposed by Hoffman, Melchers, Blair, Kleinmann, and Ladd that includes broad dimension factors, exercise factors, and a general performance factor. We evaluated the criterion-related validity of these different components and expanded their nomological network. Results showed that all components (i.e., broad dimensions, exercises, general performance) were significant predictors of training performance. Furthermore, broad dimensions showed incremental validity beyond exercises and general performance. Finally, relationships between the AC factors and individual difference constructs (e.g., Big Five, core self-evaluations, positive and negative affectivity) supported the construct-related validity of broad dimensions and provided further insights in the nature of the different AC components.     

Further evidence for the validity of the Need Inventory of Sensation Seeking

The Need Inventory of Sensation Seeking (NISS) by Roth and Hammelstein (2012) conceptualizes sensation seeking as a motivational trait, a need for stimulation that can provoke different behaviors. Extending the area of application to the field of sexuality, the results of an online survey with a convenience sample (N = 368) provided evidence that the NISS outperforms traditional measures of sensation seeking. In a series of hierarchical regression analyses, the NISS added incremental variance in predicting the number of sexual partners and the frequency of extreme sexual behaviors and fantasies. No relationship between the NISS and pornography use was found. Moreover, the NISS subscale Need for Stimulation performed better than the subscale Avoidance of Rest. Overall, the study supported the assumption that the NISS is a valid and reliable instrument in the area of sexuality.     

A further examination of the constructs underlying assessment center ratings

Previous factor analytic work (Sackett and Dreher, 1982) has found exercise factors, rather than the expected dimension factors, underlying ratings made in managerial assessment centers. The present paper responds to criticisms that inappropriate analyses were conducted and that unrepresentative assessment centers were studied. Exercise factors were found to predominate in two new sets of assessment data which we obtained and in three studies published by other authors, thus indicating that the results were not idiosyncratic to the three centers studied by Sackett and Dreher. Analyses using confirmatory, rather than exploratory, factor analysis also support the dominance of exercise factors. Although dimension factors were found in three of four data sets studied, variables loading on a dimension factor consistently loaded more highly on an exercise factor as well.     

Further evidence of reliability and validity for the driving anger expression inventory

This study provided evidence of reliability and validity for the four scales of the Driving Anger Expression Inventory. Alpha reliabilities for scales ranged from .84 to .89. Measures of aggressive anger expression while driving (Verbal Aggressive Expression, Personal Physical Aggressive Expression, and Use of the Vehicle to Express Anger scales) correlated positively with each other and negatively with the Adaptive/Constructive Expression scale. Scores on the three aggressive forms of anger expression correlated positively with trait anger and measures of driving-related anger, aggression, and risky behavior, whereas scores on the Adaptive/Constructive Expression scale correlated negatively with these variables. Reports of aggressive and risky behavior correlated most strongly with the Use of the Vehicle to Express Anger scale. Forms of anger expression were minimally or uncorrelated with rated trait anxiety and reports of moving violations, close calls, and accidents. Findings replicated earlier findings and provided further evidence for the reliability and validity of the Driving Anger Expression Inventory.     

The dynamics of assessment center validity: results of a 7-year study

The authors investigated temporal trends in the validity of an assessment center consisting of a group discussion and an analysis presentation exercise for predicting career advancement as measured by average salary growth over a 7-year period in a sample of 679 academic graduates. The validity of the overall assessment rating for persons with tenure of 7 years, corrected for initial differences in starting salaries and restriction in range, was .39. There was a considerable time variation in the validity of both the overall assessment rating and the assessment center dimensions. In accordance with findings from research in managerial effectiveness and development, the interpersonal effectiveness dimension became valid only after a number of years, whereas the firmness dimension was predictive in the whole period and increased with time. For comparison, validity trends for 2 types of interviews and a mental test were also studied.