Forming Composites of Cognitive Ability and Alternative Measures to Predict Job Performance and Reduce Adverse Impact: Corrected Estimates and Realistic Expectations

Although there has been empirical attention paid to the criterion‐related validity of predictor composites, there has been much less attention paid to the standardized ethnic group differences associated with these composites. One important area of inquiry in predictor composite research is the influence of adding predictors to a test of general mental ability. The limited empirical literature on this practice is mixed, but the prevailing expectation is that there is likely to be higher validity and less adverse impact. Unfortunately, much of the previous work is limited by the presence of inaccurate validity and standardized ethnic group difference values. In this analysis we formed meta‐analytic matrices to more accurately estimate the validity and standardized ethnic group differences of several composites that combine a measure of cognitive ability with measures of conscientiousness, a structured interview, or biodata. While results were somewhat complex, we found that adding alternative predictors does not result in a situation in which validity automatically goes up and adverse impact potential automatically goes down. In fact, the reductions in adverse impact (if any) from adding “non‐cognitive” predictors were more modest than much of the literature suggests.     

Selecting Predictor Subsets: Considering validity and adverse impact

The paper proposes a procedure for designing Pareto‐optimal selection systems considering validity, adverse impact and constraints on the number of predictors from a larger subset that can be included in an operational selection system. The procedure determines Pareto‐optimal composites of a given maximum size thereby solving the dual task of identifying the predictors that will be included in the reduced set and determining the weights with which the retained predictors will be combined to the composite predictor. Compared with earlier proposals, the simultaneous consideration of both tasks makes it possible to combine several strategies for reducing adverse impact in a single procedure. In particular, the present approach allows integrating (a) investigating a large number of possible predictors (such as multitest battery of ability tests, or a collection of ability and nonability measures); (b) explicit predictor weighting within feasible test procedures of a given limited size.     

Combining cognitive and noncognitive predictors and impact on selected individual demographics: An illustration

Research presented illustrates the impact on the demographic makeup of selected applicants when using a composite of cognitive predictors for selection compared to a composite of cognitive and noncognitive predictors. A large sample of American college applicants was used to simulate selection across a wide range of selection ratios using the two composites. Selection rates for Black individuals generally increased and Asian American selection rates generally decreased when the cognitive and noncognitive composite was used compared to the use of the cognitive composite. Results demonstrate that adding noncognitive predictors to a selection process utilizing cognitive predictors can have some beneficial impact on selection rates for some demographic groups, but does not eliminate the detrimental impact resulting from using cognitive predictors for some minority groups.     

DERIVATION AND IMPLICATIONS OF A META-ANALYTIC MATRIX INCORPORATING COGNITIVE ABILITY, ALTERNATIVE PREDICTORS, AND JOB PERFORMANCE

A variety of recent articles in the personnel selection literature have used analyses of meta-analytically derived matrices to draw general conclusions for the field. The purpose of this article is to construct a matrix that incorporates as complete information as possible on the relationships among cognitive ability measures, three sets of alternative predictors, and job performance, We build upon a starting matrix used by Schmitt, Rodgers, Chan, Sheppard, and Jennings (1997). Mean differences, by race, for each of the measures and the potential for adverse impact of predictor composites are also considered. We demonstrate that the use of alternative predictors alone to predict job performance (in the absence of cognitive ability) lowers the potential for adverse impact. However, in contrast to recent claims, adverse impact continues to occur at many commonly used selection ratios. Future researchers are encouraged to use our matrix and to expand upon it as new primary research becomes available. We also report and reaffirm many methodological lessons along the way, including the many judgment calls that appear in an effort of this magnitude and a reminder that the field could benefit from even greater conceptual care regarding what is labeled an "alternative predictor." Directions for future meta-analyses and for future primary research activities are also derived.     

Validity and Adverse Impact Potential of Predictor Composite Formation

Previous research on the validity and adverse impact (AI) of predictor composite formation focused on the merits of regression‐based or ad hoc composites. We argue for a broader focus. Ad hoc chosen composites are usually not Pareto‐optimal, whereas the regression‐based composite represents only one element from the total set of Pareto‐optimal composites and can, therefore, provide only limited information on the potential for validity and AI reduction of forming predictor composites when both validity and AI are of concern. In that case, other Pareto‐optimal composites may provide a better benchmark to decide on the merits of the predictor composite formation. We summarize a method to determine the set of Pareto‐optimal composites and apply the method to a representative collection of selection predictors. The application shows that the assessment of the AI and validity of predictor composite formation can differ substantially from the one arrived at when considering only regression‐based composites.     

Combining predictors to achieve optimal trade-offs between selection quality and adverse impact

The authors propose a procedure to determine (a) predictor composites that result in a Pareto-optimal trade-off between the often competing goals in personnel selection of quality and adverse impact and (b) the relative importance of the quality and impact objectives that correspond to each of these trade-offs. They also investigated whether the obtained Pareto-optimal composites continue to perform well under variability of the selection parameters that characterize the intended selection decision. The results of this investigation indicate that this is indeed the case. The authors suggest that the procedure be used as one of a number of potential strategies for addressing the quality-adverse impact problem in settings where estimates of the selection parameters (e.g., validity estimates, predictor intercorrelations, subgroup mean differences on the predictors and criteria) are available from either a local validation study or meta-analytic research.     

Pareto‐Optimal Predictor Composite Formation: A complementary approach to alleviating the selection quality/adverse impact dilemma

In our rejoinder to the comments of Kehoe (this issue) and Potosky, Bobko and Roth (this issue) we emphasize that our proposal on Pareto‐optimal predictor composite formation is a complementary and not a competitive alternative for reducing the tension between selection quality and adverse impact. Our work addresses the decisions to be made once one has decided to use a predictor composite. We also further clarify the basic features of Pareto‐optimal tradeoffs and Pareto‐optimal composites within the context of personnel selection. In particular, we indicate that Pareto‐optimal tradeoffs between validity and adverse impact emerge because these goals are different and not because of any dualism between them.     

Generating Race,Gender, and Other Subgroup Data in Personnel Selection Simulations: A pervasive issue with a simple solution

In the past 25 years, organizational researchers and practitioners have relied heavily on computer simulation research to understand how group mean differences and correlations affect overall validity and adverse impact in protected groups (e.g., racial/ethnic groups and gender) as they relate to personnel selection practices. We point out a multilevel issue affecting nearly all past simulations: The total correlations that these simulations intended to specify are somewhat distorted after group mean differences were introduced into the data. Although this distorting effect is minimal in most cases, it matters in some cases, and after all, the main virtue of statistical simulations is precision, both in the population parameters and sample data and statistics those parameters are supposed to generate. We demonstrate this distorting effect through one specific example, based on multiple predictors and meta‐analytic data, followed by a broader simulation for single predictors across a wide variety of selection conditions. Rather than merely point out this problem, we also provide a straightforward solution: multilevel formulas that incorporate both between‐ and within‐group correlations that always correct for this biasing problem, yielding more accurate simulation results. We conclude by discussing applications and promising extensions of this work.     

GROUP DIFFERENCES IN PERSONALITY: META‐ANALYSES COMPARING FIVE U.S. RACIAL GROUPS

This research poses 2 applied questions: How large are racial group differences on personality scales and are these differences likely to cause adverse impact in personnel selection? We examined the extent to which racial groups differ across Big 5 personality factors and facets. Large‐scale, quantitative estimates based on over 700 effect sizes were meta‐analytically summarized. Multiple personality instruments and understudied racial groups, particularly Asian Americans and American Indians, were included in the meta‐analyses. Most group comparisons and personality scales yielded negligible differences and are not likely to cause adverse impact in selection. However, facet‐level analyses produced different d‐values, with some group comparisons showing moderate differences, suggesting that the use of personality measures in selection does not uniformly circumvent adverse impact concerns. The veracity of this conclusion depends on (a) the particular trait in question, (b) the composition of the applicant pool (i.e., which groups are to be compared with one another), (c) the effect size (i.e., d‐value), and (d) the selection ratio. For practitioners, we present a quantitatively informed summary of when and where to expect adverse impact to result from personality measures used in personnel selection.     

The Role of Cognitive Ability Tests in Employment Selection

Cognitive ability tests correlate with measures of job performance across many jobs. However, cognitive ability tests produce racial differences that are 3 to 5 times larger than other predictors-such as biodata, personality inventories, and the structured interview-that are valid predictors of job performance. Given that (a) cognitive ability tests can be combined with other predictors such that adverse impact is reduced while overall validity is increased, and (b) alternative predictors with less adverse impact can produce validity coefficients comparable to those obtained with cognitive ability tests alone, sole reliance on cognitive ability tests when alternatives are available is unwarranted.     

Effects of Machiavellianism,Other-Enhancement,and Power-Position on Affect,Power Feeling,and Evaluation of the Ingratiator

The current study examined developmental differences from the age of 5 to 18 in the creation process of own- and other-race facial composites. In addition, it considered how differences in the creation process affect similarity ratings. Participants created two composites (one own- and one other-race) from memory. The complexity of the composite creation process was recorded during Phase One. In Phase Two, a separate group of participants rated the composites for similarity to the corresponding target face. Results support the cross-race effect, developmental differences (based on composite creators) in similarity ratings, and the importance of the creation process for own- and other-race facial composites. Together, these findings suggest that as children get older the process through which they create facial composites becomes more complex and their ability to create facial composites improves. Increased complexity resulted in higher rated composites. Results are discussed from a psycho-legal perspective.     

Modeling the Practical Effects of Applicant Reactions: Subgroup Differences in Test‐Taking Motivation,Test Performance,and Selection Rates

Research suggests that Black–White differences in test‐taking motivation may be related to subgroup test score differences, but this research has not shown the extent to which minimizing subgroup motivation differences will reduce subgroup differences in selection rates and adverse impact. This Monte Carlo study examined how enhancing Blacks' test‐taking motivation for cognitive ability tests might reduce adverse impact across a range of (a) subgroup test differences, (b) selection ratios, (c) subgroup differences in test‐taking motivation, and (d) relationships between motivation and test scores. The results suggest that although enhancing test‐taking motivation will consistently reduce subgroup differences in test performance and adverse impact, the effect is often small and will not eliminate adverse impact for any condition we examine. However, under some conditions the reduction may be important, and the discussion considers conditions where even these minimal reductions may be practically helpful.     

Group Differences on US Air Force Pilot Selection Tests

Sex and ethnic group differences were examined on the operational composites and tests used to select applicants for US Air Force officer commissioning programmes and for pilot training. Results showed that large mean score differences in applicant samples were substantially reduced among the pilot trainees. Despite differences in test performance, there was no evidence of differential validity for groups. When group differences in predicted pilot training completion rate were observed, performance was overestimated for the minority group relative to the majority group. When regression equations were adjusted for unreliability of the predictors, the observed differences in intercepts were reduced or eliminated. No prediction bias was observed against the minority groups.     

PRIOR SELECTION CAUSES BIASED ESTIMATES OF STANDARDIZED ETHNIC GROUP DIFFERENCES: SIMULATION AND ANALYSIS

Assessment of standardized ethnic group differences ( d ) on predictors of job performance has become an important issue for applied psychologists. A number of studies have used an experimental design in which the predictor of interest was administered after an initial screening predictor. We examined the influence of prior selection on a first predictor on observed ds for second predictors in multiple-hurdle selection systems. Results of a Monte Carlo simulation indicate observed dson the second predictor are underestimated in the presence of prior selection on another predictor. More important, "downward bias" in observed standardized ethnic group difference is substantial (30-70%) when selection ratios are low, standardized ethnic group differences on the screening predictor are high, and when the first and second predictors correlate above .30. Researchers should consider the influence of range restriction in designing studies of ethnic group differences and comparing ds across predictors, particularly when data are collected under a multiple-hurdle design.     

Understanding how and why adding valid predictors can decrease the validity of selection composites: A generalization of Sackett,Dahlke, Shewach,and Kuncel (2017)

It is usually assumed that adding more valid predictors will increase the predictive power of a selection test battery. Sackett, Dahlke, Shewach, and Kuncel showed that when selection tests are combined using unit weights, adding a valid predictor can lead to a decrease in validity. Situating the Sackett et al. approach in a more general multivariate framework I show how: (a) it is the tradeoff between predictor validity and predictor intercorrelations, and not the differences in predictor validities that determines whether adding a valid predictor to a composite will cause the validity of that composite to increase or decrease; and (b) this same dynamic applies across a wide range of non‐optimal schemes for weighting predictors and/or criteria.     

Examination of ethnic group differential responding on a biodata instrument

Biodata is a selection device that is often cited for high criterion validity and low adverse impact, yet little research examines the latter point. Further, little is known about whether differential biodata item responding by ethnic group occurs, and if so, whether this impacts ethnic group differences, predictive bias, adverse impact, and criterion validity. Data from 5,242 applicants for a government agency position indicated that elimination of biodata item response options with differential ethnic group selection rates from scoring consideration resulted in a large decrease in ethnic group differences, no predictive bias, less potential adverse impact, and a nonsignificant decrease in criterion validity. Results are compared to a general mental ability measure. Implications and future research directions are discussed.     

THE DIVERSITY–VALIDITY DILEMMA: STRATEGIES FOR REDUCING RACIOETHNIC AND SEX SUBGROUP DIFFERENCES AND ADVERSE IMPACT IN SELECTION

Pyburn, Ployhart, and Kravitz (this issue, 2008) introduced the diversity–validity dilemma: that some of the most valid predictors of job performance are also associated with large racioethnic and sex subgroup predictor score differences. This article examines 16 selection strategies hypothesized to minimize racioethnic and sex subgroup differences and adverse impact and, hence, balance diversity and validity. Rather than presenting a highly technical review, our purpose is to provide practitioners with a concise summary, paying particular attention to comparing and contrasting the effectiveness of the strategies and reporting new developments. The paper is organized around 4 key questions: (a) Which strategies are most effective for reducing subgroup differences? (b) Which strategies do not involve a validity tradeoff? (c) What are the major new developments in strategies for reducing adverse impact? (d) What are the major new developments in alternative predictor measurement methods (e.g., interviews, situational judgment tests, assessment centers) for reducing adverse impact? We then conclude with recommendations and caveats for how to best balance diversity and validity. These ideas are developed further in Kravitz (this issue, 2008) , who considers even broader approaches for solving the diversity–validity dilemma.     

Configurational information in face perception

A new facial composites technique is demonstrated, in which photographs of the top and bottom halves of different familiar faces fuse to form unfamiliar faces when aligned with each other. The perception of a novel configuration in such composite stimuli is sufficiently convincing to interfere with identification of the constituent parts (experiment 1), but this effect disappears when stimuli are inverted (experiment 2). Difficulty in identifying the parts of upright composites is found even for stimuli made from parts of unfamiliar faces that have only ever been encountered as face fragments (experiment 3). An equivalent effect is found for composites made from internal and external facial features of well-known people (experiment 4). These findings demonstrate the importance of configurational information in face perception, and that configurations are only properly perceived in upright faces.     

College grade point average as a personnel selection device: ethnic group differences and potential adverse impact

College grade point average (GPA) is often used in a variety of ways in personnel selection. Unfortunately, there is little empirical research literature in human resource management that informs researchers or practitioners about the magnitude of ethnic group differences and any potential adverse impact implications when using cumulative GPA for selection. Data from a medium-sized university in the Southeast (N = 7,498) indicate that the standardized average Black-White difference for cumulative GPA in the senior year is d = 0.78. The authors also conducted analyses at 3 GPA screens (3.00, 3.25, and 3.50) to demonstrate that employers (or educators) might face adverse impact at all 3 levels if GPA continues to be implemented as part of a selection system. Implications and future research are discussed.     

Commentary on Pareto‐Optimality as a Rationale for Adverse Impact Reduction: What would organizations do?

This paper offers an organizational perspective about the Pareto‐optimality of trade‐offs between selection quality and adverse impact reduction described by De Corte, Lievens and Sackett. Based on considerations of culture and human resource strategy, the need to understand the impact of any trade‐off, and a desire to compare this approach to alternatives, it is concluded that organizations may be strategically disinclined to implement such trade‐offs and, if interested, organizations would desire more information than is available from the accumulated research in this domain. In particular, validity differences between Pareto‐optimal composites are not likely to be good indicators of selection quality differences and the comparative effectiveness of alternative approaches is not clear.