The Role of Cognitive Ability Tests in Employment Selection

Cognitive ability tests correlate with measures of job performance across many jobs. However, cognitive ability tests produce racial differences that are 3 to 5 times larger than other predictors-such as biodata, personality inventories, and the structured interview-that are valid predictors of job performance. Given that (a) cognitive ability tests can be combined with other predictors such that adverse impact is reduced while overall validity is increased, and (b) alternative predictors with less adverse impact can produce validity coefficients comparable to those obtained with cognitive ability tests alone, sole reliance on cognitive ability tests when alternatives are available is unwarranted.     

Chasing the Title VII Holy Grail: The Pitfalls of Guaranteeing Adverse Impact Elimination

Purpose

Title VII of the Civil Rights Act of 1964 provided industrial/organizational (I/O) psychologists with a unique role as professional test developers and consultants involved in assisting organizations in establishing the job-relatedness/validity defense to charges of discrimination, specifically charges based on an adverse or disparate impact theory. However, these activities have transmogrified into the fairly common occurrence of public municipalities and organizations demanding the reduction or absence of adverse impact as part of the scope of work or contracts and for practitioners and consultants to guarantee adverse impact reduction or elimination a priori. Plaintiffs and their experts also routinely argue that the observed adverse impact could have been allayed or eliminated if the defendant had only just used alternative testing methods. This then begs the following question: “Are there well established techniques and procedures that can reduce, minimize, or eliminate adverse impact in a predictable, generalizable, and replicable fashion in the same manner that we might guarantee validity?” The present paper seeks to answer this question.

Approach and Findings

With the preceding as a backdrop, the present paper identifies and discusses four overlooked critical attributes of adverse impact that collectively and in conjunction work against and obviate adverse impact reduction and elimination guarantees.

Conclusions and Implications

We conclude that the search for guaranteed adverse impact reduction or elimination is a “Holy Grail” and that we should avoid predictions and guarantees regarding adverse impact elimination in specific situations, including those based on the inclusion of “alternative” selection devices. However, in the context of civil rights legislation, and the intersection of I/O psychologists with said legislation, what we can guarantee as a science and profession are sound and valid tests and assessment devices that can be defended accordingly should the use of said tests and devices be challenged.     

THE DIVERSITY–VALIDITY DILEMMA: STRATEGIES FOR REDUCING RACIOETHNIC AND SEX SUBGROUP DIFFERENCES AND ADVERSE IMPACT IN SELECTION

Pyburn, Ployhart, and Kravitz (this issue, 2008) introduced the diversity–validity dilemma: that some of the most valid predictors of job performance are also associated with large racioethnic and sex subgroup predictor score differences. This article examines 16 selection strategies hypothesized to minimize racioethnic and sex subgroup differences and adverse impact and, hence, balance diversity and validity. Rather than presenting a highly technical review, our purpose is to provide practitioners with a concise summary, paying particular attention to comparing and contrasting the effectiveness of the strategies and reporting new developments. The paper is organized around 4 key questions: (a) Which strategies are most effective for reducing subgroup differences? (b) Which strategies do not involve a validity tradeoff? (c) What are the major new developments in strategies for reducing adverse impact? (d) What are the major new developments in alternative predictor measurement methods (e.g., interviews, situational judgment tests, assessment centers) for reducing adverse impact? We then conclude with recommendations and caveats for how to best balance diversity and validity. These ideas are developed further in Kravitz (this issue, 2008) , who considers even broader approaches for solving the diversity–validity dilemma.     

Personality Testing in Personnel Selection: Adverse impact and differential hiring rates

Personality tests are often used in selection and have demonstrated predictive validity across a variety of occupational groups and performance criteria. Although different selection decision methods can be used to make selection decisions (e.g., compensatory top down, compensatory with sliding bands, noncompensatory) from personality test results, there is a paucity of research addressing the influence of these different selection decision methods on issues such as, adverse impact and differential hiring rates. This gap in the literature is redressed in the current study. Results from 398 bus operator candidates indicated that there may be adverse impact and differential hiring rate issues depending on the selection decision method used and the designated group being assessed. Implications and future research directions are discussed.     

Examination of ethnic group differential responding on a biodata instrument

Biodata is a selection device that is often cited for high criterion validity and low adverse impact, yet little research examines the latter point. Further, little is known about whether differential biodata item responding by ethnic group occurs, and if so, whether this impacts ethnic group differences, predictive bias, adverse impact, and criterion validity. Data from 5,242 applicants for a government agency position indicated that elimination of biodata item response options with differential ethnic group selection rates from scoring consideration resulted in a large decrease in ethnic group differences, no predictive bias, less potential adverse impact, and a nonsignificant decrease in criterion validity. Results are compared to a general mental ability measure. Implications and future research directions are discussed.     

Some Comments on Pareto Thinking,Test Validity,and Adverse Impact: When ‘and’ is optimal and ‘or’ is a trade‐off

De Corte, Lievens, and Sackett add to the literature on selection test validity and adverse impact (AI). Their Pareto‐based weighting scheme essentially asks organizations if they are willing to give up some validity to hopefully achieve some reduction in AI. We considered their approach and conclusions in relation to the regression weighting method we used, and we offer five points that reflect our observations as well as our shared goals. We hope our comments, like their work in this field, will invigorate the pursuit of new ways of examining, and one day resolving, the persistent concern regarding the AI associated with valid selection tests.     

Commentary on Pareto‐Optimality as a Rationale for Adverse Impact Reduction: What would organizations do?

This paper offers an organizational perspective about the Pareto‐optimality of trade‐offs between selection quality and adverse impact reduction described by De Corte, Lievens and Sackett. Based on considerations of culture and human resource strategy, the need to understand the impact of any trade‐off, and a desire to compare this approach to alternatives, it is concluded that organizations may be strategically disinclined to implement such trade‐offs and, if interested, organizations would desire more information than is available from the accumulated research in this domain. In particular, validity differences between Pareto‐optimal composites are not likely to be good indicators of selection quality differences and the comparative effectiveness of alternative approaches is not clear.     

LinkedIn as a new selection method: Psychometric properties and assessment approach

Various surveys suggest LinkedIn is used as a screening and selection tool by many hiring managers. Despite this widespread use, fairly little is known about whether LinkedIn meets established selection criteria, such as reliability, validity, and legality (i.e., no adverse impact). We examine the properties of LinkedIn-based assessments in two studies. Study 1 shows that raters reach acceptable levels of consistency in their assessments of applicant skills, personality, and cognitive ability. Initial ratings also correlate with subsequent ratings done 1-year later (i.e., demonstrating temporal stability), with slightly higher correlations when profile updates are taken into account. Initial LinkedIn-based ratings correlate with self-reports for more visible skills (leadership, communication, and planning) and personality traits (Extraversion), and for cognitive ability. LinkedIn-based hiring recommendations are positively associated with indicators of career success. Potential adverse impact is also limited. Profiles that are longer, include a picture, and have more connections are rated more positively. Some of those features are valid cues to applicants’ characteristics (e.g., applicants high on Conscientiousness have longer profiles). In Study 2, we show that an itemized LinkedIn assessment is more effective than a global assessment. Implications of these findings for selection and future research are discussed.     

EXAMINING SELECTION UTILITY WHERE COMPETING PREDICTORS DIFFER IN ADVERSE IMPACT

This paper discusses the roles of validity, cut score choice, and adverse impact on selection system utility using data from two concurrent validation studies. We contrast an assessment center and published aptitude test on several metrics, including validity, testing costs, adverse impact, and utility. The assessment center produced slightly lower validity than the aptitude test while costing roughly 10 times as much per candidate. In spite of these advantages for the aptitude test, the assessment center produced so much less adverse impact its operational utility would be higher given cut scores likely to be chosen in this organization. Potential concerns with applying net utility models to this type of situation are discussed in comparison to gross utility models.     

Work Sample Testing

The general evidence supporting the use of work sample tests (good criterion — related validity, positive applicant reaction and job preview capability) is reviewed, limitations are also noted. The extent to which work sample tests may be used to limit adverse impact in selection is reviewed and the extent to which work sample tests may be used to complement other selection instruments, in particular, cognitive ability tests, is discussed. Work sample tests are based on attempts to embody key tasks within the job in a selection instrument. Five distinctive (though not mutually exclusive) types of work sample tests are identified and validity evidence is reviewed. Key dimensions (bandwidth, fidelity, task specificity, necessary experience, type of tasks and mode of presentation and response) are identified and used to compare different types of work sample tests.     

Higher Cost,Lower Validity and Higher Utility: Comparing the Utilities of Two Tests that Differ in Validity,Costs and Selectivity

Traditional approaches to comparing the utility of two tests have not systematically considered the effects of different levels of selectivity that are feasible and appropriate in various selection situations. For example, employers who hope to avoid adverse impact often find they can be more selective with some tests than with others. We conducted two studies to compare the utilities of two tests that differ in costs, validity, and feasible levels of selectivity which can be employed. First, an analytical solution was conducted starting with a standard formula for utility. This analysis showed that for both fixed and variable hiring costs, a higher-cost, lower-validity procedure can have higher utility than a lower-cost, higher-validity procedure when the selection ratios permissible using the two procedures are sufficiently (yet realistically) different. Second, using a computer simulation method, several combinations of the critical variables were varied systematically to detect the limits of this effect in a finite set of specific selection situations. The results showed that the existence of more severe levels of adverse impact greatly reduced the utility of a written test with relatively high validity and low cost in comparison with an assessment center with lower validity and higher cost. Both studies showed that the consideration of selectivity can yield surprising conclusions about the comparative utility of two tests. Even if one test has lower validity and higher cost than a second test, the first may yield higher utility if it allows the organization to exercise stricter levels of selectivity.     

The relationship of specific mental ability measures compared to a general mental ability measure to quality and quantity performance on a clerical job sample

Barrett (1993) presented evidence that mental ability measures designed specifically for the prediction of performance in tasks and jobs have validity at least equal to that of tests of general mental ability, while having advantages such as lower adverse impact. The current study demonstrates that the match in specificity of predictor and criterion can be attained through simple changes in the scoring of standard mental and clerical ability tests. A sample of 51 university students completed a battery of pre-employment tests and, a week or two later, worked on a clerical job sample. The results indicated that quality of performance (number of errors on the task) was best predicted using the number of errors made on the predictor tests while quantity of work was only predicted by the number correct. A test of general mental ability did not correlate with either criterion. These results contradicted the conclusions of Schmidt (1993) that specific ability tests have no incremental validity above a general ability test.Ralph Alexander died before this report was complete. Consequently, the conclusions are those of the first two authors.     

INTEGRITY TESTING FOR PERSONNEL SELECTION: AN UPDATE

Sackett and M. M. Harris (1984) reviewed published and unpublished literature on the use of commercially available paper-and-pencil integrity tests for employee selection. This paper reviews recent developments in this area. Legal issues related to the use of the polygraph, and integrity tests are discussed. Empirical research on the reliability, criterion-related validity, construct validity, fakability, and adverse impact of integrity tests is reviewed. Major changes since the previous review include (1) the emergence of two distinct types of tests, labeled overt integrity tests and personality-oriented measures, (2) a broadening of the criteria used for test validation to include global performance, absence, turnover, and forms of counterproductivity other than theft, and (3) an increase in the use of external criteria (e.g., detected theft, turnover) rather than self-report criteria (e.g., admissions of theft) in validation research. Difficulties in interpreting research findings and practical constraints on the use of integrity tests are discussed.     

PERSONALITY TESTS AT THE CROSSROADS: A RESPONSE TO MORGESON, CAMPION, DIPBOYE, HOLLENBECK, MURPHY, AND SCHMITT (2007)

Despite widespread and growing acceptance that published personality tests are valid predictors of job performance, Morgeson et al. (2007) propose they be abandoned in personnel selection because average validity estimates are low. Our review of the literature shows that Morgeson et al.'s skepticism is unfounded. Meta-analyses have demonstrated that published personality tests, in fact, yield useful validity estimates when validation is based on confirmatory research using job analysis and taking into account the bidirectionality of trait–performance linkages. Further gains are likely by use of narrow over broad measures, multivariate prediction, and theory attuned to the complexities of trait expression and evaluation at work. Morgeson et al. also suggest that faking has little, if any, impact on personality test validity and that it may even contribute positively to job performance. Job applicant research suggests that faking under true hiring conditions attenuates personality test validity but that validity is still sufficiently strong to warrant personality test use in hiring. Contrary to Morgeson et al., we argue that the full value of published personality tests in organizations has yet to be realized, calling for programmatic theory-driven research.     

Pareto‐Optimal Predictor Composite Formation: A complementary approach to alleviating the selection quality/adverse impact dilemma

In our rejoinder to the comments of Kehoe (this issue) and Potosky, Bobko and Roth (this issue) we emphasize that our proposal on Pareto‐optimal predictor composite formation is a complementary and not a competitive alternative for reducing the tension between selection quality and adverse impact. Our work addresses the decisions to be made once one has decided to use a predictor composite. We also further clarify the basic features of Pareto‐optimal tradeoffs and Pareto‐optimal composites within the context of personnel selection. In particular, we indicate that Pareto‐optimal tradeoffs between validity and adverse impact emerge because these goals are different and not because of any dualism between them.     

RETEST EFFECTS IN OPERATIONAL SELECTION SETTINGS: DEVELOPMENT AND TEST OF A FRAMEWORK

This study proposes a framework for examining the effects of retaking tests in operational selection settings. A central feature of this framework is the distinction between within-person and between-person retest effects. This framework is used to develop hypotheses about retest effects for exemplars of 3 types of tests (knowledge tests, cognitive ability tests, and situational judgment tests) and to test these hypotheses in a high stakes selection setting (admission to medical studies in Belgium). Analyses of within-person retest effects showed that mean scores of repeat test takers were one-third of a standard deviation higher for the knowledge test and situational judgment test and one-half of a standard deviation higher for the cognitive ability test. The validity coefficients for the knowledge test differed significantly depending on whether examinees' test scores on the first versus second administration were used, with the latter being more valid. Analyses of between-person retest effects on the prediction of academic performance showed that the same test score led to higher levels of performance for those passing on the first attempt than for those passing on the second attempt. The implications of these results are discussed in light of extant retesting practice.     

The impact of corrections for faking on the validity of noncognitive measures in selection settings

In selection research and practice, there have been many attempts to correct scores on noncognitive measures for applicants who may have faked their responses somehow. A related approach with more impact would be identifying and removing faking applicants from consideration for employment entirely, replacing them with high-scoring alternatives. The current study demonstrates that under typical conditions found in selection, even this latter approach has minimal impact on mean performance levels. Results indicate about .1 SD change in mean performance across a range of typical correlations between a faking measure and the criterion. Where trait scores were corrected only for suspected faking, and applicants not removed or replaced, the minimal impact the authors found on mean performance was reduced even further. By comparison, the impact of selection ratio and test validity is much larger across a range of realistic levels of selection ratios and validities. If selection researchers are interested only in maximizing predicted performance or validity, the use of faking measures to correct scores or remove applicants from further employment consideration will produce minimal effects.     

Modeling the behavior of the 4/5ths rule for determining adverse impact: reasons for caution

The Equal Employment Opportunity Commission's 4/5ths rule has been used for over 20 years in applied psychology and employment law. The rule signals that there is adverse impact when the protected group selection ratio is less than 80% of the highest scoring group's selection ratio. We conducted several simulations and found, consistent with some previous management science literature, that the 4/5ths rule often resulted in false-positive readings of adverse impact even when there were no underlying (population) standardized group differences between subgroups. We then incorporated tests of statistical significance and found that adding such tests to the 4/5ths rule eliminated many false-positive indications of adverse impact. We also examined simulated selection systems based on meta-analytic values from the selection literature. The frequency of adverse impact signals from the 4/5ths rule increased markedly relative to simulations with no subgroup population differences. Adding statistical tests mitigated the number of indications of adverse impact to some extent.     

METAANALYSES OF VALIDITY STUDIES PUBLISHED BETWEEN 1964 AND 1982 AND THE INVESTIGATION OF STUDY CHARACTERISTICS

Review and metaanalyses of published validation studies for the years 1964-1982 of Journal of Applied Psychology and Personnel Psychology were undertaken to examine the effect of (1) research design; (2) criterion used; (3) type of selection instrument used; (4) occupational group studies; and (5) predictor-criterion combination on the level of observed validity coefficients. Results indicate that concurrent validation designs produce validity coefficients roughly equivalent to those obtained in predictive validation designs and that both of these designs produce higher validity coefficients than does a predictive design which includes use of the selection instrument. Of the criteria examined, performance rating criteria generally produced lower validity coefficients than did the use of other more "objective" criteria. In comparing the validities of various types of predictors, it was found cognitive ability tests were not superior to other predictors such as assessment centers, work samples, and supervisory/peer evaluations as has been found in previous metaanalytic work. Personality measures were clearly less valid. Compared to previous validity generalization work, much unexplained variance in validity coefficients remained after corrections for differences in sample size. Finally, the studies reviewed were deficient for our purposes with respect to the data reported. Selection ratios, standard deviations, reliabilities, predictor and criterion intercorrelations were rarely and inconsistently reported. There are also many predictor-criterion relationships for which very few validation efforts have been undertaken.     

THE IMPACT OF JOB ANALYSIS ON EMPLOYMENT TEST VALIDATION FOR MINORITY AND NONMINORITY ACCOUNTING PERSONNEL

The purposes of the present study were (a) to examine the comparative validity of a written job knowledge test constructed on the basis of a systematic job analysis with that of a commercial employment test selected in the absence of a prior job analysis for accounting positions and (b) to determine the fairness of each test for minority and nonminority job applicants. Results indicated that the job knowledge test was a valid and unbiased predictor of relevant criteria of job performance while the commercial employment test produced adverse impact and lacked validity. Implications of the results for future research studies and test validation efforts involving differential prediction are discussed.