Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

Previous studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta-analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta-analysis of the Hispanic–White mean difference (d-value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d-values for indirect range restriction and combine them with other meta-analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion-related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.     

The Use of Mobile Devices in High‐stakes Remotely Delivered Assessments and Testing

With Internet access no longer restricted to desktop and laptop computers, job applicants now have the opportunity to complete remotely delivered assessments on mobile, handheld small screen devices such as smartphones, and personal digital assistants. In this study, a large dataset is used to investigate demographic and score differences between job applicants who completed a remotely delivered high‐stakes assessment on a mobile device and those who completed it on a nonmobile device. Based on a sample of 3,575,207 job applicants who completed an unproctored Internet‐based assessment between January 2011 and April 2012, the percentage of applicants completing the assessment on a mobile device was small, 1.93%, but nevertheless represented more than 69,000 people. Overall, there were small test‐taker demographic differences in the use of mobile devices versus nonmobile devices in that mobile devices were slightly more likely to be used by women, African‐Americans and Hispanics, and younger applicants. Scores on a personality measure were similar for mobile and nonmobile devices but scores on a general mental ability test were substantially lower for mobile devices. Tests of measurement invariance also indicated equivalence across the mobile and nonmobile samples. Test taker and organizational implications for completing remotely delivered high‐stakes noncognitive and cognitive assessments on mobile versus nonmobile devices are discussed.     

The impact of corrections for faking on the validity of noncognitive measures in selection settings

In selection research and practice, there have been many attempts to correct scores on noncognitive measures for applicants who may have faked their responses somehow. A related approach with more impact would be identifying and removing faking applicants from consideration for employment entirely, replacing them with high-scoring alternatives. The current study demonstrates that under typical conditions found in selection, even this latter approach has minimal impact on mean performance levels. Results indicate about .1 SD change in mean performance across a range of typical correlations between a faking measure and the criterion. Where trait scores were corrected only for suspected faking, and applicants not removed or replaced, the minimal impact the authors found on mean performance was reduced even further. By comparison, the impact of selection ratio and test validity is much larger across a range of realistic levels of selection ratios and validities. If selection researchers are interested only in maximizing predicted performance or validity, the use of faking measures to correct scores or remove applicants from further employment consideration will produce minimal effects.     

Can Changes in Differential Dropout Rates Reduce Adverse Impact? A Computer Simulation Study of a Multi‐wave Selection System

A simulation was used to explore the effects of variations in the rate at which applicants drop out of selection processes on racial differences in selection outcomes. Archival data was used to simulate a realistic range of selection scenarios in which test score differences between groups and selection ratios varied. The basis for dropping out was manipulated in two separate studies. Study 1 simulated dropout decisions that occurred at random within racial subgroups; in this study, dropout rates of minority versus White candidates were varied. Study 2 examined dropout decisions that occurred as a function of test scores. Results from both studies showed that mean test score differences between White and minority applicants have the largest influence on adverse impact. Interventions designed to reduce the tendency of minority applicants to withdraw from selection are likely to have, at best, small effects on the adverse impact of selection tests.     

Faking on Personality Measures: Implications for selection involving multiple predictors

The potential for faking on noncognitive measures in high stakes testing situations remains a concern for many selection researchers and practitioners. However, the majority of previous research examining the practical effects of faking on noncognitive assessments has focused on these measures in isolation, rather than the more common situation in which they are used in combination with other predictors. The present simulation examined the effects of faking on a conscientiousness measure on criterion‐related validity, mean performance of those selected, and selection decision consistency when hiring decisions were based on this measure alone vs in combination with two other predictors across a range of likely selection scenarios. Overall, results indicated that including additional predictors substantially reduced – but did not eliminate – the negative effects of faking. Faking effects varied across outcomes and selection scenarios, with effects ranging from trivial to noteworthy even for multiple‐predictor selection. Implications for future research and practice are discussed.     

An Investigation of Stereotype Threat in Employment Tests1

This investigation extends stereotype threat research by examining its effect on sex and ethnicity in employment testing. Study 1 used archival data from a company‐specific cognitive ability test, demonstrating score increases (greater for some minorities) when applicants completed demographic items after a cognitive skills test, rather than before the test. The first experiment (Study 2) used college students taking the same company‐specific test with inconclusive results. Possible reasons dictated the design of a second experiment with a college population (Study 3). This study used the Wonderlic Personnel Test (WPT) and investigated priming through positioning of demographic items, as well as high/low threat. Positioning of the demographic items after the WPT was associated with an increase in test scores.     

Educational attainment as a proxy for cognitive ability in selection: effects on levels of cognitive ability and adverse impact

The authors examined the differences in mean level of cognitive ability and adverse impact that can be expected when selecting employees solely on educational attainment as a proxy for cognitive ability versus selecting employees directly on cognitive ability. Selection using cognitive ability worked as a more efficient cognitive screen. Imposing an educational attainment standard of at least 1 year of college, though, did result in noticeably higher levels of cognitive ability in potential applicant pools than did random selection, meaning that educational attainment does work as a cognitive screen. These results held not only in a nationally representative sample but also within and across 6 different occupational groups. Finally, adverse impact is examined for selection using educational attainment, compared with selection on the basis of cognitive ability.     

LinkedIn as a new selection method: Psychometric properties and assessment approach

Various surveys suggest LinkedIn is used as a screening and selection tool by many hiring managers. Despite this widespread use, fairly little is known about whether LinkedIn meets established selection criteria, such as reliability, validity, and legality (i.e., no adverse impact). We examine the properties of LinkedIn-based assessments in two studies. Study 1 shows that raters reach acceptable levels of consistency in their assessments of applicant skills, personality, and cognitive ability. Initial ratings also correlate with subsequent ratings done 1-year later (i.e., demonstrating temporal stability), with slightly higher correlations when profile updates are taken into account. Initial LinkedIn-based ratings correlate with self-reports for more visible skills (leadership, communication, and planning) and personality traits (Extraversion), and for cognitive ability. LinkedIn-based hiring recommendations are positively associated with indicators of career success. Potential adverse impact is also limited. Profiles that are longer, include a picture, and have more connections are rated more positively. Some of those features are valid cues to applicants’ characteristics (e.g., applicants high on Conscientiousness have longer profiles). In Study 2, we show that an itemized LinkedIn assessment is more effective than a global assessment. Implications of these findings for selection and future research are discussed.     

Computing Pareto‐optimal Predictor Composites for Complex Selection Decisions

A complex selection situation encompasses vacancies for several different positions and applicants that apply simultaneously for one or several of these positions. This article presents an analytic method for estimating the expected selection quality, as well as the adverse impact ratio of these complex selections, when the decisions are based on a single predictor composite score. In addition, the method is integrated within a broader decision‐making framework for designing complex selection decisions that show a Pareto‐optimal balance between the selection quality and diversity goals. Finally, the decision aid is used to demonstrate the importance of applying the appropriate selection format (either the simple or the complex format) when exploring the front of Pareto‐optimal outcomes of planned selections.     

Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability

Previous studies have indicated that as many as 25% to 50% of applicants in organizational and educational settings are retested with measures of cognitive ability. Researchers have shown that practice effects are found across measurement occasions such that scores improve when these applicants retest. In this study, the authors used meta-analysis to summarize the results of 50 studies of practice effects for tests of cognitive ability. Results from 107 samples and 134,436 participants revealed an adjusted overall effect size of .26. Moderator analyses indicated that effects were larger when practice was accompanied by test coaching and when identical forms were used. Additional research is needed to understand the impact of retesting on the validity inferences drawn from test scores.     

Noncognitive predictors of intelligence and academic achievement: An important role of confidence

Recent evidence about the relationship between cognitive tests and psychological noncognitive variables is reviewed. Noncognitive measures can be ordered with respect to their predictive validity. Many are poor predictors of intelligence and achievement. Measures of rationality, self-assessment of intelligence, Openness to Experience and self-concept correlate up to .35 with cognitive performance. Some domain-specific self-beliefs (self-efficacy and anxiety) have correlations with appropriate achievement tests that can reach .45. The best predictors of any kind of cognitive performance are measures of confidence (frequently reported correlations of .45 and above) that can capture a major part of predictive validity of the three self-beliefs. The role of self-beliefs has attracted much interest in education but their role in predicting performance on tests of fluid intelligence is likely to be low. However, self-beliefs and confidence in particular may prove to be the most potent noncognitive influences on the development of acculturated knowledge that is captured by measures of crystallized intelligence.     

The operational validity of a video-based situational judgment test for medical college admissions: illustrating the importance of matching predictor and criterion construct domains

This study is part of a trend of examining noncognitive predictors, for example, a situational judgment test (SJT), as supplements to cognitive predictors for making college admission decisions. The authors examined criterion data over multiple academic years and universities. The criterion domain was broadly conceptualized, including both cognitive and interpersonal domains. The sample consisted of 7,197 candidates of the Medical and Dental Studies Admission Exam in Belgium. Results confirmed the importance of cognitive predictors. A video-based SJT was differentially valid for predicting overall grade point average for different curricula. The SJT showed incremental validity over cognitively oriented measures for curricula that included interpersonal courses, but not for other curricula. The SJT became more valid through the years. This demonstrates the importance of carefully specifying predictor- criterion linkages and of differentiating both predictor and criterion constructs.     

Understanding subgroup differences with general mental ability tests in employment selection: Exploring socio-cultural factors across inter-generational groups

In employment selection, general mental ability (GMA) tests predict training and job performance but also lead to subgroup differences which in turn can produce adverse impact against minority groups. Although researchers have explored genetic, developmental, and environmental explanations for ethnic group differences, few studies have explored socio-cultural factors comparing immigrant and non-immigrant job applicants. Given that many ethnic job applicants may also be immigrants, understanding these factors can provide insight into GMA test score differences. The purpose of this paper is to explore the impact of individual and socio-cultural factors on GMA test scores with immigrant and non-immigrant bus driver applicants. This is the first study of its kind to our knowledge that has attempted to disentangle the effects of socio-cultural factors from race/ethnicity in the study of subgroup differences. Incorporating these variables between non-visible minority and minority groups accounted for considerable variance in GMA test scores across groups. The implications of focusing on socio-cultural variables to enhance our understanding of subgroup differences are discussed. Our results specifically suggest that practitioners attend to the issue of the intersecting grounds of potential discrimination when using GMA tests in personnel selection.     

Predicting Adaptive Performance: Further Tests of a Model of Adaptability

This research further investigates an 8-dimension taxonomy of adaptive performance developed by Pulakos, Arad, Donovan, and Plamondon (2000) and the usefulness of past experience, interest, and self-efficacy predictors developed to measure these same 8 dimensions for predicting adaptive job performance. Participants in the concurrent, criterion-related validation study included 739 military personnel. They completed the 3 adaptability measures as well as more traditional cognitive and noncognitive predictors. Supervisors of the study participants rated their adaptive job performance. Results showed support for the 8-dimension model of adaptability. In addition, cognitive ability, personality, and the new adaptability predictors were shown to predict adaptive performance, with some past experience items adding incremental validity beyond the more traditional cognitive ability and personality measures. Results are discussed.     

Noncognitive Predictors of Performance During Explosive Ordnance Disposal Training

This article identifies some noncognitive characteristics that are reliably as- sociated with performance in explosive ordnance disposal (EOD) training. In a longitudinal study of Navy students and fleet EOD divers (N = 145), a composite of three test batteries (including a vocational preference meas- ure, a personality measure, and a set of physical performance tests) correlated .51, .47, and .61, with second-class diver certification, EOD training course completion, and performance in the fleet, respectively. The personality meas- ure also predicted performance as an EOD technician in an Army sample (N = 49). Although cognitive ability test scores are used almost exclusively to select personnel for technical training in the military, these results suggest that noncognitive performance measures may also provide information useful for screening potential students and reducing attrition.     

Prior cognitive activity implicitly modulates subsequent emotional responses to subliminally presented emotional stimuli

It has been reported that engagement in several kinds of cognitive activity can successfully inhibit unpleasant emotions. In this study, we tried to replicate the previous finding that cognitive activity can modulate subsequent psychological and physiological emotional processes and to investigate whether prior cognitive activity can attenuate implicit emotional processes triggered by subliminal emotional stimuli. Sixty students were randomly divided into three groups (cognitive task group, noncognitive task group, control group). The cognitive task group was asked to engage in an n-back task, while the control group was asked to stay calm. The noncognitive task group was asked to do a handgrip-squeezing task. All participants then engaged in a version of a subliminal affective priming task where they were unconsciously exposed to affectively negative pictures. The cognitive task group showed lower negative experiences after the subliminal affective priming task and a substantial reduction in their heart rate responses, as compared with the other groups. These results provide evidence that engagement in cognitive activity can attenuate emotional processes in an automatic and unconscious manner.     

FAKING IN PERSONNEL SELECTION: TRADEOFFS IN PERFORMANCE VERSUS FAIRNESS RESULTING FROM TWO CUT-SCORE STRATEGIES

Most faking research has examined the use of personality measures when using top-down selection. We used simulation to examine the use of personality measures in selection systems using cut scores and outlined a number of issues unique to these situations. In particular, we compared the use of 2 methods of setting cut scores on personality measures: applicant-data-derived (ADD) and nonapplicant-data-derived (NADD) cut-score strategies. We demonstrated that the ADD strategy maximized mean performance resulting from the selection system in the face of applicant faking but that this strategy also resulted in the displacement of deserving applicants by fakers (which has fairness implications). On the other hand, the NADD strategy minimized displacement of deserving applicants but at the cost of some mean performance. Therefore, the use of the ADD versus NADD strategies can be viewed as a strategic decision to be made by the organization, as there is a tradeoff between the 2 strategies in effects on performance versus fairness to applicants. We quantitatively outlined these tradeoffs at various selection ratios, levels of validity, and amounts of faking in the applicant pool.     

Self‐promotion Statements in Video Resumes: Frequency,intensity, and gender effects on job applicant evaluation

Although video resumes have received a substantial amount of media attention and there seems to be a growing awareness among Human Resource professionals of video‐based job applications, little is known about the effect of video resumes on applicant evaluation. This research investigates the effectiveness of self‐promotion within the context of video resumes. Self‐promotion frequency and intensity and applicant gender were manipulated. Ratings by recruiters and college students indicate that high levels of self‐promotion in video resumes are ineffective for male applicants and potentially detrimental for female applicants. Job applicants should use caution when attempting to promote themselves using video resumes. More research is needed on impression management tactics used at the earliest stages of selection and on the mechanisms operating within video resumes that impact applicant evaluation.     

CORRESPONDENCE BETWEEN ESTIMATES OF CONTENT AND CRITERION-RELATED VALIDITY VALUES

The correspondence between inferences made using two validation strategies–content and criterion-related–were examined in a specific personnel selection application. Empirical validity values and Law-she's (1975) content validity ratios (CVR) were obtained for items from three structured interview guides used in the selection of insurance agents. Ratings of each item by over 300 field managers were used to calculate the CVR values. Statistically significant, yet modest correlations were found between empirical item validities and content validities for an interview guide used to select applicants with prior insurance sales experience. No significant differences were found among these correlations by comparing job experts of different levels of managerial experience and experience in selection. Data for the interview guide used to select experienced applicants also indicated that a content validity approach can be useful in developing a selection instrument with an empirically valid composite rating. The hypotheses were not confirmed for interview guides used to select applicants with no prior insurance sales experience. The practical importance of these results are discussed, as are plans for future research.