Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

Previous studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta-analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta-analysis of the Hispanic–White mean difference (d-value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d-values for indirect range restriction and combine them with other meta-analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion-related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.     

Investigating Retesting Effects in a Personnel Selection Context

Repeat applicants to a fire‐fighter position undertook the same cognitive ability and situational judgment tests on multiple occasions and the resultant practice effects were investigated. Practice effects of approximately two fifths of a standard deviation were observed between Times 1 and 2 on timed tests of Abstract Reasoning and Mechanical Comprehension. Smaller practice effects were observed, however, on a timed test of numerical comprehension ability and the untimed Teamwork Knowledge Skills and Abilities situational judgment test. For all four tests, the additional practice effect observed at Time 3 were smaller and nonsignificant; a result supplemented by latent growth curve analyses.     

RETEST EFFECTS IN OPERATIONAL SELECTION SETTINGS: DEVELOPMENT AND TEST OF A FRAMEWORK

This study proposes a framework for examining the effects of retaking tests in operational selection settings. A central feature of this framework is the distinction between within-person and between-person retest effects. This framework is used to develop hypotheses about retest effects for exemplars of 3 types of tests (knowledge tests, cognitive ability tests, and situational judgment tests) and to test these hypotheses in a high stakes selection setting (admission to medical studies in Belgium). Analyses of within-person retest effects showed that mean scores of repeat test takers were one-third of a standard deviation higher for the knowledge test and situational judgment test and one-half of a standard deviation higher for the cognitive ability test. The validity coefficients for the knowledge test differed significantly depending on whether examinees' test scores on the first versus second administration were used, with the latter being more valid. Analyses of between-person retest effects on the prediction of academic performance showed that the same test score led to higher levels of performance for those passing on the first attempt than for those passing on the second attempt. The implications of these results are discussed in light of extant retesting practice.     

Test‐Specificity of the Advantage of Retaking Cognitive Ability Tests

In selection settings, when people retake the same cognitive ability tests, scores are generally positively biased. Our approach aimed to test whether these previous exposure effects are test‐specific or transferable to other tests. We compared the differences between scores for first‐time test takers and retakers for 2 kinds of material: ‘old’ tests, known only to the retakers, and ‘new’ tests, unknown to both groups. The current study used data collected during 2 sessions –S and S+1 – of a selection process for entry into the French national air transport pilot training system, with at least 500 first‐time test takers and 130 retakers in each session. For Session S, on average, retakers scored higher on the ‘old’ tests, but not on the ‘new’ tests. Moreover, the material that was new to retakers at Session S was old at Session S+1, and the finding for old tests could be replicated at Session S+1. The finding that the acquired skills that led to higher scores on old tests were only test‐specific is discussed.     

Personality measurement, faking, and employment selection

Real job applicants completed a 5-factor model personality measure as part of the job application process. They were rejected; 6 months later they (n = 5,266) reapplied for the same job and completed the same personality measure. Results indicated that 5.2% or fewer improved their scores on any scale on the 2nd occasion; moreover, scale scores were as likely to change in the negative direction as the positive. Only 3 applicants changed scores on all 5 scales beyond a 95% confidence threshold. Construct validity of the personality scales remained intact across the 2 administrations, and the same structural model provided an acceptable fit to the scale score matrix on both occasions. For the small number of applicants whose scores changed beyond the standard error of measurement, the authors found the changes were systematic and predictable using measures of social skill, social desirability, and integrity. Results suggest that faking on personality measures is not a significant problem in real-world selection settings.     

Establishing the Measurement Equivalence of Online Selection Assessments Delivered on Mobile versus Nonmobile Devices

Recent usage data suggest job applicants are completing online selection assessments using mobile devices (e.g., smartphones) in greater numbers. To determine the appropriateness of this new technology, this study examined the measurement equivalence of selection assessments delivered on mobile and nonmobile devices (e.g., personal computers). Measurement invariance tests conducted with multigroup confirmatory factor analysis suggest mobile versions of a cognitive ability‐type assessment, two biodata assessments, a multimedia work simulation, and a text‐based situational judgment test appear to be equivalent to nonmobile versions. However, mobile device user latent means were half a standard deviation lower than their nonmobile counterparts for the situational judgment test. Implications for mobile device usage within selection and assessment are discussed.     

Situational Tests in Student Selection: An Examination of Predictive Validity,Adverse Impact,and Construct Validity

The Flemish Admission Exam ‘Medical and Dental Studies’ is comprised of four cognitive ability tests and four situational tests, namely two work samples (i.e., a lecture and a medical text) and two video‐based situational judgement tests (i.e., a physician–patient interaction and a medical expert discussion). On the basis of the Admission Exam scores of 941 candidates (359 men, 582 women) this study shows that situational tests significantly can predict better than cognitive ability tests, with lecture and text emerging as significant predictors. When situational tests are combined with cognitive ability tests, there are no mean gender differences. Situational tests also enable us to measure a broader range of constructs. For example, in this study, the personality factor Openness is related to better situational test performance. Overall, this study demonstrates that situational tests may be a useful complement to traditional student selection procedures.     

Applicant Perceptions of Selection Procedures: The Role of Selection Information,Belief in Tests,and Comparative Anxiety

This study addresses the effects of the provision of information on the reliability and validity of selection procedures and the effects of test–taker attitudes (i.e., belief in tests and comparative anxiety) on fairness perceptions. Prior to an actual selection process, applicants (N= 118) were given either information about the reliability and validity of various selection procedures or no information. Next, they evaluated the fairness of eight selection procedures. No significant effect of selection information was found. Belief in tests had significant effects, with applicants high on test belief giving higher fairness ratings than applicants low on test belief. In addition, an interaction effect between test belief and selection procedure was found. For example, test belief had larger effects on fairness for structured interviews, personality inventories, and cognitive ability tests. No significant effect of comparative anxiety on fairness was found.     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.     

On the Construct Validity of Integrity Tests: Individual and Situational Factors as Predictors of Test Performance

Although integrity tests are widely applied in screening job applicants, there is a need for research for examining the construct validity of these tests. In the present study, a theoretical model examining the causes of destructive behavior in organizational settings was used to develop background data measures of individual and situational variables that might be related to integrity test scores. Subsequently, 692 undergraduates were asked to complete these background data scales along with (a) two overt integrity tests – the Reid Report and the Personnel Selection Inventory, and (b) two personality‐based measures – the delinquency and socialization scales of the California Psychological Inventory. When scores of these measures were correlated with and regressed on the background data scales, it was found that relevant individual variables, such as narcissism and power motives, and relevant situational variables, such as alienation and exposure to negative peer groups, were related to scores on both types of integrity tests. However, a stronger pattern of validity evidence was obtained for the personality‐based measures and, in all cases, situational variables were found to be better predictors than individual variables. The implications of these findings for the validity of inferences drawn from overt and personality‐based integrity tests are discussed.     

Can Changes in Differential Dropout Rates Reduce Adverse Impact? A Computer Simulation Study of a Multi‐wave Selection System

A simulation was used to explore the effects of variations in the rate at which applicants drop out of selection processes on racial differences in selection outcomes. Archival data was used to simulate a realistic range of selection scenarios in which test score differences between groups and selection ratios varied. The basis for dropping out was manipulated in two separate studies. Study 1 simulated dropout decisions that occurred at random within racial subgroups; in this study, dropout rates of minority versus White candidates were varied. Study 2 examined dropout decisions that occurred as a function of test scores. Results from both studies showed that mean test score differences between White and minority applicants have the largest influence on adverse impact. Interventions designed to reduce the tendency of minority applicants to withdraw from selection are likely to have, at best, small effects on the adverse impact of selection tests.     

The Effect of Selection Ratio on Perceptions of the Fairness of a Selection Test Battery

This research examined the effects of the selection ratio on applicants’ perceptions of the fairness of a battery of tests. These fairness perceptions were also examined to assess whether they moderated the validity of the tests. A laboratory study was undertaken to examine the effects of the selection ratio on applicants going through a selection process and to examine the effects of being hired or rejected. Results revealed that the selection ratio did not have an effect on perceptions, but that hired individuals perceived greater distributive fairness. Furthermore, attitudes towards the cognitive ability test moderated the validity of the cognitive ability test, but attitudes toward the personality test did not moderate the validity of the personality test.     

Understanding subgroup differences with general mental ability tests in employment selection: Exploring socio-cultural factors across inter-generational groups

In employment selection, general mental ability (GMA) tests predict training and job performance but also lead to subgroup differences which in turn can produce adverse impact against minority groups. Although researchers have explored genetic, developmental, and environmental explanations for ethnic group differences, few studies have explored socio-cultural factors comparing immigrant and non-immigrant job applicants. Given that many ethnic job applicants may also be immigrants, understanding these factors can provide insight into GMA test score differences. The purpose of this paper is to explore the impact of individual and socio-cultural factors on GMA test scores with immigrant and non-immigrant bus driver applicants. This is the first study of its kind to our knowledge that has attempted to disentangle the effects of socio-cultural factors from race/ethnicity in the study of subgroup differences. Incorporating these variables between non-visible minority and minority groups accounted for considerable variance in GMA test scores across groups. The implications of focusing on socio-cultural variables to enhance our understanding of subgroup differences are discussed. Our results specifically suggest that practitioners attend to the issue of the intersecting grounds of potential discrimination when using GMA tests in personnel selection.     

Elaboration et validation de tests de jugement situationnel comme outil de sélection professionnelle : guide méthodologique

Current views in personnel selection recognize the necessity for situational judgements tests (SJT) which are closed to the real work context of companies. The SJTs have also become a popular selection tool across Europe. SJTs are attractive because they show good validity, positive applicants reactions and can be cost effective to test large group of applicants at once. The aim of this paper is to describe a structured methodology for the elaboration of such test. Each stage is illustrated by examples and psychometrics validation issues are discussed.     

The relationship between cognitive ability and personality scores in selection situations: A meta‐analysis

Several faking theories have identified applicants’ cognitive ability (CA) as a determinant of faking—the intentional distortion of answers by candidates—but the corresponding empirical findings in the area of personality tests are often ambiguous. Following the assumption that CA is important for faking, we expected applicants with high CA to show higher personality scores in selection situations, leading in this case to significant correlations between CA and personality scores, but not in nonselection situations. This meta‐analysis (66 studies, k = 115 individual samples, N = 46,265) showed this pattern of results as well as moderation effects for the study design (laboratory vs. field), the response format of the personality test, and the type of CA test.     

WEB‐BASED AND PAPER‐AND‐PENCIL TESTING OF APPLICANTS IN A PROCTORED SETTING: ARE PERSONALITY,BIODATA, AND SITUATIONAL JUDGMENT TESTS COMPARABLE?

This quasi-experimental study compares the equivalence of proctored Web-based tests to paper-and-pencil tests in a selection setting. The predictor battery was composed of measures of Conscientiousness, Agreeableness, Emotional Stability, a biodata form, and a situational judgment test. Three samples were administered the same test items, but differed with respect to context and test format: 425 incumbents completed a paper-and-pencil version, 2,544 applicants completed a paper-and-pencil version, and 2,356 applicants completed an identical Web-based version (although these individuals were not randomly assigned to test format, the job and organization were the same for all samples). The results found effects for both context and test format. Relative to the applicants completing the paper-and-pencil measures, the Web-based measures showed (a) better distributional properties, (b) lower means, (c) more variance, (d) higher internal consistency reliabilities, and (e) stronger intercorrelations. Although the applicant context resulted in higher mean responses than the incumbent context, this difference was about twice as large for the paper-and-pencil test than for the Web-based test. All such differences were greater for personality measures than for biodata and situational judgment tests. Overall, these results suggest that proctored Web-based testing has some positive benefits relative to paper-and-pencil measures, and we identify several implications of these findings for research and practice.     

The impact of corrections for faking on the validity of noncognitive measures in selection settings

In selection research and practice, there have been many attempts to correct scores on noncognitive measures for applicants who may have faked their responses somehow. A related approach with more impact would be identifying and removing faking applicants from consideration for employment entirely, replacing them with high-scoring alternatives. The current study demonstrates that under typical conditions found in selection, even this latter approach has minimal impact on mean performance levels. Results indicate about .1 SD change in mean performance across a range of typical correlations between a faking measure and the criterion. Where trait scores were corrected only for suspected faking, and applicants not removed or replaced, the minimal impact the authors found on mean performance was reduced even further. By comparison, the impact of selection ratio and test validity is much larger across a range of realistic levels of selection ratios and validities. If selection researchers are interested only in maximizing predicted performance or validity, the use of faking measures to correct scores or remove applicants from further employment consideration will produce minimal effects.     

Practice makes transfer of motor skills imperfect

We investigated the practice-effects on motor skill transfer and the associated representational memory changes that occur during the within-practice and between-practice phases. In two experiments, participants produced extension–flexion movements with their dominant right arm for a limited or prolonged practice session arranged in either a single- or multi-session format. We tested the ability of participants to transfer the original pattern (extrinsic transformation) or the mirrored one (intrinsic transformation) to the non-dominant left arm, 10?min and 24?h after the practice sessions. Results showed that practice induces rapid motor skill improvements that are non-transferable irrespective of the amount of acquisition trials. Furthermore, the extrinsic component of the skill develops early and remains the dominant coding system during practice. Conversely, we found distinct between-practice memory changes: a limited practice induces an off-line development of the extrinsic component, whereas a prolonged practice session subserves the off-line development of the intrinsic component (Experiment 2). We provided further evidence that the long-term representation of the motor skill also depends on the nature of the practice session itself: the parsing of practice into multiple sessions narrows the effector-transfer capacities in comparison to a single session (Experiment 1). These findings yield theoretical and practical implications that are discussed in the context of recent motor skill learning models.     

Big Five traits: Predictors of retesting propensity and score improvement

Although most high‐stakes admissions, credentialing, and pre‐employment tests allow candidates to retest, relatively little is known about the personal traits of candidates who persist in retesting upon initial failure. In this study we investigated whether Big Five traits may predispose initially unsuccessful applicants to retest and subsequently improve on high‐stakes cognitive ability and knowledge tests required for personnel selection. In this study personality measures (unlike the cognitive tests) did not affect selection outcomes and hence did not provide applicants incentive to distort their personality responses to gain entry into the organization. Applicants higher in conscientiousness were more likely to retest, and emotional stability positively predicted cognitive test score improvement. We discuss implications of these results for organizations considering retesting policies.     

Development and psychometric examination of a German video‐based situational judgment test for social competencies in medical school applicants

The present study describes the development and validation of a video‐based situational judgment test (SJT) assessing social competencies in applicants to medical school. Study 1 explored the psychometric properties of the SJT based on two applicant cohorts (N₁ = 769, N₂ = 787). Study 2 compared SJT data from applicants and 90 medical students. Tests for mean differences, correlation and factor analyses were used. In Study 1, the supposed two‐factor model for the SJT must be rejected. According to the minimum average partial test a single factor solution explaining 19–21% of the variance was suggested. The SJT showed satisfactory psychometric properties and stronger correlations to personality than to cognition as well as high acceptance. In study 2, applicants reached higher SJT scores than students.