Racial/ethnic differences in the criterion-related validity of cognitive ability tests: a qualitative and quantitative review

The correlation between cognitive ability test scores and performance was separately meta-analyzed for Asian, Black, Hispanic, and White racial/ethnic subgroups. Compared to the average White observed correlation ( = .33, N = 903,779), average correlations were lower for Black samples ( = .24, N = 112,194) and Hispanic samples ( = .30, N = 51,205) and approximately equal for Asian samples ( = .33, N = 80,705). Despite some moderating effects (e.g., type of performance criterion, decade of data collection, job complexity), validity favored White over Black and Hispanic test takers in almost all conditions that included a sizable number of studies. Black-White validity comparisons were possible both across and within the 3 broad domains that use cognitive ability tests for high-stakes selection and placement: civilian employment, educational admissions, and the military. The trend of lower Black validity was repeated in each domain; however, average Black-White validity differences were largest in military studies and smallest in educational and employment studies. Further investigation of the reasons for these validity differences is warranted.     

EMPLOYMENT TEST VALIDATION FOR MINORITY AND NONMINORITY PRODUCTION WORKERS

The purposes of the present study were (a) to examine the validity of selected employment tests potentially useful in selecting production workers engaged in the construction of boxboard containers and (b) to evaluate the applicability of the tests to minority and non-minority workers. Using data collected from 100 production workers employed by the same company but located in two different geographical regions, it was found that a short test battery was potentially useful in selecting production employees without necessarily introducing unfair racial bias. Implications of the results for future research studies and test validation efforts involving differential and single-group validity are discussed.     

THE IMPACT OF JOB ANALYSIS ON EMPLOYMENT TEST VALIDATION FOR MINORITY AND NONMINORITY ACCOUNTING PERSONNEL

The purposes of the present study were (a) to examine the comparative validity of a written job knowledge test constructed on the basis of a systematic job analysis with that of a commercial employment test selected in the absence of a prior job analysis for accounting positions and (b) to determine the fairness of each test for minority and nonminority job applicants. Results indicated that the job knowledge test was a valid and unbiased predictor of relevant criteria of job performance while the commercial employment test produced adverse impact and lacked validity. Implications of the results for future research studies and test validation efforts involving differential prediction are discussed.     

Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

Previous studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta-analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta-analysis of the Hispanic–White mean difference (d-value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d-values for indirect range restriction and combine them with other meta-analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion-related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.     

The Role of Range Restriction and Criterion Contamination in Assessing Differential Validity by Race/Ethnicity

Purpose

Berry et al.’s (J Appl Psychol 96:881–906, 2011) meta-analysis of cognitive ability test validity data across employment, college admissions, and military domains demonstrated that validity is lower for Black and Hispanic subgroups than for Asian and White subgroups. However, Berry et al. relied on observed test-criterion correlations and it is therefore not clear whether validity differences generalize beyond observed validities. The present study investigates the roles that range restriction and criterion contamination play in differential validity.

Design/Methodology/Approach

A large dataset (N > 140,000) containing SAT scores and college grades of Asian, Black, Hispanic, and White test takers was used. Within-race corrections for multivariate range restriction were applied. Differential validity analyses were carried out using freshman GPA versus individual course grades as criteria to control for the contaminating influence of individual differences between students in course choice.

Findings

Observed validities underestimated the magnitude of validity differences between subgroups relative to when range restriction and criterion contamination were controlled. Analyses also demonstrate that validity differences would translate to larger regression slope differences (i.e., differential prediction).

Implications

Subgroup differences in range restriction and/or individual differences in course choice cannot account for lower validity of the SAT for Black and Hispanic subgroups. Controlling for these factors increased subgroup validity differences. Future research must look to other explanations for subgroup validity differences.

Originality

The present study is the first differential validity study to simultaneously control for range restriction and individual differences in course choice, and answers a call to investigate potential causes of differential validity.     

Everyday spatial activities test for studying differential spatial experience and vocational behavior

Described is a four-factor, 20-item Everyday Spatial Activities Test (ESAT) developed for use with college students for studying the contribution of differential spatial experiences to educational and vocational behaviors. Item data are presented based on samples of 158 female and 113 male first-year students. Correlations with objective tests, including a test of space visualization, establishes preliminary validity for the ESAT.     

Text anxiety on cognitive ability test can result in differential predictive validity of academic performance

This paper investigates whether test anxiety leads to differential predictive validity in academic performance. Our results show that the predictive validity of a cognitive ability test, using final exam performance as a criterion, decreased a small amount as Worry (the cognitive aspect of anxiety) increased but was unaffected by Emotionality (the physiological aspect of anxiety). These results suggest that cognitive ability tests may be more useful as predictors of performance for low anxiety test-takers. These findings are discussed in the context of the interference and deficit perspectives of test anxiety.     

On the myth and the reality of the temporal validity degradation of general mental ability test scores

Claims of changes in the validity coefficients associated with general mental ability (GMA) tests due to the passage of time (i.e., temporal validity degradation) have been the focus of an on-going debate in applied psychology. To evaluate whether and, if so, under what conditions this degradation may occur, we integrate evidence from multiple sub-disciplines of psychology. The temporal stability of construct validity is considered in light of the evidence regarding the differential stability of g and the invariance of measurement properties of GMA tests over the adult life-span. The temporal stability of criterion-related validity is considered in light of evidence from long-term predictive validity studies in educational and occupational realms. The evidence gained from this broad-ranging review suggests that temporal degradation of the construct- and criterion-related validity of ability test scores may not be as ubiquitous as some have previously concluded. Rather, it appears that both construct and criterion-related validity coefficients are reasonably robust over time and that any apparent degradation of criterion-related validity coefficients has more to do with changes in the determinants of task performance and changes in the nature of the criterion domain rather temporal degradation per se (i.e., the age of the test scores). A key exception to the conclusion that temporal validity degradation is more myth than reality concerns decision validity. Although the evidence is sparse, it is likely that the utility of a given GMA test score for making diagnostic decisions about an individual deteriorates over time. Importantly, we also note several areas in need of additional and more rigorous research before strong conclusions can be supported.     

Test anxiety and the validity of cognitive tests: A confirmatory factor analysis perspective and some empirical findings

The validity of cognitive ability tests is often interpreted solely as a function of the cognitive abilities that these tests are supposed to measure, but other factors may be at play. The effects of test anxiety on the criterion related validity (CRV) of tests was the topic of a recent study by Reeve, Heggestad, and Lievens (2009) (Reeve, C. L., Heggestad, E. D., & Lievens, F. (2009). Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence, 37, 34?41.). They proposed a model on the basis of classical test theory, and concluded on the basis of data simulations that test anxiety typically decreases the CRV. In this paper, we view the effects of test anxiety on cognitive ability test scores and its implications for validity coefficients from the perspective of confirmatory factor analysis. We argue that CRV will be increased above the effect of targeted constructs if test anxiety affects both predictor and criterion performance. This prediction is tested empirically by considering convergent validity of subtests in five experimental studies of the effect of stereotype threat on test performance. Results show that the effects of test anxiety on cognitive test performance may actually enhance the validity of tests.     

PREDICTIVE EFFICIENCY AS A FUNCTION OF AMOUNT OF INFORMATION

The effect of changing the amount of information on judges' predictive efficiency in a clinical prediction task was studied. Thirty judges predicted 30 students' average achievement scores on the basis of different amount of test data. One group of judges had information about the intercorrelations among the tests and the ecologkal validity of the tests. Another group of judges had only informahion about which tests were used. The predictive efficiency was not a monotonically increasing function of amount of test data. The most marked result was that the relative predictive efficiency decreased from four to six tests in both groups.     

Knowing What NOT To Do Is a Critical Job Skill: Evidence from 10 different scoring methods

Situational judgment tests (SJTs) have become an increasingly important tool for predicting employee performance; however, at least two key areas warrant further investigation. First, prior studies of SJTs have generally relied on samples from the western world, leaving open the question of the validity of using SJTs in the developing world where the majority of the world's workforce resides. Second, there is currently no standardized, theoretically‐based method for the development and scoring of SJTs. Therefore, SJTs are highly domain‐specific and must be developed anew for each new context. We report the results of three studies, conducted in India, that aim to: (1) test the cross‐cultural validity of SJTs in a non‐western context, and (2) examine the differential validity of 10 different approaches to scoring SJTs, some of which have the potential to resolve the problem of developing a theoretically‐infused, standardized approach to scoring and future development.     

Cognitive and GMA Testing in the European Community: Issues and Evidence

This article reviews the evidence on General Mental Ability (GMA) and cognitive ability tests in connection with employment in the European Community (EC). Five themes are reviewed: prevalence, applicant reactions, testing standards, criterion validity, and recent advances. The first section shows that GMA and cognitive ability tests are used more often in Europe than in America. The second section, regarding applicant reactions, shows that the studies carried out in the EC have shown that there are no differences between Europe and America. The third section shows that there is an initiative to harmonize the standards used across the European countries. In the fourth section, we report on a validity generalization study carried out using the primary studies conducted in Great Britain and Spain. The results showed that GMA and cognitive tests are valid predictors of job performance and training success in both countries and they generalize validity across occupations. Furthermore, the size of the observed validity estimates is similar to those found in the American meta-analyses. Finally, two recent advances are mentioned: the British "Project B" and the use of virtual reality technology for predicting job performance.     

Labour market integration of refugees in Norway under changing macro-economic conditions

This paper describes the transition to ordinary employment among thirteen single-year cohorts of refugees who settled in Norway from 1987 to 1999. The refugees are observed once a year from 1992 to 1999 on the basis of register data. A central issue is how belonging to a certain settlement cohort, duration of residence in the receiving country, and observation period affect the chance of being employed. The effect of changing business cycles is also addressed. Earlier conceptions that macro-economic conditions at the time of settlement are more important for the chance of employment than duration of residence are tested and refuted.     

Updating the trainability tests literature on Black-White subgroup differences and reconsidering criterion-related validity

A number of applied psychologists have suggested that trainability test Black–White ethnic group differences are low or relatively low (e.g., Siegel & Bergman, 1975), though data are scarce. Likewise, there are relatively few estimates of criterion-related validity for trainability tests predicting job performance (cf. Robertson & Downs, 1989). We review and clarify the existing (and limited) literature on Black–White group differences on trainability tests, provide new trainability test data from a recent video-based trainability exam, and present archival data about how trainability test scores relate to cognitive ability, Black–White differences, and job performance. Consistent with hypotheses, our results suggest large correlations of trainability tests with cognitive ability (e.g., .80) and larger standardized ethnic group differences than previously thought (ds of 0.86, 1.10, and 1.21 for 3 samples). Results also suggest that trainability tests have higher validity than previously thought. Overall, our analysis provides a substantial amount of data to update our understanding of the use of trainability tests in personnel selection.     

Assessing privacy invasiveness of psychological test items: Job relevant versus clinical measures of integrity

This pilot study was conducted to determine if clinically-oriented test items are judged to be more offensive than job-related test items. Clinical tests typically ask more personal questions while employment tests usually ask job-relevant questions. A random selection of items from three employment tests was analyzed. Two of the tests evolved from clinical-personality tests, while the third test was designed specifically for employment settings. The results suggest that if companies are interested in using employment tests that are perceived as being job-relevant, inoffensive, and non-invasive, then they should consider selecting tests that include job-relevant items as opposed to tests that are derivatives of clinical assessment instruments.     

Predicting from expressed vocational choice: a review

Information on the predictive validity of expressed vocational choice was gathered in a review of large-sample longitudinal studies. These results indicate that, in general, a person's expressed vocational choice predicts his future employment about as well as interest inventories or combinations of personality and background characteristics. Expressed vocational and educational field choices are moderately stable over one- to five-year periods during and after college. Difficulties and shortcomings in past research are discussed, and recommendations are made for future research on the predictive validity of expressed choice. Implications of these results for the practicing counselor are also considered.     

A FACTOR ANALYSIS OF BRAIN DAMAGE TESTS ADMINISTERED TO NORMAL SUBJECTS WITH FACTOR SCORE COMPARISONS ACROSS AGES

This study investigated the factorial composition of brain damage tests selected on the basis of a comprehensive review of the test literature. The tests were administered to 100 normal persons between the ages of 16 and 70. Two analyses were performed. The first analysis concerned the determi- nation of the factorial composition of the brain damage tests selected. The results suggested that although the dimensionality of the brain damage test battery is relatively complex, the majority of the tests discriminate on the basis of a few dimensions, at least for normal persons. The second analysis involved the determination of factor scores for each individual on the twelve factors extracted in the 16-70 age group analysis. An analysis of the factor scores as a function of age indicated a significant difference for only two of the factors. This finding was discussed in terms of the possibility that certain of the tests would be psychometrically more efficient in differential diagnosis of brain damage in the older person.     

Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability

Previous studies have indicated that as many as 25% to 50% of applicants in organizational and educational settings are retested with measures of cognitive ability. Researchers have shown that practice effects are found across measurement occasions such that scores improve when these applicants retest. In this study, the authors used meta-analysis to summarize the results of 50 studies of practice effects for tests of cognitive ability. Results from 107 samples and 134,436 participants revealed an adjusted overall effect size of .26. Moderator analyses indicated that effects were larger when practice was accompanied by test coaching and when identical forms were used. Additional research is needed to understand the impact of retesting on the validity inferences drawn from test scores.     

VALIDITY GENERALIZATION RESULTS FOR LAW ENFORCEMENT OCCUPATIONS

The Schmidt-Hunter interactive validity generalization procedure was applied to validity data for cognitive abilities tests for law enforcement occupations. Both assumed artifact distributions, and distributions of artifacts constructed from information contained in the current sample of studies were used to test the hypothesis of situational specificity and to estimate validity generalizability. Results for studies using a criterion of performance in training programs showed that validities ranged from .41 to .71, and for four test types the hypothesis of situational specificity could be rejected using the 75% decision rule. For the remaining test types, validity was generalizable, based on 90% credibility values ranging from .37 to .71. Results for studies using a criterion of performance on the job indicated that the hypothesis of situational specificity was not tenable for three test types, which had validities between .17 and .31. For the remaining test types, estimated mean true validities ranged from .10 to .26 and were generalizable to a majority of situations. Results for both groups of studies were essentially identical for the two types of artifact distribution. Possible reasons for the apparently lower validities and lesser generalizability for job performance criteria are discussed, including possible low validity of the criterion (due to lack of opportunity by supervisors to observe behavior) and the potential role of noncognitive factors in the determination of law enforcement job success. Suggestions for specifically targeted additional research are made.     

Traditional tests and job simulations: minority and majority performance and test validities.

Performance of 565 minority and 414 majority job applicants on traditional paper-and-pencil tests and on a telephone simulation that measured similar constructs was assessed. Models of the measurement characteristics of these 2 test batteries indicated larger subgroup mean differences on the traditional tests than on the simulation. Correlations between traditional tests were lower in the majority sample than in the minority sample, and the variance of the minority candidates' scores on the traditional tests was much larger than the variance of majority applicants' scores on the same measures. The validity of the simulation was lower than the validity of the traditional tests. This study replicates previous laboratory research that has indicated smaller subgroup differences on simulations than on paper-and-pencil tests and extends this research by providing evidence of the relative validity of these 2 types of measures.