Understanding subgroup differences with general mental ability tests in employment selection: Exploring socio-cultural factors across inter-generational groups

In employment selection, general mental ability (GMA) tests predict training and job performance but also lead to subgroup differences which in turn can produce adverse impact against minority groups. Although researchers have explored genetic, developmental, and environmental explanations for ethnic group differences, few studies have explored socio-cultural factors comparing immigrant and non-immigrant job applicants. Given that many ethnic job applicants may also be immigrants, understanding these factors can provide insight into GMA test score differences. The purpose of this paper is to explore the impact of individual and socio-cultural factors on GMA test scores with immigrant and non-immigrant bus driver applicants. This is the first study of its kind to our knowledge that has attempted to disentangle the effects of socio-cultural factors from race/ethnicity in the study of subgroup differences. Incorporating these variables between non-visible minority and minority groups accounted for considerable variance in GMA test scores across groups. The implications of focusing on socio-cultural variables to enhance our understanding of subgroup differences are discussed. Our results specifically suggest that practitioners attend to the issue of the intersecting grounds of potential discrimination when using GMA tests in personnel selection.     

MULTIPLE-CHOICE AND CONSTRUCTED RESPONSE TESTS OF ABILITY: RACE-BASED SUBGROUP PERFORMANCE DIFFERENCES ON ALTERNATIVE PAPER-AND-PENCIL TEST FORMATS

We present an example of an innovative constructed response test format–a write-in/mark-in paper-and-pencil test–as an alternative to the traditional multiple-choice paper-and-pencil test, with the potential for reducing subgroup differences. We present subgroup differences data on these 2 paper-and-pencil test formats on an operational promotional exam in a sample of African American and White firefighters. The tests were designed to measure the same content domain. Using within-subjects data that compared the performance of 13 African American and 14 White fire captains, and between-subjects data that compared the performance of 21 African American and 49 White fire captains, several results were in the predicted direction such that subgroup differences were reduced on the constructed response test. However, these results did not reach statistical significance. Therefore, the study points to the need for additional research to further evaluate the promise of the constructed response test format.     

An examination of factors contributing to a reduction in subgroup differences on a constructed-response paper-and-pencil test of scholastic achievement

The authors investigated subgroup differences on a multiple-choice and constructed-response test of scholastic achievement in a sample of 197 African American and 258 White test takers. Although both groups had lower mean scores on the constructed-response test, the results showed a 39% reduction in subgroup differences compared with the multiple-choice test. The results demonstrate that the lower subgroup differences were explained by more favorable test perceptions for African Americans on the constructed-response test. In addition, the two test formats displayed comparable levels of criterion-related validity. The results suggest that the constructed-response test format may be a viable alternative to the traditional multiple-choice test format in efforts to simultaneously use valid predictors of performance and minimize subgroup differences in high-stakes testing.     

Construction and Validation of a Biodata Selection Instrument

Abstract

Biodata instruments can be used in selection for several purposes (including screening, prediction, analysis of decision strategies). This article is concerned with the construction and validation of a biodata instrument for the selection of intermediaries by a profit-making employment agency in the Netherlands. The instrument was constructed according to the sample approach and measures work experiences with regard to relevant tasks and job dimensions. A field study gathered biodata scores from applicants (n £ 300) and present employees (n £ 70), interview ratings from the applicants, and ratings of job performance for the employees. The results obtained indicated that the instrument was a potentially useful selection device. The homogeneity of scale scores was sufficient and the scores show little overlap with other selection devices (interview ratings). In addition, particular scores correlate at a statistically significant level (P < 0.01) with ratings of job performance of present employees (r £ 0.30 to 0.40). Estimates of the utility (financial gains) of using the instrument in selection revealed that the instrument may yield considerable pay-offs. It is concluded that the construction and validation of the instrument yielded several positive results, and that the sample approach in constructing biodata selection instruments is an option worth considering.     

Modeling the Practical Effects of Applicant Reactions: Subgroup Differences in Test‐Taking Motivation,Test Performance,and Selection Rates

Research suggests that Black–White differences in test‐taking motivation may be related to subgroup test score differences, but this research has not shown the extent to which minimizing subgroup motivation differences will reduce subgroup differences in selection rates and adverse impact. This Monte Carlo study examined how enhancing Blacks' test‐taking motivation for cognitive ability tests might reduce adverse impact across a range of (a) subgroup test differences, (b) selection ratios, (c) subgroup differences in test‐taking motivation, and (d) relationships between motivation and test scores. The results suggest that although enhancing test‐taking motivation will consistently reduce subgroup differences in test performance and adverse impact, the effect is often small and will not eliminate adverse impact for any condition we examine. However, under some conditions the reduction may be important, and the discussion considers conditions where even these minimal reductions may be practically helpful.     

Ethnicity,school attitudes,intelligence and academic achievement

Regression surface analysis was used to examine relations between school-related attitudes and measures of academic achievement at different levels of intellectual ability, for children from different Australian ethnic groups. The sample included 660 eleven-year-old children from lower social status families, with 250 Anglo Australian children, 170 Greek, 120 Southern Italian, and 120 English children. Regression surfaces were constructed from models that examined possible linear, interaction, and curvilinear relations between the variables. The findings suggest the propositions that: (a) at each level of intelligence, increases in the affective component of school attitudes are not associated with changes in academic achievement, (b) at each level of the affective component of school attitudes, increments in intelligence test scores are related to increases in academic achievement scores, and (c) the relations between academic achievement, intelligence, and the cognitive-behavioral component of school attitudes vary for children from different ethnic groups. The results suggest that the manipulation of the cognitive-behavioral aspects of school attitudes may influence the academic performance of low social status children from different ethnic groups but educational programs that focus on the affective components of school attitudes may not be effective in changing children's academic achievement.     

Test‐Specificity of the Advantage of Retaking Cognitive Ability Tests

In selection settings, when people retake the same cognitive ability tests, scores are generally positively biased. Our approach aimed to test whether these previous exposure effects are test‐specific or transferable to other tests. We compared the differences between scores for first‐time test takers and retakers for 2 kinds of material: ‘old’ tests, known only to the retakers, and ‘new’ tests, unknown to both groups. The current study used data collected during 2 sessions –S and S+1 – of a selection process for entry into the French national air transport pilot training system, with at least 500 first‐time test takers and 130 retakers in each session. For Session S, on average, retakers scored higher on the ‘old’ tests, but not on the ‘new’ tests. Moreover, the material that was new to retakers at Session S was old at Session S+1, and the finding for old tests could be replicated at Session S+1. The finding that the acquired skills that led to higher scores on old tests were only test‐specific is discussed.     

Effects of Individual Differences on the Perceived Job Relatedness of a Cognitive Ability Test and a Multimedia Situational Judgment Test

Although there is a growing number of publications concerning applicant reactions to different selection instruments, the relationships between individual differences and applicant reactions have largely remained unexplored. The aim of the present study was to examine the effects of several testing‐related and general individual differences (anxiety, self‐evaluations, and personality) on the most commonly studied dimension of applicant reactions, namely the perceived job relatedness of selection instruments. Participants were 153 psychology students, who completed a cognitive ability test and a multimedia situational judgment test as part of their educational program. Our results indicated that computer anxiety negatively affected perceived job relatedness and core self‐evaluations, subjective well‐being, agreeableness, emotional stability, and openness to experience positively affected perceived job relatedness. Openness to experience was the most consistent predictor of perceived job relatedness. The results of our study suggest that certain individuals may be more predisposed to react positively to selection instruments. Therefore, we concluded that the nature of the applicant pool should be carefully considered when designing interventions to improve applicant reactions.     

WORK SAMPLE TESTS IN PERSONNEL SELECTION: A META‐ANALYSIS OF BLACK–WHITE DIFFERENCES IN OVERALL AND EXERCISE SCORES

Work sample exams are generally thought to have either low or comparatively low levels of ethnic group differences when used for personnel selection. Such exams are sometimes called “simulation exercises” and involve having applicants perform a set of tasks that are similar to those performed on the job. The nearly ubiquitous meta‐analytic value of Black–White subgroup differences in the literature is d= .38. Unfortunately, this estimate is plagued by a variety of problems (e.g., range restriction, inclusion of nonwork sample tests). Further, there are virtually no analyses that examine how the saturation of different constructs influence work sample tests. We gathered available data for Black–White ethnic group differences and found that overall work sample differences were markedly larger for samples of job applicants (d= .73) than previously thought. We also examined how different exercises and saturation of different constructs influenced work sample ds. For example, work sample test ratings of cognitive and job knowledge skills were associated with a mean observed d= .80, whereas ratings of various social skills were associated with mean observed ds that varied from .21 to .27. We urge scientists and practitioners to consider both the method and the constructs that are targeted when forecasting predictor ds.     

Adolescent Self-Esteem: Differences by Race/Ethnicity, Gender, and Age

Large-scale representative surveys of 8th-, 10th-, and 12th-grade students in the United States show high self-esteem scores for all groups. African-American students score highest, Whites score slightly higher than Hispanics, and Asian Americans score lowest. Males score slightly higher than females. Multivariate controls for grades and college plans actually heighten these race/ethnic/gender differences. A truncated scoring method, designed to counter race/ethnic differences in extreme response style, reduced but did not eliminate the subgroup differences. Age differences in self-esteem are modest, with 12th graders reporting the highest scores. The findings are highly consistent across 18 annual surveys from 1991 through 2008, and self-esteem scores show little overall change during that period.     

Reviewing,Categorizing, and Analyzing the Literature on Black–White Mean Differences for Predictors of Job Performance: Verifying Some perceptions and Updating/Correcting Others

In both theoretical and applied literatures, there is confusion regarding accurate values for expected Black–White subgroup differences in personnel selection test scores. Much confusion arises because empirical estimates of standardized subgroup differences (d) are subject to many of the same biasing factors associated with validity coefficients (i.e., d is functionally related to a point‐biserial r). To address such issues, we review/cumulate, categorize, and analyze a systematic set of many predictor‐specific meta‐analyses in the literature. We focus on confounds due to general use of concurrent, versus applicant, samples in the literature on Black–White d. We also focus on potential confusion due to different constructs being assessed within the same selection test method, as well as the influence of those constructs on d. It is shown that many types of predictors (such as biodata inventories or assessment centers) can have magnitudes of d that are much larger than previously thought. Indeed, some predictors (such as work samples) can have ds similar to that associated with paper‐and‐pencil tests of cognitive ability. We present more realistic values of d for both researcher and practitioner use. Implications for practice and future research are noted.     

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

To promote diversity in organizations it is important to have accurate knowledge about subgroup differences associated with selection procedures. However, current estimates of subgroup differences in situational judgment tests (SJTs) are overwhelmingly based on range‐restricted incumbent samples that are downwardly biased. This study provides much‐needed applicant level estimates of SJT subgroup differences (N = 37,530). As a key finding, Black‐White differences (d = 0.66) were higher than in incumbent samples (d = 0.38). Overall, sex differences were small. Females scored higher for management jobs (d = ?0.13) and males scored higher for administrative jobs (d = 0.15). By analyzing applicant samples that do not suffer from range restriction, this study adds knowledge about subgroup differences in SJTs.     

International comparison of group differences in general mental ability for immigrants versus non‐immigrants

Globalization has led to increased migration and labor mobility over the past several decades and immigrants generally seek jobs in their new countries. Tests of general mental ability (GMA) are common in personnel selection systems throughout the world. Unfortunately, GMA test scores often display differences between majority groups and ethnic subgroups that may represent a barrier to employment for immigrants. The purpose of this study was to examine differences in GMA based on immigrant status in 29 countries (or jurisdictions of countries) throughout the world using an existing database that employs high‐quality measurement and sampling methodologies with large sample sizes. The primary findings were that across countries, non‐immigrants (n = 139,464) scored approximately half of a standard deviation (d = .53) higher than first‐generation immigrants (n = 22,162) but only one‐tenth of a standard deviation (d = .12) higher than second‐generation immigrants (n = 6,428). Considerable variability in effect sizes was found across countries as Nordic European and Germanic European countries evidenced the highest non‐immigrant/first‐generation immigrant mean differences and Anglo countries the smallest. Countries with the lowest income inequality tended to evidence the highest differences in GMA between non‐immigrants and first‐generation immigrants. Implications for GMA testing as a potential barrier to immigrant employment success and the field's current understanding of group differences in GMA test scores will be discussed.     

Does test anxiety bias scholastic aptitude test performance by gender and sociocultural group?

This study examines the commonly held contention that test anxiety may serve as a source of bias in the scholastic aptitude test performance of gender and ethnic minority groups. In addition, this study examines sex and sociocultural group differences in the level and pattern of test anxiety among Israeli college students. The sample was composed of 163 male and 198 female students sitting for scholastic aptitude tests routinely administered to all student applicants as part of their college admissions procedures. About 67% were of Western cultural extraction whereas the remainder were of Eastern extraction. Significant differences in text anxiety scores for males and females were observed, with greater sex group differentiation on the Emotionality than on the Worry scale. Test anxiety scores were not discernible by ethnicity or social class. Furthermore, test anxiety was not differentially related to aptitude test scores by sex or sociocultural group membership. Thus, this study lends little evidence to the common contention that test anxiety differentially debilitates the aptitude test scores of females and ethnic minority student candidates.     

Does Test Anxiety Bias Scholastic Aptitude Test Performance by Gender and Sociocultural Group?

This study examines the commonly held contention that test anxiety may serve as a source of bias in the scholastic aptitude test performance of gender and ethnic minority groups. In addition, this study examines sex and sociocultural group differences in the level and pattern of test anxiety among Israeli college students. The sample was composed of 163 male and 198 female students sitting for scholastic aptitude tests routinely administered to all student applicants as part of their college admissions procedures. About 67% were of Western cultural extraction whereas the remainder were of Eastern extraction. Significant differences in text anxiety scores for males and females were observed, with greater sex group differentiation on the Emotionality than on the Worry scale. Test anxiety scores were not discernible by ethnicity or social class. Furthermore, test anxiety was not differentially related to aptitude test scores by sex or sociocultural group membership. Thus, this study lends little evidence to the common contention that test anxiety differentially debilitates the aptitude test scores of females and ethnic minority student candidates.     

Same‐gender and same‐race bias in assessment center ratings: A rating error approach to understanding subgroup differences

This study investigated leniency and similar‐to‐me bias as mechanisms underlying demographic subgroup differences among assessees in assessors’ initial dimension ratings from three assessment center (AC) simulation exercises used as part of high‐stakes promotional testing. It examined whether even small individual‐level effects can accumulate (i.e., “trickle‐up”) to produce larger subgroup‐level differences. Individual‐level analyses were conducted using cross‐classified multilevel modeling and conducted separately for each exercise. Results demonstrated weak evidence of leniency toward White assessees and similar‐to‐me bias among non‐White assessee–assessor pairs. Similar leniency was found toward female assessees, but no statistically significant effects were found for assessee or assessor gender or assessee–assessor gender similarity. Using traditional d effect size estimates, weak individual level assessee effects translated into small but consistent subgroup differences favoring White and female assessees. Generally small but less consistent subgroup differences indicated that non‐White and male assessors gave higher ratings. Moreover, analyses of overall promotion decisions indicate the absence of adverse impact. Findings from this AC provide some support for the “trickle‐up” effect, but the effect on subgroup differences is trivial. The results counter recent reviews of AC studies suggesting larger than previously assumed subgroup differences. Consequently, the findings demonstrate the importance of following established best practices when developing and implementing the AC method for selection purposes to minimize subgroup differences.     

Are cognitive differences between immigrant and majority groups diminishing?

A review is given of scores on various cognitive measures, comparing groups of ethnic Dutch and non‐Western immigrants using a large number of datasets. The research shows that there are large group differences in school results, work proficiency, and g for Turks, Moroccans, Surinamese, Netherlands Antilleans, and Indonesians from the Moluccans compared with ethnic Dutch. However, South‐East Asians score higher, and persons with one immigrant and one ethnic Dutch parent score only slightly below the mean of the Dutch. When comparing first‐generation disadvantaged immigrant groups with later generations the data show substantial improvements for g, a remarkable stability of educational differences for younger children, and a clear improvement in educational achievement at the end of primary school. Indirect data on intergenerational improvements in work proficiency appear suggestive of a trend of closing gaps. Some of the data reflect higher cognitive capacities over time, and this enhances integration of immigrants into Dutch society. Causes of group differences and improvements in mean level of g are discussed. Copyright © 2004 John Wiley & Sons, Ltd.     

Validation of a clerical test using work samples

zWork simulation performance tests of filing and proofreading were the principal criteria in a validation study of a paper and pencil clerical selection test constructed by content-oriented methods; supervisory ratings were also used. Experiment 1 was a concurrent study using as subjects 59 provisional employees. Experiment 2 was a predictive study using as subjects 184 employees actually selected on the basis of test scores. In both studies, substantial correlations were found between scores on the selection test and performance on the work samples: Estimates of correlations in the original selection group ranged from .4 to .8. Correlations between the selection test and the supervisor ratings were significant in Experiment 1, but not in Experiment 2; even when significant, they were much lower than the correlations with the work samples. These results suggest the value of using work samples as criteria for validation studies. Implications for other validation efforts are considered.     

Exploring Differential Subgroup Functioning on SAT Writing Items: What Happens When English Is Not a Test Taker's Best Language?

The purpose of this study is to explore patterns in model-data fit related to subgroups of test takers from a large-scale writing assessment. Using data from the SAT, a calibration group was randomly selected to represent test takers who reported that English was their best language from the total population of test takers (N = 322,011). A reference scale for the items was constructed based on EBL responses. Response behaviors of test takers who reported that English was not their best language (ENBL) were examined in relationship to this reference scale. This study illustrates the use of differential subgroup analyses to identify patterns related to person misfit within subgroups, as well as subsets of items, that may affect the validity of writing scores for ENBL test takers. The methodology described here offers an approach that can be used to explore, understand, and improve the validity of scores obtained from ENBL test takers in large-scale writing assessments.     

Cohen's d Corrected for Case IV Range Restriction: A More Accurate Procedure for Evaluating Subgroup Differences in Organizational Research

Organizational and staffing researchers are often interested in evaluating whether subgroup differences exist (e.g., between Caucasian and African‐American individuals) on predictors of job performance. To investigate subgroup differences, researchers often will collect data from current employees to make inferences about subgroup differences among job applicants. However, the magnitude of subgroup differences (i.e., Cohen's d) within incumbent samples may be different (i.e., smaller) than the magnitude of subgroup differences in applicant samples because selection of applicants typically reduces the variance of scores on the predictors (i.e., because lower scoring applicants are not selected). If researchers seek to generalize a d value in an incumbent sample to the applicant population, they may use Bobko, Roth, and Bobko's (correcting the effect size of d for range restriction and unreliability, 2001) Case II or III correction. By extension, Hunter, Schmidt, and Le (implications of direct and indirect range restriction for meta‐analysis methods and findings, 2006) have proposed a Case IV correction, which is more realistic than Bobko et al.'s approach. Therefore, this paper develops a Case IV correction for d (i.e., d_c4). The simulation results showed that the d_c4 was generally accurate across 6,000 simulation conditions. Moreover, 2 published datasets were reanalyzed to show the influence of the Case IV correction on d. In addition, implications and future directions of the d_c4 are discussed.