Range Shrinkage of Cognitive Ability Test Scores in Applicant Pools for German Governmental Jobs: Implications for range restriction corrections

Range restriction corrections require the predictor standard deviation in the applicant pool of interest. Unfortunately, this information is frequently not available in applied contexts. The common strategy in this type of situations is to use national‐norm standard deviation estimates. This study used data from 8,276 applicants applying to nine jobs in German governmental organizations to compare applicant pool standard deviations for two cognitive ability tests with national‐norm standard deviation estimates, and standard deviations for the total group of governmental applicants. Results revealed that job‐ and organizational context‐specific applicant pool standard deviations were on average about 10–12% smaller than estimates from national norms, and about 4–6% smaller than standard deviations for the total group of governmental applicants.     

Application patterns when applicants know the odds: implications for selection research and practice

Unlike previous research that found small differences between population standard deviations and applicant pool standard deviations (P. R. Sackett & D. J. Ostgaard, 1994; D. S. Ones & C. Viswesvaran, 2003), this study revealed a 23% disparity between Law School Admission Test (LSAT) scores of all LSAT test takers and those of LSAT test takers who applied to law school. This study also illustrated robust applicant self-selection behavior across different law school ranks. These findings are important, because predictor scores of applicants who know their scores in advance and perceive small selection ratios necessitate substantially smaller range restriction corrections than those that would be required by population standard deviations. Furthermore, these findings more generally reveal that applicants who know their scores in advance behave quite differently from applicants who do not.     

Faking and selection: considering the use of personality from select-in and select-out perspectives

The effects of faking on criterion-related validity and the quality of selection decisions are examined in the present study by combining the control of an experiment with the realism of an applicant setting. Participants completed an achievement motivation measure in either a control group or an incentive group and then completed a performance task. With respect to validity, greater prediction error was found in the incentive condition among those with scores at the high end of the predictor distribution. When selection ratios were small, those in the incentive condition were more likely to be selected and had lower mean performance than those in the control group. Implications for using personality assessments from select-in and select-out strategies are discussed.     

MODELING THE EFFECTS OF BANDING IN PERSONNEL SELECTION

Selection outcomes under banding are affected by characteristics of the selection system and the applicant pool. This study examined the effects of eight parameters on the proportions hired from higher- and lower-scoring groups: (a) selection ratio; (b) reliability; (b) fixed vs. sliding bands; (d) top-down vs. random within-band selection; (e) preferential vs. nonpreferential selection; (f) mean differences; (g) standard deviation differences; and (h) proportion of applicants from the lower-scoring group. Simulation results were analyzed in a fully-crossed eight-way ANOVA. Higher-order interactions among selection system and applicant pool characteristics had virtually no effect on selection outcomes; the proportion of the applicant pool from the lower-scoring group accounted for nearly half the variance in out-comes. Other important effects are, in order, the effects of standard deviation differences, mean differences, preferential hiring, and the selection ratio. Applicant pool characteristics have considerably more influence on selection outcomes than do selection system characteristics.     

A Monte Carlo Exanimation of Banding and Rank Order Methods of Test Score use in Personnel Selection

Cascio, Outtz, Zedeck, and Goldstein (1991-this issue) examined the effects of various selection rules, including topdown, topdown within-group, fixed- band, and sliding-band approaches, on the mean test score of selected appli- cants and the proportion of minority-group members among those selected using one illustrative data set. A Monte Carlo simulation was conducted to examine the generalizability of their findings across various selection ratios, proportions of minority-group members in the applicant pool, magnitudes of majority-minority mean test score differences, and majority-minority test standard deviation differences. These characteristics were found to have large effects on minority-group hiring; thus in applied setting, values of these char- acteristics must be known before one can determine the consequences of vari- ous selection rules for minority-group hiring.     

Predicting adverse impact and mean criterion performance in multistage selection

The authors present an analytical method to assess the average criterion performance of the selected candidates as well as the adverse impact and the cost of general multistage selection decisions. The method extends previous work on the analytical estimation of multistage selection outcomes to the case in which the applicant pool is a mixture of applicant populations that differ in their average performance on the selection predictors. Next, the method was used to conduct 3 studies of important issues practitioners and researchers have with multistage selection processes. Finally, the authors indicate how the method can be integrated into a broader analytical framework to design multistage selection decisions that achieve intended levels of selection cost, workforce quality, and workforce diversity.     

ETHNIC GROUP DIFFERENCES IN COGNITIVE ABILITY IN EMPLOYMENT AND EDUCATIONAL SETTINGS: A META-ANALYSIS

The cognitive ability levels of different ethnic groups have interested psychologists for over a century. Many narrative reviews of the empirical literature in the area focus on the Black-White differences, and the reviews conclude that the mean difference in cognitive ability ( g ) is approximately 1 standard deviation; that is, the generally accepted effect size is about 1.0. We conduct a meta-analytic review that suggests that the one standard deviation effect size accurately summarizes Black-White differences for college application tests (e.g., SAT) and overall analyses of tests of g for job applicants in corporate settings. However, the 1 standard deviation summary of group differences fails to capture many of the complexities in estimating ethnic group differences in employment settings. For example, our results indicate that job complexity, the use of within job versus across job study design, focus on applicant versus incumbent samples, and the exact construct of interest are important moderators of standardized group differences. In many instances, standardized group differences are less than 1 standard deviation. We conduct similar analyses for Hispanics, when possible, and note that Hispanic-White differences are somewhat less than Black-White differences.     

The biasing effects of selection and attrition on estimating the mean

Organizational and validation researchers often work with data that has been subjected to selection on the predictor and attrition on the criterion. These researchers often use the data observed under these conditions to estimate either the predictor or criterion's restricted population means. We show that the restricted means due to direct or indirect selection are a function of the population means plus the selection ratios. Thus, any difference between selected mean groups reflects the population difference plus the selection ratio difference. When there is also attrition on the criterion, the estimation of group differences becomes even more complicated. The effect of selection and attrition induces measurement bias when estimating the restricted population mean of either the predictor or criterion. A sample mean observed under selection and attrition does not estimate either the population mean or the restricted population mean. We propose several procedures under normality that yield unbiased estimates of the mean. The procedures focus on correcting the effects of selection and attrition. Each procedure was evaluated with a Monte Carlo simulation to ascertain its strengths and weaknesses. Given appropriate sample size and conditions, we show that these procedures yield unbiased estimators of the restricted and unrestricted population means for both predictor and criterion. We also show how our findings have implications for replicating selected group differences.     

Effects of Individual Differences on the Perceived Job Relatedness of a Cognitive Ability Test and a Multimedia Situational Judgment Test

Although there is a growing number of publications concerning applicant reactions to different selection instruments, the relationships between individual differences and applicant reactions have largely remained unexplored. The aim of the present study was to examine the effects of several testing‐related and general individual differences (anxiety, self‐evaluations, and personality) on the most commonly studied dimension of applicant reactions, namely the perceived job relatedness of selection instruments. Participants were 153 psychology students, who completed a cognitive ability test and a multimedia situational judgment test as part of their educational program. Our results indicated that computer anxiety negatively affected perceived job relatedness and core self‐evaluations, subjective well‐being, agreeableness, emotional stability, and openness to experience positively affected perceived job relatedness. Openness to experience was the most consistent predictor of perceived job relatedness. The results of our study suggest that certain individuals may be more predisposed to react positively to selection instruments. Therefore, we concluded that the nature of the applicant pool should be carefully considered when designing interventions to improve applicant reactions.     

Assuming preferential selection when the admissions policy is unknown: the effects of gender rarity

One hundred thirty-five undergraduates indicated the degree to which they believed gender played a role in the selection of an applicant for a graduate degree program. Both the gender composition of the cohort and the selection policy (explicitly merit-based, explicitly affirmative action, or ambiguous) were varied. Results indicated that preferential selection on the basis of gender was assumed when women were solos and explicit information about the selection policy was not provided and that these assumptions were as strong as when an affirmative action policy was explicitly stated. This did not occur when the female selectee was not a solo or when a male selectee was a solo. Evaluations of qualifications and prediction of success paralleled the preferential selection assumptions.     

POTENTIAL EFFECTS OF BANDING AS A FUNCTION OF TEST RELIABILITY

Cascio, Outtz, Zedeck, and Goldstein (1991) described the application of a number of test score banding procedures in personnel selection. Equations are developed illustrating the relationship between the width of test score bands and test reliability. When reliability is moderate to low, bands are likely to be larger than the standard deviation of the test, and are likely to include a large proportion of the applicant pool. The relationships between band widths and the differences between higher scoring and lower scoring groups are also examined. When the band is smaller than the differences between groups (which may happen when highly reliable tests are used), banding may not by itself prove effective as a means of reducing the adverse impact of tests, even when banding systems that maximize opportunities for members of the lower scoring group are used.     

Perceptions of Justice in Employment Selection Decisions: The Role of Applicant Gender

In contrast to the vast majority of justice literature that controls for applicant gender, the present study investigated the role of applicant gender in relation to applicant procedural and distributive justice perceptions after being informed of an organization's reject/accept decision. A sample of 503 students completed a selection test, believing the results would be used to make initial selection decisions for an organization recruiting from a university. Two weeks later, participants were given selection decisions (randomly assigned), and procedural and distributive justice perceptions were assessed. Hierarchical regression analyses indicated gender moderated the relationship between selection decision favorability and organizational justice perceptions. As hypothesized, in comparison with rejected males, rejected female applicants reacted most negatively to both forms of justice. On the other hand, selected female applicants had a more positive reaction than selected male applicants to both procedural and distributive justice. Potential implications for these and other findings are discussed.     

Standard Errors of Multivariate Range-Corrected Validities

The multivariate rather than the univariate range correction is used for estimating unrestricted applicant population validities in many military test validity studies but not uniformly. A Monte Carlo approach compared the standard errors of range-corrected validities under various experimental conditions adhering to the assumptions underlying correction accuracy. The multivariate corrected validities had smaller standard errors than both the univariate-corrected validities and the unrestricted validities. We conclude that using the univariate correction could fail to reveal the most valid selection instrument and that the multivariate correction should be used when scores for relevant predictors are available for the unrestricted population.     

A NOTE ON DIRECT RANGE RESTRICTION CORRECTIONS IN UTILITY ANALYSIS

Brogden's equations for employment test utility analysis have been used in conjunction with direct range restriction correction formulas, because the data used typically come from selected applicant (incumbent) samples. We show that the use of direct range restriction corrections in utility analysis is not needed. Identical results can be obtained using a simpler equation that employs a regression coefficient based on the employee (i.e., restricted or job incumbent) population. This utility estimation procedure, which does not employ range restriction corrections, is derived and discussed. We recommend that this new equation be used in the practice of selection utility analysis because it not only provides identical utility estimates to those obtained using current equations that employ direct range restriction correction formulas, but also is computationally simpler.     

Administrative Assumptions in Top‐Down Selection: A Test in Graduate School Admission Decisions

Although top‐down selection is the gold standard for making personnel decisions, several administrative assumptions must be met for it to be effective. We discuss three of these assumptions and test two of them: (1) top applicants will accept an offer, and (2) the time organisations give applicants to consider an offer will not influence the availability of next‐tier applicants. We also examine the effectiveness of top‐down selection by comparing it to an administratively simpler procedure, random selection above a threshold. Using archival admissions data from three university graduate psychology programs, we found that top applicants were less likely to accept an offer; however, waiting time did not influence applicant availability. In comparing the quality of applicants actually selected (with a top‐down procedure) with the quality of applicants selected at random (from above five progressively stringent thresholds), we found that at higher admission thresholds, random selection resulted in better or equal quality applicants as top‐down selection, depending on the criteria. We discuss implications for future research and practice.     

Better explaining the benefits why AI? Analyzing the impact of explaining the benefits of AI-supported selection on applicant responses

Despite the increasing popularity of AI-supported selection tools, knowledge about the actions that can be taken by organizations to increase AI acceptance is still in its infancy, even though multiple studies point out that applicants react negatively to the implementation of AI-supported selection tools. Therefore, this study investigates ways to alter applicant reactions to AI-supported selection. Using a scenario-based between-subject design with participants from the working population (N = 200), we varied the information provided by the organization about the reasons for using an AI-supported selection process (no additional information vs. written information vs. video information) in comparison to a human selection process. Results show that the use of AI without information and with written information decreased perceived fairness, personableness perception, and increased emotional creepiness. In turn, perceived fairness, personableness perceptions, and emotional creepiness mediated the association between an AI-supported selection process, organizational attractiveness, and the intention to further proceed with the selection process. Moreover, results did not differ for applicants who were provided video explanations of the benefits of AI-supported selection tools and those who participated in an actual human selection process. Important implications for research and practice are discussed.     

The Writing on the (Facebook) Wall: The Use of Social Networking Sites in Hiring Decisions

The popular media has reported an increase in the use of social networking sites (SNSs) such as Facebook by hiring managers and human resource professionals attempting to find more detailed information about job applicants. Within the peer-reviewed literature, cursory empirical evidence exists indicating that others’ judgments of characteristics or attributes of an individual based on information obtained from SNSs may be accurate. Although this predictor method provides a potentially promising source of applicant information on predictor constructs of interest, it is also fraught with potential limitations and legal challenges. The level of publicly available data obtainable by employers is highly unstandardized across applicants, as some applicants will choose not to use SNSs at all while those choosing to use SNSs customize the degree to which information they share is made public to those outside of their network. It is also unclear how decision makers are currently utilizing the available information. Potential discrimination may result through employer’s access to publicly available pictures, videos, biographical information, or other shared information that often allows easy identification of applicant membership to a protected class. For the practice to progress in a positive direction, evidence for the validity and job-relevance of information obtained from SNSs needs to be established. Organizational researchers and practitioners also need to promote awareness and attempt to create safeguards against the potential negative outcomes related to misuse of SNSs by employers.     

TWO-STAGE SEQUENTIAL SELECTION PROCEDURES USING ABILITY AND TRAINING PERFORMANCE: INCREMENTAL VALIDITY OF BEHAVIORAL CONSISTENCY MEASURES

A behavioral consistency model was used to evaluate the predictive validity of ability and training performance measures as components in a two-stage pre-reject sequential selection procedure. Participants were 91 college students who completed ability tests and training relevant to an air intercept and traffic control operator task. A simulation study was conducted so that two groups could be examined: a high ability group, analogous to a screened hired employee group in an organization; and a quasi-random ability group, analogous to an applicant pool as a validation sample. The incremental validity of training performance was practically and statistically significant in the prediction of component and overall task performance after including ability as a predictor in both groups. Adding ability to the prediction of task performance after training performance had been used as a predictor had no practical effects in either sample. The validity and likely utility of using sequential procedures to select employees as well as implications of behavioral consistency measures for privacy and discrimination in employee testing are discussed. Future research directions using different types of training and employee samples are also described.     

结构化面试中评分量表对评分准确性的影响

采用2(组内变量:量尺大小(25分和9分))×2(组间变量:评分方法(相对和绝对))的混合实验设计探讨评分量表对115名大学生新手评委评分准确性的影响。对于评分准确性,采用Cronbach1955年提出的四个指标,Elevation(EL)、Differential elevation(DE)、Stereotype accuracy(SA)、Differential Accuracy(DA)。结果发现,评分方法只在SA上主效应显著,量尺大小在只在DA上主效应边缘显著,评分方法和量尺大小在DE、SA和DA三个指标上均有交互作用。总体上看,在结构化面试评分中,对于评分准确性,相对评分量表优于绝对评分量表,小量尺量表优于大量尺量表。     

Perceived Job Discrimination: Going beyond a science of mean effects

This comment responds to the major issues raised by the papers of Anseel (2011) and Patterson and Zibarras (2011) as discussant commentaries to my keynote paper on perceived job discrimination (PJD) and applicant propensity to case initiation in employee selection. I respond first to the two main themes noted by these authors – (i) theoretical explanations for PJD, and, (ii) methodological and research design imperatives. In conclusion, this brief paper extends these issues to propose that applicant reactions research needs to develop from being a science of mean reactions to becoming a paradigm that can account for pragmatically important outlier reactions and responses as well.