APPLYING RANGE RESTRICTION CORRECTIONS USING PUBLISHED NORMS: THREE CASE STUDIES

This study evaluated the accuracy of range restriction corrections. Initial range restriction corrections used published norms as a proxy for unrestricted applicant data. The accuracy of the initial correction was evaluated using two large independent follow-up samples ( n = 983 and n = 1,389). The author found the initial norm-based correction to be very consistent with empirical findings. The application of these findings to range restriction corrections with customized (i.e., non-published) selection tests is also discussed.     

Implications of direct and indirect range restriction for meta-analysis methods and findings

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.     

Cohen's d Corrected for Case IV Range Restriction: A More Accurate Procedure for Evaluating Subgroup Differences in Organizational Research

Organizational and staffing researchers are often interested in evaluating whether subgroup differences exist (e.g., between Caucasian and African‐American individuals) on predictors of job performance. To investigate subgroup differences, researchers often will collect data from current employees to make inferences about subgroup differences among job applicants. However, the magnitude of subgroup differences (i.e., Cohen's d) within incumbent samples may be different (i.e., smaller) than the magnitude of subgroup differences in applicant samples because selection of applicants typically reduces the variance of scores on the predictors (i.e., because lower scoring applicants are not selected). If researchers seek to generalize a d value in an incumbent sample to the applicant population, they may use Bobko, Roth, and Bobko's (correcting the effect size of d for range restriction and unreliability, 2001) Case II or III correction. By extension, Hunter, Schmidt, and Le (implications of direct and indirect range restriction for meta‐analysis methods and findings, 2006) have proposed a Case IV correction, which is more realistic than Bobko et al.'s approach. Therefore, this paper develops a Case IV correction for d (i.e., d_c4). The simulation results showed that the d_c4 was generally accurate across 6,000 simulation conditions. Moreover, 2 published datasets were reanalyzed to show the influence of the Case IV correction on d. In addition, implications and future directions of the d_c4 are discussed.     

INCREASING THE ACCURACY OF CORRECTIONS FOR RANGE RESTRICTION: IMPLICATIONS FOR SELECTION PROCEDURE VALIDITIES AND OTHER RESEARCH RESULTS

The common practice in meta-analyses and in individual studies of correcting for direct range restriction even though range restriction is actually indirect has long been known to lead to undercorrection, but this error has been assumed to be small. Using validity generalization data sets for 4 jobs, this study calibrated this error by comparing meta-analysis results based on corrections for direct range restriction with the more accurate results from a recently developed method of correcting for indirect range restriction. It was found that, on average, correction for direct range restriction resulted in substantial underestimation of operational validities for both job performance measures (21%) and training performance measures (28%). In addition, 90% credibility values were on average underestimated by 38%–40%. In addition to the implications for personnel selection, these findings suggest that similar underestimation of important relationships has occurred in other areas of research, with potential implications for theory development.     

The proper sequence for correcting correlation coefficients for range restriction and unreliability

Corrections of correlations for range restriction (i.e., selection) and unreliability are common in psychometric work. The current rule of thumb for determining the order in which to apply these corrections looks to the nature of the reliability estimate (i.e., restricted or unrestricted). While intuitive, this rule of thumb is untenable when the correction includes the variable upon which selection is made, as is generally the case. Using classical test theory, we show that it is the nature of the range restriction, not the nature of the available reliability coefficient, that determines the sequence for applying corrections for range restriction and unreliability.We would like to thank Malcolm James Ree for his encouragement and helpful comments as well as those of the editors, associate editor, and reviewers.     

An Improved Correction for Range Restricted Correlations Under Extreme,Monotonic Quadratic Nonlinearity and Heteroscedasticity

Standardized tests are frequently used for selection decisions, and the validation of test scores remains an important area of research. This paper builds upon prior literature about the effect of nonlinearity and heteroscedasticity on the accuracy of standard formulas for correcting correlations in restricted samples. Existing formulas for direct range restriction require three assumptions: (1) the criterion variable is missing at random; (2) a linear relationship between independent and dependent variables; and (3) constant error variance or homoscedasticity. The results in this paper demonstrate that the standard approach for correcting restricted correlations is severely biased in cases of extreme monotone quadratic nonlinearity and heteroscedasticity. This paper offers at least three significant contributions to the existing literature. First, a method from the econometrics literature is adapted to provide more accurate estimates of unrestricted correlations. Second, derivations establish bounds on the degree of bias attributed to quadratic functions under the assumption of a monotonic relationship between test scores and criterion measurements. New results are presented on the bias associated with using the standard range restriction correction formula, and the results show that the standard correction formula yields estimates of unrestricted correlations that deviate by as much as 0.2 for high to moderate selectivity. Third, Monte Carlo simulation results demonstrate that the new procedure for correcting restricted correlations provides more accurate estimates in the presence of quadratic and heteroscedastic test score and criterion relationships.     

Moving Forward Indirectly: Reanalyzing the validity of employment interviews with indirect range restriction methodology

This study provides updated estimates of the criterion‐related validity of employment interviews, incorporating indirect range restriction methodology. Using a final dataset of 92 coefficients (N = 7,389), we found corrected estimates by structural level of .20 (Level 1), .46 (Level 2), .71 (Level 3), and .70 (Level 4). The latter values are noticeably higher than in previous interview meta‐analyses where the assumption was made that all restriction was direct. These results highlight the importance of considering indirect range restriction in selection. However, we found a number of studies involving both indirect and direct restriction, which calls into question the viability of assuming all restriction is now indirect. We found preliminary empirical support for correction of one of these multiple restriction patterns, indirect then direct.     

Correction for Range Restriction in Meta‐Analysis Revisited: Improvements and Implications for Organizational Research

In this study, we present a more accurate method for correcting for range restriction (Case V) that expands upon Bryant and Gokhale's (1972) method. We further present detailed steps to incorporate the Case V method into Schmidt and Hunter's (2015) psychometric meta‐analysis methods (both individual correction and artifact distribution approaches). We then evaluate the accuracy of the Case V method vis‐à‐vis existing methods. Monte‐Carlo simulation results indicate that the Case V method provides very accurate estimates for the mean true score correlation and reasonably accurate estimates for the true standard deviation. More important, Case V almost always provides more accurate results than alternative methods (particularly, Case IV). To illustrate how the Case V method works with real data, we conduct a reanalysis of Judge, Heller, and Mount's (2002) meta‐analysis examining the relationships between the Big 5 personality traits and job satisfaction. Results indicate that the true score correlations between the Big 5 traits and job satisfaction have been underestimated, whereas their true standard deviations have been overestimated. Implications for range restriction corrections in organizational research are discussed.     

Correcting the misinformation effect

Decades of memory research have demonstrated a dire need for effective methods of correcting misinformation, particularly once it has been encoded. However, much of this research has exposed participants to misinformation first then provided a correction, and used indirect memory questions. Using a misinformation effect (ME) paradigm, in which participants' memory is distorted by misleading postevent information, we examine whether corrections can combat memory distortion on direct memory tests. Experiment 1 showed corrections greatly reduced the ME. Experiment 2 replicated this finding with a longer lag time (3 min) between exposure to misinformation and its correction, except for participants that read only the misinformation and its correction. Experiment 3 ruled out the possibility that participants reported the most recent information they read, suggesting participants evaluate the correction's veracity. Finally, a meta‐analysis of the three experiments reiterates that corrections may be effective in combating misinformation.     

Using individual differences to predict job performance: correcting for direct and indirect restriction of range

The present study investigates the relationship between individual differences, indicated by personality (FFM) and general mental ability (GMA), and job performance applying two different methods of correction for range restriction. The results, derived by analyzing meta-analytic correlations, show that the more accurate method of correcting for indirect range restriction increased the operational validity of individual differences in predicting job performance and that this increase primarily was due to general mental ability being a stronger predictor than any of the personality traits. The estimates for single traits can be applied in practice to maximize prediction of job performance. Further, differences in the relative importance of general mental ability in relation to overall personality assessment methods was substantive and the estimates provided enables practitioners to perform a correct utility analysis of their overall selection procedure.     

Adjusting Incremental Fit Indices for Nonnormality

A variety of indices are commonly used to assess model fit in structural equation modeling. However, fit indices obtained from the normal theory maximum likelihood fit function are affected by the presence of nonnormality in the data. We present a nonnormality correction for 2 commonly used incremental fit indices, the comparative fit index and the Tucker-Lewis index. This correction uses the Satorra-Bentler scaling constant to modify the sample estimate of these fit indices but does not affect the population value. We argue that this type of nonnormality correction is superior to the correction that changes the population value of the fit index implemented in some software programs. In a simulation study, we demonstrate that our correction performs well across a variety of sample sizes, model types, and misspecification types.     

Correcting correlation coefficients for group heterogeneity when one variable is a dichotomy

A method is illustrated for estimating correlation coefficients and mean criterion scores in a full-range population from bivariate distributions available in a selected sample when the criterion is a dichotomy. The proposed method requires only the assumptions needed for use of formulas for correcting correlation coefficients for restriction of range when both variables are continuous and is suitable for use when restriction is due to either direct or indirect selection.The research reported in this paper was sponsored by the 6570th Personnel Research Laboratory, AMD, under AFSC Project 7717.     

The optimal correction for estimating extreme discriminability

Discriminability measures such asd’ and logd become infinite when performance is extremely accurate and no errors are recorded. Different arbitrary corrections can be applied to produce finite values, but how well do these values estimate true performance? To answer this question, we directly calculated the effects of a range of different corrections on the sampling distributions of\(\hat d'\) and log\(\hat d\). Many arbitrary corrections produced better estimates of discriminability than did the intuitively plausible technique of rerunning problem conditions. We concluded that when it is not possible to run more trials and when other techniques are not appropriate, the best correction overall is to add a correction constant between 0.25 and 0.5 to all response counts, regardless of their value.     

Concise formulas for the standard errors of component loading estimates

Concise formulas for the asymptotic standard errors of component loading estimates were derived. The formulas cover the cases of principal component analysis for unstandardized and standardized variables with orthogonal and oblique rotations. The formulas can be used under any distributions for observed variables as long as the asymptotic covariance matrix for sample covariances/correlations is available. The estimated standard errors in numerical examples were shown to be equivalent to those by the methods using information matrices.The author is indebted to anonymous reviewers for the corrections and suggestions on this study, which have led to improvements of earlier versions of this article.     

Adjustments to the correction for attenuation

With respect to the often-present covariance between error terms of correlated variables, D. W. Zimmerman and R. H. Williams's (1977) adjusted correction for attenuation estimates the strength of the pairwise correlation between true scores without assuming independence of error scores. This article focuses on the derivation and analysis of formulas that perform the same function for partial and part correlation coefficients. Values produced by these formulas lie closer to the actual true-score coefficient than do the observed-score coefficients or those obtained by using C. Spearman's (1904) correction for attenuation. The new versions of the formulas thus allow analysts to use hypothetical values for error-score correlations to estimate values for the partial and part correlations between true scores while disregarding the independence-of-errors assumption.     

Asymptotic biases in exploratory factor analysis and structural equation modeling

Formulas for the asymptotic biases of the parameter estimates in structural equation models are provided in the case of the Wishart maximum likelihood estimation for normally and nonnormally distributed variables. When multivariate normality is satisfied, considerable simplification is obtained for the models of unstandardized variables. Formulas for the models of standardized variables are also provided. Numerical examples with Monte Carlo simulations in factor analysis show the accuracy of the formulas and suggest the asymptotic robustness of the asymptotic biases with normality assumption against nonnormal data. Some relationships between the asymptotic biases and other asymptotic values are discussed.The author is indebted to the editor and anonymous reviewers for their comments, corrections, and suggestions on this paper, and to Yutaka Kano for discussion on biases.     

Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure

Using computer simulation, the authors assessed the accuracy of J. E. Hunter, F. L. Schmidt, and H. Le's (2006) procedure for correcting for indirect range restriction, the most common type of range restriction, in comparison with the conventional practice of applying the Thorndike Case II correction for direct range restriction. Hunter et al.'s procedure produced more accurate estimates of both the mean and standard deviation in meta-analysis than the conventional procedure. Even when its key assumption that the effect of selection on a 3rd variable is fully mediated by the independent variable was violated, Hunter et al.'s procedure was still relatively more accurate than the conventional procedure. When applied to data from a previously published meta-analysis, the new procedure yielded results that led to different substantive conclusions.     

Theory discovery from data with mixed quantifiers

Convergent realists desire scientific methods that converge reliably to informative, true theories over a wide range of theoretical possibilities. Much attention has been paid to the problem of induction from quantifier-free data. In this paper, we employ the techniques of formal learning theory and model theory to explore the reliable inference of theories from data containing alternating quantifiers. We obtain a hierarchy of inductive problems depending on the quantifier prefix complexity of the formulas that constitute the data, and we provide bounds relating the quantifier prefix complexity of the data to the quantifier prefix complexity of the theories that can be reliably inferred from such data without background knowledge. We also examine the question whether there are theories with mixed quantifiers that can be reliably inferred with closed, universal formulas in the data, but not without.     

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

To promote diversity in organizations it is important to have accurate knowledge about subgroup differences associated with selection procedures. However, current estimates of subgroup differences in situational judgment tests (SJTs) are overwhelmingly based on range‐restricted incumbent samples that are downwardly biased. This study provides much‐needed applicant level estimates of SJT subgroup differences (N = 37,530). As a key finding, Black‐White differences (d = 0.66) were higher than in incumbent samples (d = 0.38). Overall, sex differences were small. Females scored higher for management jobs (d = ?0.13) and males scored higher for administrative jobs (d = 0.15). By analyzing applicant samples that do not suffer from range restriction, this study adds knowledge about subgroup differences in SJTs.     

INTERVIEW VALIDITY FOR SELECTING SALES CLERKS

Interviewer judgments, gender, and age data were collected for job applicants interviewing for seasonal retail sales clerk positions in two separate years. Job performance data were matched with interviewer judgments for 312 employees in Year One and for 205 employees in Year Two. The correlation between interview judgments and job performance was .34 in Year One; after correction for criterion attenuation and restriction of range, the correlation was estimated at .42. The second year results yielded a correlation of .51, which, after corrections, was estimated at .61. Analyses of the data for differential prediction as a function of sex and age for both years revealed that females and older applicants received higher average interview evaluations, but regression lines did not differ significantly among the age and gender subgroups.