A NOTE ON DIRECT RANGE RESTRICTION CORRECTIONS IN UTILITY ANALYSIS

Brogden's equations for employment test utility analysis have been used in conjunction with direct range restriction correction formulas, because the data used typically come from selected applicant (incumbent) samples. We show that the use of direct range restriction corrections in utility analysis is not needed. Identical results can be obtained using a simpler equation that employs a regression coefficient based on the employee (i.e., restricted or job incumbent) population. This utility estimation procedure, which does not employ range restriction corrections, is derived and discussed. We recommend that this new equation be used in the practice of selection utility analysis because it not only provides identical utility estimates to those obtained using current equations that employ direct range restriction correction formulas, but also is computationally simpler.     

INCREASING THE ACCURACY OF CORRECTIONS FOR RANGE RESTRICTION: IMPLICATIONS FOR SELECTION PROCEDURE VALIDITIES AND OTHER RESEARCH RESULTS

The common practice in meta-analyses and in individual studies of correcting for direct range restriction even though range restriction is actually indirect has long been known to lead to undercorrection, but this error has been assumed to be small. Using validity generalization data sets for 4 jobs, this study calibrated this error by comparing meta-analysis results based on corrections for direct range restriction with the more accurate results from a recently developed method of correcting for indirect range restriction. It was found that, on average, correction for direct range restriction resulted in substantial underestimation of operational validities for both job performance measures (21%) and training performance measures (28%). In addition, 90% credibility values were on average underestimated by 38%–40%. In addition to the implications for personnel selection, these findings suggest that similar underestimation of important relationships has occurred in other areas of research, with potential implications for theory development.     

Implications of direct and indirect range restriction for meta-analysis methods and findings

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.     

The proper sequence for correcting correlation coefficients for range restriction and unreliability

Corrections of correlations for range restriction (i.e., selection) and unreliability are common in psychometric work. The current rule of thumb for determining the order in which to apply these corrections looks to the nature of the reliability estimate (i.e., restricted or unrestricted). While intuitive, this rule of thumb is untenable when the correction includes the variable upon which selection is made, as is generally the case. Using classical test theory, we show that it is the nature of the range restriction, not the nature of the available reliability coefficient, that determines the sequence for applying corrections for range restriction and unreliability.We would like to thank Malcolm James Ree for his encouragement and helpful comments as well as those of the editors, associate editor, and reviewers.     

Correction for Range Restriction in Meta‐Analysis Revisited: Improvements and Implications for Organizational Research

In this study, we present a more accurate method for correcting for range restriction (Case V) that expands upon Bryant and Gokhale's (1972) method. We further present detailed steps to incorporate the Case V method into Schmidt and Hunter's (2015) psychometric meta‐analysis methods (both individual correction and artifact distribution approaches). We then evaluate the accuracy of the Case V method vis‐à‐vis existing methods. Monte‐Carlo simulation results indicate that the Case V method provides very accurate estimates for the mean true score correlation and reasonably accurate estimates for the true standard deviation. More important, Case V almost always provides more accurate results than alternative methods (particularly, Case IV). To illustrate how the Case V method works with real data, we conduct a reanalysis of Judge, Heller, and Mount's (2002) meta‐analysis examining the relationships between the Big 5 personality traits and job satisfaction. Results indicate that the true score correlations between the Big 5 traits and job satisfaction have been underestimated, whereas their true standard deviations have been overestimated. Implications for range restriction corrections in organizational research are discussed.     

The Effect of Practice on the Control of Sequential and Simultaneous Actions

The effect of practice on the control of sequential and simultaneous multilimb aiming responses was studied. In one group (n = l0), subjects pushed hand levers and foot pedals 20 deg. (plus or minus 1.5 deg.) forward in the following order: left hand, right hand, left foot, right foot. In the other group (n = l0), all subjects pushed both levers and pedals forward simultaneously for the same distance. The goal for both groups was to reduce movement time (MT). Analysis of the displacement and velocity records showed that sequential movements were made quicker and with fewer movement corrections than simultaneous movements. With practice, both groups reduced the MT and the number of movement corrections, suggesting that the accuracy of the initial propulsive phase of the movement became more precise. In addition, temporal interlimb correlations increased with practice, suggesting the development of a coordination factor related to multilimb movements, but little support was shown for the gear-shift analogy.     

Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure

Using computer simulation, the authors assessed the accuracy of J. E. Hunter, F. L. Schmidt, and H. Le's (2006) procedure for correcting for indirect range restriction, the most common type of range restriction, in comparison with the conventional practice of applying the Thorndike Case II correction for direct range restriction. Hunter et al.'s procedure produced more accurate estimates of both the mean and standard deviation in meta-analysis than the conventional procedure. Even when its key assumption that the effect of selection on a 3rd variable is fully mediated by the independent variable was violated, Hunter et al.'s procedure was still relatively more accurate than the conventional procedure. When applied to data from a previously published meta-analysis, the new procedure yielded results that led to different substantive conclusions.     

INCREASED ACCURACY FOR RANGE RESTRICTION CORRECTIONS: IMPLICATIONS FOR THE ROLE OF PERSONALITY AND GENERAL MENTAL ABILITY IN JOB AND TRAINING PERFORMANCE

A recently developed procedure produces substantial improvements in the accuracy of corrections for range restriction and reveals that predictive validities of employment selection methods are higher than previously thought. This study applied this procedure to meta‐analytic validities of 2 personality measures (Conscientiousness and Emotional Stability) and general cognitive ability measures. Results show that the increases in validity estimates are greater for cognitive ability than for personality, reducing the relative validity of personality. In addition, the incremental validity of these personality measures over that provided by cognitive ability is smaller than previously estimated. Path analyses based on the more accurate data reveal relatively smaller path coefficients from these personality traits to job and training performance. These findings have implications for theories of job performance and for the practical value in selection of personality measures relative to cognitive ability measures.     

Moving Forward Indirectly: Reanalyzing the validity of employment interviews with indirect range restriction methodology

This study provides updated estimates of the criterion‐related validity of employment interviews, incorporating indirect range restriction methodology. Using a final dataset of 92 coefficients (N = 7,389), we found corrected estimates by structural level of .20 (Level 1), .46 (Level 2), .71 (Level 3), and .70 (Level 4). The latter values are noticeably higher than in previous interview meta‐analyses where the assumption was made that all restriction was direct. These results highlight the importance of considering indirect range restriction in selection. However, we found a number of studies involving both indirect and direct restriction, which calls into question the viability of assuming all restriction is now indirect. We found preliminary empirical support for correction of one of these multiple restriction patterns, indirect then direct.     

Correction and use of biomedical literature affected by scientific misconduct

The purpose of this study was to identify and describe published research articles that were named in official findings of scientific misconduct and to investigate compliance with the administrative actions contained in these reports for corrections and retractions, as represented in PubMed. Between 1993 and 2001, 102 articles were named in either the NIH Guide for Grants and Contracts ("Findings of Scientific Misconduct") or the U.S. Office of Research Integrity annual reports as needing retraction or correction. In 2002, 98 of the 102 articles were indexed in PubMed. Eighty-five of these 98 articles had indexed corrections: 47 were retracted; 26 had an erratum; 12 had a correction described in the "comment" field. Thirteen had no correction, but 10 were linked to the NIH Guide "Findings of Scientific Misconduct", leaving only 3 articles with no indication of any sort of problem. As of May 2005, there were 5,393 citations to the 102 articles, with a median of 26 citations per article (range 0-592). Researchers should be alert to "Comments" linked to the NIH Guide as these are open access, and the "Findings of Scientific Misconduct' reports are often more informative than the statements about the retraction or correction found in the journals.     

INTERVIEW VALIDITY FOR SELECTING SALES CLERKS

Interviewer judgments, gender, and age data were collected for job applicants interviewing for seasonal retail sales clerk positions in two separate years. Job performance data were matched with interviewer judgments for 312 employees in Year One and for 205 employees in Year Two. The correlation between interview judgments and job performance was .34 in Year One; after correction for criterion attenuation and restriction of range, the correlation was estimated at .42. The second year results yielded a correlation of .51, which, after corrections, was estimated at .61. Analyses of the data for differential prediction as a function of sex and age for both years revealed that females and older applicants received higher average interview evaluations, but regression lines did not differ significantly among the age and gender subgroups.     

Self-enhancement and theory-based correction processes

Two studies examined bias correction by manipulating a perceived chronic judgmental bias (i.e., overestimator/underestimator) using a modified dot estimation task. In Experiment 1, participants corrected for this perceived estimation bias by making adjustments away from the arbitrary feedback about their personal bias tendencies. In Experiment 2, the perceived desirability of the same estimation bias was manipulated. Results indicated that self-enhancement concerns impacted perceivers’ motivation to correct, at a cost to accuracy. These studies expand our current understanding of theory-based correction by including self-enhancement motives as causes of correction, demonstrating that such corrections can decrease rather than increase judgment accuracy, and illustrating the usefulness of a new perceived bias manipulation in theory testing.     

Criterion Validity and Practical Utility of the Minnesota Multiphasic Personality Inventory–2–Restructured Form (MMPI–2–RF) in Assessments of Police Officer Candidates

Minnesota Multiphasic Personality Inventory–2–Restructured Form scores for 145 male police officer candidates were compared with supervisor ratings of field performance and problem behaviors during their initial probationary period. Results indicated that the officers produced meaningfully lower and less variant substantive scale scores compared to the general population. After applying a statistical correction for range restriction, substantive scale scores from all domains assessed by the inventory demonstrated moderate to large correlations with performance criteria. The practical significance of these results was assessed with relative risk ratio analyses that examined the utility of specific cutoffs on scales demonstrating associations with performance criteria.     

An Improved Correction for Range Restricted Correlations Under Extreme,Monotonic Quadratic Nonlinearity and Heteroscedasticity

Standardized tests are frequently used for selection decisions, and the validation of test scores remains an important area of research. This paper builds upon prior literature about the effect of nonlinearity and heteroscedasticity on the accuracy of standard formulas for correcting correlations in restricted samples. Existing formulas for direct range restriction require three assumptions: (1) the criterion variable is missing at random; (2) a linear relationship between independent and dependent variables; and (3) constant error variance or homoscedasticity. The results in this paper demonstrate that the standard approach for correcting restricted correlations is severely biased in cases of extreme monotone quadratic nonlinearity and heteroscedasticity. This paper offers at least three significant contributions to the existing literature. First, a method from the econometrics literature is adapted to provide more accurate estimates of unrestricted correlations. Second, derivations establish bounds on the degree of bias attributed to quadratic functions under the assumption of a monotonic relationship between test scores and criterion measurements. New results are presented on the bias associated with using the standard range restriction correction formula, and the results show that the standard correction formula yields estimates of unrestricted correlations that deviate by as much as 0.2 for high to moderate selectivity. Third, Monte Carlo simulation results demonstrate that the new procedure for correcting restricted correlations provides more accurate estimates in the presence of quadratic and heteroscedastic test score and criterion relationships.     

GROUP HOMOGENEITY, RANGE RESTRICTION, AND RANGE ENHANCEMENT EFFECTS ON CORRELATIONS

Recently, the issue of range enhancement effects on correlations has been raised as a potential criticism of correction for range restriction in validity generalization. While adding to or deleting from a sample can affect both the variance and correlation in the sample, increases or decreases in individual differences in an intact sample resulting from such factors as history, maturation, training, and experience will not necessarily affect correlations. If such changes in individual differences do occur in concurrent validation studies, the most likely result will be an underestimate of the unrestricted validity.     

Empirical validation of a procedure to correct position and stimulus biases in matching-to-sample

The development of position and stimulus biases often occurs during initial training on matching-to-sample tasks. Furthermore, without intervention, these biases can be maintained via intermittent reinforcement provided by matching-to-sample contingencies. The present study evaluated the effectiveness of a correction procedure designed to eliminate both position and stimulus biases. Following key-peck training, a group of 6 pigeons had extended exposure to matching-to-sample contingencies without a correction procedure, a group of 4 pigeons was briefly exposed to a simultaneous matching-to-sample procedure to assess biases prior to exposure to the correction procedure, and a group of 5 pigeons was exposed directly to the correction procedure. The correction procedure arranged that every time an incorrect match was made, the trial configuration was repeated on the subsequent trial until a correct match was made. Extended exposure to matching-to-sample contingencies without a correction procedure was associated with reduced biases eventually for most subjects, but rapid development of near-perfect accuracy and bias-free performance was observed upon the implementation of the correction procedure regardless of the type of bias. Bias-free performance was maintained following subsequent exposure to a zero-delay MTS procedure.     

Application patterns when applicants know the odds: implications for selection research and practice

Unlike previous research that found small differences between population standard deviations and applicant pool standard deviations (P. R. Sackett & D. J. Ostgaard, 1994; D. S. Ones & C. Viswesvaran, 2003), this study revealed a 23% disparity between Law School Admission Test (LSAT) scores of all LSAT test takers and those of LSAT test takers who applied to law school. This study also illustrated robust applicant self-selection behavior across different law school ranks. These findings are important, because predictor scores of applicants who know their scores in advance and perceive small selection ratios necessitate substantially smaller range restriction corrections than those that would be required by population standard deviations. Furthermore, these findings more generally reveal that applicants who know their scores in advance behave quite differently from applicants who do not.     

CORRECTING THE 16PF FOR FAKING: EFFECTS ON CRITERION-RELATED VALIDITY AND INDIVIDUAL HIRING DECISIONS

The effects of correcting a personality measure for faking were evaluated within an organizational context. Two possible repercussions of score correction were studied using the 16PF personality inventory: the effect on criterion-related validity and the effect on individual hiring decisions (i.e., which applicants would or would not be hired). Results indicated that correction for faking had little effect on criterion-related validity. However, depending on the selection ratio, correction of scores would have resulted in different hiring decisions than those that would have been made on the basis of uncorrected scores. Implications for organizations using personality measures for selection and recommendations regarding the use of faking corrections are discussed.     

Cohen's d Corrected for Case IV Range Restriction: A More Accurate Procedure for Evaluating Subgroup Differences in Organizational Research

Organizational and staffing researchers are often interested in evaluating whether subgroup differences exist (e.g., between Caucasian and African‐American individuals) on predictors of job performance. To investigate subgroup differences, researchers often will collect data from current employees to make inferences about subgroup differences among job applicants. However, the magnitude of subgroup differences (i.e., Cohen's d) within incumbent samples may be different (i.e., smaller) than the magnitude of subgroup differences in applicant samples because selection of applicants typically reduces the variance of scores on the predictors (i.e., because lower scoring applicants are not selected). If researchers seek to generalize a d value in an incumbent sample to the applicant population, they may use Bobko, Roth, and Bobko's (correcting the effect size of d for range restriction and unreliability, 2001) Case II or III correction. By extension, Hunter, Schmidt, and Le (implications of direct and indirect range restriction for meta‐analysis methods and findings, 2006) have proposed a Case IV correction, which is more realistic than Bobko et al.'s approach. Therefore, this paper develops a Case IV correction for d (i.e., d_c4). The simulation results showed that the d_c4 was generally accurate across 6,000 simulation conditions. Moreover, 2 published datasets were reanalyzed to show the influence of the Case IV correction on d. In addition, implications and future directions of the d_c4 are discussed.     

Extending the activity restriction model of depressed affect: evidence from a sample of breast cancer patients.

Survey data from 95 women with Stage 1 (n = 36), Stage 2 (n = 49), or Stage 3 (n = 10) breast cancer both confirm and extend prior research indicating that restriction of normal activities is an important factor in depressed affect. Illness severity was directly related to more restricted routine activities, and more activity restriction was associated with higher public self-consciousness and less social support. Beyond the effects of age, self-consciousness, illness severity, and social support, activity restriction explained significant additional variance in symptoms of depression. Moreover, activity restriction mediated the impact of pain, public self-consciousness, and social support on depressed affect, which implies that these factors foster symptoms of depression by disrupting normal activities.