INCREASING THE ACCURACY OF CORRECTIONS FOR RANGE RESTRICTION: IMPLICATIONS FOR SELECTION PROCEDURE VALIDITIES AND OTHER RESEARCH RESULTS

The common practice in meta-analyses and in individual studies of correcting for direct range restriction even though range restriction is actually indirect has long been known to lead to undercorrection, but this error has been assumed to be small. Using validity generalization data sets for 4 jobs, this study calibrated this error by comparing meta-analysis results based on corrections for direct range restriction with the more accurate results from a recently developed method of correcting for indirect range restriction. It was found that, on average, correction for direct range restriction resulted in substantial underestimation of operational validities for both job performance measures (21%) and training performance measures (28%). In addition, 90% credibility values were on average underestimated by 38%–40%. In addition to the implications for personnel selection, these findings suggest that similar underestimation of important relationships has occurred in other areas of research, with potential implications for theory development.     

INTERPRETING THE RESULTS OF META-ANALYTIC RESEARCH: A COMMENT ON SCHMITT, GOODING, NOE, AND KIRSCH (1984)

This comment shows that the conclusion of Schmitt, Gooding, Noe, and Kirsch (1984) that their meta-analytic findings are inconsistent with earlier validity generalization work is in error. The findings in their study that less variance than previously reported was due to sampling error are a result of their larger average sample sizes. Their claim that, after sampling error variance was accounted for, much unexplained variance remained, is incorrect. This error is demonstrated to be a result of their exclusive concentration on percentages and consequent failure to examine amount of observed and residual variance.     

ESTIMATION OF EMPLOYMENT VALIDITIES BY LESS EXPERIENCED JUDGES

In a previous study, Schmidt, Hunter, Croll and McKenzie (1983) demonstrated that estimates of the validity of cognitive tests made by highly trained and experienced judges are more accurate than empirical estimates obtained from small-sample validity studies. The present study examined whether less experienced judges could also produce accurate estimates. Twenty-eight recent Ph.D.'s in I/O Psychology estimated observed validities for the same 54 job-test combinations used by Schmidt et al. (1983). The estimates of these judges contained about twice as much random error as the experts' estimates. Systematic error of the less experienced judges was also greater than that of the experts (.0732 vs .019). The systematic errors of the two sets of judges were in opposite directions: less experienced judges overestimated validities, on average, while experts underestimated them. The results show that the estimates of less experienced judges contain less information than those of experts, but also that averages of estimates of several less experienced judges are as accurate as those obtained from small-sample empirical studies.     

A WITHIN SETTING EMPIRICAL TEST OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION

The situational specificity hypothesis of selection procedure validity makes two predictions. The first is that variation in observed validities across settings is caused by real differences in what constitutes job performance. Validity generalization studies to date have provided disconfirming evidence for this prediction by showing that the observed variation is due to statistical and measurement artifacts. The second prediction is that if the situation (i.e., the organization, the setting, the job, the test and the criterion) is held constant, then validity findings will not vary from study to study. This article tests this prediction empirically using data from a unique series of studies reported by Bender and Loveless (1958). The results are contrary to the situational specificity hypothesis and consistent with the hypothesis that variation in observed validities is due to statistical and measurement artifacts that are unrelated to situations and settings. It is concluded that both major predictions of the situational specificity hypothesis have now been empirically disconfirmed.     

THE FUTURE OF CRITERION-RELATED VALIDITY1

As a result of rejection by personnel psychologists of the erroneous law of small numbers and of the adoption of correct inferential procedures, the future of criterion-related validity promises to be bright. Probable future developments include: (a) widespread appreciation of the low statistical power characteristic of small samples; (b) rejection of the traditional belief that validities are situationally specific; (c) widespread validity generalization based on new methods; (d) research demonstrations that rational estimates of validity are often superior in accuracy to empirical estimates; (e) widespread acceptance of research findings showing that single-group and differential validity by race are artifactual rather than substantive in origin and that test unfairness by race is a nonproblem; (f) increased application of decision-theoretic dollar utility analyses to selection programs; and (g) progress in the establishment of general principles and theories about trait-performance relations in the world of work.