MODERATOR RESEARCH AND THE LAW OF SMALL NUMBERS1

The thesis of this paper is that many proposed moderators in personnel psychology are probably illusory, having been created solely by belief in the law of small numbers. Evidence is presented that race as a moderator of test validity is one such illusory moderator. In addition, a model for validity generalization is described which, in addition to eliminating the need for criterion-related validity studies under certain circumstances, strongly calls into question the idea that situations moderate test validity, i.e., the traditional doctrine of situational specificity of test validities. Calculations are presented which show that adequate statistical power in moderator research requires much larger sample sizes than have typically been employed. This requirement is illustrated empirically using validity data for the Army Classification Battery for 35 jobs and 21,000 individuals. These analyses show that (1) even when a moderator is generally assumed to be large, large samples are required to gauge its effect reliably and (2) large sample research may show that moderators that appear plausible and important a priori are nonexistent or trivial in magnitude. The practice of pooling across numerous small sample studies to obtain statistical power equivalent to that of large sample studies is recommended. In light of the evidence that many proposed moderators may not exist, the authors hypothesize that the true structure of underlying relationships in personnel psychology is considerably simpler than personnel psychologists have generally imagined it to be.     

THE ECONOMIC IMPACT OF JOB SELECTION METHODS ON SIZE, PRODUCTIVITY, AND PAYROLL COSTS OF THE FEDERAL WORK FORCE: AN EMPIRICALLY BASED DEMONSTRATION

In this study, job performance increases resulting from improved selection validity were measured empirically rather than estimated from the standard linear regression utility equations. Selection utility analyses based on these empirical measurements were carried out for most white-collar jobs in the federal government. Results indicate that selection of a one-year cohort based on valid measures of cognitive ability, rather than on non-test procedures (mostly evaluations of education and experience), produces increases in output worth up to $600 million for each year that the new employees remain employed by the government. Newly hired federal employees remain with the government an average of approximately 13 years, resulting in a total gain in output of almost $8 billion over this period. This gain represents a 9.7% increase in output among new hires. If total output is held constant rather than increased, new hiring can be reduced by up to 20,044 per year (a 9% decrease), resulting in payroll savings of $272 million for every year the new cohort of employees remains on the job. The percentage of new hires in the bottom decile of the non-test-selected job performance distribution     

ASSESSING THE ECONOMIC IMPACT OF PERSONNEL PROGRAMS ON WORKFORCE PRODUCTIVITY

This article describes and illustrates the adaptation of the linear-regression-based decision-theoretic equations used to estimate the dollar impact of valid selection procedures on workforce productivity to the evaluation of intervention programs designed to improve job performance. The appropriate equations are derived and explained, methods for estimating equation parameters are discussed, and the use of these equations is illustrated by means of a hypothetical example. It is concluded that in the future these methods and equations will allow psychologists to make more accurate assessments of the impact of intervention programs on workforce productivity than has heretofore been the case.     

FURTHER WITHIN-SETTING EMPIRICAL TESTS OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION

The situational specificity hypothesis in personnel selection holds that variation in observed validity coefficients across studies for the same test and job is due to subtle variations from setting to setting in what constitutes job performance. This hypothesis therefore predicts that, if the setting does not vary, validity will not vary. Using data from a single large-sample validity study ( N = 1,455), this research generated numerous small-sample studies for which the setting (organization, job, test, criterion measure, applicant pool, time period, and sample size) was held constant. It was found that even under these circumstances there was substantial variability across studies in (a) observed validity coefficients, (b) significance levels, and (c) (using traditional data analytic methods) conclusions about the presence or absence of validity. These findings disconfirm the situational specificity hypothesis and argue strongly against traditional data-analytic procedures and the practice of reliance on single small-sample studies. In contrast to the erroneous conclusions produced by traditional data-analytic procedures, meta-analytic methods correctly estimated the population observed validity at .22 and correctly indicated that all between-study variance in observed validities was due to sampling error alone.     

ANALYZING UTTERANCES AS THE OBSERVATIONAL UNIT

If utterances are the observational unit of analysis and there are no sequential patterns to the interaction, two alternative statistical models may be applied. A hierarchical design in which utterances are nested within subjects and subjects are nested within treatment condition is considered. One method of analysis pools utterances within treatment condition; the other method collapses across utterances to obtain subject means. Inappropriate application of the pooling model instead of the subject means model can lead to Type I errors, decreased generalizability, and inflated variance estimates that attenuate univariate and multivariate correlations. The effect of utterance reliability on the apparent unidimensionality and parallelism of multiple indicators is also discussed.