FURTHER TESTS OF THE SCHMIDT-HUNTER BAYESIAN VALIDITY GENERALIZATION PROCEDURE1

Using an improved version of the Bayesian validity generalization model presented in Schmidt and Hunter (1977), this study showed that most of the between-study variation in observed validity coefficients of various aptitude and ability tests for two types of clerical work and for the job of first-line supervisor is artifactual in nature. These results cast additional doubt on the traditional belief that employment test validities are situationally specific. In addition, the results showed that generalization of validities to new settings was justified even where the hypothesis of situational specificity could not be rejected under the standards used in this study. Implications for the use of aptitude and ability tests in selection and for theory development in industrial-organizational psychology are discussed.     

VALIDITY GENERALIZATION RESULTS FOR LAW ENFORCEMENT OCCUPATIONS

The Schmidt-Hunter interactive validity generalization procedure was applied to validity data for cognitive abilities tests for law enforcement occupations. Both assumed artifact distributions, and distributions of artifacts constructed from information contained in the current sample of studies were used to test the hypothesis of situational specificity and to estimate validity generalizability. Results for studies using a criterion of performance in training programs showed that validities ranged from .41 to .71, and for four test types the hypothesis of situational specificity could be rejected using the 75% decision rule. For the remaining test types, validity was generalizable, based on 90% credibility values ranging from .37 to .71. Results for studies using a criterion of performance on the job indicated that the hypothesis of situational specificity was not tenable for three test types, which had validities between .17 and .31. For the remaining test types, estimated mean true validities ranged from .10 to .26 and were generalizable to a majority of situations. Results for both groups of studies were essentially identical for the two types of artifact distribution. Possible reasons for the apparently lower validities and lesser generalizability for job performance criteria are discussed, including possible low validity of the criterion (due to lack of opportunity by supervisors to observe behavior) and the potential role of noncognitive factors in the determination of law enforcement job success. Suggestions for specifically targeted additional research are made.     

A WITHIN SETTING EMPIRICAL TEST OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION

The situational specificity hypothesis of selection procedure validity makes two predictions. The first is that variation in observed validities across settings is caused by real differences in what constitutes job performance. Validity generalization studies to date have provided disconfirming evidence for this prediction by showing that the observed variation is due to statistical and measurement artifacts. The second prediction is that if the situation (i.e., the organization, the setting, the job, the test and the criterion) is held constant, then validity findings will not vary from study to study. This article tests this prediction empirically using data from a unique series of studies reported by Bender and Loveless (1958). The results are contrary to the situational specificity hypothesis and consistent with the hypothesis that variation in observed validities is due to statistical and measurement artifacts that are unrelated to situations and settings. It is concluded that both major predictions of the situational specificity hypothesis have now been empirically disconfirmed.     

The Within‐Setting Variability of Validity in Cognitive Ability Tests

This study was directed to check the second prediction of the situational specificity hypothesis (SSH). According to this prediction, if the setting is held constant, then validity coefficients will not vary from study to study. This prediction was tested using data from a series of studies in which cognitive ability tests were used. These tests are subjected to a cultural bias that could affect the validity variability. The results are contrary to the second prediction of the SSH because a great variability was found in the validity coefficients within‐setting. However, the findings are consistent with the hypothesis of validity generalization (VGH) that predicts similar variability in both across settings and within‐setting. Also, the cultural context does not seem to affect the variability of validity and this variability is similar to that observed in previous studies of validity generalization.     

Implications of empirical Bayes meta-analysis for test validation.

Empirical Bayes meta-analysis provides a useful framework for examining test validation. The fixed-effects case in which rho has a single value corresponds to the inference that the situational specificity hypothesis can be rejected in a validity generalization study. A Bayesian analysis of such a case provides a simple and powerful test of rho = 0; such a test has practical implications for significance testing in test validation. The random-effects case in which sigma2rho > 0 provides an explicit method with which to assess the relative importance of local validity studies and previous meta-analyses. Simulated data are used to illustrate both cases. Results of published meta-analyses are used to show that local validation becomes increasingly important as sigma2rho increases. The meaning of the term validity generalization is explored, and the problem of what can be inferred about test transportability in the random-effects case is described.     

Comparison of two random-effects methods of meta-analysis

Two studies compared the Schmidt-Hunter method of meta-analysis (J. E. Hunter & F. L. Schmidt, 1990) with the method described by L. V. Hedges and J. L. Vevea (1998). Study 1 evaluated estimates of rho, sigmarho, and resulting credibility intervals for both models through Monte Carlo methods. Results showed slight differences between the 2 methods. In Study 2, a reanalysis of published meta-analyses using both methods with several artifact distributions showed that although both choice of technique and type of correction could matter, the technique of meta-analysis used is less influential on the study outcome than is the choice of artifact correction.     

EXPLORING OPTIONS FOR SUPPORTING TEST USE IN SITUATIONS PRECLUDING LOCAL VALIDATION

This paper examines some of the options available to practitioners interested in supporting the use of selection measures in an organization, including test transportability, validity generalization (VG), and synthetic validation, reviewing some of the advantages, disadvantages and requirements of each approach. Results of four unpublished, proprietary validation studies are reported which compare validity estimates provided by the job component validation (JCV; a type of synthetic validation) routine inherent in the Position Analysis Questionnaire (PAQ) with observed validity coefficients for a variety of criterion measures. We then examine the accuracy of the JCV procedure in predicting validity coefficients for 51 clerical classifications extracted from an existing PAQ job evaluation database of a large utility company. Predicted JCVs are compared to mean observed validity coefficients for five DOT clerical categories provided by Pearlman, Schmidt, and Hunter (1980). The VG and JCV methods provided highly similar and converging estimates of the validity of cognitive ability tests for predicting performance in clerical occupations. Implications for practice are discussed, particularly the need to use multiple, converging lines of evidence to support test use.     

USING RANDOM RATHER THAN FIXED EFFECTS MODELS IN META-ANALYSIS: IMPLICATIONS FOR SITUATIONAL SPECIFICITY AND VALIDITY GENERALIZATION

Combining statistical information across studies (i.e., meta-analysis) is a standard research tool in applied psychology. The most common meta-analytic approach in applied psychology, the fixed effects approach, assumes that individual studies are homogeneous and are sampled from the same population. This model assumes that sampling error alone explains the majority of observed differences in study effect sizes and its use has lead some to challenge the notion of situational specificity in favor of validity generalization. We critique the fixed effects methodology and propose an advancement–the random effects model (RE) which provides estimates of how between-study differences influence the relationships under study. RE models assume that studies are heterogeneous since they are often conducted by different investigators under different settings. Parameter estimates of both models are compared and evidence in favor of the random effects approach is presented. We argue against use of the fixed effects model because it may lead to misleading conclusions about situational specificity.     

Meta‐Analysis of Clerical Performance Predictors: Still stable after all these years

Pearlman, Schmidt, and Hunter (1980) published a seminal paper on the validity of various measures for predicting performance in clerical jobs. They concluded that for both job and training performance criteria, 10 types of tests (e.g., perceptual speed, clerical aptitude, verbal ability) predicted performance across 5 clerical job families. This paper describes a psychometric meta‐analysis of validity studies using similar measures for clerical jobs conducted since 1980 to examine the stability of Pearlman and colleagues' validity estimates. This paper contributes to the literature by investigating the long‐term stability of validity generalization estimates over a period of several decades. Clerical jobs provide a compelling case study because these jobs have changed considerably due to the increased use of computers and technology in the office environment. Results showed that the mean population estimates in the present study were consistent with, or higher than, those obtained by Pearlman and colleagues, thus demonstrating the long‐term stability of meta‐analytic estimates of validity. The relative stability of the validity estimates also supports the robustness of g as a predictor, even as jobs change over time.     

FURTHER WITHIN-SETTING EMPIRICAL TESTS OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION

The situational specificity hypothesis in personnel selection holds that variation in observed validity coefficients across studies for the same test and job is due to subtle variations from setting to setting in what constitutes job performance. This hypothesis therefore predicts that, if the setting does not vary, validity will not vary. Using data from a single large-sample validity study ( N = 1,455), this research generated numerous small-sample studies for which the setting (organization, job, test, criterion measure, applicant pool, time period, and sample size) was held constant. It was found that even under these circumstances there was substantial variability across studies in (a) observed validity coefficients, (b) significance levels, and (c) (using traditional data analytic methods) conclusions about the presence or absence of validity. These findings disconfirm the situational specificity hypothesis and argue strongly against traditional data-analytic procedures and the practice of reliance on single small-sample studies. In contrast to the erroneous conclusions produced by traditional data-analytic procedures, meta-analytic methods correctly estimated the population observed validity at .22 and correctly indicated that all between-study variance in observed validities was due to sampling error alone.     

Impact of Assessments of Validity Generalization and Situational Specificity on the Science and Practice of Personnel Selection

The application of meta-analysis, in particular validity generalization (VG) analysis, to the cumulative literature on the validity of selection tests has fundamentally changed the science and practice of personnel selection. VG analyses suggest that the validities of standardized tests and other structured assessments are both higher and more consistent across jobs and organizations than was previously believed. As a result, selection researchers and practitioners can draw on the research literature to make reasonably accurate forecasts about the validity and usefulness of different tests in particular applications. Distinctions between tests of validity generalization and tests of situational specificity are described, and difficulties in demonstrating that validity is constant across the different settings where tests are used are outlined.     

The Situational Specificity of Validities: Correcting for Statistical Artifacts Does Not Always Reduce the Trans-Situational Variability of Correlation Coefficients

Validity generalization research suggests that much of the variability in validity coefficients is due to statistical artifacts, such as sampling error. Correcting for these artifacts usually reduces the trans-situational variability in validity coefficients, but the reverse can happen. Conditions under which corrections for statistical artifacts can increase the variability in validity coefficients are described, examples of meta-analyses reporting larger corrected than observed variance in validity are cited, and implications for the situational specificity of validity coefficients are discussed. In general, the true variability in validity coefficients based on large samples is often larger than the observed variability in test validities.     

Factors and fictions in person perception: A reply to lamiell,Foss, and Cavenee1

The systematic distortion hypothesis states that under difficult memory conditions judges infer what “must” have happened from their general model of what the world is like and/or find it easier to retrieve conceptually affiliated memory items. The hypothesis further states that lay conceptual schemes “tend to be innacurate with respect to how behaviors covary, confusing ‘what is like what’ with ‘what goes with what’” (Shweder & D'Andrade, 1979), and that interbehavior correlations derived from memory-based ratings cannot be considered valid evidence for the objective existence of proposed personality traits, factors or syndromes. The systematic distortion hypothesis is clarified with reference to the Lamiell, Foss, and Cavenee “critique” (1980). That “critique,” it is argued, is neither a test of the systematic distortion hypothesis nor a critique. What Lamiell, Foss, and Cavenee demonstrate is that the validity of the systematic distortion hypothesis does not depend upon the unrealistic assumption that observers are capable of reporting only what they expect to see.     

效度概化的理论研究述评

效度概化（validity generalization：VG）是心理测量理论和元分析的结合体,也是元分析在人事测评领域的重要应用。它深刻地挑战了人事测评效度的情景特异论,在工业与组织心理学的发展历程当中具有里程碑意义。该文系统地介绍了效度概化理论的逻辑思路、主要模型、关键理论语与应用,以及研究趋势。     

Explanatory Consolidation: From ‘Best’ to ‘Good Enough’

In science and everyday life, we often infer that something is true because it would explain some set of facts better than any other hypothesis we can think of. But what if we have reason to believe that there is a better way to explain these facts that we just haven’t thought of? Wouldn’t that undermine our warrant for believing the best available explanation? Many philosophers have assumed that we can solve such underconsideration problems by stipulating that a hypothesis should not only be ‘the best’ explanation available; rather, it should also be ‘good enough’. Unfortunately, however, the only current suggestion for what it might mean to say that an explanation is ‘good enough’ is, well, not good enough. This paper aims to provide a better account of what is required for an explanatory hypothesis to be considered ‘good enough’. In brief, the account holds that a ‘good enough’ hypothesis is one that has gone through a process that I call explanatory consolidation, in which accumulating evidence and failed attempts to formulate better alternatives gradually make it more plausible that the explanation we currently have is better than any other that could be formulated.     

Empowerment,agency, and power

As ‘empowerment’ and ‘agency’ have received wider usage within development research and policy, ambiguities and variant meanings have proliferated. Amidst this conceptual drift, there has also been a tendency to assimilate the two concepts. This tendency is problematic in a number of ways. First, ‘agency’ has various meanings, and the weakest of these captures little of the concept of empowerment. Second, empowerment has a conceptual link with well-being that agency cannot have. Third, when empowerment is assimilated with expanded agency, that agency is not considered in a relational way: the focus is on how the agency of a group or individual becomes greater than it was, not on the degree to which their agency is dependent on or dominated by the agency of others. If ‘empowerment’ no longer refers to social relations, it loses its direct relevance to the transformation of those relations and, as some critics have claimed, it ceases to be a ‘transformative’ concept. After showing that there are cases of empowerment that cannot be captured by conceptions of empowerment that ‘take power out’, I draw upon the capability approach to propose relational conceptions of agency and empowerment that ‘bring power back in’.     

How Do You Assess a Manager’s Decision-Making Abilities? The Use of Situational Inventories

Decision-making capabilities are absolutely crucial to a manager. Unfortunately, existing methods of assessing managers in this area for selection and development purposes (in-tray exercises, situational interviews, ability tests, etc.) leave a lot to be desired. This article focuses on an alternative for assessing managerial decision making – the situational inventory – and presents research findings and information on practical applications. Detailed findings are also presented for ‘Scenarios’, the UK’s first published situational measure of managerial judgement. Situational inventories work by presenting participants with realistic but difficult real-life management scenarios. Each scenario is accompanied by a number of possible responses which participants rate for effectiveness in dealing with the scenario. Participants’ ratings are then scored against a set of ideal answers, producing an assessment of current decision-making ability. Decision-making ability can be developed in individuals, making feedback invaluable to participants. Evidence that has been accumulating in the United States and the UK for at least 15 years is presented to support the general situational inventory approach. Additionally, specific evidence is presented for ‘Scenarios’. It was found to correlate significantly with a number of managerial performance and responsibility indicators while appearing to be largely separate from existing psychometric (ability and personality) tests.     

Cognitive Tests,Constructs, and Content Validity: A commentary on Schmidt ()

This commentary discusses a number of issues that build on Schmidt's (International Journal of Selection and Assessment, 20, 1‐13 (2012)) perspective on content validity and cognitive tests. First, it elaborates on the relationship between the treatment of content validity in various professional standards and government guidelines. Second, it offers a differing perspective on the definition of ‘construct’ than that taken by Schmidt. Third, it elaborates on the settings in which content validity can and cannot be used to support the use of a given test in the cognitive ability domain.     

MODERATOR RESEARCH AND THE LAW OF SMALL NUMBERS1

The thesis of this paper is that many proposed moderators in personnel psychology are probably illusory, having been created solely by belief in the law of small numbers. Evidence is presented that race as a moderator of test validity is one such illusory moderator. In addition, a model for validity generalization is described which, in addition to eliminating the need for criterion-related validity studies under certain circumstances, strongly calls into question the idea that situations moderate test validity, i.e., the traditional doctrine of situational specificity of test validities. Calculations are presented which show that adequate statistical power in moderator research requires much larger sample sizes than have typically been employed. This requirement is illustrated empirically using validity data for the Army Classification Battery for 35 jobs and 21,000 individuals. These analyses show that (1) even when a moderator is generally assumed to be large, large samples are required to gauge its effect reliably and (2) large sample research may show that moderators that appear plausible and important a priori are nonexistent or trivial in magnitude. The practice of pooling across numerous small sample studies to obtain statistical power equivalent to that of large sample studies is recommended. In light of the evidence that many proposed moderators may not exist, the authors hypothesize that the true structure of underlying relationships in personnel psychology is considerably simpler than personnel psychologists have generally imagined it to be.     

Revisiting Case IV: A reassessment of bias and standard errors of Case IV under range restriction

In 2004, Hunter and Schmidt proposed a correction (called Case IV) that seeks to estimate disattenuated correlations when selection is made on an unmeasured variable. Although Case IV is an important theoretical development in the range restriction literature, it makes an untestable assumption, namely that the partial correlation between the unobserved selection variable and the performance measure is zero. We show in this paper why this assumption may be difficult to meet and why previous simulations have failed to detect the full extent of bias. We use meta‐analytic literature to investigate the plausible range of bias. We also show how Case IV performs in terms of standard errors. Finally, we give practical recommendations about how the contributions of Hunter and Schmidt (2004) can be extended without making such stringent assumptions.