Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure

Using computer simulation, the authors assessed the accuracy of J. E. Hunter, F. L. Schmidt, and H. Le's (2006) procedure for correcting for indirect range restriction, the most common type of range restriction, in comparison with the conventional practice of applying the Thorndike Case II correction for direct range restriction. Hunter et al.'s procedure produced more accurate estimates of both the mean and standard deviation in meta-analysis than the conventional procedure. Even when its key assumption that the effect of selection on a 3rd variable is fully mediated by the independent variable was violated, Hunter et al.'s procedure was still relatively more accurate than the conventional procedure. When applied to data from a previously published meta-analysis, the new procedure yielded results that led to different substantive conclusions.     

Cohen's d Corrected for Case IV Range Restriction: A More Accurate Procedure for Evaluating Subgroup Differences in Organizational Research

Organizational and staffing researchers are often interested in evaluating whether subgroup differences exist (e.g., between Caucasian and African‐American individuals) on predictors of job performance. To investigate subgroup differences, researchers often will collect data from current employees to make inferences about subgroup differences among job applicants. However, the magnitude of subgroup differences (i.e., Cohen's d) within incumbent samples may be different (i.e., smaller) than the magnitude of subgroup differences in applicant samples because selection of applicants typically reduces the variance of scores on the predictors (i.e., because lower scoring applicants are not selected). If researchers seek to generalize a d value in an incumbent sample to the applicant population, they may use Bobko, Roth, and Bobko's (correcting the effect size of d for range restriction and unreliability, 2001) Case II or III correction. By extension, Hunter, Schmidt, and Le (implications of direct and indirect range restriction for meta‐analysis methods and findings, 2006) have proposed a Case IV correction, which is more realistic than Bobko et al.'s approach. Therefore, this paper develops a Case IV correction for d (i.e., d_c4). The simulation results showed that the d_c4 was generally accurate across 6,000 simulation conditions. Moreover, 2 published datasets were reanalyzed to show the influence of the Case IV correction on d. In addition, implications and future directions of the d_c4 are discussed.     

Bootstrap standard error and confidence intervals for the correlations corrected for indirect range restriction

The standard Pearson correlation coefficient, r, is a biased estimator of the population correlation coefficient, ρ(XY) , when predictor X and criterion Y are indirectly range-restricted by a third variable Z (or S). Two correction algorithms, Thorndike's (1949) Case III, and Schmidt, Oh, and Le's (2006) Case IV, have been proposed to correct for the bias. However, to our knowledge, the two algorithms did not provide a procedure to estimate the associated standard error and confidence intervals. This paper suggests using the bootstrap procedure as an alternative. Two Monte Carlo simulations were conducted to systematically evaluate the empirical performance of the proposed bootstrap procedure. The results indicated that the bootstrap standard error and confidence intervals were generally accurate across simulation conditions (e.g., selection ratio, sample size). The proposed bootstrap procedure can provide a useful alternative for the estimation of the standard error and confidence intervals for the correlation corrected for indirect range restriction.     

The danger of conflating level-specific effects of control variables when primary interest lies in level-2 effects

In the multilevel modelling literature, methodologists widely acknowledge that a level-1 variable can have distinct within-cluster and between-cluster effects, and that failing to disaggregate these can yield a slope estimate that is an uninterpretable, conflated blend of the two. Methodologists have stated, however, that including conflated slopes of level-1 variables in a model is not problematic if substantive interest lies only in effects of level-2 predictors. Researchers commonly follow this advice and use methods that do not disaggregate effects of level-1 control variables (e.g., grand mean centering) when examining effects of level-2 predictors. The primary purpose of this paper is to show that this is a dangerous practice. When level-specific effects of level-1 variables differ, failing to disaggregate them can severely bias estimation of level-2 predictor slopes. We show mathematically why this is the case and highlight factors that can exacerbate such bias. We corroborate these findings with simulations and present an empirical example, showing how such distortions can severely alter substantive conclusions. We ultimately recommend that simply including the cluster mean of the level-1 variable as a control will alleviate the problem.     

Correction for Range Restriction in Meta‐Analysis Revisited: Improvements and Implications for Organizational Research

In this study, we present a more accurate method for correcting for range restriction (Case V) that expands upon Bryant and Gokhale's (1972) method. We further present detailed steps to incorporate the Case V method into Schmidt and Hunter's (2015) psychometric meta‐analysis methods (both individual correction and artifact distribution approaches). We then evaluate the accuracy of the Case V method vis‐à‐vis existing methods. Monte‐Carlo simulation results indicate that the Case V method provides very accurate estimates for the mean true score correlation and reasonably accurate estimates for the true standard deviation. More important, Case V almost always provides more accurate results than alternative methods (particularly, Case IV). To illustrate how the Case V method works with real data, we conduct a reanalysis of Judge, Heller, and Mount's (2002) meta‐analysis examining the relationships between the Big 5 personality traits and job satisfaction. Results indicate that the true score correlations between the Big 5 traits and job satisfaction have been underestimated, whereas their true standard deviations have been overestimated. Implications for range restriction corrections in organizational research are discussed.     

An Improved Correction for Range Restricted Correlations Under Extreme,Monotonic Quadratic Nonlinearity and Heteroscedasticity

Standardized tests are frequently used for selection decisions, and the validation of test scores remains an important area of research. This paper builds upon prior literature about the effect of nonlinearity and heteroscedasticity on the accuracy of standard formulas for correcting correlations in restricted samples. Existing formulas for direct range restriction require three assumptions: (1) the criterion variable is missing at random; (2) a linear relationship between independent and dependent variables; and (3) constant error variance or homoscedasticity. The results in this paper demonstrate that the standard approach for correcting restricted correlations is severely biased in cases of extreme monotone quadratic nonlinearity and heteroscedasticity. This paper offers at least three significant contributions to the existing literature. First, a method from the econometrics literature is adapted to provide more accurate estimates of unrestricted correlations. Second, derivations establish bounds on the degree of bias attributed to quadratic functions under the assumption of a monotonic relationship between test scores and criterion measurements. New results are presented on the bias associated with using the standard range restriction correction formula, and the results show that the standard correction formula yields estimates of unrestricted correlations that deviate by as much as 0.2 for high to moderate selectivity. Third, Monte Carlo simulation results demonstrate that the new procedure for correcting restricted correlations provides more accurate estimates in the presence of quadratic and heteroscedastic test score and criterion relationships.     

Case and Reanalysis

In this paper we discuss an asymmetry in the Case system of German and its implications for human sentence processing: the asymmetry between nominative/accusative and dative case. Starting from the assumption that dative case has a distinct grammatical representation--dative DPs are embedded into an extra structural layer KP--the results of two experiments will be presented, which show that dative assignment during reanalysis is accompanied by additional processing operations that are not needed when accusative or nominative are assigned. In particular, we show that dative assignment during reanalysis triggers reaccess to the mental lexicon, giving rise to greater processing difficulty. We conclude with a discussion of empirical and theoretical consequences of our findings.     

Updating meta-analytic research findings: Bayesian approaches versus the medical model

The authors examine 3 methods of combining new studies into existing meta-analyses: (a) adding the new study or studies to the database and recalculating the meta-analysis (the medical model); (b) using the Bayesian procedure advocated by F. L. Schmidt and J. E. Hunter (1977) and F. L. Schmidt, J. E. Hunter, K. Pearlman, and G. S. Shane (1979) to update the meta-analysis; and (c) using the Bayesian methods advocated by these authors and M. T. Brannick (2001) and M. T. Brannick, S. M. Hall, and Y. Liu (2002) to estimate study-specific parameters. Method b was found to severely overweight new studies relative to the previous studies contained in the meta-analysis, and Method c was found to do the same while also requiring an assumption with a low prior probability of being correct, causing the method to violate Bayesian principles. The authors present an alternative Bayesian procedure that does not suffer from these drawbacks and yields meta-analytic results very similar to those obtained with the medical model. They recommend use of the medical model or this alternative Bayesian procedure.     

Complicating Counterspaces: Intersectionality and the Michigan Womyn's Music Festival

The counterspaces framework articulated by Case and Hunter (2012), follows from community psychology's long‐standing interest in the potential for settings to promote well‐being and liberatory responses to oppression. This framework proposes that certain settings (i.e., “counterspaces”) facilitate a specific set of processes that promote the well‐being of marginalized groups. We argue that an intersectional analysis is crucial to understand whether and how counterspaces achieve these goals. We draw from literature on safe spaces and present a case study of the Michigan Womyn's Music Festival (Michfest) to illustrate the value of an intersectional analysis and explore how these processes operate. Based on 20 in‐person interviews, 23 responses to an online survey, and ethnographic field notes, we show how Michfest was characterized by a particular intersection of identities at the setting level, and intersectional diversity complicated experiences at the individual level. Moreover, intersectional identities provided opportunities for dialogue and change at the setting level, including the creation of counterspaces within counterspaces. Overall, we demonstrate the need to attend to intersectionality in counterspaces, and more broadly in how we conceptualize settings in community psychology.     

Validity generalization: Some critical remarks on the Schmidt-Hunter procedure*

This article presents some critical comments on the validity generalization procedure which has been presented by Schmidt, Hunter and others. They have put forward a method for testing the hypothesis that the variance in validity coefficients across situations for job-test combinations is due to what they consider to be statistical artifacts. The Schmidt-Hunter approach is criticized on the following points: the compilation of validity data, the use of criterion measures, and the test of the hypothesis of no situational specificity. Further, the relation between the concepts ‘situational specificity’ and ‘validity generalization’ is considered. In addition, it is noted that Schmidt, Hunter and others have defined the concept ‘situation’ in a different way than classical writers. It is concluded that the Schmidt-Hunter approach to validity generalization shows fundamental shortcomings. As a consequence their far-reaching conclusions for the practice of personnel selection should be considered premature.     

The ecological rationality of state-dependent valuation

Laboratory studies on a range of animals have identified a bias that seems to violate basic principles of rational behavior: a preference is shown for feeding options that previously provided food when reserves were low, even though another option had been found to give the same reward with less delay. The bias presents a challenge to normative models of decision making (which only take account of expected rewards and the state of the animal at the decision time). To understand the behavior, we take a broad ecological perspective and consider how valuation mechanisms evolve when the best action depends upon the environment being faced. We show that in a changing and uncertain environment, state-dependent valuation can be favored by natural selection: Individuals should allow their hunger to affect learning for future decisions. The valuation mechanism that typically evolves produces the kind of behavior seen in standard laboratory tests. By providing an insight into why learning should be affected by the state of an individual, we provide a basis for understanding psychological principles in terms of an animal's ecology.     

ITEM BIAS INDICES BASED ON TOTAL TEST SCORE AND JOB PERFORMANCE ESTIMATES OF ABILITY

Researchers (e.g., Ironson, 1982; Tenopyr, 1990) have suggested that item bias investigators equate subgroups on external criteria such as job performance rather than total test scores before considering subgroup passing rates on test items. In a study comparing these two approaches to studies of item bias, we found little evidence of bias using total test score as the estimate of overall examinee ability, but nearly all items were biased in comparisons of white and African-American subgroups on Numerical, Verbal, and Mechanical Reasoning tests and in male-female comparisons on a Mechanical Reasoning test when job performance was used to select "equally able" examinees. However, the use of job performance as the ability index is analogous to performance-based approaches to test bias (Hartigan & Wigdor, 1989; Thorndike, 1971) and directly equivalent to the Darlington (1971) and Cole (1973) test bias definition, the logical inconsistencies of which have been previously described (Hunter & Schmidt, 1976; Peterson & Novick, 1976). We conclude that performance matching as a basis of forming "equal ability" groups is inappropriate.     

Case method and casuistry: The problem of bias

Case methods of reasoning are persuasive, but we need to address problems of bias in order to use them to reach morally justifiable conclusions. A bias is an unwarranted inclination or a special perspective that disposes us to mistaken or one-sided judgments. The potential for bias arises at each stage of a case method of reasoning including in describing, framing, selecting and comparing of cases and paradigms. A problem of bias occurs because to identify the relevant features for such purposes, we must use general views about what is relevant; but some of our general views are biased, both in the sense of being unwarranted inclinations and in the sense that they are one of many viable perspectives. This reliance upon general views to determine relevancy creates additional difficulties for defenders who maintain that case methods of moral reasoning are not only useful, but more basic, reliable or prior to other forms of moral reasoning. If we cannot identify the case's relevant features and issues independently of our general views or biases, we need further explanation about why a case method or casuistry should be viewed as prior to or more basic or reliable than other forms of moral reasoning. Problems of bias also arise for other methods of reasoning. In medical science, case reviews are regarded as an unreliable way to form generalizations, and methods such as clinical trials are used to address bias.     

FURTHER TESTS OF THE SCHMIDT-HUNTER BAYESIAN VALIDITY GENERALIZATION PROCEDURE1

Using an improved version of the Bayesian validity generalization model presented in Schmidt and Hunter (1977), this study showed that most of the between-study variation in observed validity coefficients of various aptitude and ability tests for two types of clerical work and for the job of first-line supervisor is artifactual in nature. These results cast additional doubt on the traditional belief that employment test validities are situationally specific. In addition, the results showed that generalization of validities to new settings was justified even where the hypothesis of situational specificity could not be rejected under the standards used in this study. Implications for the use of aptitude and ability tests in selection and for theory development in industrial-organizational psychology are discussed.     

Wishful Thinking about the Future: Does Desire Impact Optimism?

The notion that desire for an outcome inflates optimism about that outcome has been dubbed the desirability bias or wishful thinking. In this paper, we discuss the importance of distinguishing wishful thinking from the more general concept of motivated reasoning, and we explain why documenting overoptimism or correlations between preferences and optimism is not sufficient to infer a desirability bias. Then, we discuss results from a review and meta-analysis of the experimental literature on wishful thinking. These findings, in conjunction with more recent work, not only highlight important moderators and mediators of the desirability bias but also point out limitations of the empirical research on the bias. These results also reveal an important difference between how likelihood judgments and discrete outcome predictions respond to desirability of outcomes. We conclude by presenting avenues for future research useful for understanding wishful thinking's manifestation in everyday environments and its integration with related phenomena.     

The Kuder-Richardson formula (21) as a split-half coefficient,and some remarks on its basic assumption

Case IV of the Kuder-Richardson series, their formula (21), is derived as a generalized split-half Spearman-Brown coefficient. The basic assumption employed is shown to be sufficient to justify the various assumptions used in derivations by other authors. Some of the implications of this assumption are discussed.     

A solution for case iii of the law of comparative judgment

A solution for Case III of the Law of Comparative Judgment, modeled after Thurstone's solution for Case IV but eliminating the restrictive assumption of relatively equal discriminal dispersions, is developed.     

Adjusting researchers’ approach to adjustment: On the use of covariates when testing interactions

A common design in social psychology involves the use of two independent variables, an experimental manipulation and a measured individual difference, and the interest is in the interaction between them. In such designs, there are often obvious covariate(s), correlated with the measured independent variable, which the researcher wishes to control. Typically this is done by including the covariate in the analytic model. We show that in most cases this is an inadequate model. In general, the interaction between the two independent variables will be estimated without bias only when the interaction between the covariate and the manipulated independent variable is included in the analysis. We present simulations showing the factors affecting the magnitude of the bias and provide a survey of recent social psychological literature illustrating the frequency of the problem. Finally, we discuss cases where both independent variables are manipulated and the covariate is a potential mediator.     

Party Support,Values, and Perceptions of Electoral Integrity

The legitimacy of the electoral process is crucial for the consolidation of democracy. We here focus on individual perceptions of electoral integrity (IPEI) and seek to understand what factors can explain different degrees of IPEI. In particular, we use the sixth wave of the World Values Survey (2010–14) to examine how antiauthoritarian values affect individuals' directional bias, driven by political party support, in evaluating electoral integrity. The results show that IPEI do depend on an interaction of political party support and the strength of antiauthoritarian values. However, the addition of the latter does not lead to a convergence of integrity evaluations among winners and losers, as may be expected under the assumption that antiauthoritarian values drive voters to more carefully monitor and evaluate the electoral process. Instead, it leads to greater polarization between electoral winners and losers. We explain the result with reference to the motivated reasoning literature on biased information processing: While antiauthoritarian convictions lead people to obtain more information on the electoral process, their political leanings bias their reading of this information, which in effect leads to stronger polarization in perceptions.     

ECONOMIC CONSIDERATIONS IN ESTIMATING THE UTILITY OF HUMAN RESOURCE PRODUCTIVITY IMPROVEMENT PROGRAMS

Three economic concepts (variable costs, taxes, and discounting) are applied and incorporated into the previous utility formulas proposed by Brogden (1946, 1949), Cronbach and Gleser (1965), and Schmidt, et al. (1979, 1982). The resulting utility model indicates that the previous formulas are deficient and can produce upwardly biased utility estimates. Empirical examples based on published research (e.g., Schmidt, et al., 1982) are presented indicating the substantial magnitude of the bias given realistic levels of variable costs, taxes, and discount rates. The present utility model is used to adjust for such bias and is shown to provide a more complete and precise utility definition. Implications for future research are discussed.