Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure

Using computer simulation, the authors assessed the accuracy of J. E. Hunter, F. L. Schmidt, and H. Le's (2006) procedure for correcting for indirect range restriction, the most common type of range restriction, in comparison with the conventional practice of applying the Thorndike Case II correction for direct range restriction. Hunter et al.'s procedure produced more accurate estimates of both the mean and standard deviation in meta-analysis than the conventional procedure. Even when its key assumption that the effect of selection on a 3rd variable is fully mediated by the independent variable was violated, Hunter et al.'s procedure was still relatively more accurate than the conventional procedure. When applied to data from a previously published meta-analysis, the new procedure yielded results that led to different substantive conclusions.     

INCREASING THE ACCURACY OF CORRECTIONS FOR RANGE RESTRICTION: IMPLICATIONS FOR SELECTION PROCEDURE VALIDITIES AND OTHER RESEARCH RESULTS

The common practice in meta-analyses and in individual studies of correcting for direct range restriction even though range restriction is actually indirect has long been known to lead to undercorrection, but this error has been assumed to be small. Using validity generalization data sets for 4 jobs, this study calibrated this error by comparing meta-analysis results based on corrections for direct range restriction with the more accurate results from a recently developed method of correcting for indirect range restriction. It was found that, on average, correction for direct range restriction resulted in substantial underestimation of operational validities for both job performance measures (21%) and training performance measures (28%). In addition, 90% credibility values were on average underestimated by 38%–40%. In addition to the implications for personnel selection, these findings suggest that similar underestimation of important relationships has occurred in other areas of research, with potential implications for theory development.     

Using individual differences to predict job performance: correcting for direct and indirect restriction of range

The present study investigates the relationship between individual differences, indicated by personality (FFM) and general mental ability (GMA), and job performance applying two different methods of correction for range restriction. The results, derived by analyzing meta-analytic correlations, show that the more accurate method of correcting for indirect range restriction increased the operational validity of individual differences in predicting job performance and that this increase primarily was due to general mental ability being a stronger predictor than any of the personality traits. The estimates for single traits can be applied in practice to maximize prediction of job performance. Further, differences in the relative importance of general mental ability in relation to overall personality assessment methods was substantive and the estimates provided enables practitioners to perform a correct utility analysis of their overall selection procedure.     

Publication bias in research synthesis: sensitivity analysis using a priori weight functions

Publication bias, sometimes known as the "file-drawer problem" or "funnel-plot asymmetry," is common in empirical research. The authors review the implications of publication bias for quantitative research synthesis (meta-analysis) and describe existing techniques for detecting and correcting it. A new approach is proposed that is suitable for application to meta-analytic data sets that are too small for the application of existing methods. The model estimates parameters relevant to fixed-effects, mixed-effects or random-effects meta-analysis contingent on a hypothetical pattern of bias that is fixed independently of the data. The authors illustrate this approach for sensitivity analysis using 3 data sets adapted from a commonly cited reference work on research synthesis (H. M. Cooper & L. V. Hedges, 1994).     

Methodological issues in meta-analyzing standard deviations: Comment on Bond and DePaulo (2008)

In this comment on C. F. Bond and B. M. DePaulo, the authors raise methodological concerns about the approach used to analyze the data. The authors suggest further refinement of the procedures used, and they compare the approach taken by Bond and DePaulo with standard methods for meta-analysis.     

A cautionary note on the effects of range restriction on predictor intercorrelations

The purpose of this research report is to highlight a unique set of issues that arise when considering the effects of range restriction in the context of estimating predictor intercorrelations. Three approaches are used to illustrate the issue: simulation, a concrete applied example, and a reanalysis of a meta-analysis of ability-interview correlations. The general conclusion is that a predictor intercorrelation can differ dramatically from the population value when both predictors are used in a composite that is used operationally for selection. The compensatory nature of a composite means that low scorers on one predictor can only obtain high scores on the composite if they obtain very high scores on the other predictor; this phenomenon distorts the correlation between the predictors.     

Moving Forward Indirectly: Reanalyzing the validity of employment interviews with indirect range restriction methodology

This study provides updated estimates of the criterion‐related validity of employment interviews, incorporating indirect range restriction methodology. Using a final dataset of 92 coefficients (N = 7,389), we found corrected estimates by structural level of .20 (Level 1), .46 (Level 2), .71 (Level 3), and .70 (Level 4). The latter values are noticeably higher than in previous interview meta‐analyses where the assumption was made that all restriction was direct. These results highlight the importance of considering indirect range restriction in selection. However, we found a number of studies involving both indirect and direct restriction, which calls into question the viability of assuming all restriction is now indirect. We found preliminary empirical support for correction of one of these multiple restriction patterns, indirect then direct.     

Working memory and intelligence--their correlation and their relation: comment on Ackerman, Beier, and Boyle (2005)

On the basis of a meta-analysis of pairwise correlations between working memory tasks and cognitive ability measures, P. L. Ackerman, M. E. Beier, and M. O. Boyle (2005; see record 2004-22408-002) claimed that working memory capacity (WMC) shares less than 25% of its variance with general intelligence (g) and with reasoning ability. In this comment, the authors argue that this is an underestimation because of several methodological shortcomings and biases. A reanalysis of the data reported in Ackerman et al. using the correct statistical procedures demonstrates that g and WMC are very highly correlated. On a conceptual level, the authors point out that WMC should be regarded as an explanatory construct for intellectual abilities. Theories of working memory do not claim that WMC is isomorphic with intelligence factors but that it is a very strong predictor of reasoning ability and also predicts general fluid intelligence and g.     

The critical role of the research question, inclusion criteria, and transparency in meta-analyses of integrity test research: a reply to Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012)

We clear up a number of misconceptions from the critiques of our meta-analysis (Van Iddekinge, Roth, Raymark, & Odle-Dusseau, 2012). We reiterate that our research question focused on the criterion-related validity of integrity tests for predicting individual work behavior and that our inclusion criteria flowed from this question. We also reviewed the primary studies we could access from Ones, Viswesvaran, and Schmidt's (1993) meta-analysis of integrity tests and found that only about 30% of the studies met our inclusion criteria. Further, analyses of some of the types of studies we had to exclude revealed potentially inflated validity estimates (e.g., corrected validities as high as .80 for polygraph studies). We also discuss our experience trying to obtain primary studies and other information from authors of Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012). In addition, we address concerns raised about certain decisions we made and values we used, and we demonstrate how such concerns would have little or no effect on our results or conclusions. Finally, we discuss some other misconceptions about our meta-analysis, as well as some divergent views about the integrity test literature in general. Overall, we stand by our research question, methods, and results, which suggest that the validity of integrity tests for criteria such as job performance and counterproductive work behavior is weaker than the authors of the critiques appear to believe.     

INCREASED ACCURACY FOR RANGE RESTRICTION CORRECTIONS: IMPLICATIONS FOR THE ROLE OF PERSONALITY AND GENERAL MENTAL ABILITY IN JOB AND TRAINING PERFORMANCE

A recently developed procedure produces substantial improvements in the accuracy of corrections for range restriction and reveals that predictive validities of employment selection methods are higher than previously thought. This study applied this procedure to meta‐analytic validities of 2 personality measures (Conscientiousness and Emotional Stability) and general cognitive ability measures. Results show that the increases in validity estimates are greater for cognitive ability than for personality, reducing the relative validity of personality. In addition, the incremental validity of these personality measures over that provided by cognitive ability is smaller than previously estimated. Path analyses based on the more accurate data reveal relatively smaller path coefficients from these personality traits to job and training performance. These findings have implications for theories of job performance and for the practical value in selection of personality measures relative to cognitive ability measures.     

Discrimination and Eating Disorder Psychopathology: A Meta-Analysis

Eating disorders occur in diverse populations, and discrimination may be a specific factor that is related to higher eating disorder psychopathology among marginalized individuals. To evaluate the current evidence on this topic, a meta-analysis was used to quantitatively synthesize the literature on discrimination and eating disorder psychopathology across a heterogeneous range of studies. Searches were conducted in peer-reviewed journals and accessible unpublished dissertations of all years through January 2020. Studies were coded by two authors using a tailored coding form, and zero-order bivariate correlations were used as effect size measures. There were 55 cross-sectional studies extracted for inclusion in the meta-analysis. Results showed a small-to-medium association between discrimination and eating disorder psychopathology that was consistent across domains. Effect sizes were typically higher for weight discrimination. For binge eating and general eating disorder pathology, effects were smaller in studies that had larger proportions of women, and for binge eating only, effects were higher in college samples. These findings could suggest that discrimination represents a contributory factor related to eating disorder psychopathology across types of discrimination and eating disorder psychopathology. Implications are discussed for future research on discrimination and psychopathology including possible mechanisms.     

Working memory and intelligence: the same or different constructs?

Several investigators have claimed over the past decade that working memory (WM) and general intelligence (g) are identical, or nearly identical, constructs, from an individual-differences perspective. Although memory measures are commonly included in intelligence tests, and memory abilities are included in theories of intelligence, the identity between WM and intelligence has not been evaluated comprehensively. The authors conducted a meta-analysis of 86 samples that relate WM to intelligence. The average correlation between true-score estimates of WM and g is substantially less than unity (p=.479). The authors also focus on the distinction between short-term memory and WM with respect to intelligence with a supplemental meta-analysis. The authors discuss how consideration of psychometric and theoretical perspectives better informs the discussion of WM-intelligence relations.     

Intelligence and leadership: a quantitative review and test of theoretical propositions

Meta-analysis was used to aggregate results from studies examining the relationship between intelligence and leadership. One hundred fifty-one independent samples in 96 sources met the criteria for inclusion in the meta-analysis. Results indicated that the corrected correlation between intelligence and leadership is.21 (uncorrected for range restriction) and.27 (corrected for range restriction). Perceptual measures of intelligence showed stronger correlations with leadership than did paper-and-pencil measures of intelligence. Intelligence correlated equally well with objective and perceptual measures of leadership. Additionally, the leader's stress level and the leader's directiveness moderated the intelligence-leadership relationship. Overall, results suggest that the relationship between intelligence and leadership is considerably lower than previously thought. The results also provide meta-analytic support for both implicit leadership theory and cognitive resource theory.     

The validity and appropriateness of methods, analyses, and conclusions in Rind et al. (1998): A rebuttal of victimological critique from Ondersma et al. (2001) and Dallam et al. (2001).

The authors respond to 2 victimological critiques of their 1998 meta-analysis on child sexual abuse (CSA). S. J. Dallam et al. (2001) claimed that B. Rind, P. Tromovitch, and R. Bauserman (1998) committed numerous methodological and statistical errors, and often miscoded and misinterpreted data. The authors show all these claims to be invalid. To the contrary, they demonstrate frequent bias in Dallam et al.'s criticisms. S. J. Ondersma et al. (2001) claimed that Rind et al.'s study is part of a backlash against psychotherapists, that its suggestions regarding CSA definitions were extrascientific, and that the moral standard is needed to understand CSA scientifically. The authors show their suggestions to have been scientific and argue that it is Ondersma et al.'s issue-framing and moral standard that are extrascientific. This reply supports the original methods, analyses, recommendations, and conclusions of Rind et al.     

An Evidence-Based Approach to Managing Suicidal Patients in the Patient-Centered Medical Home

Primary care is a critical setting for suicide prevention because it is often the first and only source of mental health care for the U.S. general population. It is also important because suicidal patients report a greater number of somatic complaints and make more frequent medical visits compared to nonsuicidal patients. Models for managing suicide within primary care have recently arisen, yet no models have been proposed for use within the patient-centered medical home (PCMH), a primary care model that integrates behavioral health into its practice. The authors suggest a chronic disease model for the management of suicide risk in the PCMH along with collaborative strategies that may include suicide screening and targeted assessment, warm hand-offs, cognitive-behavioral interventions, routine collaborative medication management, and means restriction counseling. The current paper advises how those within the PCMH can adapt and implement evidence-based practices to manage suicide. Finally, the authors discuss a case example illustrating these evidence-based and collaborative methods.     

Correction for Range Restriction in Meta‐Analysis Revisited: Improvements and Implications for Organizational Research

In this study, we present a more accurate method for correcting for range restriction (Case V) that expands upon Bryant and Gokhale's (1972) method. We further present detailed steps to incorporate the Case V method into Schmidt and Hunter's (2015) psychometric meta‐analysis methods (both individual correction and artifact distribution approaches). We then evaluate the accuracy of the Case V method vis‐à‐vis existing methods. Monte‐Carlo simulation results indicate that the Case V method provides very accurate estimates for the mean true score correlation and reasonably accurate estimates for the true standard deviation. More important, Case V almost always provides more accurate results than alternative methods (particularly, Case IV). To illustrate how the Case V method works with real data, we conduct a reanalysis of Judge, Heller, and Mount's (2002) meta‐analysis examining the relationships between the Big 5 personality traits and job satisfaction. Results indicate that the true score correlations between the Big 5 traits and job satisfaction have been underestimated, whereas their true standard deviations have been overestimated. Implications for range restriction corrections in organizational research are discussed.     

Negative affect and sexual risk behavior: comment on Crepaz and Marks (2001).

In this commentary, the authors highlight the findings of the meta-analysis by N. Crepaz and G. Marks (2001). The role of affect in sexual risk behavior, although intuitively obvious, is not well understood and has been largely ignored by HIV prevention researchers in favor of social-cognitive models of behavior. Crepaz and Marks synthesized the results from studies that have examined the relation of negative affect (e.g., depression, anxiety, anger) to sexual risk behavior and concluded that in the literature to date, these variables appear unrelated. The authors suggest that the Crepaz and Marks findings are not surprising given the methods used in the reviewed studies and suggest methodological approaches that will allow more sensitive analyses of the association between affect and sexual risk behavior.     

A Meta-Analysis of the Relative Effects of Tabular and Graphic Display Formats on Decision-Making Performance

The results of 21 between-subjects experiments assessing the relative effects of tabular and graphic formats of data presentation on decision-making accu- racy were quantitatively reviewed using the meta-analysis procedure developed by Hunter, Schmidt, and Jackson (1982). Sampling error and differential range restriction accounted for variability in the average differences between tabular and graphic presentations, and there was no difference in the effects of the two presentation formats. These results counter recent assertions that task complexity moderates the effect of presentation format. The findings should also discourage further investigation of the effects of tabular and graphic formats using the conventional methodology.     

A NOTE ON DIRECT RANGE RESTRICTION CORRECTIONS IN UTILITY ANALYSIS

Brogden's equations for employment test utility analysis have been used in conjunction with direct range restriction correction formulas, because the data used typically come from selected applicant (incumbent) samples. We show that the use of direct range restriction corrections in utility analysis is not needed. Identical results can be obtained using a simpler equation that employs a regression coefficient based on the employee (i.e., restricted or job incumbent) population. This utility estimation procedure, which does not employ range restriction corrections, is derived and discussed. We recommend that this new equation be used in the practice of selection utility analysis because it not only provides identical utility estimates to those obtained using current equations that employ direct range restriction correction formulas, but also is computationally simpler.     

Correcting correlation coefficients for group heterogeneity when one variable is a dichotomy

A method is illustrated for estimating correlation coefficients and mean criterion scores in a full-range population from bivariate distributions available in a selected sample when the criterion is a dichotomy. The proposed method requires only the assumptions needed for use of formulas for correcting correlation coefficients for restriction of range when both variables are continuous and is suitable for use when restriction is due to either direct or indirect selection.The research reported in this paper was sponsored by the 6570th Personnel Research Laboratory, AMD, under AFSC Project 7717.