Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure

Using computer simulation, the authors assessed the accuracy of J. E. Hunter, F. L. Schmidt, and H. Le's (2006) procedure for correcting for indirect range restriction, the most common type of range restriction, in comparison with the conventional practice of applying the Thorndike Case II correction for direct range restriction. Hunter et al.'s procedure produced more accurate estimates of both the mean and standard deviation in meta-analysis than the conventional procedure. Even when its key assumption that the effect of selection on a 3rd variable is fully mediated by the independent variable was violated, Hunter et al.'s procedure was still relatively more accurate than the conventional procedure. When applied to data from a previously published meta-analysis, the new procedure yielded results that led to different substantive conclusions.     

Comparison of two random-effects methods of meta-analysis

Two studies compared the Schmidt-Hunter method of meta-analysis (J. E. Hunter & F. L. Schmidt, 1990) with the method described by L. V. Hedges and J. L. Vevea (1998). Study 1 evaluated estimates of rho, sigmarho, and resulting credibility intervals for both models through Monte Carlo methods. Results showed slight differences between the 2 methods. In Study 2, a reanalysis of published meta-analyses using both methods with several artifact distributions showed that although both choice of technique and type of correction could matter, the technique of meta-analysis used is less influential on the study outcome than is the choice of artifact correction.     

A Meta-Analysis of the Relative Effects of Tabular and Graphic Display Formats on Decision-Making Performance

The results of 21 between-subjects experiments assessing the relative effects of tabular and graphic formats of data presentation on decision-making accu- racy were quantitatively reviewed using the meta-analysis procedure developed by Hunter, Schmidt, and Jackson (1982). Sampling error and differential range restriction accounted for variability in the average differences between tabular and graphic presentations, and there was no difference in the effects of the two presentation formats. These results counter recent assertions that task complexity moderates the effect of presentation format. The findings should also discourage further investigation of the effects of tabular and graphic formats using the conventional methodology.     

FURTHER TESTS OF THE SCHMIDT-HUNTER BAYESIAN VALIDITY GENERALIZATION PROCEDURE1

Using an improved version of the Bayesian validity generalization model presented in Schmidt and Hunter (1977), this study showed that most of the between-study variation in observed validity coefficients of various aptitude and ability tests for two types of clerical work and for the job of first-line supervisor is artifactual in nature. These results cast additional doubt on the traditional belief that employment test validities are situationally specific. In addition, the results showed that generalization of validities to new settings was justified even where the hypothesis of situational specificity could not be rejected under the standards used in this study. Implications for the use of aptitude and ability tests in selection and for theory development in industrial-organizational psychology are discussed.     

ESTIMATION OF SAMPLING VARIANCE OF CORRELATIONS IN META-ANALYSIS

Monte Carlo simulations were conducted to compare the performance of the traditional (Fisher, 1954) and mean (Hunter & Schmidt, 1990) estimators of the sampling variance of correlations in meta-analysis. The mean estimator differs from the traditional estimator in that it uses the mean observed correlation, averaged across studies, in the sampling variance formula. The simulations investigated the homogeneous (i.e., no true correlation variance across studies) and heterogeneous case (i.e., true correlation variance across studies). Results reveal that, compared to the traditional estimator, the mean estimator provides less negatively biased estimates of sampling variance in the homogeneous and heterogeneous cases and more positively biased estimates in the heterogenous case. Thus, results support the use of the mean estimator unless strong, theory-based hypotheses regarding moderating effects exist.     

Longitudinal factor structure of posttraumatic stress symptoms related to intimate partner violence

Confirmatory factor analysis (CFA) studies have suggested that a model of posttraumatic stress disorder (PTSD) that is characterized by 4 factors is preferable to competing models. However, the composition of these 4 factors has varied across studies, with 1 model splitting avoidance and numbing symptoms (e.g., D. W. King, G. A. Leskin, L. A. King, & F. W. Weathers, 1998) and the other including a dysphoria factor that combines numbing and nonspecific hyperarousal symptoms (L. J. Simms, D. Watson, & B. N. Doebbeling, 2002). Using the PTSD Checklist (F. W. Weathers, B. T. Litz, D. S. Herman, J. A. Huska, & T. M. Keane, 1993) and CFA, the authors compared these models with competing models. A model of PTSD with 4 intercorrelated factors of Intrusions, Avoidance, Dysphoria, and Hyperarousal was found superior among 396 medical patients who screened positive for intimate partner violence (IPV) and 405 women seeking services for IPV. Structural invariance testing indicated that this 4-factor model remains stable across service setting and time.     

Hebbian learning of cognitive control: dealing with specific and nonspecific adaptation

The conflict monitoring model of M. M. Botvinick, T. S. Braver, D. M. Barch, C. S. Carter, and J. D. Cohen (2001) triggered several research programs investigating various aspects of cognitive control. One problematic aspect of the Botvinick et al. model is that there is no clear account of how the cognitive system knows where to intervene when conflict is detected. As a result, recent findings of task-specific and context-specific (e.g., item-specific) adaptation are difficult to interpret. The difficulty with item-specific adaptation was recently pointed out by C. Blais, S. Robidoux, E. F. Risko, and D. Besner (2007), who proposed an alternative model that could account for this. However, the same problem of where the cognitive system should intervene resurfaces in a different shape in this model, and it has difficulty in accounting for the Gratton effect, a hallmark item-nonspecific effect. The authors of the current article show how these problems can be solved when cognitive control is implemented as a conflict-modulated Hebbian learning rule.     

Publication bias in psychological science: comment on Ferguson and Brannick (2012)

It is well documented that studies reporting statistically significant results are more likely to be published than are studies reporting nonsignificant results--a phenomenon called publication bias. Publication bias in meta-analytic reviews should be identified and reduced when possible. Ferguson and Brannick (2012) argued that the inclusion of unpublished articles is ineffective and possibly counterproductive as a means of reducing publication bias in meta-analyses. We show how idiosyncratic choices on the part of Ferguson and Brannick led to an erroneous conclusion. We demonstrate that their key finding--that publication bias was more likely when unpublished studies were included--may be an artifact of the way they assessed publication bias. We also point out how the lack of transparency about key choices and the absence of information about critical features of Ferguson and Brannick's sample and procedures might have obscured readers' ability to assess the validity of their claims. Furthermore, we demonstrate that many of the claims they made are without empirical support, even though they could have tested these claims empirically, and that these claims may be misleading. With their claim that addressing publication bias introduces subjectivity and bias into meta-analysis, they ignored a large body of evidence showing that including unpublished studies that meet the inclusion criteria of a meta-analysis decreases (rather than increases) publication bias. Rather than exclude unpublished studies, we recommend that meta-analysts code study characteristics related to methodological quality (e.g., experimental vs. nonexperimental design) and test whether these factors influence the meta-analytic results.     

Does Psi exist? Lack of replication of an anomalous process of information transfer.

D. J. Bem and C. Honorton (1994) recently presented in this journal a set of ganzfeld extrasensory perception (ESP) experiments conducted by C. Honorton that appeared to support the existence of a communication anomaly. In this article, the authors present a meta-analysis of 30 ganzfeld ESP studies from 7 independent laboratories adhering to the same stringent methodological guidelines that C. Honorton followed. The studies failed to confirm his main effect of participants scoring above chance on the ESP task, Stouffer z = 0.70, p = .24, one-tailed; M effect size (z/N1/2) = 0.013, SD = 0.23. The new studies included replication attempts of 3 out of 5 internal effects reported as statistically significant by D. J. Bem and C. Honorton. Only 1 was confirmed, and the authors found that D. J. Bem and C. Honorton were mistaken in describing the original effect as being statistically significant. The authors conclude that the ganzfeld technique does not at present offer a replicable method for producing ESP in the laboratory.     

Effects of age and ability on self-reported memory functioning and knowledge of memory aging

The authors examined the effects of age and ability (as measured by education and verbal ability) on self-reported memory functioning in adulthood. In Study 1, the age and ability groups responded similarly to the Cognitive Failures Questionnaire (D. E. Broadbent, P. F. Cooper, P. Fitzgerald, & K. R. Parkes, 1982), but differences emerged when the authors examined specific items tapping prospective and retrospective memory. In Study 2, the authors found age and ability differences in self-reported memory functioning, as measured by the Memory Functioning Questionnaire (M. J. Gilewski, E. M. Zelinski, & K. W. Schaie, 1990) and in knowledge of memory aging, as measured by the Knowledge of Memory Aging Questionnaire (K. E. Cherry, R. L. West, C. M. Reese, M. P. Santa Maria, & M. Yassuda, 2000). In both studies, differences in self-reported functioning were not related to objective memory performance. In the second study, knowledge was positively related to performance. Implications for conceptions of metamemory and memory aging are considered.     

Is the meta-analysis of correlation coefficients accurate when population correlations vary?

One conceptualization of meta-analysis is that studies within the meta-analysis are sampled from populations with mean effect sizes that vary (random-effects models). The consequences of not applying such models and the comparison of different methods have been hotly debated. A Monte Carlo study compared the efficacy of Hedges and Vevea's random-effects methods of meta-analysis with Hunter and Schmidt's, over a wide range of conditions, as the variability in population correlations increases. (a) The Hunter-Schmidt method produced estimates of the average correlation with the least error, although estimates from both methods were very accurate; (b) confidence intervals from Hunter and Schmidt's method were always slightly too narrow but became more accurate than those from Hedges and Vevea's method as the number of studies included in the meta-analysis, the size of the true correlation, and the variability of correlations increased; and (c) the study weights did not explain the differences between the methods.     

Construct validity of the posttraumatic stress disorder checklist in cancer survivors: analyses based on two samples

The measurement of posttraumatic stress disorder (PTSD) is critically important for the identification and treatment of this disorder. The PTSD Checklist (PCL; F. W. Weathers and J. Ford, 1996) is a self-report measure that is increasingly used. In this study, the authors investigated the factorial validity of the PCL with data from 236 cancer survivors who received a bone marrow or stem cell transplantation. The authors examined the fit of these data with the clinical model of 3 symptom clusters for PTSD, as proposed in the Diagnostic and Statistical Manual of Mental Disorders, and alternative models tested in prior research. By using confirmatory factor analysis the authors found that a 4-first-order-factor model of PTSD provided the best fit. The relations of PTSD symptoms with sociodemographic and medical variables were also explored.     

THEORY TESTING: COMBINING PSYCHOMETRIC META-ANALYSIS AND STRUCTURAL EQUATIONS MODELING

This paper presents an overview of a useful approach for theory testing in the social sciences that combines the principles of psychometric meta-analysis and structural equations modeling. In this approach to theory testing, the estimated true score correlations between the constructs of interest are established through the application of meta-analysis (Hunter & Schmidt, 1990), and structural equations modeling is then applied to the matrix of estimated true score correlations. The potential advantages and limitations of this approach are presented. The approach enables researchers to test complex theories involving several constructs that cannot all be measured in a single study. Decision points are identified, the options available to a researcher are enumerated, and the potential problems as well as the prospects of each are discussed.     

Theories of word naming interact with spelling-sound consistency

In a previous study (E. Strain, K. Patterson, & M. S. Seidenberg, 1995), the authors concluded that word naming is characterized by an interaction between spelling-sound typicality and word imageability, thus implicating a role for word meaning in the naming process. J. Monaghan and A. W. Ellis (2002) reject E. Strain et al.'s conclusion, arguing that it is age of acquisition (AoA) and not imageability that interacts with spelling-sound typicality. In this article, the authors question their alternative interpretation (a) by raising a number of conceptual and methodological issues germane to this debate and (b) by presenting new data that confirm a significant interaction between spelling-sound typicality and imageability in word-naming latencies, an interaction that is reliable when word AoA is controlled in a regression analysis.     

Male physical attractiveness in Britain and Greece: a cross-cultural study

The waist-to-chest ratio (WCR), body mass index (BMI), and waist-to-hip ratio (WHR) are the major cues to women's ratings of men's bodily attractiveness (J. T. Fan, W. Dai, F. Liu, & J. Wu, 2005; D. M. Maisey, E. L. E. Vale, P. L. Cornelissen, & M. J. Tovée, 1999; V. Swami & M. J. Tovée, 2005b). The authors examined the relative importance of each of these cues cross-culturally in Greece and Britain. Participants were 36 British and 40 Greek women who rated a set of images of real men with known WCR, BMI, and WHR. The results showed that, regardless of the cultural setting, WCR was the primary determinant of men's physical attractiveness to women, with BMI playing a minor role. However, there were also cross-cultural differences: The Greek women showed a stronger preference for a lower WCR and smaller overall body weight than did the British women. The authors considered possible explanations of these findings.     

Emotional Security and Cognitive Appraisals Mediate the Relationship Between Parents' Marital Conflict and Adjustment in Older Adolescents

In this study, the authors evaluated a model of the effects of parental conflict witnessed in childhood on psychosocial adjustment in 175 college students. Using this model, which integrated the cognitive-contextual framework (J. H. Grych & F. D. Fincham, 1990) and the emotional-security hypothesis (P. T. Davies & E. M. Cummings, 1994), the authors proposed that both cognitive appraisal of parents' conflict and emotional security mediate the conflict-adjustment link in late adolescence. Furthermore, the authors proposed that cognitive appraisals are associated with emotional security. The results suggested that appraisals and security were both important mediators in the model but that they were not related to each other. Instead, there appeared to be parallel, yet separate, cognitive and emotional channels through which exposure to parental conflict in childhood can have detrimental effects on later adjustment.     

Emotional security and cognitive appraisals mediate the relationship between parents' marital conflict and adjustment in older adolescents

In this study, the authors evaluated a model of the effects of parental conflict witnessed in childhood on psychosocial adjustment in 175 college students. Using this model, which integrated the cognitive-contextual framework (J. H. Grych & F. D. Fincham, 1990) and the emotional-security hypothesis (P. T. Davies & E. M. Cummings, 1994), the authors proposed that both cognitive appraisal of parents' conflict and emotional security mediate the conflict-adjustment link in late adolescence. Furthermore, the authors proposed that cognitive appraisals are associated with emotional security. The results suggested that appraisals and security were both important mediators in the model but that they were not related to each other. Instead, there appeared to be parallel, yet separate, cognitive and emotional channels through which exposure to parental conflict in childhood can have detrimental effects on later adjustment.     

Validity generalization: Some critical remarks on the Schmidt-Hunter procedure*

This article presents some critical comments on the validity generalization procedure which has been presented by Schmidt, Hunter and others. They have put forward a method for testing the hypothesis that the variance in validity coefficients across situations for job-test combinations is due to what they consider to be statistical artifacts. The Schmidt-Hunter approach is criticized on the following points: the compilation of validity data, the use of criterion measures, and the test of the hypothesis of no situational specificity. Further, the relation between the concepts ‘situational specificity’ and ‘validity generalization’ is considered. In addition, it is noted that Schmidt, Hunter and others have defined the concept ‘situation’ in a different way than classical writers. It is concluded that the Schmidt-Hunter approach to validity generalization shows fundamental shortcomings. As a consequence their far-reaching conclusions for the practice of personnel selection should be considered premature.     

Allowing for correlations between correlations in random-effects meta-analysis of correlation matrices

Practical meta-analysis of correlation matrices generally ignores covariances (and hence correlations) between correlation estimates. The authors consider various methods for allowing for covariances, including generalized least squares, maximum marginal likelihood, and Bayesian approaches, illustrated using a 6-dimensional response in a series of psychological studies concerning prediction of exercise behavior change. Quantities of interest include the overall population mean correlation matrix, the contrast between the mean correlations, the predicted correlation matrix in a new study, and the conflict between the existing studies and a new correlation matrix. The authors conclude that accounting for correlations between correlations is unnecessary when interested in individual correlations but potentially important if concerned with a composite measure involving 2 or more correlations. A simulation study indicates the asymptotic normal assumption appears reasonable. Because of potential instability in the generalized least squares methods, they recommend a model-based approach, either the maximum marginal likelihood approach or a full Bayesian analysis.