Meta-analytic Reviews in the Organizational Sciences: Two Meta-analytic Schools on the Way to MARS (the Meta-analytic Reporting Standards)

Purpose

The purpose of this study was to review the Meta-Analysis Reporting Standards (MARS) of the American Psychological Association (APA) and highlight opportunities for improvement of meta-analytic reviews in the organizational sciences.

Design/Methodology/Approach

The paper reviews MARS, describes “best” meta-analytic practices across two schools of meta-analysis, and shows how implementing such practices helps achieve the aims set forth in MARS. Examples of best practices are provided to aid readers in finding models for their own research.

Implications/Value

Meta-analytic reviews are a primary avenue for the accumulation of knowledge in the organizational sciences as well as many other areas of science. Unfortunately, many meta-analytic reviews in the organizational sciences do not fully follow professional guidelines and standards as closely as they should. Such deviations from best practice undermine the transparency and replicability of the reviews and thus their usefulness for the generation of cumulative knowledge and evidence-based practice. This study shows how implementing “best” meta-analytic practices helps to achieve the aims set forth in MARS. Although the paper is written primarily for organizational scientists, the paper’s recommendations are not limited to any particular scientific domain.     

Meta-analysis for single-case consultation outcomes: Applications to research and practice

The application of meta-analysis holds much appeal for single-case consultation outcome research. We review a meta-analytic method for using within-study treatment effect sizes in reporting consultation outcomes. The strengths and limitations of traditional group design meta-analysis are examined. Various methods for analyzing single-case outcomes are discussed briefly, followed by an examination of the use of meta-analysis in single-case reviews across independent studies. Within-study meta-analytic results are presented that were derived from treatments implemented in consultations in natural settings. To conclude the article, an illustration is offered of a single-case data analysis display that incorporates meta-analytic results along with other indices of treatment outcome. Recommendations are provided for using meta-analytic methods to evaluate outcomes of single-case consultation treatment.     

On the Multilevel Nature of Meta-Analysis: A Tutorial,Comparison of Software Programs,and Discussion of Analytic Choices

The term “multilevel meta-analysis” is encountered not only in applied research studies, but in multilevel resources comparing traditional meta-analysis to multilevel meta-analysis. In this tutorial, we argue that the term “multilevel meta-analysis” is redundant since all meta-analysis can be formulated as a special kind of multilevel model. To clarify the multilevel nature of meta-analysis the four standard meta-analytic models are presented using multilevel equations and fit to an example data set using four software programs: two specific to meta-analysis (metafor in R and SPSS macros) and two specific to multilevel modeling (PROC MIXED in SAS and HLM). The same parameter estimates are obtained across programs underscoring that all meta-analyses are multilevel in nature. Despite the equivalent results, not all software programs are alike and differences are noted in the output provided and estimators available. This tutorial also recasts distinctions made in the literature between traditional and multilevel meta-analysis as differences between meta-analytic choices, not between meta-analytic models, and provides guidance to inform choices in estimators, significance tests, moderator analyses, and modeling sequence. The extent to which the software programs allow flexibility with respect to these decisions is noted, with metafor emerging as the most favorable program reviewed.     

Publication bias in psychological science: comment on Ferguson and Brannick (2012)

It is well documented that studies reporting statistically significant results are more likely to be published than are studies reporting nonsignificant results--a phenomenon called publication bias. Publication bias in meta-analytic reviews should be identified and reduced when possible. Ferguson and Brannick (2012) argued that the inclusion of unpublished articles is ineffective and possibly counterproductive as a means of reducing publication bias in meta-analyses. We show how idiosyncratic choices on the part of Ferguson and Brannick led to an erroneous conclusion. We demonstrate that their key finding--that publication bias was more likely when unpublished studies were included--may be an artifact of the way they assessed publication bias. We also point out how the lack of transparency about key choices and the absence of information about critical features of Ferguson and Brannick's sample and procedures might have obscured readers' ability to assess the validity of their claims. Furthermore, we demonstrate that many of the claims they made are without empirical support, even though they could have tested these claims empirically, and that these claims may be misleading. With their claim that addressing publication bias introduces subjectivity and bias into meta-analysis, they ignored a large body of evidence showing that including unpublished studies that meet the inclusion criteria of a meta-analysis decreases (rather than increases) publication bias. Rather than exclude unpublished studies, we recommend that meta-analysts code study characteristics related to methodological quality (e.g., experimental vs. nonexperimental design) and test whether these factors influence the meta-analytic results.     

Choosing the best method for local validity estimation: relative accuracy of meta-analysis versus a local study versus Bayes-analysis

This study assessed the relative accuracy of 3 techniques--local validity studies, meta-analysis, and Bayesian analysis--for estimating test validity, incremental validity, and adverse impact in the local selection context. Bayes-analysis involves combining a local study with nonlocal (meta-analytic) validity data. Using tests of cognitive ability and personality (conscientiousness) as predictors, an empirically driven selection scenario illustrates conditions in which each of the 3 estimation techniques performs best. General recommendations are offered for how to estimate local parameters, based on true population variability and the number of studies in the meta-analytic prior. Benefits of empirical Bayesian analysis for personnel selection are demonstrated, and equations are derived to help guide the choice of a local validity technique (i.e., meta-analysis vs. local study vs. Bayes-analysis).     

Successfully Combining Meta-analysis and Structural Equation Modeling: Recommendations and Strategies

During the past two decades, organizational researchers have combined the techniques of meta-analysis (MA) and structural equation modeling (SEM) with the intention of building on the strengths of these approaches to address unique research questions. Though these integrative analyses can involve the use of SEM to conduct MA, the focus of the current article is on those situations in which meta-analytic correlations are used as input for testing structural models not previously evaluated in any single, primary study. The purpose of this paper is to provide a summary of the salient choices that must be made by researchers interested in integrating these methods and offering several recommendations for those undertaking such analytic strategies. Overall, the combination of MA and SEM offers researchers unique opportunities, but caution must be exercised when drawing inferences from results.     

Aggregating and Testing Intra-Individual Correlations: Methods and Comparisons

From a longitudinal study, we have repeatedly measured data from multiple individuals at multiple occasions. For each individual, the relation between 2 variables can be measured by the Pearson’s correlation. The question is how to aggregate the multiple correlations and conduct statistical inference on the aggregated intra-individual correlation. Several methods are proposed to aggregate and test intra-individual correlations: (a) a meta-analysis method based on Fisher’s Z transformed correlations, (b) a meta-analysis method based on the Pearson’s correlations, and (c) a multilevel modeling method using data standardized within each individual. The performance of the methods after bias corrections was compared using simulations with considering factors including numbers of individuals, numbers of time points, population effect sizes, and their distribution forms (homogeneous vs heterogeneous). The results from the simulation studies show that estimation biases were found using the meta-analytic methods and suggestions on when and how to correct biases were provided based on the simulation results. Furthermore, the performance of the 3 methods after necessary bias corrections was found to be comparable and reasonably good, indicating that all 3 methods worked for aggregating and testing intra-individual correlations. An empirical daily diary data set was then used to illustrate the applications of the 3 methods. The assumptions, advantages and disadvantages, and possible extensions of the 3 methods were discussed.     

Methods for second order meta-analysis and illustrative applications

This paper presents methods for second order meta-analysis along with several illustrative applications. A second order meta-analysis is a meta-analysis of a number of statistically independent and methodologically comparable first order meta-analyses examining ostensibly the same relationship in different contexts. First order meta-analysis greatly reduces sampling error variance but does not eliminate it. The residual sampling error is called second order sampling error. The purpose of a second order meta-analysis is to estimate the proportion of the variance in mean meta-analytic effect sizes across multiple first order meta-analyses attributable to second order sampling error and to use this information to improve accuracy of estimation for each first order meta-analytic estimate. We present equations and methods based on the random effects model for second order meta-analysis for three situations and three empirical applications of second order meta-analysis to illustrate the potential value of these methods to the pursuit of cumulative knowledge.     

The Influence of Judgment Calls on Meta-Analytic Findings

Previous research has suggested that judgment calls (i.e., methodological choices made in the process of conducting a meta-analysis) have a strong influence on meta-analytic findings and question their robustness. However, prior research applies case study comparison or reanalysis of a few meta-analyses with a focus on a few selected judgment calls. These studies neglect the fact that different judgment calls are related to each other and simultaneously influence the outcomes of a meta-analysis, and that meta-analytic findings can vary due to non–judgment call differences between meta-analyses (e.g., variations of effects over time). The current study analyzes the influence of 13 judgment calls in 176 meta-analyses in marketing research by applying a multivariate, multilevel meta-meta-analysis. The analysis considers simultaneous influences from different judgment calls on meta-analytic effect sizes and controls for alternative explanations based on non–judgment call differences between meta-analyses. The findings suggest that judgment calls have only a minor influence on meta-analytic findings, whereas non–judgment call differences between meta-analyses are more likely to explain differences in meta-analytic findings. The findings support the robustness of meta-analytic results and conclusions.     

On Conducting Construct Validity Meta-Analyses for the Rorschach: A Reply to Tibon Czopp and Zeligman (2016)

We respond to Tibon Czopp and Zeligman's (2016) critique of our systematic reviews and meta-analyses of 65 Rorschach Comprehensive System (CS) variables published in Psychological Bulletin (2013). The authors endorsed our supportive findings but critiqued the same methodology when used for the 13 unsupported variables. Unfortunately, their commentary was based on significant misunderstandings of our meta-analytic method and results, such as thinking we used introspectively assessed criteria in classifying levels of support and reporting only a subset of our externally assessed criteria. We systematically address their arguments that our construct label and criterion variable choices were inaccurate and, therefore, meta-analytic validity for these 13 CS variables was artificially low. For example, the authors created new construct labels for these variables that they called “the customary CS interpretation,” but did not describe their methodology nor provide evidence that their labels would result in better validity than ours. They cite studies they believe we should have included; we explain how these studies did not fit our inclusion criteria and that including them would have actually reduced the relevant CS variables’ meta-analytic validity. Ultimately, criticisms alone cannot change meta-analytic support from negative to positive; Tibon Czopp and Zeligman would need to conduct their own construct validity meta-analyses.     

Meta-Analysis of Pragmatic and Theoretical Research: A Critique

Meta-analysis refers to a set of statistical procedures used to summarize and integrate many empirical studies that focus on one issue. This numerical method of integrating research findings is said to be superior to the narrative type of reviews because it is more objective, reliable, and rigorous. Moreover, the meta-analytic approach is supposedly capable of resolving research controversies, strengthening empirical hypotheses, and discovering new relationships among variables. In this study, these claims are examined and found to be wanting. Some objections to the use of meta-analysis as a means of substantiating theoretical assertions are raised with reference to the rationale of experimentation and to how knowledge evolves. It is concluded that it is inappropriate to apply meta-analysis to integrate theoretical research.     

Leaders and followers behaving badly: A meta-analytic examination of curvilinear relationships between destructive leadership and followers’ workplace behaviors

We draw from social psychological and resource-based theories to meta-analytically examine curvilinear relationships between destructive leadership and followers’ workplace behaviors (i.e., job performance, organizational citizenship behaviors, and workplace deviance). Overall, our meta-analytic results demonstrate that relationships between destructive leadership and followers’ workplace behaviors are essentially linear. The limited evidence of curvilinear relationships we did find supports the application of social psychological theories when examining high levels of destructive leadership. Overall, this study's meta-analytic regression, relative weight, and semipartial correlation results have important implications for how to interpret the conclusions drawn from prior destructive leadership research, how to conduct future studies that examine destructive leadership, and practitioners’ attempts to limit the effects of destructive leadership on followers’ workplace behaviors.     

Sequential meta-analysis of single-case experimental data

In this article, sequential meta-analysis is presented as a method for determining the sufficiency of cumulative knowledge in single-case research synthesis. Sufficiency addresses the question of whether there is enough cumulative knowledge on a topic to yield convincing statistical evidence. The method combines cumulative meta-analysis of single-case experimental data with formal sequential testing. After describing the underlying statistical techniques, a strategy for conducting a sequential single-case meta-analysis is illustrated using a real meta-analytic database. The sequential methodology may serve as a valuable tool for behavioral researchers to guide them in making optimal use of limited resources.     

Meta-analysis of coefficient alpha

The meta-analysis of coefficient alpha across many studies is becoming more common in psychology by a methodology labeled reliability generalization. Existing reliability generalization studies have not used the sampling distribution of coefficient alpha for precision weighting and other common meta-analytic procedures. A framework is provided for a statistically grounded meta-analysis of coefficient alpha using its sampling distribution. Two empirical examples are offered to illustrate these methods, and limitations of reliability generalization are described.     

元分析的特点、方法及其应用的现状分析

元分析是心理、教育及其他科学领域内新近出现的一种重要研究方法,它主要是借助统计方法,对针对同一问题的大量研究结果进行综合分析与评价,从而概括出其研究结果所反映的共同效应,即普遍性的结论。但这种方法在目前国内心理与教育研究中仍不多见。本文首先将通过与其他整合研究结果的分析方法的比较,归纳出元分析的主要特点及其局限性,其次本文着重介绍目前应用较为广泛的三种重要的元分析方法,并对三种方法作出比较分析;最后本文还将对元分析技术在国内心理学研究中的应用现状作分析与评价。     

Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

Previous studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta-analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta-analysis of the Hispanic–White mean difference (d-value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d-values for indirect range restriction and combine them with other meta-analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion-related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.     

Intervenciones psicológicas para cuidadores con síntomas depresivos: revisión sistemática y metanálisis

It has been well determined that nonprofessional caregiving, which is a fundamental component of care to dependent persons, may lead to an increased risk for mental disorders in caregivers, especially depression. This paper sought to conduct a systematic review and meta-analysis to determine the efficacy of psychological interventions to reduce depressive symptoms aimed at nonprofessional caregivers. A comprehensive search of studies published from 1980 to 2013 was conducted in Medline and PsycInfo databases, as well as in the references of other reviews, meta-analysis and relevant book chapters, thereby shortlisting 13 papers. Although most of the interventions reached positive results in decreasing depressive symptomatology, their effects in general were moderate (ES = −0.49, 95% CI [−0.62, −0.36]). Important advances have been produced regarding the quality of the studies, but some limitations are still found which restrict the scope of their findings. It is necessary to conduct further studies that meet excellence standards and, more concretely, regarding indicated prevention of depression.     

基于IPO模型的团队多样性-冲突-绩效权变模型的元分析

本文建立基于IPO模型的团队多样性-冲突-绩效的权变模型。通过对国内外的112篇实证文章（n=9263）进行元分析,本文定量地梳理近年来团队多样性-团队冲突-团队绩效之间关系的研究文献,并重点考察团队冲突与团队绩效之间的调节变量。元分析结果发现：团队深层多样性是引起团队冲突的重要因素,表层多样性并不会显著增加团队冲突;与美国的研究不同,中国情境下任务冲突将显著降低团队绩效;被试类型、回收率、团队规模和团队冲突的不对称感知等变量调节团队冲突与团队绩效之间的关系。