On the Multilevel Nature of Meta-Analysis: A Tutorial,Comparison of Software Programs,and Discussion of Analytic Choices

The term “multilevel meta-analysis” is encountered not only in applied research studies, but in multilevel resources comparing traditional meta-analysis to multilevel meta-analysis. In this tutorial, we argue that the term “multilevel meta-analysis” is redundant since all meta-analysis can be formulated as a special kind of multilevel model. To clarify the multilevel nature of meta-analysis the four standard meta-analytic models are presented using multilevel equations and fit to an example data set using four software programs: two specific to meta-analysis (metafor in R and SPSS macros) and two specific to multilevel modeling (PROC MIXED in SAS and HLM). The same parameter estimates are obtained across programs underscoring that all meta-analyses are multilevel in nature. Despite the equivalent results, not all software programs are alike and differences are noted in the output provided and estimators available. This tutorial also recasts distinctions made in the literature between traditional and multilevel meta-analysis as differences between meta-analytic choices, not between meta-analytic models, and provides guidance to inform choices in estimators, significance tests, moderator analyses, and modeling sequence. The extent to which the software programs allow flexibility with respect to these decisions is noted, with metafor emerging as the most favorable program reviewed.     

The Influence of Judgment Calls on Meta-Analytic Findings

Previous research has suggested that judgment calls (i.e., methodological choices made in the process of conducting a meta-analysis) have a strong influence on meta-analytic findings and question their robustness. However, prior research applies case study comparison or reanalysis of a few meta-analyses with a focus on a few selected judgment calls. These studies neglect the fact that different judgment calls are related to each other and simultaneously influence the outcomes of a meta-analysis, and that meta-analytic findings can vary due to non–judgment call differences between meta-analyses (e.g., variations of effects over time). The current study analyzes the influence of 13 judgment calls in 176 meta-analyses in marketing research by applying a multivariate, multilevel meta-meta-analysis. The analysis considers simultaneous influences from different judgment calls on meta-analytic effect sizes and controls for alternative explanations based on non–judgment call differences between meta-analyses. The findings suggest that judgment calls have only a minor influence on meta-analytic findings, whereas non–judgment call differences between meta-analyses are more likely to explain differences in meta-analytic findings. The findings support the robustness of meta-analytic results and conclusions.     

Analysis and meta-analysis of single-case designs: An introduction

The last 10 years have seen great progress in the analysis and meta-analysis of single-case designs (SCDs). This special issue includes five articles that provide an overview of current work on that topic, including standardized mean difference statistics, multilevel models, Bayesian statistics, and generalized additive models. Each article analyzes a common example across articles and presents syntax or macros for how to do them. These articles are followed by commentaries from single-case design researchers and journal editors. This introduction briefly describes each article and then discusses several issues that must be addressed before we can know what analyses will eventually be best to use in SCD research. These issues include modeling trend, modeling error covariances, computing standardized effect size estimates, assessing statistical power, incorporating more accurate models of outcome distributions, exploring whether Bayesian statistics can improve estimation given the small samples common in SCDs, and the need for annotated syntax and graphical user interfaces that make complex statistics accessible to SCD researchers. The article then discusses reasons why SCD researchers are likely to incorporate statistical analyses into their research more often in the future, including changing expectations and contingencies regarding SCD research from outside SCD communities, changes and diversity within SCD communities, corrections of erroneous beliefs about the relationship between SCD research and statistics, and demonstrations of how statistics can help SCD researchers better meet their goals.     

Meta-Analysis of Single-Case Design Research: Introduction to the Special Issue

Single-case design (SCD) research focuses on finding powerful effects, but the influence of this methodology on the evidence-based practice (EBP) movement is questionable. Meta-analytic procedures may help facilitate the role of SCD research in the EBP movement, but meta-analyses of SCDs are controversial. The current article provides an introduction to the special issue on meta-analyses of SCD research by discussing concerns regarding the internal and external validity of these designs. Specific considerations for increasing the validity of SCD meta-analyses are provided, as are brief overviews of the articles included in the special issue.     

It has been said that, “There are three degrees of falsehoods: Lies,damn lies,and statistics”

Applied behavior analysis (ABA) researchers have historically eschewed population-based, inferential statistics, preferring to conduct and analyze repeated observations of each participant's responding under carefully controlled and manipulated experimental conditions using single-case designs (SCDs). In addition, early attempts to adapt traditional statistical procedures for use with SCDs often involved trade-offs between experimental and statistical control that most ABA researchers have found undesirable. The statistical methods recommended for use with SCDs in the current special issue represent a welcome departure from such prior suggestions in that the current authors are proposing that SCD researchers add statistical methods to their current practices in ways that do not alter traditional SCD practices. Further refinement and use of such methods would (a) facilitate the inclusion of research using SCDs in meta-analyses and (b) aid in the development and planning of grant-funded research using SCD methods. Collaboration between SCD researchers and statisticians, particularly on research that demonstrates the benefit of these methods, may help promote their acceptance and use in ABA.     

Single-case design standards: An update and proposed upgrades

In this paper, we provide a critique focused on the What Works Clearinghouse (WWC) Standards for Single-Case Research Design (Standards 4.1). Specifically, we (a) recommend the use of visual-analysis to verify a single-case intervention study's design standards and to examine the study's operational issues, (b) identify limitations of the design-comparable effect-size measure and discuss related statistical matters, (c) review the applicability and practicality of Standards 4.1 to single-case designs (SCDs), and (d) recommend inclusion of content pertaining to diversity, equity, and inclusion in future standards. Within the historical context of the WWC Pilot Standards for Single-Case Design (1.0), we suggest that Standards 4.1 may best serve as standards for meta-analyses of SCDs but will need to make clear distinctions among the various types of SCD studies that are included in any research synthesis. In this regard, we argue for transparency in SCD studies that meet design standards and those that do not meet design standards in any meta-analysis emanating from the WWC. The intent of these recommendations is to advance the science of SCD research both in research synthesis and in promoting evidence-based practices.     

Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model

Numerous ways to meta-analyze single-case data have been proposed in the literature; however, consensus has not been reached on the most appropriate method. One method that has been proposed involves multilevel modeling. For this study, we used Monte Carlo methods to examine the appropriateness of Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach for the meta-analysis of single-case data. Specifically, we examined the fixed effects (e.g., the overall average treatment effect) and the variance components (e.g., the between-person within-study variance in the treatment effect) in a three-level multilevel model (repeated observations nested within individuals, nested within studies). More specifically, bias of the point estimates, confidence interval coverage rates, and interval widths were examined as a function of the number of primary studies per meta-analysis, the modal number of participants per primary study, the modal series length per primary study, the level of autocorrelation, and the variances of the error terms. The degree to which the findings of this study are supportive of using Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach to meta-analyzing single-case data depends on the particular parameter of interest. Estimates of the average treatment effect tended to be unbiased and produced confidence intervals that tended to overcover, but did come close to the nominal level as Level-3 sample size increased. Conversely, estimates of the variance in the treatment effect tended to be biased, and the confidence intervals for those estimates were inaccurate.     

Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications

This article presents a d-statistic for single-case designs that is in the same metric as the d-statistic used in between-subjects designs such as randomized experiments and offers some reasons why such a statistic would be useful in SCD research. The d has a formal statistical development, is accompanied by appropriate power analyses, and can be estimated using user-friendly SPSS macros. We discuss both advantages and disadvantages of d compared to other approaches such as previous d-statistics, overlap statistics, and multilevel modeling. It requires at least three cases for computation and assumes normally distributed outcomes and stationarity, assumptions that are discussed in some detail. We also show how to test these assumptions. The core of the article then demonstrates in depth how to compute d for one study, including estimation of the autocorrelation and the ratio of between case variance to total variance (between case plus within case variance), how to compute power using a macro, and how to use the d to conduct a meta-analysis of studies using single-case designs in the free program R, including syntax in an appendix. This syntax includes how to read data, compute fixed and random effect average effect sizes, prepare a forest plot and a cumulative meta-analysis, estimate various influence statistics to identify studies contributing to heterogeneity and effect size, and do various kinds of publication bias analyses. This d may prove useful for both the analysis and meta-analysis of data from SCDs.     

Modelling partially cross‐classified multilevel data

This article proposes an approach to modelling partially cross‐classified multilevel data where some of the level‐1 observations are nested in one random factor and some are cross‐classified by two random factors. Comparisons between a proposed approach to two other commonly used approaches which treat the partially cross‐classified data as either fully nested or fully cross‐classified are completed with a simulation study. Results show that the proposed approach demonstrates desirable performance in terms of parameter estimates and statistical inferences. Both the fully nested model and the fully cross‐classified model suffer from biased estimates of some variance components and statistical inferences of some fixed effects. Results also indicate that the proposed model is robust against cluster size imbalance.     

Methods for second order meta-analysis and illustrative applications

This paper presents methods for second order meta-analysis along with several illustrative applications. A second order meta-analysis is a meta-analysis of a number of statistically independent and methodologically comparable first order meta-analyses examining ostensibly the same relationship in different contexts. First order meta-analysis greatly reduces sampling error variance but does not eliminate it. The residual sampling error is called second order sampling error. The purpose of a second order meta-analysis is to estimate the proportion of the variance in mean meta-analytic effect sizes across multiple first order meta-analyses attributable to second order sampling error and to use this information to improve accuracy of estimation for each first order meta-analytic estimate. We present equations and methods based on the random effects model for second order meta-analysis for three situations and three empirical applications of second order meta-analysis to illustrate the potential value of these methods to the pursuit of cumulative knowledge.     

An effect size measure and Bayesian analysis of single-case designs

This article describes a linear modeling approach for the analysis of single-case designs (SCDs). Effect size measures in SCDs have been defined and studied for the situation where there is a level change without a time trend. However, when there are level and trend changes, effect size measures are either defined in terms of changes in R² or defined separately for changes in slopes and intercept coefficients. We propose an alternate effect size measure that takes into account changes in slopes and intercepts in the presence of serial dependence and provides an integrated procedure for the analysis of SCDs through estimation and inference based directly on the effect size measure. A Bayesian procedure is described to analyze the data and draw inferences in SCDs. A multilevel model that is appropriate when several subjects are available is integrated into the Bayesian procedure to provide a standardized effect size measure comparable to effect size measures in a between-subjects design. The applicability of the Bayesian approach for the analysis of SCDs is demonstrated through an example.     

On reconciling conflicting meta-analytic findings regarding integrity test validity

We react to the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012a) meta-analysis of the relationship between integrity test scores and work-related criteria, the earlier Ones, Viswesvaran, and Schmidt (1993) meta-analysis of those relationships, the Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012) responses, and the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012b) rebuttal. We highlight differences between the findings of the 2 meta-analyses by focusing on studies that used predictive designs, applicant samples, and non-self-report criteria. We conclude that study exclusion criteria, correction for artifacts, and second order sampling error are not likely explanations for the differences in findings. The lack of detailed documentation of all effect size estimates used in either meta-analysis makes it impossible to ascertain the bases for the differences in findings. We call for increased detail in meta-analytic reporting and for better information sharing among the parties producing and meta-analytically integrating validity evidence.     

Using response ratios for meta-analyzing single-case designs with behavioral outcomes

Methods for meta-analyzing single-case designs (SCDs) are needed to inform evidence-based practice in clinical and school settings and to draw broader and more defensible generalizations in areas where SCDs comprise a large part of the research base. The most widely used outcomes in single-case research are measures of behavior collected using systematic direct observation, which typically take the form of rates or proportions. For studies that use such measures, one simple and intuitive way to quantify effect sizes is in terms of proportionate change from baseline, using an effect size known as the log response ratio. This paper describes methods for estimating log response ratios and combining the estimates using meta-analysis. The methods are based on a simple model for comparing two phases, where the level of the outcome is stable within each phase and the repeated outcome measurements are independent. Although auto-correlation will lead to biased estimates of the sampling variance of the effect size, meta-analysis of response ratios can be conducted with robust variance estimation procedures that remain valid even when sampling variance estimates are biased. The methods are demonstrated using data from a recent meta-analysis on group contingency interventions for student problem behavior.     

Examining Estimates of Effect Using Non-Overlap of All Pairs in Multiple Baseline Studies of Academic Intervention

The inclusion of single-case design (SCD) studies in meta-analytic research is an important consideration in identifying effective evidence-based practices. Various SCD effect sizes have been previously suggested; non-overlap of all pairs (NAP) is a recently introduced effect size. Preliminary field tests investigating the adequacy of NAP are promising, but no analyses have been conducted using only multiple baseline designs (MBDs). This preliminary study investigated typical values of NAP in MBDs, investigated agreement with visual analysis, and suggested cut scores for interpreting a NAP effect size. Typical values of NAP in MBDs were larger compared to a previous meta-analysis of studies using AB, MBD, or ABAB withdrawal designs, and agreement of suggested cut scores and visual analysis was moderate.     

REVISITING INTERVIEW–COGNITIVE ABILITY RELATIONSHIPS: ATTENDING TO SPECIFIC RANGE RESTRICTION MECHANISMS IN META-ANALYSIS

This study revisits the relationship between interviews and cognitive ability tests, finding lower magnitudes of correlation than have previous meta-analyses; a finding that has implications for both the construct and incremental validity of the interview. Our lower estimates of this relationship than previous meta-analyses were mainly due to (a) an updated set of studies, (b) exclusion of samples in which interviewers potentially had access to applicants' cognitive test scores, and (c) attention to specific range restriction mechanisms that allowed us to identify a sizable subset of studies for which range restriction could be accurately accounted. Moderator analysis results were similar to previous meta-analyses, but magnitudes of correlation were generally lower than in previous meta-analyses. Findings have implications for the construct and incremental validity of interviews, and meta-analytic methodology in general.     

Examining the “Pros” and “Cons” of Team Conflict: A Team-Level Meta-Analysis of Task,Relationship, and Process Conflict

Jehn (e.g., 1997) offered three distinct types of team conflict, namely, task conflict, relationship conflict, and process conflict. Despite existing meta-analyses, there remain important and ongoing issues that warrant further meta-analytic investigation. Our contribution is threefold. First, we report novel meta-analytic findings involving moderators of the conflict–team performance relationship. Second, we report on meta-analytic correlations involving all three conflict types and team innovation. Third, we report on the relations involving task conflict and relationship conflict with previously unexamined, but critical, teamwork variables: team potency, cooperative behaviors, competitive behaviors, and avoidance behaviors. Input for the current meta-analysis included 89 independent samples, 6,122 teams, and approximately 28,000 team members.     

Meta-analyses in sales research

Meta-analytic inquiries enable researchers to synthesize empirical findings obtained over the evolution of a topic area and identify boundary conditions affecting the associations of key variables. Sales researchers can employ meta-analytic techniques to amalgamate empirical work conducted in a given topic area, and several sales researchers have effectively used meta-analyses to advance our understanding of the field. This article provides an exposition of this research design by analyzing its application in sales research. The authors review the meta-analytic studies in sales research and advance key considerations in topical foci, article selection, data coding and evaluation, and analytic approaches. An empirical example is provided to illustrate the power of meta-analysis in substantiating or refuting findings that diverge from accumulated insight in sales research. Results provide support for a positive effort–job satisfaction association in contrast to findings evidencing a negative association between these variables. A second empirical example is used to evaluate the nomological and discriminant validity of two related constructs: job involvement and organizational commitment. Results suggest that these two constructs may be empirically redundant. The authors also provide guidance for the future in regard to new substantive research areas, construct assessment, and additional types of meta-analytic approaches.     

Cohesion and performance in groups: a meta-analytic clarification of construct relations

Previous meta-analytic examinations of group cohesion and performance have focused primarily on contextual factors. This study examined issues relevant to applied researchers by providing a more detailed analysis of the criterion domain. In addition, the authors reinvestigated the role of components of cohesion using more modern meta-analytic methods and in light of different types of performance criteria. The results of the authors' meta-analyses revealed stronger correlations between cohesion and performance when performance was defined as behavior (as opposed to outcome), when it was assessed with efficiency measures (as opposed to effectiveness measures), and as patterns of team workflow became more intensive. In addition, and in contrast to B. Mullen and C. Copper's (1994) meta-analysis, the 3 main components of cohesion were independently related to the various performance domains. Implications for organizations and future research on cohesion and performance are discussed.     

Successfully Combining Meta-analysis and Structural Equation Modeling: Recommendations and Strategies

During the past two decades, organizational researchers have combined the techniques of meta-analysis (MA) and structural equation modeling (SEM) with the intention of building on the strengths of these approaches to address unique research questions. Though these integrative analyses can involve the use of SEM to conduct MA, the focus of the current article is on those situations in which meta-analytic correlations are used as input for testing structural models not previously evaluated in any single, primary study. The purpose of this paper is to provide a summary of the salient choices that must be made by researchers interested in integrating these methods and offering several recommendations for those undertaking such analytic strategies. Overall, the combination of MA and SEM offers researchers unique opportunities, but caution must be exercised when drawing inferences from results.     

Effects of psychotherapy with people who have been sexually assaulted: A meta-analysis

This paper presents the results of a meta-analysis of the treatment outcome studies of different types of psychotherapeutic approaches for sexual assault victims experiencing PTSD or rape trauma symptoms. There were 15 outcome studies identified for inclusion in the meta-analysis dating from 1988–2005, and these studies comprised 25 treatment conditions. Separate meta-analyses were conducted according to study design (independent samples and repeated measures), in keeping with meta-analytic conventions. The overall results for the two meta-analyses were highly consistent, and effect sizes were in the large range for independent samples (g = .91) and repeated measures treatments (g = .90). Effects were maintained at follow-up from 6–12 months after treatment. Studies represented diverse treatment approaches, and most treatments were effective in improving outcome according to symptom reduction. A number of moderating variables were examined. Better outcomes were achieved with individual therapy compared to group approaches. The use of semi-structured approaches and homework techniques were positively related to the magnitude of effect size.