USING RANDOM RATHER THAN FIXED EFFECTS MODELS IN META-ANALYSIS: IMPLICATIONS FOR SITUATIONAL SPECIFICITY AND VALIDITY GENERALIZATION

Combining statistical information across studies (i.e., meta-analysis) is a standard research tool in applied psychology. The most common meta-analytic approach in applied psychology, the fixed effects approach, assumes that individual studies are homogeneous and are sampled from the same population. This model assumes that sampling error alone explains the majority of observed differences in study effect sizes and its use has lead some to challenge the notion of situational specificity in favor of validity generalization. We critique the fixed effects methodology and propose an advancement–the random effects model (RE) which provides estimates of how between-study differences influence the relationships under study. RE models assume that studies are heterogeneous since they are often conducted by different investigators under different settings. Parameter estimates of both models are compared and evidence in favor of the random effects approach is presented. We argue against use of the fixed effects model because it may lead to misleading conclusions about situational specificity.     

Methods for second order meta-analysis and illustrative applications

This paper presents methods for second order meta-analysis along with several illustrative applications. A second order meta-analysis is a meta-analysis of a number of statistically independent and methodologically comparable first order meta-analyses examining ostensibly the same relationship in different contexts. First order meta-analysis greatly reduces sampling error variance but does not eliminate it. The residual sampling error is called second order sampling error. The purpose of a second order meta-analysis is to estimate the proportion of the variance in mean meta-analytic effect sizes across multiple first order meta-analyses attributable to second order sampling error and to use this information to improve accuracy of estimation for each first order meta-analytic estimate. We present equations and methods based on the random effects model for second order meta-analysis for three situations and three empirical applications of second order meta-analysis to illustrate the potential value of these methods to the pursuit of cumulative knowledge.     

Causal graphical views of fixed effects and random effects models

Despite the long-standing discussion on fixed effects (FE) and random effects (RE) models, how and under what conditions both methods can eliminate unmeasured confounding bias has not yet been widely understood in practice. Using a simple pretest–posttest design in a linear setting, this paper translates the conventional algebraic formalization of FE and RE models into causal graphs and provides intuitively accessible graphical explanations about their data-generating and bias-removing processes. The proposed causal graphs highlight that FE and RE models consider different data-generating models. RE models presume a data-generating model that is identical to a randomized controlled trial, while FE models allow for unobserved time-invariant treatment–outcome confounding. Augmenting regular causal graphs that describe data-generating processes by adding the computational structures of FE and RE estimators, the paper visualizes how FE estimators (gain score and deviation score estimators) and RE estimators (quasi-deviation score estimators) offset unmeasured confounding bias. In contrast to standard regression or matching estimators that reduce confounding bias by blocking non-causal paths via conditioning, FE and RE estimators offset confounding bias by deliberately creating new non-causal paths and associations of opposite sign. Though FE and RE estimators are similar in their bias-offsetting mechanisms, the augmented graphs reveal their subtle differences that can result in different biases in observational studies.     

Some recommended statistical analytic practices when reliability generalization studies are conducted

Precursors of the reliability generalization (RG) meta‐analytic approach have not established a single preferred analytic method. By means of five real RG examples, we examine how using different statistical methods to integrate coefficients alpha can influence results in RG studies. Specifically, we compare thirteen different statistical models for averaging reliability coefficients and searching for moderator variables that differ in terms of: (a) whether to transform or not the coefficients alpha, and (b) the statistical model assumed, distinguishing between ordinary least squares methods, the fixed‐effect (FE) model, the varying coefficient (VC) model, and several versions of the random‐effects (RE) model. The results obtained with the different methods exhibited important discrepancies, especially regarding moderator analyses. The main criterion for the model choice should be the extent to which the meta‐analyst intends to generalize the results. RE models are the most appropriate when the meta‐analyst aims to generalize to a hypothetical population of past or future studies, while FE and VC models are the most appropriate when the interest focuses on generalizing the results to a population of studies identical to those included in the meta‐analysis. Finally, some guidelines are proposed for selecting the statistical model when conducting an RG study.     

Meta-analysis of raw mean differences

This article discusses the meta-analysis of raw mean differences. It presents a rationale for cumulating psychological effects in a raw metric and compares raw mean differences to standardized mean differences. Some limitations of standardization are noted, and statistical techniques for raw meta-analysis are described. These include a graphical device for decomposing effect sizes. Several illustrative data sets are analyzed.     

A model for integrating fixed-, random-, and mixed-effects meta-analyses into structural equation modeling

Meta-analysis and structural equation modeling (SEM) are two important statistical methods in the behavioral, social, and medical sciences. They are generally treated as two unrelated topics in the literature. The present article proposes a model to integrate fixed-, random-, and mixed-effects meta-analyses into the SEM framework. By applying an appropriate transformation on the data, studies in a meta-analysis can be analyzed as subjects in a structural equation model. This article also highlights some practical benefits of using the SEM approach to conduct a meta-analysis. Specifically, the SEM-based meta-analysis can be used to handle missing covariates, to quantify the heterogeneity of effect sizes, and to address the heterogeneity of effect sizes with mixture models. Examples are used to illustrate the equivalence between the conventional meta-analysis and the SEM-based meta-analysis. Future directions on and issues related to the SEM-based meta-analysis are discussed.     

The power of statistical tests for moderators in meta-analysis

Calculation of the statistical power of statistical tests is important in planning and interpreting the results of research studies, including meta-analyses. It is particularly important in moderator analyses in meta-analysis, which are often used as sensitivity analyses to rule out moderator effects but also may have low statistical power. This article describes how to compute statistical power of both fixed- and mixed-effects moderator tests in meta-analysis that are analogous to the analysis of variance and multiple regression analysis for effect sizes. It also shows how to compute power of tests for goodness of fit associated with these models. Examples from a published meta-analysis demonstrate that power of moderator tests and goodness-of-fit tests is not always high.     

Is the meta-analysis of correlation coefficients accurate when population correlations vary?

One conceptualization of meta-analysis is that studies within the meta-analysis are sampled from populations with mean effect sizes that vary (random-effects models). The consequences of not applying such models and the comparison of different methods have been hotly debated. A Monte Carlo study compared the efficacy of Hedges and Vevea's random-effects methods of meta-analysis with Hunter and Schmidt's, over a wide range of conditions, as the variability in population correlations increases. (a) The Hunter-Schmidt method produced estimates of the average correlation with the least error, although estimates from both methods were very accurate; (b) confidence intervals from Hunter and Schmidt's method were always slightly too narrow but became more accurate than those from Hedges and Vevea's method as the number of studies included in the meta-analysis, the size of the true correlation, and the variability of correlations increased; and (c) the study weights did not explain the differences between the methods.     

The nature and severity of cognitive impairment associated with adjuvant chemotherapy in women with breast cancer: a meta-analysis of the current literature

OBJECTIVE: Several studies have identified that adjuvant chemotherapy for breast cancer is associated with cognitive impairment; however, the magnitude of this impairment is unclear. This study assessed the severity and nature of cognitive impairment associated with adjuvant chemotherapy by conducting a meta-analysis of the published literature to date. METHOD: Six studies (five cross-sectional and one prospective) meeting the inclusion criteria provided a total of 208 breast cancer patients who had undergone adjuvant chemotherapy, 122 control participants and 122 effect sizes (Cohen's d) falling into six cognitive domains. First, the mean of all the effect sizes within each cognitive domain was calculated (separately for cross-sectional and prospective studies); second, a mean effect size was calculated for all of the effect sizes in each cross-sectional study; and third, regression analyses were conducted to determine any relationships between effect size for each study and four different variables. RESULTS: For the cross-sectional studies, each of the cognitive domains assessed (besides attention) showed small to moderate effect sizes (-0.18 to -0.51). The effect sizes for each study were small to moderate (-0.07 to -0.50) and regression analysis detected a significant negative logarithmic relationship (R2 = .63) between study effect size and the time since last receiving chemotherapy. For the prospective study, effect sizes ranged from small to large (0.11-1.09) and indicated improvements in cognitive function from the beginning of chemotherapy treatment to 3 weeks and even 1 year following treatment. CONCLUSION: This meta-analysis suggests that cognitive impairment occurs reliably in women who have undergone adjuvant chemotherapy for breast cancer but that the magnitude of this impairment depends on the type of design that was used (i.e., cross-sectional or prospective). Thus, more prospective studies are required before definite conclusions about the effects of adjuvant chemotherapy on cognition can be made.     

Synthesizer 1.0: A varying-coefficient meta-analytic tool

Meta-analysis has become an indispensable tool for reaching accurate and representative conclusions about phenomena of interest within a research literature. However, in order for meta-analytic computations to provide accurate estimates of population parameters (e.g., a population correlation), underlying statistical models need to be both efficient and unbiased. Current fixed-effect (i.e., constant-coefficient) models that assume a common effect for all research results perform poorly under conditions of effect size heterogeneity, whereas current random-effects (i.e., random-coefficient) models require unrealistic assumptions about random sampling of observed effect sizes from a normally distributed superpopulation. This article describes a free statistical software tool that employs a varying-coefficient model recently proposed by Bonett (2008, 2009). The software (Synthesizer 1.0) employs procedures that do not require effect homogeneity or random sampling of effect sizes from a normal distribution. It may be used to meta-analyze correlations, alpha reliabilities, and standardized mean differences. The Synthesizer tool for Microsoft Excel 2007 may be downloaded from the author at www .psychology.iastate.edu/~zkrizan/Synthesizer.htm or as a supplement to the article at http://brm.psychonomic-journals.org/content/supplemental.     

从元分析看传统心理统计的局限性

当前,心理学研究文献中占主导地位的数据分析和解释方法是传统的统计方法。元分析方法已经表明传统的统计方法延缓了心理学理论的创新和知识累积的增长。该文从元分析方法的角度阐述了传统的心理统计在处理研究结果方面的局限性。     

道德外语效应及其调节变量：来自元分析的证据

同一个道德决策情景使用外语(相比母语)呈现时,个体会表现出更强的功利性倾向,即道德外语效应。随着研究的深入,结论并不一致。本研究运用元分析方法首次探讨了语言类型(母语vs.外语)对道德判断中功利性倾向的影响,并分析了相关的调节变量。通过文献检索及梳理,共有19篇文献46个独立样本97个效应量符合元分析标准(N=9672)。结果显示存在较小但稳定的道德外语效应(g=0.23);调节效应分析表明,道德外语效应受故事类型的影响,在个人道德两难故事中存在较小但稳定的外语效应(g=0.32),但在非个人道德两难故事(g=0.11)与日常道德评价故事中(g=0.12)不存在外语效应;非个人道德两难故事中的外语效应受记分方式的影响,多点记分在该故事类型下存在效应(g=0.27),二点记分不存在效应(g=0.05);性别和语系类型没有显著的调节效应。这些结果表明语言类型对个体面对道德困境时的选择倾向有一定程度的影响,道德故事类型和记分方式在未来的研究中需要加以考虑。     

Inclusion of theory-relevant moderators yield the same conclusions as Sedikides, Gaertner, and Vevea (2005): A meta-analytical reply to Heine, Kitayama, and Hamamura (2007)

Heine, Kitayama and Hamamura (2007) attributed the Sedikides, Gaertner and Vevea (2005) findings to the exclusion of six papers. We report a meta-analysis that includes those six papers. The Heine et al . conclusions are faulty, because of a misspecified meta-analysis that failed to consider two moderators central to the theory. First, some of their effect sizes originated from studies that did not empirically validate comparison dimensions. Inclusion of this moderator evidences pancultural self-enhancement: Westerners enhance more strongly on individualistic dimensions, Easterners on collectivistic dimensions. Second, some of their effect sizes were irrelevant to whether enhancement is correlated with dimension importance. Inclusion of this moderator evidences pancultural self-enhancement: Both Westerners and Easterners enhance on personally important dimensions. The Sedikides et al . conclusions are valid: Tactical self-enhancement is pancultural.     

The effects of psychological therapies under clinically representative conditions: a meta-analysis

Recently, concern has arisen that meta-analyses overestimate the effects of psychological therapies and that those therapies may not work under clinically representative conditions. This meta-analysis of 90 studies found that therapies are effective over a range of clinical representativeness. The projected effects of an ideal study of clinically representative therapy are similar to effect sizes in past meta-analyses. Effects increase with larger dose and when outcome measures are specific to treatment. Some clinically representative studies used self-selected treatment clients who were more distressed than available controls, and these quasi-experiments underestimated therapy effects. This study illustrates the joint use of fixed and random effects models, use of pretest effect sizes to study selection bias in quasi-experiments, and use of regression analysis to project results to an ideal study in the spirit of response surface modeling.     

Estimating the mean effect size in meta-analysis: Bias,precision, and mean squared error of different weighting methods

Although use of the standardized mean difference in meta-analysis is appealing for several reasons, there are some drawbacks. In this article, we focus on the following problem: that a precision-weighted mean of the observed effect sizes results in a biased estimate of the mean standardized mean difference. This bias is due to the fact that the weight given to an observed effect size depends on this observed effect size. In order to eliminate the bias, Hedges and Olkin (1985) proposed using the mean effect size estimate to calculate the weights. In the article, we propose a third alternative for calculating the weights: using empirical Bayes estimates of the effect sizes. In a simulation study, these three approaches are compared. The mean squared error (MSE) is used as the criterion by which to evaluate the resulting estimates of the mean effect size. For a meta-analytic dataset with a small number of studies, theMSE is usually smallest when the ordinary procedure is used, whereas for a moderate or large number of studies, the procedures yielding the best results are the empirical Bayes procedure and the procedure of Hedges and Olkin, respectively.     

Sequential meta-analysis of single-case experimental data

In this article, sequential meta-analysis is presented as a method for determining the sufficiency of cumulative knowledge in single-case research synthesis. Sufficiency addresses the question of whether there is enough cumulative knowledge on a topic to yield convincing statistical evidence. The method combines cumulative meta-analysis of single-case experimental data with formal sequential testing. After describing the underlying statistical techniques, a strategy for conducting a sequential single-case meta-analysis is illustrated using a real meta-analytic database. The sequential methodology may serve as a valuable tool for behavioral researchers to guide them in making optimal use of limited resources.     

A Meta-Analysis of Research on Hindsight Bias

We conducted a meta-analysis of research on hindsight bias to gain an up-to-date summary of the overall strength of hindsight effects and to test hypotheses about potential moderators of hindsight distortion. A total of 95 studies (83 published and 12 unpublished) were included, and 252 independent effect sizes were coded for moderator variables in 3 broad categories involving characteristics of the study, of measurement, and of the experimental manipulation. When excluding missing effect sizes, the overall mean effect size was Md = .39 with a 95% confidence interval of .36 to .42. Five main findings emerged: (a) effect sizes calculated from objective probability estimates were larger than effect sizes calculated from subjective probability estimates; (b) effect sizes of studies that used almanac questions were larger than effect sizes of studies that used real-world events or case histories; (c) studies that included neutral outcomes resulted in larger effect sizes than studies that used positive or negative outcomes; (d) studies that included manipulations to increase hindsight bias resulted in significantly larger effect sizes than studies in which there were no manipulations to reduce or increase hindsight bias; and (e) studies that included manipulations to reduce hindsight bias did not produce lower effect sizes. These findings contribute to our understanding of hindsight bias by updating the state of knowledge, widening the range of known moderator variables, identifying factors that may activate different mediating processes, and highlighting critical gaps in the research literature.     

Effects of psychotherapy for depression in children and adolescents: a meta-analysis

Serious sequelae of youth depression, plus recent concerns over medication safety, prompt growing interest in the effects of youth psychotherapy. In previous meta-analyses, effect sizes (ESs) have averaged .99, well above conventional standards for a large effect and well above mean ES for other conditions. The authors applied rigorous analytic methods to the largest study sample to date and found a mean ES of .34, not superior but significantly inferior to mean ES for other conditions. Cognitive treatments (e.g., cognitive-behavioral therapy) fared no better than noncognitive approaches. Effects showed both generality (anxiety was reduced) and specificity (externalizing problems were not), plus short- but not long-term holding power. Youth depression treatments appear to produce effects that are significant but modest in their strength, breadth, and durability.     

Meta-analysis for single-case consultation outcomes: Applications to research and practice

The application of meta-analysis holds much appeal for single-case consultation outcome research. We review a meta-analytic method for using within-study treatment effect sizes in reporting consultation outcomes. The strengths and limitations of traditional group design meta-analysis are examined. Various methods for analyzing single-case outcomes are discussed briefly, followed by an examination of the use of meta-analysis in single-case reviews across independent studies. Within-study meta-analytic results are presented that were derived from treatments implemented in consultations in natural settings. To conclude the article, an illustration is offered of a single-case data analysis display that incorporates meta-analytic results along with other indices of treatment outcome. Recommendations are provided for using meta-analytic methods to evaluate outcomes of single-case consultation treatment.     

Implications of the restricted range of family environments for estimates of heritability and nonshared environment in behavior-genetic adoption studies.

Group and individual-difference adoption designs lead to opposite conclusions concerning the importance of shared environment (SE) for the child outcomes of IQ and antisocial behavior. This paradox could be due to the range restriction (RR) of family environments (FE) that goes with adoption studies. Measures of FE from 2 of the most recent adoption studies indicate that RR is substantial, about 67%, which corresponds to the top half of a normal FE distribution. RR of 67% cuts effect sizes and R2 statistics by factors of 3 and 2-2.5, respectively. Because selection into an adoption study in inherently a between-family process and assuming that comparable restriction of genetic (G) influences are absent, estimates of SE, G, and nonshared influences will be substantially biased, respectively, down, up, and up by RR. Corrections for RR applied to adoption studies indicate that SE could account for as much as 50% of the variance in IQ.