Three-level meta-analysis of dependent effect sizes

Although dependence in effect sizes is ubiquitous, commonly used meta-analytic methods assume independent effect sizes. We describe and illustrate three-level extensions of a mixed effects meta-analytic model that accounts for various sources of dependence within and across studies, because multilevel extensions of meta-analytic models still are not well known. We also present a three-level model for the common case where, within studies, multiple effect sizes are calculated using the same sample. Whereas this approach is relatively simple and does not require imputing values for the unknown sampling covariances, it has hardly been used, and its performance has not been empirically investigated. Therefore, we set up a simulation study, showing that also in this situation, a three-level approach yields valid results: Estimates of the treatment effects and the corresponding standard errors are unbiased.     

A meta-analysis of estimated effect sizes for group versus individual control treatments

A meta-analysis was conducted combining results from nine studies, using seventy-five outcome measures of 349 subjects. Only studies that contrasted group, individual, and control treatments with a pretest-posttest design were selected. Estimates of effect sizes were calculated for group, individual, and control treatments across all studies, according to treatment modality (psychotherapy, counseling, or psychoeducation), type of control (no treatment or some treatment), date of the study (1955-1969 or 1977-1982), and the subjects' age (adult, adolescent, or child). Results indicated that both group and individual treatments had a measurable effect that was consistently greater than that of controls. Further, (1) counseling produced much more of an effect than either therapy or psychoeducation; (2) the more recent treatments produced larger effects than the older studies; and (3) group treatments were more effective than individual methods in the treatment of adolescents, but were less effective in the treatment of children.     

Decision rules for selecting effect sizes in meta-analysis: a review and reanalysis of psychotherapy outcome studies

This study deals with some of the judgmental factors involved in selecting effect sizes from within the studies that enter a meta-analysis. Particular attention is paid to the conceptual redundancy rule that Smith, Glass, and Miller (1980) used in their study of the effectiveness of psychotherapy for deciding which effect sizes should and should not be counted in determining an overall effect size. Data from a random sample of 25 studies from Smith et al.'s (1980) population of psychotherapy outcome studies were first recoded and then reanalyzed meta-analytically. Using the conceptual redundancy rule, three coders independently coded effect sizes and identified more than twice as many of them per study as did Smith et al. Moreover, the treatment effect estimates associated with this larger sample of effects ranged between .30 and .50, about half the size claimed by Smith et al. Analyses of other rules for selecting effect sizes showed that average effect estimates also varied with these rules. Such results indicate that the average effect estimates derived from meta-analyses may depend heavily on judgmental factors that enter into how effect sizes are selected within each of the individual studies considered relevant to a meta-analysis.     

When effect sizes disagree: the case of r and d

The increased use of effect sizes in single studies and meta-analyses raises new questions about statistical inference. Choice of an effect-size index can have a substantial impact on the interpretation of findings. The authors demonstrate the issue by focusing on two popular effect-size measures, the correlation coefficient and the standardized mean difference (e.g., Cohen's d or Hedges's g), both of which can be used when one variable is dichotomous and the other is quantitative. Although the indices are often practically interchangeable, differences in sensitivity to the base rate or variance of the dichotomous variable can alter conclusions about the magnitude of an effect depending on which statistic is used. Because neither statistic is universally superior, researchers should explicitly consider the importance of base rates to formulate correct inferences and justify the selection of a primary effect-size statistic.     

The overjustification effect: A meta-analysis

This article reports the results of a meta-analysis using 50 experimental studies (256 effect sizes) of the overjustification effect. In general, we found support for the effect across age, dependent measure, and design type in specific situations where it is predicted to occur. Findings are less supportive of the theory for situations where there should be no difference between experimental and control groups (e. g. non-contingent or unexpected reward). Suggestions for future research are made.     

A meta-analysis of team-efficacy,potency, and performance: interdependence and level of analysis as moderators of observed relationships

Meta-analytic techniques were used to examine level of analysis and interdependence as moderators of observed relationships between task-specific team-efficacy, generalized potency, and performance. Sixty-seven empirical studies yielding 256 effect sizes were identified and meta-analyzed. Results demonstrated that relationships are moderated by level of analysis. Effect sizes were stronger at the team level (p = .39) than at the individual level (p = .20). At the team level, both team-efficacy and potency had positive relationships with performance (ps = .41 and .37, respectively). Interdependence significantly moderated the relationship between team-efficacy and performance, but not between potency and performance. The relationship between team-efficacy and performance was stronger when interdependence was high (p = .45) than when it was low (p = .34).     

Individual differences in the social facilitation effect: A review and meta-analysis

The present study reviews the social facilitation literature and identifies two overarching responses to social presence: negative-apprehensive and positive-self-assured. These responses correspond to two general orientations toward the social environment described in current personality literature: a negative orientation, comprised of neuroticism and low self-esteem; and a positive orientation, comprised of extraversion and high self-esteem. A meta-analysis reveals that social presence is associated with performance impairment for negatively oriented individuals, and with performance improvement for positively oriented individuals. Additional analysis reveals that personality is a more substantial moderator of the effect of social presence than is task complexity. The results of this study open a new avenue in the research on social facilitation, encouraging a closer inspection of the meaning that various individuals attribute to social presence.     

Quantifying the face inversion effect in nonhuman primates: a phylogenetic meta-analysis

Animal Cognition - Face recognition is important for primate social cognition, enabling rapid discrimination between faces and objects. In humans, face recognition is characterized by certain...     

Hierarchical linear models for the quantitative integration of effect sizes in single-case research

In this article, the calculation of effect size measures in single-case research and the use of hierarchical linear models for combining these measures are discussed. Special attention is given to meta-analyses that take into account a possible linear trend in the data. We show that effect size measures that have been proposed for this situation appear to be systematically affected by the duration of the experiment and fail to distinguish between effects on level and slope. To avoid these flaws, we propose to perform a multivariate meta-analysis on the standardized ordinary least squares regression coefficients from the study-specific regression equations describing the response variable.     

BootES: An R package for bootstrap confidence intervals on effect sizes

Bootstrap Effect Sizes (bootES; Gerlanc & Kirby, 2012) is a free, open-source software package for R (R Development Core Team, 2012), which is a language and environment for statistical computing. BootES computes both unstandardized and standardized effect sizes (such as Cohen’s d, Hedges’s g, and Pearson’s r) and makes easily available for the first time the computation of their bootstrap confidence intervals (CIs). In this article, we illustrate how to use bootES to find effect sizes for contrasts in between-subjects, within-subjects, and mixed factorial designs and to find bootstrap CIs for correlations and differences between correlations. An appendix gives a brief introduction to R that will allow readers to use bootES without having prior knowledge of R.     

The meta‐analysis of partial effect sizes

This paper discusses the synthesis of partial effect sizes derived from multivariate settings. The general statistical properties of the d‐effect size are derived, extending Hedges's statement of zero‐order properties. These general properties have direct relevance in the synthesis of a set of independent effect sizes arising from empirical studies predicated on a common theoretical model. We discuss possible solutions to the problem of comparing effect sizes arising from models employing differing sets of covariates. We apply the general statistical properties in the synthesis of gender performance differences in first‐level economics courses at three New Zealand universities. The model of academic performance is based on Spearman's conception of general academic ability and specific ability in economics.     

The production effect benefits performance in between-subject designs: A meta-analysis

Producing (e.g., saying, mouthing) some items and silently reading others has been shown to result in a reliable advantage favoring retention of the produced compared to non-produced items at test. However, evidence has been mixed as to whether the benefits of production are limited to within- as opposed to between-subject designs. It has even been suggested that the within-subjects nature of the production effect may be one of its defining characteristics. Meta-analytic techniques were applied to evaluate this claim. Findings indicated a moderate effect of production on recognition memory when varied between-subjects (g = 0.37). This outcome suggests that the production effect is not defined as an exclusively within-subject occurrence.     

Interval estimates of weighted effect sizes in the one‐way heteroscedastic ANOVA

A framework for comparing normal population means in the presence of heteroscedasticity and outliers is provided. A single number called the weighted effect size summarizes the differences in population means after weighting each according to the difficulty of estimating their respective means, whether the difficulty is due to unknown population variances, unequal sample sizes or the presence of outliers. For an ANOVA weighted for unequal variances, we find interval estimates for the weighted effect size. In addition, the weighted effect size is shown to be a monotone function of a suitably defined weighted coefficient of determination, which means that interval estimates of the former are readily transformed into interval estimates of the latter. Extensive simulations demonstrate the accuracy of the nominal 95% coverage of these intervals for a wide range of parameters.     

Age differences and the identity negative priming effect: an updated meta-analysis

A recent meta-analysis (P. Verhaeghen & L. De Meersman, 1998a) revealed that older adults show a reliable but significantly reduced negative priming effect compared with young adults. The present study provides an updated quantitative review on the effect of aging on the magnitude of the negative priming effect in identity tasks. This analysis demonstrated that the negative priming effect was not significantly different between young and old adults. This result differs from P. Verhaeghen and L. De Meersman's study. The implications of this finding for inhibitory-based theories of cognitive aging are discussed.     

Confidence intervals for the overall effect size in random-effects meta-analysis

One of the main objectives in meta-analysis is to estimate the overall effect size by calculating a confidence interval (CI). The usual procedure consists of assuming a standard normal distribution and a sampling variance defined as the inverse of the sum of the estimated weights of the effect sizes. But this procedure does not take into account the uncertainty due to the fact that the heterogeneity variance (tau2) and the within-study variances have to be estimated, leading to CIs that are too narrow with the consequence that the actual coverage probability is smaller than the nominal confidence level. In this article, the performances of 3 alternatives to the standard CI procedure are examined under a random-effects model and 8 different tau2 estimators to estimate the weights: the t distribution CI, the weighted variance CI (with an improved variance), and the quantile approximation method (recently proposed). The results of a Monte Carlo simulation showed that the weighted variance CI outperformed the other methods regardless of the tau2 estimator, the value of tau2, the number of studies, and the sample size.     

On the interdependence of cognition and emotion

Affect and cognition have long been treated as independent entities, but in the current review we suggest that affect and cognition are in fact highly interdependent. We open the article by discussing three classic views for the independence of affect. These are (i) the affective independence hypothesis, that emotion is processed independently from cognition, (ii) the affective primacy hypothesis, that evaluative processing precedes semantic processing, and (iii) the affective automaticity hypothesis, that affectively potent stimuli commandeer attention and evaluation is automatic. We argue that affect is not independent from cognition, that affect is not primary to cognition, nor is affect automatically elicited. The second half of the paper discusses several instances of how affect influences cognition. We review experiments showing affective involvement in perception, semantic activation, and attitude activation. We conclude that one function of affect is to regulate cognitive processing.     

焦虑障碍的认知偏向矫正元分析:效果量、影响因素及中介检验

焦虑障碍的认知偏向矫正效果及作用机制是当前一个研究热点且有争议的主题。采用元分析、元回归分析并且组合两步结构方程元分析技术,探索认知偏向矫正对焦虑障碍干预的临床和亚临床有效性,分析可能的影响因素并考察认知偏向改变在认知偏向矫正(Cognitive Bias Modification,CBM)和焦虑症状之间的中介作用。元分析共纳入46篇文献,CBM对焦虑症状干预具有虽小但显著的效果量(包含非标准化测量:g=0.29;标准化测量:g=0.26,p0.001;压力任务后焦虑症状g=0.34,p0.001),对认知偏向有中等效果量(g=0.48,p0.001);干预范式、出版特征、实验环境是焦虑症状CBM效果量的调节因素,焦虑障碍类型和实验环境对CBM认知偏向效果量有调节作用;认知偏向改变在CBM干预与焦虑障碍改变之间起显著的中介效应(z=3.13,SE=0.014,p0.001),为部分中介。将来研究可关注提升CBM症状测量或者认知偏向测量工具的信效度、发展新干预范式,尝试CBM同其他干预方式相结合,进一步探索CBM理论基础和干预机制,提升干预效果。     

Effects of directionality of significance tests on the bias of accessible effect sizes

The proportion of studies that use one-tailed statistical significance tests (pi) in a population of studies targeted by a meta-analysis can affect the bias of the sample effect sizes (sample ESs, or ds) that are accessible to the meta-analyst. H. C. Kraemer, C. Gardner, J. O. Brooks, and J. A. Yesavage (1998) found that, assuming pi = 1.0, for small studies (small Ns) the overestimation bias was large for small population ESs (delta < or = 0.2) and reached a maximum for the smallest population ES (viz., delta = 0). The present article shows (with a minor modification of H. C. Kraemer et al.'s model) that when pi = 0, the small-N bias of accessible sample ESs is relatively small for delta < or = 0.2, and a minimum (in fact, nonexistent) for delta = 0. Implications are discussed for interpretations of meta-analyses of (a) therapy efficacy and therapy effectiveness studies, (b) comparative outcome studies, and (c) studies targeting small but important population ESs.     

The false consensus effect: A meta-analysis of 115 hypothesis tests

False consensus refers to an egocentric bias that occurs when people estimate consensus for their own behaviors. Specifically, the false consensus hypothesis holds that people who engage in a given behavior will estimate that behavior to be more common than it is estimated to be by people who engage in alternative behaviors. A meta-analysis was conducted upon 115 tests of this hypothesis. The combined effects of the tests of the false consensus hypothesis were highly statistically significant and of moderate magnitude. Further, the 115 tests of false consensus appear to be relatively heterogeneous in terms of significance levels and effect sizes. Correlational analyses and focused comparisons indicate that the false consensus effect does not appear to be influenced by the generality of the reference population, nor by the difference between alternative choices in actual consensus. However, the significance and magnitude of the false consensus effect was significantly predicted by the number of behavioral choices/estimates subjects had to make, and the sequence of measurement of choices and estimates. These patterns of results are interpreted as being inconsistent with the self-presentational, motivational explanation for the false consensus effect.     

The effect of goal setting on group performance: a meta-analysis

Updating and extending the work of O'Leary-Kelly, Martocchio, and Frink (1994), with this meta-analysis on goal setting and group performance we show that specific difficult goals yield considerably higher group performance compared with nonspecific goals (d = 0.80 ± 0.35, k = 23 effect sizes). Moderately difficult and easy goals were also associated with performance benefits relative to nonspecific goals, but these effects were smaller. The overall effect size for all group goals was d = 0.56 ± 0.19 (k = 49). Unexpectedly, task interdependence, task complexity, and participation did not moderate the effect of group goals. Our inventory of multilevel goals in interdependent groups indicated that the effect of individual goals in groups on group performance was contingent upon the focus of the goal: "Egocentric" individual goals, aimed at maximizing individual performance, yielded a particularly negative group-performance effect (d = -1.75 ± 0.60, k = 6), whereas "groupcentric" goals, aimed at maximizing the individual contribution to the group's performance, showed a positive effect (d = 1.20 ± 1.03, k = 4). These findings demonstrate that group goals have a robust effect on group performance. Individual goals can also promote group performance but should be used with caution in interdependent groups. Future research might explore the role of multilevel goals for group performance in more detail. The striking lack of recent field studies in organizational settings that emerged from our brief review of trends in group goal-setting research should be taken into account when designing future studies in this domain.