Decision rules for selecting effect sizes in meta-analysis: a review and reanalysis of psychotherapy outcome studies

This study deals with some of the judgmental factors involved in selecting effect sizes from within the studies that enter a meta-analysis. Particular attention is paid to the conceptual redundancy rule that Smith, Glass, and Miller (1980) used in their study of the effectiveness of psychotherapy for deciding which effect sizes should and should not be counted in determining an overall effect size. Data from a random sample of 25 studies from Smith et al.'s (1980) population of psychotherapy outcome studies were first recoded and then reanalyzed meta-analytically. Using the conceptual redundancy rule, three coders independently coded effect sizes and identified more than twice as many of them per study as did Smith et al. Moreover, the treatment effect estimates associated with this larger sample of effects ranged between .30 and .50, about half the size claimed by Smith et al. Analyses of other rules for selecting effect sizes showed that average effect estimates also varied with these rules. Such results indicate that the average effect estimates derived from meta-analyses may depend heavily on judgmental factors that enter into how effect sizes are selected within each of the individual studies considered relevant to a meta-analysis.     

Meta-Analysis of Interventions for Basic Mathematics Computation in Single-case Research

This study examined interventions for addition and subtraction that were implemented through single-case design (SCD) research studies. We attempted to extend prior SCD meta-analyses by examining differences in effect sizes across several moderating variables and by including a novel index of effect, improvement rate difference (IRD). We also examined the extent to which effect sizes differed by degree of experimental control achieved in the studies. Forty-seven effect sizes were obtained across 11 studies. IRD effect sizes ranged from .59 to .90 and suggested a moderate to large effect for the math interventions. Variables that appeared to moderate the effects were student age, time spent in intervention, and intervention type. We also identified a relationship between experimental control and the obtained effect sizes. Findings indicated that further SCD research in basic arithmetic and rigorous experiments are necessary to establish an evidence base that accurately characterizes intervention effectiveness.     

Rorschach assessment of changes following psychotherapy: a meta-analytic review

I examined Rorschach assessment of personality changes following psychotherapy. I conducted a comprehensive literature search to find all studies using the Rorschach method at least twice for the same participant in connection with psychotherapy. I conducted meta-analyses for 38 samples, and I performed regression analyses to identify moderating factors. Across all Rorschach scores, the total weighted sample effect size was r = .26, and nearly half the variables obtained effect sizes higher than .30. Several moderating factors were found. Most important, effect sizes increased with longer and more intensive therapy. More concern for interscorer reliability was associated with larger effect sizes, whereas a higher degree of scorer blinding was associated with smaller effect size magnitudes. Predicted levels of change based on the regression models indicated substantial increases in effect size with longer therapies. The data indicate that many elements in the Rorschach are valid indicators of change despite the poor reputation the method has acquired within psychotherapy research.     

The lab versus the clinic. Effects of child and adolescent psychotherapy.

Four recent meta-analyses, involving more than 200 controlled outcome studies, have shown consistent evidence of beneficial therapy effects with children and adolescents. However, most of the studies involved experimental procedures, nonreferred subjects, specially trained therapists with small caseloads, and other features that may not represent conventional clinic therapy. Research focused on more representative treatment of referred clients in clinics has shown more modest effects; in fact, most clinic studies have not shown significant effects. Interpretation studies have not shown significant effects. Interpretation of these findings requires caution; such studies are few and most could profit from improved methodology. The clinic studies do raise questions as to whether the positive lab findings can be generalized to the clinics where most therapy occurs; however, the lab interventions that have worked so well may point the way to enhanced therapy effects in clinics.     

A Systematic Review of the Effectiveness of Self-Compassion-Related Interventions for Individuals With Chronic Physical Health Conditions

Self-compassion, defined as a mindful way of coping with pain and suffering by showing kindness, care, and concern towards the self, may improve psychological adjustment in people living with a chronic physical health condition (CPHC). Various studies illustrate that self-compassion is associated with positive outcomes in general. The aim of this systematic review is to establish the effect of compassion-related therapies on self-compassion specifically in people with CPHCs. Secondary aims are to (a) establish the effect on other psychological and physiological outcomes and (b) explore the relative effectiveness of different therapy types among those identified. Cochrane, Embase, Medline, PsycINFO, and CINAHL databases were searched using “compassion” AND “chronic disease” AND “psychological outcomes” and their synonyms, from 2004 to March 2019. Eligible studies had an experimental design using a self-compassion scale with an adult population. Risk of bias (RoB) was assessed using the Cochrane RoB tool. Effect sizes were calculated for study outcomes. Fifteen studies, including a total of 1,190 participants, 7 different CPHCs, and 11 types of therapies, were included in the review. Nearly all included therapies significantly increased self-compassion with medium to large effect sizes, and reported positive outcomes, such as decreased depression. None of the therapy types appeared clearly superior to the others. Findings from this review show that included therapies increased self-compassion and improved various outcomes, which may represent clinically significant benefits for patients. However, there is a need to further understand how self-compassion exerts its benefits and determine the best methods to increase self-compassion.     

Effects of psychotherapy with people who have been sexually assaulted: A meta-analysis

This paper presents the results of a meta-analysis of the treatment outcome studies of different types of psychotherapeutic approaches for sexual assault victims experiencing PTSD or rape trauma symptoms. There were 15 outcome studies identified for inclusion in the meta-analysis dating from 1988–2005, and these studies comprised 25 treatment conditions. Separate meta-analyses were conducted according to study design (independent samples and repeated measures), in keeping with meta-analytic conventions. The overall results for the two meta-analyses were highly consistent, and effect sizes were in the large range for independent samples (g = .91) and repeated measures treatments (g = .90). Effects were maintained at follow-up from 6–12 months after treatment. Studies represented diverse treatment approaches, and most treatments were effective in improving outcome according to symptom reduction. A number of moderating variables were examined. Better outcomes were achieved with individual therapy compared to group approaches. The use of semi-structured approaches and homework techniques were positively related to the magnitude of effect size.     

Durability of Cognitive Behavioral Therapy Effects for Youth and Adolescents With Anxiety,Depression, or Traumatic Stress:A Meta-Analysis on Long-Term Follow-Ups

Cognitive behavioral therapies (CBT) for youth with anxiety, traumatic stress, and depression have demonstrated strong effects in individual studies and meta-analyses. Relatively more attention has been given to posttreatment effects, though, and assessment of follow-up effects has been limited at the meta-analytic level. The current meta-analysis aimed to (a) examine the effects of youth CBT at posttreatment, 1-month, 3-month, 6-month, 1-year, and long-term (2 + years) follow-up as well as (b) identify research-related variables (e.g., measure respondent type) that relate to effects. Using a random effects model across 110 child and adolescent CBT groups, within-group effect sizes were large at posttreatment (g = 1.24) and from 1-month through long-term follow-up (g = 1.23–1.82), and effect sizes did not significantly differ by treatment target (i.e., anxiety, traumatic stress, depression). However, availability of outcome data for effect sizes diminished across later follow-up assessments. Moreover, effect sizes were significantly associated with outcome respondent type across assessment timing, with outcome measures from caregiver and youth respondents associated with smaller effect sizes (B = -0.97, p < 0.001) relative to outcome measures that were evaluator-reported. Results provide initial support for the durability of treatment effects for youth CBTs and highlight the importance of some confounding variables. Implications for improving treatment research standards and prioritizing assessment of long-term follow-up assessment are discussed.     

Effect sizes in ANCOVA and difference-in-differences designs

It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate and—in non-randomized designs—its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thereby making it comparable to other interventions and studies. Curiously, the estimation of this effect size, including covariate adjustment, has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.     

The crime prevention value of hot spots policing

This paper reviews the available research evidence on the effectiveness of hot spots policing programs in reducing crime and disorder. The research identified five randomized controlled experiments and four non-equivalent control group quasi-experiments evaluating the effects of hot spots policing interventions on crime. Seven of nine selected evaluations reported noteworthy crime and disorder reductions. Meta-analyses of the randomized experiments revealed statistically significant mean effect sizes favoring hot spots policing interventions in reducing citizen calls for service in treatment places relative to control places. When immediate spatial displacement was measured, it was very limited and unintended crime prevention benefits were associated with the hot spots policing programs. The results of this review suggest that hot spots policing is an effective crime prevention strategy.     

Ist psychodynamische Kurztherapie eine wirksame Behandlungsform bei psychischen Störungen?

To test the efficacy of short-term psychodynamic psychotherapy (STPP) in specific psychiatric disorders, a meta-analysis of more recent studies was performed. Its importance is grounded on the inconsistent results of previous meta-analyses regarding short-term psychodynamic psychotherapy (STPP) and the controversial discussion regarding the efficacy of psychodynamic therapy in general. In a computer-based search, studies of STPP published between 1970 and 2004 were identified. Studies of interpersonal therapy were excluded and rigorous inclusion criteria were applied. Seventeen studies fulfilled these criteria. Effect sizes were calculated for target problems, general psychiatric symptoms, and social functioning. In order to examine the stability of outcome, effect sizes were assessed separately for end of therapy and for follow-up assessment. The effect sizes of STPP were compared to those of waiting list control patients, treatments as usual, and of other forms of psychotherapy. STPP yielded significant and large pre-post effect sizes, which were not only stable, but tended to increase at follow-up. The effect sizes of STPP significantly exceeded those of waiting list and treatments as usual, but no differences were found between STPP and other forms of psychotherapy. STPP proved to be an effective treatment in different psychiatric disorders. However, further research of STPP in specific psychiatric disorders is needed, including a study of the active ingredients of STPP. Effectiveness studies should be included.     

Pharmacological and Behavioral Strategies for Smoking Cessation

After over two decades of intensive research, a variety of smoking cessation strategies, both behavioral and pharmacotherapeutic, have been developed and clinically validated. In this review, pharmacological products available for smoking cessation are profiled and examined for efficacy. Results of a recent comprehensive review and meta-analysis by the Agency for Health Care Policy and Research on the effects of behavior therapy on smoking cessation are also discussed. Further, meta-analyses and studies exploring the important interaction between pharmacological and behavioral therapies are considered. Applications of the growing wealth of smoking cessation strategies are discussed with respect to the needs of some special subpopulations of smokers including highly nicotine-dependent smokers and smokers with comorbid psychiatric diagnoses.     

On the use of beta coefficients in meta-analysis

This research reports an investigation of the use of standardized regression (beta) coefficients in meta-analyses that use correlation coefficients as the effect-size metric. The investigation consisted of analyzing more than 1,700 corresponding beta coefficients and correlation coefficients harvested from published studies. Results indicate that, under certain conditions, using knowledge of corresponding beta coefficients to input missing correlations (effect sizes) generally produces relatively accurate and precise population effect-size estimates. Potential benefits from applying this knowledge include smaller sampling errors because of increased numbers of effect sizes and smaller non-sampling errors because of the inclusion of a broader array of research designs.     

The impact of methodological factors on child psychotherapy outcome research: A meta-analysis for researchers

Two recent meta-analyses have generated evidence for child and adolescent psychotherapy effects. However, critics note that such meta-analyses often include studies with methodological shortcomings which might invalidate their results. In the present study, we explored whether the results of the most extensive child/adolescent meta-analysis might have been influenced by such methodological variables, focusing on internal validity and external validity factors. Together, these factors accounted for two-thirds as much variance as the substantive factors (e.g., type of therapy, age) in the original metaanalysis. This suggests that relative to these therapy and child-characteristic variables, methodological factors have a substantial, though smaller, impact on metaanalysis results. In general, increased experimental rigor was related to larger effect sizes; this argues against the hypothesis that methodologically weak studies have led to an overestimate of therapy effects. No significant interactive relations were found between validity factors and predictors of outcome; this suggests that the relations noted in previous metaanalyses between outcome and various variables were notdistorted by the validity factors tested here.The research described here was supported in part by grants from the North Carolina Department of Human Resources, Division of Mental Health, Mental Retardation, and Substance Abuse Services (41626), and from the National Institute of Mental Health (1 RO3 MH38450). The authors gratefully acknowledge the assistance of Todd Morton, who provided reliability coding.     

The Findings of a Cochrane Meta-Analysis of Couple Therapy in Adult Depression: Implications for Research and Clinical Practice

The association between depressive symptoms and distressed intimate relationships supported the assumption that couple therapy, by focusing on the interpersonal context of depression, might be more effective as a treatment for depression than individual psychotherapy or drug therapy. This issue was addressed by a Cochrane meta-analysis assessing the evidence from clinical trials of couple therapy for depression in comparison with individual psychotherapy, drug therapy, and no/minimal intervention, including fourteen studies with 651 participants. No study was found free of bias and the quality of the evidence was low, with major problems of small sample sizes, missing outcome data, selective reporting, lack of information on random sequence generation and allocation concealment, recruitment of people not representative of clinical practice, and allegiance bias. The meta-analysis showed that both couple therapy and individual psychotherapy improved depressive symptoms at end of treatment and after 6 months or longer, with moderate effect sizes, without any difference between the two treatments. Couple therapy was more effective in reducing couple distress. This effect was larger in studies with distressed couples only and should be considered as relevant in its own right. Couple therapy is a viable option for the treatment of a depressed partner, especially in discordant couples. Future research should address several issues left open to provide a sound empirical foundation for clinical practice.     

Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses

The issue of publication bias in psychological science is one that has remained difficult to address despite decades of discussion and debate. The current article examines a sample of 91 recent meta-analyses published in American Psychological Association and Association for Psychological Science journals and the methods used in these analyses to identify and control for publication bias. Of the 91 studies analyzed, 64 (70%) made some effort to analyze publication bias, and 26 (41%) reported finding evidence of bias. Approaches to controlling publication bias were heterogeneous among studies. Of these studies, 57 (63%) attempted to find unpublished studies to control for publication bias. Nonetheless, those studies that included unpublished studies were just as likely to find evidence for publication bias as those that did not. Furthermore, authors of meta-analyses themselves were overrepresented in unpublished studies acquired, as compared with published studies, suggesting that searches for unpublished studies may increase rather than decrease some sources of bias. A subset of 48 meta-analyses for which study sample sizes and effect sizes were available was further analyzed with a conservative and newly developed tandem procedure of assessing publication bias. Results indicated that publication bias was worrisome in about 25% of meta-analyses. Meta-analyses that included unpublished studies were more likely to show bias than those that did not, likely due to selection bias in unpublished literature searches. Sources of publication bias and implications for the use of meta-analysis are discussed.     

A systematic meta-review of evaluations of youth violence prevention programs: Common and divergent findings from 25 years of meta-analyses and systematic reviews

Violence among youth is a pervasive public health problem. In order to make progress in reducing the burden of injury and mortality that result from youth violence, it is imperative to identify evidence-based programs and strategies that have a significant impact on violence. There have been many rigorous evaluations of youth violence prevention programs. However, the literature is large, and it is difficult to draw conclusions about what works across evaluations from different disciplines, contexts, and types of programs. The current study reviews the meta-analyses and systematic reviews published prior to 2009 that synthesize evaluations of youth violence prevention programs. This meta-review reports the findings from 37 meta-analyses and 15 systematic reviews; the included reviews were coded on measures of the social ecology, prevention approach, program type, and study design. A majority of the meta-analyses and systematic reviews were found to demonstrate moderate program effects. Meta-analyses yielded marginally smaller effect sizes compared to systematic reviews, and those that included programs targeting family factors showed marginally larger effects than those that did not. In addition, there are a wide range of individual/family, program, and study moderators of program effect sizes. Implications of these findings and suggestions for future research are discussed.     

Psychotherapeutische Behandlung von depressiven Kindern und Jugendlichen

Background

Depression is a psychiatric disorder with debilitating symptoms (e.g. suicidal behavior) with a high prevalence rate even in children and adolescents and the disorder shows a chronic course in many cases. According to psychiatric guidelines, cognitive behavioral therapy and interpersonal therapy are the psychotherapeutic methods of choice.

Aim

This article gives an overview of the current studies on cognitive behavioral and interpersonal group therapy programs for the treatment of depression in children and adolescents as well as a short illustration of the most prevalent therapy programs.

Material and methods

A literature research (PsycInfo, Psyndex, Pubmed) revealed 280 hits. After a review of all titles and abstracts 25 studies were included in this study.

Results

Efficacy studies mainly exist for cognitive behavioral group therapy programs (effect sizes ranged from 0.02 to 1.34) from English-speaking countries. There are only a few German programs available. With respect to interpersonal group therapies there are only few articles published in English and to the best of our knowledge none in German.

Conclusion

There is a great need for further studies that investigate the efficacy of group therapies for the treatment of depression in children and adolescents especially in German-speaking countries.     

Accuracy of effect size estimates from published psychological research

A Monte-Carlo simulation was used to model the biasing of effect sizes in published studies. The findings from the simulation indicate that, when a predominant bias to publish studies with statistically significant results is coupled with inadequate statistical power, there will be an overestimation of effect sizes. The consequences such an effect size overestimation will then have on meta-analyses and power analyses are highlighted and discussed along with measures which can be taken to reduce the problem.     

The Influence of Judgment Calls on Meta-Analytic Findings

Previous research has suggested that judgment calls (i.e., methodological choices made in the process of conducting a meta-analysis) have a strong influence on meta-analytic findings and question their robustness. However, prior research applies case study comparison or reanalysis of a few meta-analyses with a focus on a few selected judgment calls. These studies neglect the fact that different judgment calls are related to each other and simultaneously influence the outcomes of a meta-analysis, and that meta-analytic findings can vary due to non–judgment call differences between meta-analyses (e.g., variations of effects over time). The current study analyzes the influence of 13 judgment calls in 176 meta-analyses in marketing research by applying a multivariate, multilevel meta-meta-analysis. The analysis considers simultaneous influences from different judgment calls on meta-analytic effect sizes and controls for alternative explanations based on non–judgment call differences between meta-analyses. The findings suggest that judgment calls have only a minor influence on meta-analytic findings, whereas non–judgment call differences between meta-analyses are more likely to explain differences in meta-analytic findings. The findings support the robustness of meta-analytic results and conclusions.     

A Meta-analysis of Wilderness Therapy Outcomes for Private Pay Clients

Wilderness therapy is becoming a more widely used intervention for adolescents, but there have not been any meta-analyses focused solely on its clinical effectiveness for private pay clients. This study’s objective was to conduct outcome-based meta-analyses of private-pay wilderness therapy programs, benchmark primary features of this approach, and educate the clinical community as to its effectiveness. The authors conducted a review of all available databases, as well as manual searches. Searches resulted in a meta-analysis based on 36 studies, totaling 2399 participants receiving wilderness therapy. Our meta-analyses found medium effect sizes for all six constructs assessed: self-esteem (g = 0.49), locus of control (g = 0.55), behavioral observations (g = 0.75), personal effectiveness (g = 0.46), clinical measures (g = 0.50) and interpersonal measures (g = 0.54). Subgroup analyses included age of participants, duration of program, open or closed model, presence of a mental health practitioner, and publication year.