Eta Squared,Partial Eta Squared,and Misreporting of Effect Size in Communication Research

Communication researchers, along with social scientists from a variety of disciplines, are increasingly recognizing the importance of reporting effect sizes to augment significance tests. Serious errors in the reporting of effect sizes, however, have appeared in recently published articles. This article calls for accurate reporting of estimates of effect size. Eta squared (η²) is the most commonly reported estimate of effect sized for the ANOVA. The classical formulation of eta squared (Pearson, 1911; Fisher, 1928) is distinguished from the lesser known partial eta squared (Cohen, 1973), and a mislabeling problem in the statistical software SPSS (1998) is identified. What SPSS reports as eta squared is really partial eta squared. Hence, researchers obtaining estimates of eta squared from SPSS are at risk of reporting incorrect values. Several simulations are reported to demonstrate critical issues. The strengths and limitations of several estimates of effect size used in ANOVA are discussed, as are the implications of the reporting errors. A list of suggestions for researchers is then offered.     

Sex Differences in Clinical Pain: A Multisample Study

A recent meta-analysis of the experimental pain literature revealed effect sizes of .55 for pain threshold and .57 for pain tolerance, indicating a moderate difference in pain perception between men and women, with women reporting an increased sensitivity to pain. The current study investigated the relationship between sex and clinical pain ratings, in patients seeking care at a tertiary care facility. Five samples of chronic pain patients were recruited from several diverse clinics associated with the University of Florida. Analyses of clinical pain ratings revealed similar effect sizes for all samples, ranging from –.07 to –.25, indicating small differences, with women reporting higher levels of clinical pain. This is the first paper to report effect sizes for differences in report of pain in samples of chronic pain patients presenting for treatment at a tertiary care facility.     

A meta-analysis of the survival-processing advantage in memory

The survival-processing advantage occurs when processing words for their survival value improves later performance on a memory test. Due to the interest in this topic, we conducted a meta-analysis to review the literature regarding the survival-processing advantage, in order to estimate a bias-corrected effect size. Traditional meta-analytic methods were used, as well as the test of excess significance, p-curve, p-uniform, trim and fill, PET–PEESE, and selection models, to reevaluate previous effect sizes while controlling for forms of small-study-size effects. The average effect sizes for survival processing ranged between η _p ² = .06 and .09 for between-subjects experiments and between η _p ² = .15 and .18 for within-subjects experiments, after correcting for potential bias and selective reporting. Overall, researchers can expect to find medium to large survival-processing effects, with selective reporting and bias-correcting techniques typically estimating lower effect sizes than traditional meta-analytic techniques.     

Role of Effect Sizes in Contemporary Research in Counseling

Effect sizes (e.g., Cohen's d, Glass's Δ, η², adjusted R², ω²) quantify the extent to which sample results diverge from the expectations specified in the null hypothesis. The present article addresses 5 related questions. First, is the advocacy for reporting and interpreting effect sizes part of the controversy over statistical significance testing? Second, why cannot p values be used as effect sizes? Third, what are the various categories of effect sizes and some commonly used examples of each type? Fourth, how should effect sizes be interpreted? Fifth, what are some recommendations for further reading?     

The consistency test may be too weak to be useful: Its systematic application would not improve effect size estimation in meta-analyses

If the consistency test were used to select papers for inclusion in meta-analysis, the resulting estimates of true effect sizes would be no less biased. Increasing its detection rate at the risk of a higher false alarm rate biases the pooled effect size estimates more—not less—because papers reporting large effect sizes are less likely to be judged inconsistent.     

Effect size estimates: current use, calculations, and interpretation

The Publication Manual of the American Psychological Association (American Psychological Association, 2001, American Psychological Association, 2010) calls for the reporting of effect sizes and their confidence intervals. Estimates of effect size are useful for determining the practical or theoretical importance of an effect, the relative contributions of factors, and the power of an analysis. We surveyed articles published in 2009 and 2010 in the Journal of Experimental Psychology: General, noting the statistical analyses reported and the associated reporting of effect size estimates. Effect sizes were reported for fewer than half of the analyses; no article reported a confidence interval for an effect size. The most often reported analysis was analysis of variance, and almost half of these reports were not accompanied by effect sizes. Partial η2 was the most commonly reported effect size estimate for analysis of variance. For t tests, 2/3 of the articles did not report an associated effect size estimate; Cohen's d was the most often reported. We provide a straightforward guide to understanding, selecting, calculating, and interpreting effect sizes for many types of data and to methods for calculating effect size confidence intervals and power analysis.     

On effect size

The call for researchers to report and interpret effect sizes and their corresponding confidence intervals has never been stronger. However, there is confusion in the literature on the definition of effect size, and consequently the term is used inconsistently. We propose a definition for effect size, discuss 3 facets of effect size (dimension, measure/index, and value), outline 10 corollaries that follow from our definition, and review ideal qualities of effect sizes. Our definition of effect size is general and subsumes many existing definitions of effect size. We define effect size as a quantitative reflection of the magnitude of some phenomenon that is used for the purpose of addressing a question of interest. Our definition of effect size is purposely more inclusive than the way many have defined and conceptualized effect size, and it is unique with regard to linking effect size to a question of interest. Additionally, we review some important developments in the effect size literature and discuss the importance of accompanying an effect size with an interval estimate that acknowledges the uncertainty with which the population value of the effect size has been estimated. We hope that this article will facilitate discussion and improve the practice of reporting and interpreting effect sizes.     

To adjust or not adjust: Nonparametric effect sizes,confidence intervals,and real-world meaning

ObjectivesThe main objectives of this article are to: (a) investigate if there are any meaningful differences between adjusted and unadjusted effect sizes (b) compare the outcomes from parametric and non-parametric effect sizes to determine if the potential differences might influence the interpretation of results, (c) discuss the importance of reporting confidence intervals in research, and discuss how to interpret effect sizes in terms of practical real-world meaning.DesignReview.MethodA review of how to estimate and interpret various effect sizes was conducted. Hypothetical examples were then used to exemplify the issues stated in the objectives.ResultsThe results from the hypothetical research designs showed that: (a) there is a substantial difference between adjusted and non-adjusted effect sizes especially in studies with small sample sizes, and (b) there are differences in outcomes between the parametric and non-parametric effect size formulas that may affect interpretations of results.ConclusionsThe different hypothetical examples in this article clearly demonstrate the importance of treating data in ways that minimize potential biases and the central issues of how to discuss the meaningfulness of effect sizes in research.     

Reexamining psychokinesis: comment on Bösch, Steinkamp, and Boller (2006)

H. B?sch, F. Steinkamp, and E. Boller's review of the evidence for psychokinesis confirms many of the authors' earlier findings. The authors agree with B?sch et al. that existing studies provide statistical evidence for psychokinesis, that the evidence is generally of high methodological quality, and that effect sizes are distributed heterogeneously. B?sch et al. postulated the heterogeneity is attributable to selective reporting and thus that psychokinesis is "not proven." However, B?sch et al. assumed that effect size is entirely independent of sample size. For these experiments, this assumption is incorrect; it also guarantees heterogeneity. The authors maintain that selective reporting is an implausible explanation for the observed data and hence that these studies provide evidence for a genuine psychokinetic effect.     

Say something nice: A meta-analytic review of peer reporting interventions

Peer reporting interventions (i.e., Positive Peer Reporting and tootling) are commonly used peer-mediated interventions in schools. These interventions involve training students to make reports about peers' prosocial behaviors, whether in oral or written form. Although peer reporting interventions have been included in meta-analyses of group contingencies, this study is the first meta-analytic review of single-case research focusing exclusively on peer reporting interventions. The literature search and application of inclusion criteria yielded 21 studies examining the impact of a peer reporting intervention on student behavior compared to baseline conditions. All studies used single-case experimental designs including at least three demonstrations of an effect and at least three data points per phase. Several aspects of studies, participants, and interventions were coded. Log response ratios and Tau were calculated as effect size estimates. Effect size estimates were synthesized in a multi-level meta-analysis with random effects for (a) studies and (b) cases within studies. Overall results indicated peer reporting interventions had a non-zero and positive impact on student outcomes. This was also true when data were subset by outcome (i.e., disruptive behavior, academically engaged behavior, and social behavior). Results were suggestive of more between- than within-study variability. Moderator analyses were conducted to identify aspects of studies, participants, or peer reporting interventions associated with differential effectiveness. Moderator analyses suggested published studies were associated with higher effect sizes than unpublished studies (i.e., theses/dissertations). This meta-analysis suggests peer reporting interventions are effective in improving student behavior compared to baseline conditions. Implications and directions for future investigation are discussed.     

Nonrelativist Ethical Standards for Goal Setting in Psychotherapy

A number of authors have commented on the topic of mandated reporting in cases of suspected child maltreatment and the application of this requirement to researchers. Most of these commentaries focus on the interpretation of current legal standards and offer opinions for or against the imposition of mandated reporting laws on research activities. Authors on both sides of the issue offer ethical arguments, although a direct comparison and analysis of these opposing arguments is rare. This article critically examines the ethical arguments made by authors on both sides of the debate. The conclusion is reached that researchers should be mandated reporters of child maltreatment because the current arguments do not justify their exclusion from current ethical and legal standards. The author makes recommendations for the ethically responsible conduct of research in regard to this topic and legal implications are discussed.     

The value of RCT evidence depends on the quality of statistical analysis

The authors examined statistical practices in 193 randomized controlled trials (RCTs) of psychological therapies published in prominent psychology and psychiatry journals during 1999-2003. Statistical significance tests were used in 99% of RCTs, 84% discussed clinical significance, but only 46% considered-even minimally-statistical power, 31% interpreted effect size and only 2% interpreted confidence intervals. In a second study, 42 respondents to an email survey of the authors of RCTs analyzed in the first study indicated they consider it very important to know the magnitude and clinical importance of the effect, in addition to whether a treatment effect exists. The present authors conclude that published RCTs focus on statistical significance tests ("Is there an effect or difference?"), and neglect other important questions: "How large is the effect?" and "Is the effect clinically important?" They advocate improved statistical reporting of RCTs especially by reporting and interpreting clinical significance, effect sizes and confidence intervals.     

Convergence of scores on the interview and questionnaire versions of the Eating Disorder Examination: a meta-analytic review

Significant discrepancies have been found between interview- and questionnaire-based assessments of psychopathology; however, these studies have typically compared instruments with unmatched item content. The Eating Disorder Examination (EDE), a structured interview, and the questionnaire version of the EDE (EDE-Q) are considered the preeminent assessments of eating disorder symptoms and provide a unique opportunity to examine the concordance of interview- and questionnaire-based instruments with matched item content. The convergence of EDE and EDE-Q scores has been examined previously; however, past studies have been limited by small sample sizes and have not compared the convergence of scores across diagnostic groups. A meta-analysis of 16 studies was conducted to compare the convergence of EDE and EDE-Q scores across studies and diagnostic groups. With regard to the EDE and EDE-Q subscale scores, the overall correlation coefficient effect sizes ranged from .68 to .76. The overall Cohen's d effect sizes ranged from .31 to .62, with participants consistently scoring higher on the questionnaire. For the items measuring behavior frequency, the overall correlation coefficient effect sizes ranged from .37 to .55 for binge eating and .90 to .92 for compensatory behaviors. The overall Cohen's d effect sizes ranged from -0.16 to -0.22, with participants reporting more binge eating on the interview than in the questionnaire in 70% of the studies. These results suggest the interview and questionnaire assess similar constructs but should not be used interchangeably. Additional research is needed to examine the inconsistencies between binge frequency scores on the 2 instruments.     

Treatment Effect on Recidivism for Juveniles Who Have Sexually Offended: a Multilevel Meta-Analysis

The current study investigated the effect on recidivism of treatment aimed at juveniles who have sexually offended. It also assessed the potential moderating effect of type of recidivism, and several treatment, participant and study characteristics. In total, 14 published and unpublished primary studies, making use of a comparison group and reporting on official recidivism rates, were included in a multilevel meta-analysis. This resulted in the use of 77 effect sizes, and 1726 participants. A three-level meta-analytic model was used to calculate the combined effect sizes (Cohens d) and to perform moderator analyses. Study quality was assessed with the EPHPP Quality Assessment Tool for Quantitative Studies. A moderate effect size was found (d = 0.37), indicating that the treatment groups achieved an estimated relative reduction in recidivism of 20.5% as compared to comparison groups. However, after controlling for publication bias, a significant treatment effect was no longer found. Type of recidivism did not moderate the effect of treatment, indicating that treatment groups were equally effective for all types of recidivism. Also, no moderating effects of participant or treatment characteristics were found. Regarding study characteristics, a shorter follow up time showed a trend for larger effect sizes, and the effect size calculation based on proportions yielded larger effect sizes than calculation via mean frequency of offending. Implications for future research and clinical practice are discussed.     

有中介的调节模型的拓展及其效应量

传统的有中介的调节(mediated moderation, meMO)模型关于误差方差齐性的假设经常被违背, 应用研究中也缺乏测量meMO效应大小的指标。对于单层数据, 本文借助于两层建模的思想, 提出了一种可用于处理方差非齐性的两层有中介的调节(2meMO)模型; 给出了用于测量meMO分析中总调节效应、直接调节效应和有中介调节效应大小的效应量。通过Monte Carlo模拟研究, 比较了meMO和2meMO模型在参数和效应量估计上的表现。并通过实际案例解释了2meMO模型的应用以及效应量的计算和解释。     

Post-Traumatic Reactions in Adolescents: How Well Do the DSM-IV PTSD Criteria Fit the Real Life Experience of Trauma Exposed Youth?

This study examined the structure and symptom specific patterns of post traumatic distress in a sample of 1,581 adolescents who reported exposure to at least one traumatic event. Symptom reporting patterns are consistent with past literature in that females reported more symptoms than males and older youth reported more symptoms than did their younger peers. Young people reporting exposure to exclusively violent type traumas were also found to be more likely to endorse symptoms than peers exposed exclusively to non violent type traumas. Confirmatory factor analysis provided stronger support for a four-factor model of PTSD than either the DSM-IV model or an alternate model. Further examination of the four factor model revealed gender differences in factor loadings with small to moderate effect sizes for recurrent, distressing memories, flashbacks, restricted affect, difficulty remember details, detachment, limited future orientation, hypervigilance and startle symptoms. Differences in factor loadings with the four factor model were also noted between younger and older adolescents, with medium to large effect sizes on the arousal items. In contract, comparison of the factor loadings revealed only small differences between youth exposed exclusively to violent traumatic stressors and those exposed exclusively to non violent traumatic stressors, suggesting relative similarity between these two groups.     

Negative estimate of variance-accounted-for effect size: How often it is obtained,and what happens if it is treated as zero

Researchers recommend reporting of bias-corrected variance-accounted-for effect size estimates such as omega squared instead of uncorrected estimates, because the latter are known for their tendency toward overestimation, whereas the former mostly correct this bias. However, this argument may miss an important fact: A bias-corrected estimate can take a negative value, and of course, a negative variance ratio does not make sense. Therefore, it has been a common practice to report an obtained negative estimate as zero. This article presents an argument against this practice, based on a simulation study investigating how often negative estimates are obtained and what are the consequences of treating them as zero. The results indicate that negative estimates are obtained more often than researchers might have thought. In fact, they occur more than half the time under some reasonable conditions. Moreover, treating the obtained negative estimates as zero causes substantial overestimation of even bias-corrected estimators when the sample size and population effect are not large, which is often the case in psychology. Therefore, the recommendation is that researchers report obtained negative estimates as is, instead of reporting them as zero, to avoid the inflation of effect sizes in research syntheses, even though zero can be considered the most plausible value when interpreting such a result. R code to reproduce all of the described results is included as supplemental material.     

The Five Factor Model of personality and social support: A meta-analysis

ObjectiveThe aim of the study was to evaluate the relation between the Big Five personality traits and social support.MethodData for the meta-analysis were collected from 72 studies, which included 84 independent samples, 624 effect sizes, and 37 678 participants.ResultsLower neuroticism and higher extraversion, openness to experience, agreeableness, and conscientiousness were associated with greater perceived availability of social support. Higher extraversion was related to greater perceived received social support. The personality traits-social support relationship was stronger for samples reporting perceived availability of social support from many people than it was for samples reporting perceived availability of social support from concrete people.ConclusionThe study extends current knowledge on the associations between personality traits and social support.