Eta Squared,Partial Eta Squared,and Misreporting of Effect Size in Communication Research

Communication researchers, along with social scientists from a variety of disciplines, are increasingly recognizing the importance of reporting effect sizes to augment significance tests. Serious errors in the reporting of effect sizes, however, have appeared in recently published articles. This article calls for accurate reporting of estimates of effect size. Eta squared (η²) is the most commonly reported estimate of effect sized for the ANOVA. The classical formulation of eta squared (Pearson, 1911; Fisher, 1928) is distinguished from the lesser known partial eta squared (Cohen, 1973), and a mislabeling problem in the statistical software SPSS (1998) is identified. What SPSS reports as eta squared is really partial eta squared. Hence, researchers obtaining estimates of eta squared from SPSS are at risk of reporting incorrect values. Several simulations are reported to demonstrate critical issues. The strengths and limitations of several estimates of effect size used in ANOVA are discussed, as are the implications of the reporting errors. A list of suggestions for researchers is then offered.     

A meta-analysis of the survival-processing advantage in memory

The survival-processing advantage occurs when processing words for their survival value improves later performance on a memory test. Due to the interest in this topic, we conducted a meta-analysis to review the literature regarding the survival-processing advantage, in order to estimate a bias-corrected effect size. Traditional meta-analytic methods were used, as well as the test of excess significance, p-curve, p-uniform, trim and fill, PET–PEESE, and selection models, to reevaluate previous effect sizes while controlling for forms of small-study-size effects. The average effect sizes for survival processing ranged between η _p ² = .06 and .09 for between-subjects experiments and between η _p ² = .15 and .18 for within-subjects experiments, after correcting for potential bias and selective reporting. Overall, researchers can expect to find medium to large survival-processing effects, with selective reporting and bias-correcting techniques typically estimating lower effect sizes than traditional meta-analytic techniques.     

A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models

Group-level variance estimates of zero often arise when fitting multilevel or hierarchical linear models, especially when the number of groups is small. For situations where zero variances are implausible a priori, we propose a maximum penalized likelihood approach to avoid such boundary estimates. This approach is equivalent to estimating variance parameters by their posterior mode, given a weakly informative prior distribution. By choosing the penalty from the log-gamma family with shape parameter greater than 1, we ensure that the estimated variance will be positive. We suggest a default log-gamma(2,λ) penalty with λ→0, which ensures that the maximum penalized likelihood estimate is approximately one standard error from zero when the maximum likelihood estimate is zero, thus remaining consistent with the data while being nondegenerate. We also show that the maximum penalized likelihood estimator with this default penalty is a good approximation to the posterior median obtained under a noninformative prior. Our default method provides better estimates of model parameters and standard errors than the maximum likelihood or the restricted maximum likelihood estimators. The log-gamma family can also be used to convey substantive prior information. In either case—pure penalization or prior information—our recommended procedure gives nondegenerate estimates and in the limit coincides with maximum likelihood as the number of groups increases.     

Abstract: A Meta-Analysis of the Autocorrelation in Single Case Designs

Single case design (SCD) experiments in the behavioral sciences utilize just one participant from whom data is collected over time. This design permits causal inferences to be made regarding various intervention effects, often in clinical or educational settings, and is especially valuable when between-participant designs are not feasible or when interest lies in the effects of an individualized treatment. Regression techniques are the most common quantitative practice for analyzing time series data and provide parameter estimates for both treatment and trend effects. However, the presence of serially correlated residuals, known as autocorrelation, can severely bias inferences made regarding these parameter estimates. Despite the severity of the issue, few researchers test or correct for the autocorrelation in their analyses.

Shadish and Sullivan (in press) recently conducted a meta-analysis of over 100 studies in order to assess the prevalence of the autocorrelation in the SCD literature. Although they found that the meta-analytic weighted average of the autocorrelation was close to zero, the distribution of autocorrelations was found to be highly heterogeneous. Using the same set of SCDs, the current study investigates various factors that may be related to the variation in autocorrelation estimates (e.g., study and outcome characteristics). Multiple moderator variables were coded for each study and then used in a metaregression in order to estimate the impact these predictor variables have on the autocorrelation.

This current study investigates the autocorrelation using a multilevel meta-analytic framework. Although meta-analyses involve nested data structures (e.g., effect sizes nested within studies nested within journals), there are few instances of meta-analysts utilizing multilevel frameworks with more than two levels. This is likely attributable to the fact that very few software packages allow for meta-analyses to be conducted with more than two levels and those that do allow this provide sparse documentation on how to implement these models. The proposed presentation discusses methods for carrying out a multilevel meta-analysis. The presentation also discusses the findings from the metaregression on the autocorrelation and the implications these findings have on SCDs.     

Sample Size Planning for Detecting Mediation Effects: A Power Analysis Procedure Considering Uncertainty in Effect Size Estimates

Abstract

When planning mediation studies, researchers are often interested in the sample size needed to achieve adequate power for testing mediation. Power depends on population effect sizes, which are unknown in practice. In conventional power analysis, effect size estimates, however, are often used as population values, which could result in underpowered studies. Uncertainty in effect size estimates has been considered in other sample size planning contexts (e.g., t-test, ANOVA), but has not been handled properly for planning mediation studies. In the current study, we proposed an easy-to-use sample size planning method for testing mediation with uncertainty in effect size estimates considered. We conducted simulation studies to demonstrate the impact of uncertainty in effect size estimates on power of testing mediation, and to provide sample size suggestions under different levels of uncertainty. Empirical examples were provided to illustrate the application of our method. R functions and a web application were developed to facilitate implementation.     

Adults see vision to be more informative than it is

Humans gain a wide range of knowledge through interacting with the environment. Each aspect of our perceptual experiences offers a unique source of information about the world—colours are seen, sounds heard and textures felt. Understanding how perceptual input provides a basis for knowledge is thus central to understanding one's own and others' epistemic states. Developmental research suggests that 5-year-olds have an immature understanding of knowledge sources and that they overestimate the knowledge to be gained from looking. Without evidence from adults, it is not clear whether the mature reasoning system outgrows this overestimation. The current study is the first to investigate whether an overestimation of the knowledge to be gained from vision occurs in adults. Novel response time paradigms were adapted from developmental studies. In two experiments, participants judged whether an object or feature could be identified by performing a specific action. Adult participants found it disproportionately easy to accept looking as a proposed action when it was informative, and difficult to reject looking when it was not informative. This suggests that adults, like children, overestimate the informativeness of vision. The origin of this overestimation and the implications that the current findings bear on the interpretation of children's overestimation are discussed.     

Effect size estimates: current use, calculations, and interpretation

The Publication Manual of the American Psychological Association (American Psychological Association, 2001, American Psychological Association, 2010) calls for the reporting of effect sizes and their confidence intervals. Estimates of effect size are useful for determining the practical or theoretical importance of an effect, the relative contributions of factors, and the power of an analysis. We surveyed articles published in 2009 and 2010 in the Journal of Experimental Psychology: General, noting the statistical analyses reported and the associated reporting of effect size estimates. Effect sizes were reported for fewer than half of the analyses; no article reported a confidence interval for an effect size. The most often reported analysis was analysis of variance, and almost half of these reports were not accompanied by effect sizes. Partial η2 was the most commonly reported effect size estimate for analysis of variance. For t tests, 2/3 of the articles did not report an associated effect size estimate; Cohen's d was the most often reported. We provide a straightforward guide to understanding, selecting, calculating, and interpreting effect sizes for many types of data and to methods for calculating effect size confidence intervals and power analysis.     

Story Problems: Where Do the Agonists of the Dialogue Model of Argument Interact?

When discussing dialogue, argumentation researchers rarely draw the distinction between the story world and interactional world. While mediators often help to shape the interactions among agonists in the emerging flow of spoken discourse, writers of postulated dialogues narrate them, constructing a story world that depicts the agonists, depicts their utterances and their circumstances. In this paper, I ask where the agonists of the dialogue model of argument interact, and I show that they often interact in the story world of postulated dialogues. Postulated dialogues are story problem conversations. Common in textbooks, exams, and standardized tests, story problems create hypothetical situations to illustrate formal relationships among variables, and are designed to be read in a theoretical attitude that treats the characters, objects, and circumstances that they depict as given. When argumentation researchers examine postulated dialogues, they tend to adopt a theoretical attitude, limiting their analysis to the conversation between the agonists depicted in the story world. Reading this way makes it easier to overlook the interactional world where the writing and reading of the texts takes place, obscuring the fact that they are narrated dialogues, often written by researchers. Reading this way also makes it easier to confirm the traditional participation framework of the dialogue model of argument.     

基于结构方程模型的有调节的中介效应分析

有调节的中介模型是中介过程受到调节变量影响的模型。指出了目前有调节的中介效应分析普遍存在的问题：当前有调节的中介效应检验大多使用多元线性回归分析,忽略了测量误差;而基于结构方程模型(SEM)的有调节的中介效应分析需要产生乘积指标,又会面临乘积指标生成和乘积项非正态分布的问题。在简介潜调节结构方程(LMS)方法后,建议使用LMS方法得到偏差校正的bootstrap置信区间来进行基于SEM的有调节的中介效应分析。总结出一个有调节的中介SEM分析流程,并有示例和相应的Mplus程序。文末展望了LMS和有调节的中介模型的发展方向。     

Configural face processing: a meta-analytic survey

In the field of face processing, the configural hypothesis is defended by many researchers. It is often claimed that this thesis is robustly supported by a large number of experiments exploring the face-inversion effect, the composite face effect, the face superiority effect, and the negative face effect. However, this claim is generally based on a rudimentary and approximate vote-counting approach. In this paper, I use meta-analyses to examine the relevant literature in more depth. The analysis supports the vote-counting argument.     

Explanation of Two Anomalous Results in Statistical Mediation Analysis

Previous studies of different methods of testing mediation models have consistently found two anomalous results. The first result is elevated Type I error rates for the bias-corrected and accelerated bias-corrected bootstrap tests not found in nonresampling tests or in resampling tests that did not include a bias correction. This is of special concern as the bias-corrected bootstrap is often recommended and used due to its higher statistical power compared with other tests. The second result is statistical power reaching an asymptote far below 1.0 and in some conditions even declining slightly as the size of the relationship between X and M, a, increased. Two computer simulations were conducted to examine these findings in greater detail. Results from the first simulation found that the increased Type I error rates for the bias-corrected and accelerated bias-corrected bootstrap are a function of an interaction between the size of the individual paths making up the mediated effect and the sample size, such that elevated Type I error rates occur when the sample size is small and the effect size of the nonzero path is medium or larger. Results from the second simulation found that stagnation and decreases in statistical power as a function of the effect size of the a path occurred primarily when the path between M and Y, b, was small. Two empirical mediation examples are provided using data from a steroid prevention and health promotion program aimed at high school football players (Athletes Training and Learning to Avoid Steroids; Goldberg et al., 1996), one to illustrate a possible Type I error for the bias-corrected bootstrap test and a second to illustrate a loss in power related to the size of a. Implications of these findings are discussed.     

Competing Explanations and Explaining-Away Arguments

An “explaining-away argument” [EAA] aims to discredit some explanatory hypothesis by appealing to the explanatory power of an alternative hypothesis. Nietzsche's genealogical argument against theism and Darwin's case against Paley's “old argument of design in nature” are famous examples. In order for EAAs to have their negative force, they must satisfy several conditions. After clarifying these conditions, I focus in on one in particular: the two hypotheses in question offer potential explanations that compete with one another. I develop a formal account of what it takes for potential explanations to compete, and I use this account to argue that EAAs are often misapplied today. This is due to the fact that philosophers often fail to appreciate the subtle line dividing competing from non-competing explanations.     

Using a calendar in event dating

Three experiments examined Thompson's reconstructive account for estimating the temporal location of an event. Thompson's day‐of‐week (DOW) model suggests that people know the DOW because diary studies have consistently produced dating errors at 7‐day intervals (e.g. 0, 7, 14, 21, etc.). This study determined whether this DOW effect was produced by a calendar. It also examined whether time estimations were primarily reconstructed. A calendar improved both date and DOW estimates, but neither estimate depended on the use of a calendar. In addition, both date and DOW estimates were primarily reconstructed, but a large portion of DOW estimates were reproduced. Therefore, the argument was made that theories explaining memory for temporal location should account for both reproductive and reconstructive processes. Copyright © 2001 John Wiley & Sons, Ltd.     

基于结构方程模型的多重中介效应分析

多重中介模型是指存在多个中介变量的模型。多重中介模型可以分析特定中介效应、总的中介效应和对比中介效应。指出了目前多重中介模型分析普遍存在的问题,包括分析不完整、使用Sobel检验带来的局限。建议通过增加辅助变量的方法进行完整的多重中介效应分析,使用偏差校正的Bootstrap方法进行中介检验。总结出一个多重中介SEM分析流程,并有示例和相应的MPLUS程序。随后展望了辅助变量和中介效应检验方法的发展方向。     

Mindfulness, Big Five personality, and affect: A meta-analysis

Mindfulness is purposefully and nonjudgmentally paying attention to the present moment. The primary purpose of this study is to provide a more precise empirical estimate of the relationship between mindfulness and the Big Five personality traits as well as trait affect. Current research results present inconsistent or highly variable estimates of these relationships. Meta-analysis was used to synthesize findings from 32 samples in 29 studies. Results indicate that, although all of the traits display appreciable relationships with mindfulness, the strongest relationships are found with neuroticism, negative affect, and conscientiousness. Conscientiousness, in particular, is often ignored by mindfulness researchers; results here indicate it deserves stronger consideration. Although the results provide a clearer picture of how mindfulness relates to these traits, they also highlight the need to ensure an appropriate conceptualization and measurement of mindfulness.     

Estimating Incremental Validity Under Missing Data

A common form of missing data is caused by selection on an observed variable (e.g., Z). If the selection variable was measured and is available, the data are regarded as missing at random (MAR). Selection biases correlation, reliability, and effect size estimates when these estimates are computed on listwise deleted (LD) data sets. On the other hand, maximum likelihood (ML) estimates are generally unbiased and outperform LD in most situations, at least when the data are MAR. The exception is when we estimate the partial correlation. In this situation, LD estimates are unbiased when the cause of missingness is partialled out. In other words, there is no advantage of ML estimates over LD estimates in this situation. We demonstrate that under a MAR condition, even ML estimates may become biased, depending on how partial correlations are computed. Finally, we conclude with recommendations about how future researchers might estimate partial correlations even when the cause of missingness is unknown and, perhaps, unknowable.     

Practical Guide for Reporting Effect Size in Quantitative Research in the Journal of Counseling & Development

The purpose of this article is to assist researchers in meeting the requirement of reporting effect sizes in quantitative research studies submitted to the Journal of Counseling & Development. This requirement is detailed in the “Guidelines for Authors” included in this issue. The authors provide practical information on generating, reporting, and interpreting effect size estimates for various types of statistical analyses. Information is provided on the meaning of effect sizes within the larger knowledge base.     

Asymptotic bias of normal-distribution-based maximum likelihood estimates of moderation effects with data missing at random

Moderation analysis is useful for addressing interesting research questions in social sciences and behavioural research. In practice, moderated multiple regression (MMR) models have been most widely used. However, missing data pose a challenge, mainly because the interaction term is a product of two or more variables and thus is a non-linear function of the involved variables. Normal-distribution-based maximum likelihood (NML) has been proposed and applied for estimating MMR models with incomplete data. When data are missing completely at random, moderation effect estimates are consistent. However, simulation results have found that when data in the predictor are missing at random (MAR), NML can yield inaccurate estimates of moderation effects when the moderation effects are non-null. Simulation studies are subject to the limitation of confounding systematic bias with sampling errors. Thus, the purpose of this paper is to analytically derive asymptotic bias of NML estimates of moderation effects with MAR data. Results show that when the moderation effect is zero, there is no asymptotic bias in moderation effect estimates with either normal or non-normal data. When the moderation effect is non-zero, however, asymptotic bias may exist and is determined by factors such as the moderation effect size, missing-data proportion, and type of missingness dependence. Our analytical results suggest that researchers should apply NML to MMR models with caution when missing data exist. Suggestions are given regarding moderation analysis with missing data.     

Sensation seeking and estimation of numerosity

In a sample of 81 subjects, significant negative correlations were obtained between scores on Zuckerman's Sensation Seeking Scale and estimates of numerosity. A cognitive view of sensation seeking was suggested, according to which high scores are a function of an underestimation of the amount of stimulation experienced and low scores a function of overestimation. This approach was discussed in relation to motivational theories and those involving strength of the nervous system.