Recommended effect size statistics for repeated measures designs

Investigators, who are increasingly implored to present and discuss effect size statistics, might comply more often if they understood more clearly what is required. When investigators wish to report effect sizes derived from analyses of variance that include repeated measures, past advice has been problematic. Only recently has a generally useful effect size statistic been proposed for such designs: generalized eta squared (η_G²; Olejnik & Algina, 2003). Here, we present this method, explain that η_G² is preferred to eta squared and partial eta squared because it provides comparability across between-subjects and within-subjects designs, show that it can easily be computed from information provided by standard statistical packages, and recommend that investigators provide it routinely in their research reports when appropriate.     

方差分析效果大小报告的新指标

心理学期刊论文中可重复性不高的现象,原因之一是研究结果的效果普遍较小。并且在报告效果大小的文章中,效果大小指标使用不当。在方差分析中最经常报告的是η2和η2_p,但是在不同的研究设计中,这些效果大小是无法直接进行比较的。广义eta方(η²_G)是近年来新出现的一种新的效果大小指标,可克服的η2和η2_p不足,灵活处理重复测量等多种研究设计下个体差异的计算问题,实现跨研究设计效果大小的可比性。论文结合实例介绍了η²_G的原理和计算方法,并对其优缺点、使用和报告等问题进行了讨论。研究人员在报告效果大小时要考虑到不同的研究设计和研究假设,并选择恰当的指标防止过高估计效果大小。     

Eta Squared,Partial Eta Squared,and Misreporting of Effect Size in Communication Research

Communication researchers, along with social scientists from a variety of disciplines, are increasingly recognizing the importance of reporting effect sizes to augment significance tests. Serious errors in the reporting of effect sizes, however, have appeared in recently published articles. This article calls for accurate reporting of estimates of effect size. Eta squared (η²) is the most commonly reported estimate of effect sized for the ANOVA. The classical formulation of eta squared (Pearson, 1911; Fisher, 1928) is distinguished from the lesser known partial eta squared (Cohen, 1973), and a mislabeling problem in the statistical software SPSS (1998) is identified. What SPSS reports as eta squared is really partial eta squared. Hence, researchers obtaining estimates of eta squared from SPSS are at risk of reporting incorrect values. Several simulations are reported to demonstrate critical issues. The strengths and limitations of several estimates of effect size used in ANOVA are discussed, as are the implications of the reporting errors. A list of suggestions for researchers is then offered.     

THE ETA COEFFICIENT IN MANOVA

Methods for the calculation of eta squared have recently been presented for examining the strength of relationship in univariate analysis of variance. This paper extends them to the multivariate used in which the effects of independent variables may be examined in relation to two or more dependent variables and presents considerations for the calculation, use, and interpretation of the coefficients. The concepts of eta squared are generalized to the rnultivaziate case through the application of the multivariate general linear hypothesis Both orthogronal and nonorthogonal solutions are considered and an example of the former is presented.     

Bias and precision of some classical ANOVA effect sizes when assumptions are violated

Previous simulation research has focused on evaluating the impact of analytic assumption violations on statistics related to the F test and associated p _CALCULATED values. The present article evaluated the bias of classical estimates of practical significance (i.e., effect size sample estimators $ {\widehat{\eta}^2} $ , $ {\widehat{\varepsilon}^2} $ , and $ {\widehat{\omega}^2} $ ) in a one-way between-subjects univariate ANOVA when assumptions are violated. The simulation conditions modeled were selected on the basis of prior empirical research. Estimated (1) sampling error bias and (2) precision computed for each of the three effect size estimates for the 5,000 samples drawn for each of the 270 (5 parameter Cohen's d values × 3 group size ratios × 3 population distribution shapes × 3 variance ratios × 2 total ns) conditions were modeled for each of the k = 2, 3, and 4 group analyses. Our results corroborate the limited previous related research and suggest that $ {\widehat{\eta}^2} $ should not be used as an ANOVA effect size estimator, even though $ {\widehat{\eta}^2} $ is the only available choice in the menus in most commonly available software.     

Changes in Liberalism-Conservatism of College Students since the Depression

Summary

One hundred and twenty female subjects took part in an experiment to explore the necessity of deceiving subjects about the purpose of a conformity experiment. The subjects were randomly assigned to one of three groups: no information, partial information, or complete information about the purpose of the experiment. In addition, group pressure to conform was given to half of the subjects on trials on which the correct response was fairly obvious and to the other half on trials on which the correct response was somewhat ambiguous. Anova revealed a significant effect only for item difficulty. The statistic omega squared indicated that none of the variance in the conformity scores could be attributed to amount of information about the purpose of the experiment, either alone or in interaction with item difficulty. It was suggested that these results raise severe doubts about methodological assumptions commonly employed in social Psychology     

Negative estimate of variance-accounted-for effect size: How often it is obtained,and what happens if it is treated as zero

Researchers recommend reporting of bias-corrected variance-accounted-for effect size estimates such as omega squared instead of uncorrected estimates, because the latter are known for their tendency toward overestimation, whereas the former mostly correct this bias. However, this argument may miss an important fact: A bias-corrected estimate can take a negative value, and of course, a negative variance ratio does not make sense. Therefore, it has been a common practice to report an obtained negative estimate as zero. This article presents an argument against this practice, based on a simulation study investigating how often negative estimates are obtained and what are the consequences of treating them as zero. The results indicate that negative estimates are obtained more often than researchers might have thought. In fact, they occur more than half the time under some reasonable conditions. Moreover, treating the obtained negative estimates as zero causes substantial overestimation of even bias-corrected estimators when the sample size and population effect are not large, which is often the case in psychology. Therefore, the recommendation is that researchers report obtained negative estimates as is, instead of reporting them as zero, to avoid the inflation of effect sizes in research syntheses, even though zero can be considered the most plausible value when interpreting such a result. R code to reproduce all of the described results is included as supplemental material.     

Measures of effect size

Two different approaches have been used to derive measures of effect size. One approach is based on the comparison of treatment means. The standardized mean difference is an appropriate measure of effect size when one is merely comparing two treatments, but there is no satisfactory analogue for comparing more than two treatments. The second approach is based on the proportion of variance in the dependent variable that is explained by the independent variable. Estimates have been proposed for both fixed-factor and random-factor designs, but their sampling properties are not well understood. Nevertheless, measures of effect size can allow quantitative comparisons to be made across different studies, and they can be a useful adjunct to more traditional outcome measures such as test statistics and significance levels.     

What spelling tells us about the orthographic development and word study instruction with emergent bilingual secondary students

Educators need ways to assess orthographic knowledge and differentiate word study instruction for secondary, emergent bilingual learners. In this study, the spelling of 199 students in grades 7–12 across eight features and four spelling stages was examined to understand students' orthographic development; all but two were learning Spanish and English. Spelling data were collected three times over a nine-month period. There were significant changes across the time periods that show students progressing across four stages of spelling development, with a large effect size (partial eta squared = .498). Data from a standardized assessment of language proficiency were examined to study the relationship between students' orthographic knowledge and their language scores. This study suggests ways to use spelling assessments to plan differentiated vocabulary and word study instruction.     

客体特征统计属性表征机制的特异性

通过两个实验就视觉系统能否像计算平均数那样高效地计算其他统计量的问题进行了探讨。实验—保持—组圆的平均大小不变而改变众数,考察平均数估计是否受众数变化的影响;实验二采用与实验一相同的刺激,直接考察被试估计众数的绩效。结果发现：（1）对平均数的估计不受总体众数变化的影响;（2）对众数的估计往往不如对平均数的估计准确,估计值受平均数变化的影响;（3）在估计众数的任务中,被试成绩受总体极大值的影响。上述结果表明,视觉系统不存在针对众数的自动化加工机制。根据本实验的结果可以进一步推测,视觉系统并非对所有统计量都可做高效加工,而可能存在针对平均数的特异加工机制。     

Method Variance in Organizational Behavior and Human Resources Research: Effects on Correlations,Path Coefficients,and Hypothesis Testing

This research simulates the effects of method variance on correlations, standardized regression (path) coefficients, and squared multiple correlation coefficients. The results show that method variance can have extreme effects on these measures of association, depending on assumptions made about the nature of the method factors. The analysis also indicated that method variance can have strong effects on the probability of obtaining significant findings in the absence of true relationships. The link between findings of the present study and current developments on method variance in organizational behavior and human resources management research is discussed.     

Analysis and meta-analysis of single-case designs: An introduction

The last 10 years have seen great progress in the analysis and meta-analysis of single-case designs (SCDs). This special issue includes five articles that provide an overview of current work on that topic, including standardized mean difference statistics, multilevel models, Bayesian statistics, and generalized additive models. Each article analyzes a common example across articles and presents syntax or macros for how to do them. These articles are followed by commentaries from single-case design researchers and journal editors. This introduction briefly describes each article and then discusses several issues that must be addressed before we can know what analyses will eventually be best to use in SCD research. These issues include modeling trend, modeling error covariances, computing standardized effect size estimates, assessing statistical power, incorporating more accurate models of outcome distributions, exploring whether Bayesian statistics can improve estimation given the small samples common in SCDs, and the need for annotated syntax and graphical user interfaces that make complex statistics accessible to SCD researchers. The article then discusses reasons why SCD researchers are likely to incorporate statistical analyses into their research more often in the future, including changing expectations and contingencies regarding SCD research from outside SCD communities, changes and diversity within SCD communities, corrections of erroneous beliefs about the relationship between SCD research and statistics, and demonstrations of how statistics can help SCD researchers better meet their goals.     

Differences of Mean Scores on the Preliminary Scholastic Aptitude Test (PSAT) for Classical Christian Schools Compared to Non-Classical Christian Schools

Abstract

This causal-comparative quantitative study compares mean scores on the Preliminary Scholastic Aptitude Test (PSAT) between Classical Christian schools and non-Classical Christian schools using data randomly selected from survey responses. The sample consisted of 4,486 mean scores from the 2003–2004 school year through 2012–2013. Welch’s t-tests for unequal variances was used with an alpha set at 0.05 and 0.017 for Bonferroni correction and returned statistically significant results for all three academic areas: reading, math, and writing. Effect size measured by Cohen’s d and eta squared indicated Classical Christian methodology should have a large, positive effect on PSAT scores. Demographic and between school comparison limitations noted. Future research and associated correlational studies are recommended.     

Using generalizability theory to estimate the relative effect of class size and number of items on the dependability of student ratings of instruction

Previous research on the effect of class size on student ratings of instruction has primarily investigated the effect of class size on the favorableness of these ratings rather than its effect on their reliability (dependability). A few studies have used "generalizability theory" to demonstrate the relative effect of class size on the dependability of student ratings of instruction. The purpose of the present study was to test the validity of the findings of these studies in a different cultural setting using a different student ratings questionnaire. Using a random-effect analysis of variance to estimate the variance components for a design in which students were nested within classes and crossed with items, it was found that the variance component for class size was appreciably larger than that for items. At least 20 students were needed to obtain a generalizability coefficient for relative decisions of .70 or more. Increasing the number of students has a greater effect on generalizability coefficients than increasing the number of items.     

Some rao-guttman relationships

An examination of the determinantal equation associated with Rao's canonical factors suggests that Guttman's best lower bound for the number of common factors corresponds to the number of positive canonical correlations when squared multiple correlations are used as the initial estimates of communality. When these initial communality estimates are used, solving Rao's determinantal equation (at the first stage) permits expressing several matrices as functions of factors that differ only in the scale of their columns; these matrices include the correlation matrix with units in the diagonal, the correlation matrix with squared multiple correlations as communality estimates, Guttman's image covariance matrix, and Guttman's anti-image covariance matrix. Further, the factor scores associated with these factors can be shown to be either identical or simply related by a scale change. Implications for practice are discussed, and a computing scheme which would lead to an exhaustive analysis of the data with several optional outputs is outlined.     

The Influence of Differing the Paths to an Incentive on Third Graders’ Reading Achievement and Attitudes

The purpose of this study was to examine the impact of four incentive paths on third graders’ reading vocabulary and comprehension achievement and recreational and academic reading attitude. One hundred and twenty third-grade students were assigned to one of four incentive path treatment conditions. Data were analyzed using analysis of covariance (ANCOVA) with Fisher's least significant difference (LSD) post hoc comparisons. An effect size of .09 was calculated using a partial η² (eta squared) statistic. Findings indicated that the four incentive path treatment conditions did not differentially impact student vocabulary, comprehension, and recreational reading. A significant difference for students’ academic reading attitudes was identified across the four incentive path treatment conditions.     

Better than average: alternatives to statistical summary representations for rapid judgments of average size

People can perceive the individual features of an object by focusing attention on it and binding the features together at a location. Some perceptual processing can occur without focusing attention on each object, though; people may even be able to extract summary information about the sizes of all the objects in a display, essentially computing the mean size at a glance. Evidence that people can judge the mean size of an array efficiently and accurately has been used to support the strong claim that people use a global, parallel process to extract a statistical summary of the average size of the objects in the display. Such claims are based both on the accuracy of performance and on the supposition that performance exceeds what would be possible with serial, focused attention. However, these studies typically have not examined the limits of performance with focused-attention strategies. Through experiments and simulations, we show that existing evidence for mean size perception can be explained through various focused-attention strategies, without appealing to a new mechanism of average size perception. Although our evidence does not eliminate the possibility that people do perceive the average size of all the objects in a display, it suggests that simpler mechanisms can accommodate the existing data.     

Latin square analysis of variance using summary statistics

The method for computing a Latin square analysis of variance using summary statistics is presented.     

Scaling Variances of Latent Variables by Standardizing Loadings: Applications to Working Memory and the Position Effect

The standardization of loadings gives a metric to the corresponding latent variable and thus scales the variance of this latent variable. By assigning an appropriately estimated weight to all the loadings on the same latent variable it can be achieved that the average squared loading is 1 as the result of standardization. As a consequence, there is comparability of the variances of the latent variables of a confirmatory factor model. A precondition of comparability is that the latent variables must have loadings of the same manifest variables and that the variances are estimated with respect to the same covariance matrix. The usefulness of this standardization method is demonstrated by applying it for the evaluation of the sources of performance in a working memory task and for the evaluation of the impact of the position effect on performance in completing a reasoning measure. In these examples the scaled variances of the latent variables provided useful information.