Recommended effect size statistics for repeated measures designs

Investigators, who are increasingly implored to present and discuss effect size statistics, might comply more often if they understood more clearly what is required. When investigators wish to report effect sizes derived from analyses of variance that include repeated measures, past advice has been problematic. Only recently has a generally useful effect size statistic been proposed for such designs: generalized eta squared (η_G²; Olejnik & Algina, 2003). Here, we present this method, explain that η_G² is preferred to eta squared and partial eta squared because it provides comparability across between-subjects and within-subjects designs, show that it can easily be computed from information provided by standard statistical packages, and recommend that investigators provide it routinely in their research reports when appropriate.     

Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs

When a meta-analysis on results from experimental studies is conducted, differences in the study design must be taken into consideration. A method for combining results across independent-groups and repeated measures designs is described, and the conditions under which such an analysis is appropriate are discussed. Combining results across designs requires that (a) all effect sizes be transformed into a common metric, (b) effect sizes from each design estimate the same treatment effect, and (c) meta-analysis procedures use design-specific estimates of sampling variance to reflect the precision of the effect size estimates.     

Accounting for common method variance in cross-sectional research designs

Cross-sectional studies of attitude-behavior relationships are vulnerable to the inflation of correlations by common method variance (CMV). Here, a model is presented that allows partial correlation analysis to adjust the observed correlations for CMV contamination and determine if conclusions about the statistical and practical significance of a predictor have been influenced by the presence of CMV. This method also suggests procedures for designing questionnaires to increase the precision of this adjustment.     

A measure of effect size for experimental designs with heterogeneous variances

A recent trend in the psychological literature has been to include measures of effect size when reporting probability values. The several measures of effect size associated with the Student t test for two independent samples are appropriate only when the variances are homogeneous. In this paper, commonly used measures of effect size are considered and compared, using four data sets. A chance-corrected measure of effect size is provided for two or more treatment groups characterized by either homogeneous or heterogeneous variances.     

An effect size measure and Bayesian analysis of single-case designs

This article describes a linear modeling approach for the analysis of single-case designs (SCDs). Effect size measures in SCDs have been defined and studied for the situation where there is a level change without a time trend. However, when there are level and trend changes, effect size measures are either defined in terms of changes in R² or defined separately for changes in slopes and intercept coefficients. We propose an alternate effect size measure that takes into account changes in slopes and intercepts in the presence of serial dependence and provides an integrated procedure for the analysis of SCDs through estimation and inference based directly on the effect size measure. A Bayesian procedure is described to analyze the data and draw inferences in SCDs. A multilevel model that is appropriate when several subjects are available is integrated into the Bayesian procedure to provide a standardized effect size measure comparable to effect size measures in a between-subjects design. The applicability of the Bayesian approach for the analysis of SCDs is demonstrated through an example.     

Maximum-corrected and chance-corrected measures of effect size for the Mantel-Haenszel test

Two measures of effect size are described for the Mantel-Haenszel test. Both measures belong to the r-family of effect size measures. One measure is based on a maximum-corrected model, and the second measure is based on a chance-corrected model.     

Generalized sample size determination formulas for experimental research with hierarchical data

Hierarchical data sets arise when the data for lower units (e.g., individuals such as students, clients, and citizens) are nested within higher units (e.g., groups such as classes, hospitals, and regions). In data collection for experimental research, estimating the required sample size beforehand is a fundamental question for obtaining sufficient statistical power and precision of the focused parameters. The present research extends previous research from Heo and Leon (2008) and Usami (2011b), by deriving closed-form formulas for determining the required sample size to test effects in experimental research with hierarchical data, and by focusing on both multisite-randomized trials (MRTs) and cluster-randomized trials (CRTs). These formulas consider both statistical power and the width of the confidence interval of a standardized effect size, on the basis of estimates from a random-intercept model for three-level data that considers both balanced and unbalanced designs. These formulas also address some important results, such as the lower bounds of the needed units at the highest levels.     

Methods and measures in developmental emotions research: some assembly required

The remarkable contributors to this special issue highlight the importance of developmental research on emotion and its regulation, as well as its conceptual and methodological challenges. This commentary offers some additional thoughts, especially concerning alternative views of the convergence of multiple measures of emotional responding, the conceptualization of emotion and emotion regulation, and future directions for work in this field. In the end, in light of the complex construction of emotion and its development, we may learn from studying the divergence among multiple components of emotional responding as we do from expectations of their convergence. In each case, some assembly is required.     

The role of the couch in psychoanalysis: proposed research designs and some preliminary data

Ever since Freud, the couch has been viewed as an important--some would argue essential--component of psychoanalysis. Although many theoretical papers and case reports have addressed the use of the couch in psychoanalysis, no empirical study has investigated its effect on psychoanalytic process or outcome. After a review of the literature, a number of research designs are proposed that might be used in such an investigation. Finally, preliminary empirical data are presented from archived audiotapes of two psychoanalyses: one in which the patient switched from lying down to sitting up, and one in which the opposite occurred. The aim is to stimulate research-oriented psychoanalysts to undertake empirical investigations of the theoretical concepts underlying use of the couch and, more generally, to present a specific example of research as a paradigm for a broader research agenda for empirical investigation of the key theoretical ideas underlying psychoanalysis.     

Randomization tests for ABAB designs: comparing-data-division- specific and common distributions

Monte Carlo simulations were used to generate data for ABAB designs of different lengths. The points of change in phase are randomly determined before gathering behaviour measurements, which allows the use of a randomization test as an analytic technique. Data simulation and analysis can be based either on data-division-specific or on common distributions. Following one method or another affects the results obtained after the randomization test has been applied. Therefore, the goal of the study was to examine these effects in more detail. The discrepancies in these approaches are obvious when data with zero treatment effect are considered and such approaches have implications for statistical power studies. Data-division-specific distributions provide more detailed information about the performance of the statistical technique.     

Useful effect size interpretations for single case research

An obstacle to broader acceptability of effect sizes in single case research is their lack of intuitive and useful interpretations. Interpreting Cohen's d as "standard deviation units difference" and R2 as "percent of variance accounted for" do not resound with most visual analysts. In fact, the only comparative analysis widely supported in single case research (SCR) is "percent of nonoverlapping data." This article explores five alternative interpretations of Cohen's d and R2 effect sizes that may be more acceptable to the SCR field. They are: (a) Cohen's (Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum) "Percent of Nonoverlapping Data" (CPND), (b) Parker et al.'s (Parker, R.I., Cryer, J., Byrns, G., 2006. Controlling trend in single case research. School Psychology Quarterly, 21, 418-440, Parker, R.I., Hagan-Burke, S., Vannest, K., in press. Percent of all non-overlapping data (PAND): An alternative to PND. Journal of Special Education) "Percent of All Nonoverlapping Data" (PAND), (c) Rosenthal et al. (Rosenthal, R., Rosnow, R., & Rubin, D. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge: Cambridge University Press.) "Binomial Effect Size Display" (BESD), (d) "Percentile Rank in Control Group" (PR), and (e) McGraw and Wong's (McGraw, K. O., & Wong, S. P. (1992). A common language effect-size statistic. Psychological Bulletin, 111, 361-365) "Common Language Effect Size" (CLES). Each of the five interpretation schemes are applied to a published data set and are evaluated according to (a) intuitive appeal, (b) relevance to visual analysis, (c) ease of calculation, and (d) technical adequacy. Three of the five appear to be improvements over prevailing practice.     

Estimating the mean effect size in meta-analysis: Bias,precision, and mean squared error of different weighting methods

Although use of the standardized mean difference in meta-analysis is appealing for several reasons, there are some drawbacks. In this article, we focus on the following problem: that a precision-weighted mean of the observed effect sizes results in a biased estimate of the mean standardized mean difference. This bias is due to the fact that the weight given to an observed effect size depends on this observed effect size. In order to eliminate the bias, Hedges and Olkin (1985) proposed using the mean effect size estimate to calculate the weights. In the article, we propose a third alternative for calculating the weights: using empirical Bayes estimates of the effect sizes. In a simulation study, these three approaches are compared. The mean squared error (MSE) is used as the criterion by which to evaluate the resulting estimates of the mean effect size. For a meta-analytic dataset with a small number of studies, theMSE is usually smallest when the ordinary procedure is used, whereas for a moderate or large number of studies, the procedures yielding the best results are the empirical Bayes procedure and the procedure of Hedges and Olkin, respectively.     

Behavioral history: A definition and some common findings from two areas of research

Behavioral history research includes studies that (a) permit assessment of a prior experimental condition on a subsequent one, (b) show either short-lived or permanent effects, and (c) produce effects that are observable in ongoing behavior or that may be unobservable until special test conditions are introduced. We review experiments within both the conventional experimental analysis of behavior and behavioral pharmacology in order to identify commonalities and differences in the outcomes of conceptually similar experiments. We suggest that a deeper understanding of the necessary and sufficient conditions for producing history effects will emerge from these complementary research efforts.     

Effect of non-normality on test statistics for one-way independent groups designs

The data obtained from one‐way independent groups designs is typically non‐normal in form and rarely equally variable across treatment populations (i.e. population variances are heterogeneous). Consequently, the classical test statistic that is used to assess statistical significance (i.e. the analysis of variance F test) typically provides invalid results (e.g. too many Type I errors, reduced power). For this reason, there has been considerable interest in finding a test statistic that is appropriate under conditions of non‐normality and variance heterogeneity. Previously recommended procedures for analysing such data include the James test, the Welch test applied either to the usual least squares estimators of central tendency and variability, or the Welch test with robust estimators (i.e. trimmed means and Winsorized variances). A new statistic proposed by Krishnamoorthy, Lu, and Mathew, intended to deal with heterogeneous variances, though not non‐normality, uses a parametric bootstrap procedure. In their investigation of the parametric bootstrap test, the authors examined its operating characteristics under limited conditions and did not compare it to the Welch test based on robust estimators. Thus, we investigated how the parametric bootstrap procedure and a modified parametric bootstrap procedure based on trimmed means perform relative to previously recommended procedures when data are non‐normal and heterogeneous. The results indicated that the tests based on trimmed means offer the best Type I error control and power when variances are unequal and at least some of the distribution shapes are non‐normal.     

N-Watch: A program for deriving neighborhood size and other psycholinguistic statistics

This article describes a Windows program that enables users to obtain a broad range of statistics concerning the properties of word and nonword stimuli, including measures of word frequency, orthographic similarity, orthographic and phonological structure, age of acquisition, and imageability. It is designed for use by researchers in psycholinguistics, particularly those concerned with recognition of isolated words. The program computes measures of orthographic similarity on line, either with respect to a default vocabulary of 30,605 words or to a vocabulary specified by the user. In addition to providing standard orthographic neighborhood measures, the program can be used to obtain information about other forms of orthographic similarity, such as transposed-letter similarity and embedded-word similarity. It is available, free of charge, from the following Web site: http://www.maccs.mq.edu.au/≈colin/N-Watch/.     

Planned missing-data designs in experience-sampling research: Monte Carlo simulations of efficient designs for assessing within-person constructs

Experience-sampling research involves trade-offs between the number of questions asked per signal, the number of signals per day, and the number of days. By combining planned missing-data designs and multilevel latent variable modeling, we show how to reduce the items per signal without reducing the number of items. After illustrating different designs using real data, we present two Monte Carlo studies that explored the performance of planned missing-data designs across different within-person and between-person sample sizes and across different patterns of response rates. The missing-data designs yielded unbiased parameter estimates but slightly higher standard errors. With realistic sample sizes, even designs with extensive missingness performed well, so these methods are promising additions to an experience-sampler’s toolbox.     

A priori tests in repeated measures designs: Effects of nonsphericity

The validity conditions for univariate repeated measures designs are described. Attention is focused on the sphericity requirement. For av degree of freedom family of comparisons among the repeated measures, sphericity exists when all contrasts contained in thev dimensional space have equal variances. Under nonsphericity, upper and lower bounds on test size and power of a priori, repeated measures,F tests are derived. The effects of nonsphericity are illustrated by means of a set of charts. The charts reveal that small departures from sphericity (.97 <1.00) can seriously affect test size and power. It is recommended that separate rather than pooled error term procedures be routinely used to test a priori hypotheses.Appreciation is extended to Milton Parnes for his insightful assistance.