Power and Sample Size Calculations for Contrast Analysis in ANCOVA

Analysis of covariance (ANCOVA) is commonly used in behavioral and educational research to reduce the error variance and improve the power of analysis of variance by adjusting the covariate effects. For planning and evaluating randomized ANCOVA designs, a simple sample-size formula has been proposed to account for the variance deflation factor in the comparison of two treatment groups. The objective of this article is to highlight an overlooked and potential problem of the exiting approximation and to provide an alternative and exact solution of power and sample size assessments for testing treatment contrasts. Numerical investigations are conducted to reveal the relative performance of the two procedures as a reliable technique to accommodate the covariate features that make ANCOVA design particularly distinctive. The described approach has important advantages over the current method in general applicability, methodological justification, and overall accuracy. To enhance the practical usefulness, computer algorithms are presented to implement the recommended power calculations and sample-size determinations.     

Sample Size Planning for Detecting Mediation Effects: A Power Analysis Procedure Considering Uncertainty in Effect Size Estimates

Abstract

When planning mediation studies, researchers are often interested in the sample size needed to achieve adequate power for testing mediation. Power depends on population effect sizes, which are unknown in practice. In conventional power analysis, effect size estimates, however, are often used as population values, which could result in underpowered studies. Uncertainty in effect size estimates has been considered in other sample size planning contexts (e.g., t-test, ANOVA), but has not been handled properly for planning mediation studies. In the current study, we proposed an easy-to-use sample size planning method for testing mediation with uncertainty in effect size estimates considered. We conducted simulation studies to demonstrate the impact of uncertainty in effect size estimates on power of testing mediation, and to provide sample size suggestions under different levels of uncertainty. Empirical examples were provided to illustrate the application of our method. R functions and a web application were developed to facilitate implementation.     

Abstract: Sample Size Planning for Latent Curve Models

When designing a study that uses structural equation modeling (SEM), an important task is to decide an appropriate sample size. Historically, this task is approached from the power analytic perspective, where the goal is to obtain sufficient power to reject a false null hypothesis. However, hypothesis testing only tells if a population effect is zero and fails to address the question about the population effect size. Moreover, significance tests in the SEM context often reject the null hypothesis too easily, and therefore the problem in practice is having too much power instead of not enough power.

An alternative means to infer the population effect is forming confidence intervals (CIs). A CI is more informative than hypothesis testing because a CI provides a range of plausible values for the population effect size of interest. Given the close relationship between CI and sample size, the sample size for an SEM study can be planned with the goal to obtain sufficiently narrow CIs for the population model parameters of interest.

Latent curve models (LCMs) is an application of SEM with mean structure to studying change over time. The sample size planning method for LCM from the CI perspective is based on maximum likelihood and expected information matrix. Given a sample, to form a CI for the model parameter of interest in LCM, it requires the sample covariance matrix S, sample mean vector , and sample size N. Therefore, the width (w) of the resulting CI can be considered a function of S, , and N. Inverting the CI formation process gives the sample size planning process. The inverted process requires a proxy for the population covariance matrix Σ, population mean vector μ, and the desired width ω as input, and it returns N as output. The specification of the input information for sample size planning needs to be performed based on a systematic literature review. In the context of covariance structure analysis, Lai and Kelley (2011) discussed several practical methods to facilitate specifying Σ and ω for the sample size planning procedure.     

Exact Interval Estimation, Power Calculation, and Sample Size Determination in Normal Correlation Analysis

This paper considers the problem of analysis of correlation coefficients from a multivariate normal population. A unified theorem is derived for the regression model with normally distributed explanatory variables and the general results are employed to provide useful expressions for the distributions of simple, multiple, and partial-multiple correlation coefficients. The inversion principle and monotonicity property of the proposed formulations are used to describe alternative approaches to the exact interval estimation, power calculation, and sample size determination for correlation coefficients. The author thanks the referees for their constructive comments and helpful suggestions and especially the associate editor for drawing attention to several critical results which led to substantial improvements of the exposition. The work for this paper was initiated while the author was visiting the Department of Statistics, Stanford University. This research was partially supported by National Science Council Grant NSC-94-2118-M-009-004. Request for reprints should be sent to Gwowen Shieh, Department of Management Science, National Chiao Tung University, Hsinchu, Taiwan 30050, ROC.     

Sample Size in Factor Analysis: The Role of Model Error

This article examines effects of sample size and other design features on correspondence between factors obtained from analysis of sample data and those present in the population from which the samples were drawn. We extend earlier work on this question by examining these phenomena in the situation in which the common factor model does not hold exactly in the population. We present a theoretical framework for representing such lack of fit and examine its implications in the population and sample. Based on this approach we hypothesize that lack of fit of the model in the population will not, on the average, influence recovery of population factors in analysis of sample data, regardless of degree of model error and regardless of sample size. Rather, such recovery will be affected only by phenomena related to sampling error which have been studied previously. These hypotheses are investigated and verified in two sampling studies, one using artificial data and one using empirical data.     

A Unified Approach to Power Calculation and Sample Size Determination for Random Regression Models

The underlying statistical models for multiple regression analysis are typically attributed to two types of modeling: fixed and random. The procedures for calculating power and sample size under the fixed regression models are well known. However, the literature on random regression models is limited and has been confined to the case of all variables having a joint multivariate normal distribution. This paper presents a unified approach to determining power and sample size for random regression models with arbitrary distribution configurations for explanatory variables. Numerical examples are provided to illustrate the usefulness of the proposed method and Monte Carlo simulation studies are also conducted to assess the accuracy. The results show that the proposed method performs well for various model specifications and explanatory variable distributions. The author would like to thank the editor, the associate editor, and the referees for drawing attention to pertinent references that led to improved presentation. This research was partially supported by National Science Council grant NSC-94-2118-M-009-004.     

Sample Size Planning for the Squared Multiple Correlation Coefficient: Accuracy in Parameter Estimation via Narrow Confidence Intervals

Methods of sample size planning are developed from the accuracy in parameter approach in the multiple regression context in order to obtain a sufficiently narrow confidence interval for the population squared multiple correlation coefficient when regressors are random. Approximate and exact methods are developed that provide necessary sample size so that the expected width of the confidence interval will be sufficiently narrow. Modifications of these methods are then developed so that necessary sample size will lead to sufficiently narrow confidence intervals with no less than some desired degree of assurance. Computer routines have been developed and are included within the MBESS R package so that the methods discussed in the article can be implemented. The methods and computer routines are demonstrated using an empirical example linking innovation in the health services industry with previous innovation, personality factors, and group climate characteristics.     

Effect sizes in ANCOVA and difference-in-differences designs

It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate and—in non-randomized designs—its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thereby making it comparable to other interventions and studies. Curiously, the estimation of this effect size, including covariate adjustment, has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.     

Sample Size Determination for Rasch Model Tests

This paper is concerned with supplementing statistical tests for the Rasch model so that additionally to the probability of the error of the first kind (Type I probability) the probability of the error of the second kind (Type II probability) can be controlled at a predetermined level by basing the test on the appropriate number of observations. An approach to determining a practically meaningful extent of model deviation is proposed, and the approximate distribution of the Wald test is derived under the extent of model deviation of interest.     

Measurement error and ANCOVA: Functional and structural relationship approaches

This article discusses alternative procedures to the standardF-test for ANCOVA in case the covariate is measured with error. Both a functional and a structural relationship approach are described. Examples of both types of analysis are given for the simple two-group design. Several cases are discussed and special attention is given to issues of model identifiability. An approximate statistical test based on the functional relationship approach is described. On the basis of Monte Carlo simulation results it is concluded that this testing procedure is to be preferred to the conventionalF-test of the ANCOVA null hypothesis. It is shown how the standard null hypothesis may be tested in a structural relationship approach. It is concluded that some knowledge of the reliability of the covariate is necessary in order to obtain meaningful results.     

Minimum Sample Size Recommendations for Conducting Factor Analyses

There is no shortage of recommendations regarding the appropriate sample size to use when conducting a factor analysis. Suggested minimums for sample size include from 3 to 20 times the number of variables and absolute ranges from 100 to over 1,000. For the most part, there is little empirical evidence to support these recommendations. This simulation study addressed minimum sample size requirements for 180 different population conditions that varied in the number of factors, the number of variables per factor, and the level of communality. Congruence coefficients were calculated to assess the agreement between population solutions and sample solutions generated from the various population conditions. Although absolute minimums are not presented, it was found that, in general, minimum sample sizes appear to be smaller for higher levels of communality; minimum sample sizes appear to be smaller for higher ratios of the number of variables to the number of factors; and when the variables-to-factors ratio exceeds 6, the minimum sample size begins to stabilize regardless of the number of factors or the level of communality.     

知道与不知道要学多少：类别学习中样例量的预期作用

采用“5/4模型”类别结构探讨了类别学习中样例量的预期作用。设置了两种学习条件(“知道样例量”和“不知道样例量”), 分别探讨两种学习条件下的学习效率、学习策略以及所形成的类别表征。106名大学生参加了实验, 结果表明：在类别学习中, 样例量的预期作用显著, 知道样例量组的学习效率高于不知道样例量组; 样例量的预期作用对类别学习效率的影响是通过影响学习过程中使用的策略来实现的; 样例量的预期作用不影响两种学习条件的学习后形成的类别表征, 且两种学习条件的被试自始至终表现出样例学习的表征模式。     

Analysis Of Quasi-Experimental Time-Series Designs

Multivariate procedures, based on the general linear hypothesis, are outlined for inferring treatment effects in quasi-experimental time-series designs. These procedures, which take into account the dependent nature of the data obtained in time-series experiments and which require comparing the fitted regression curves for the pre-treatment and post-treatment observations, provide exact tests of significance for the various hypotheses of interest.     

ANCOVA Versus CHANGE From Baseline in Nonrandomized Studies: The Difference

The pretest-posttest control group design can be analyzed with the posttest as dependent variable and the pretest as covariate (ANCOVA) or with the difference between posttest and pretest as dependent variable (CHANGE). These 2 methods can give contradictory results if groups differ at pretest, a phenomenon that is known as Lord's paradox. Literature claims that ANCOVA is preferable if treatment assignment is based on randomization or on the pretest and questionable for preexisting groups. Some literature suggests that Lord's paradox has to do with measurement error in the pretest. This article shows two new things: First, the claims are confirmed by proving the mathematical equivalence of ANCOVA to a repeated measures model without group effect at pretest. Second, correction for measurement error in the pretest is shown to lead back to ANCOVA or to CHANGE, depending on the assumed absence or presence of a true group difference at pretest. These two new theoretical results are illustrated with multilevel (mixed) regression and structural equation modeling of data from two studies.     

A Bayesian Power Analysis Procedure Considering Uncertainty in Effect Size Estimates from a Meta-analysis

In conventional frequentist power analysis, one often uses an effect size estimate, treats it as if it were the true value, and ignores uncertainty in the effect size estimate for the analysis. The resulting sample sizes can vary dramatically depending on the chosen effect size value. To resolve the problem, we propose a hybrid Bayesian power analysis procedure that models uncertainty in the effect size estimates from a meta-analysis. We use observed effect sizes and prior distributions to obtain the posterior distribution of the effect size and model parameters. Then, we simulate effect sizes from the obtained posterior distribution. For each simulated effect size, we obtain a power value. With an estimated power distribution for a given sample size, we can estimate the probability of reaching a power level or higher and the expected power. With a range of planned sample sizes, we can generate a power assurance curve. Both the conventional frequentist and our Bayesian procedures were applied to conduct prospective power analyses for two meta-analysis examples (testing standardized mean differences in example 1 and Pearson's correlations in example 2). The advantages of our proposed procedure are demonstrated and discussed.     

Bottom-Up Analysis of Single-Case Research Designs

This paper defines and promotes the qualities of a ??bottom-up?? approach to single-case research (SCR) data analysis. Although ??top-down?? models, for example, multi-level or hierarchical linear models, are gaining momentum and have much to offer, interventionists should be cautious about analyses that are not easily understood, are not governed by a ??wide lens?? visual analysis, do not yield intuitive results, and remove the analysis process from the interventionist, who alone has intimate understanding of the design logic and resulting data patterns. ??Bottom-up?? analysis possesses benefits which fit well with SCR, including applicability to designs with few data points and few phases, customization of analyses based on design and data idiosyncrasies, conformation with visual analysis, and directly meaningful effect sizes. Examples are provided to illustrate these benefits of bottom-up analyses.     

Sample Size Requirements for the Capron and Duyme Balanced Fostering Study of IQ

A simplified method is used to estimate the appropriate sample sizes needed to detect main effects and an interaction effect in analysis of variance, using the IQ data from the Capron and Duyme (1991) adoption study as an example. To achieve power of 80% to reject an hypothesis of no interaction when there is in reality a modest interaction requires about 215 children in each of four groups in a 2 × 2 design, whereas only 9 to 10 children per group are needed to detect main effects. Only a transnational collaborative study could hope to find this many children in the condition where a child from high socioeconomic status background is adopted into a low status family.     

ANOVA and ANCOVA of pre- and post-test,ordinal data

With random assignment to treatments and standard assumptions, either a one-way ANOVA of post-test scores or a two-way, repeated measures ANOVA of pre- and post-test scores provides a legitimate test of the equal treatment effect null hypothesis for latent variable . In an ANCOVA for pre- and post-test variablesX andY which are ordinal measures of and , respectively, random assignment and standard assumptions ensure the legitimacy of inferences about the equality of treatment effects on latent variable . Sample estimates of adjustedY treatment means are ordinal estimators of adjusted post-test means on latent variable .     

Sample Size Calculation for Estimating or Testing a Nonzero Squared Multiple Correlation Coefficient

The problems of hypothesis testing and interval estimation of the squared multiple correlation coefficient of a multivariate normal distribution are considered. It is shown that available one-sided tests are uniformly most powerful, and the one-sided confidence intervals are uniformly most accurate. An exact method of calculating sample size to carry out one-sided tests (null hypothesis may involve a nonzero value for the multiple correlation coefficient) to attain a specified power is given. Sample size calculation for computing confidence intervals for the squared multiple correlation coefficient with a specified expected width is also provided. Sample sizes for powers and confidence intervals are tabulated for a wide range of parameter configurations and dimensions. The results are illustrated using the empirical data from Timm (1975) Timm, N. H. 1975. Multivariate analysis with applications in education and psychology, Belmont, CA: Wadsworth.  [Google Scholar] that related scores from the Peabody Picture Vocabulary Test to four proficiency measures.