Common Misconceptions Concerning The Analysis Of Covariance

Four misconceptions about the requirements for proper use of analysis of covariance (ANCOVA) are examined by means of Monte Carlo simulation. Conclusions are that ANCOVA does not require covariates to be measured without error, that ANCOVA can be used effectively to adjust for initial group differences that result from nonrandom assignment which is dependent on observed covariate scores, that ANCOVA does not provide unbiased estimates of true treatment effects where initial group differences are due to nonrandom assignment which is dependent on the true latent covariable if the covariate contains measurement error, and that ANCOVA requires no assumption concerning the equality of within-groups and between-groups regression. Where treatments actually influence covariate scores, the hypothesis tested by ANCOVA concerns a weighted combination of effects on covariate and dependent variables.     

ANCOVA Versus CHANGE From Baseline in Nonrandomized Studies: The Difference

The pretest-posttest control group design can be analyzed with the posttest as dependent variable and the pretest as covariate (ANCOVA) or with the difference between posttest and pretest as dependent variable (CHANGE). These 2 methods can give contradictory results if groups differ at pretest, a phenomenon that is known as Lord's paradox. Literature claims that ANCOVA is preferable if treatment assignment is based on randomization or on the pretest and questionable for preexisting groups. Some literature suggests that Lord's paradox has to do with measurement error in the pretest. This article shows two new things: First, the claims are confirmed by proving the mathematical equivalence of ANCOVA to a repeated measures model without group effect at pretest. Second, correction for measurement error in the pretest is shown to lead back to ANCOVA or to CHANGE, depending on the assumed absence or presence of a true group difference at pretest. These two new theoretical results are illustrated with multilevel (mixed) regression and structural equation modeling of data from two studies.     

A comparison of subset selection and analysis of covariance for the adjustment of confounders

Two common methods for adjusting group comparisons for differences in the distribution of confounders, namely analysis of covariance (ANCOVA) and subset selection, are compared using real examples from neuropsychology, theory, and simulations. ANCOVA has potential pitfalls, but the blanket rejection of the method in some areas of empirical psychology is not justified. Assumptions of the methods are reviewed, with issues of selection bias, nonlinearity, and interaction emphasized. Advantages of ANCOVA include better power, improved ability to detect and estimate interactions, and the availability of extensions to deal with measurement error in the covariates. Forms of ANCOVA are advocated that relax the standard assumption of linearity between the outcome and covariates. Specifically, a version of ANCOVA that models the relationship between the covariate and the outcome through cubic spline with fixed knots outperforms other methods in simulations.     

Power Analysis and Sample Size Planning in ANCOVA Designs

The analysis of covariance (ANCOVA) has notably proven to be an effective tool in a broad range of scientific applications. Despite the well-documented literature about its principal uses and statistical properties, the corresponding power analysis for the general linear hypothesis tests of treatment differences remains a less discussed issue. The frequently recommended procedure is a direct application of the ANOVA formula in combination with a reduced degrees of freedom and a correlation-adjusted variance. This article aims to explicate the conceptual problems and practical limitations of the common method. An exact approach is proposed for power and sample size calculations in ANCOVA with random assignment and multinormal covariates. Both theoretical examination and numerical simulation are presented to justify the advantages of the suggested technique over the current formula. The improved solution is illustrated with an example regarding the comparative effectiveness of interventions. In order to facilitate the application of the described power and sample size calculations, accompanying computer programs are also presented.

     

Using analysis of covariance (ANCOVA) with fallible covariates

Analysis of covariance (ANCOVA) is used widely in psychological research implementing nonexperimental designs. However, when covariates are fallible (i.e., measured with error), which is the norm, researchers must choose from among 3 inadequate courses of action: (a) know that the assumption that covariates are perfectly reliable is violated but use ANCOVA anyway (and, most likely, report misleading results); (b) attempt to employ 1 of several measurement error models with the understanding that no research has examined their relative performance and with the added practical difficulty that several of these models are not available in commonly used statistical software; or (c) not use ANCOVA at all. First, we discuss analytic evidence to explain why using ANCOVA with fallible covariates produces bias and a systematic inflation of Type I error rates that may lead to the incorrect conclusion that treatment effects exist. Second, to provide a solution for this problem, we conduct 2 Monte Carlo studies to compare 4 existing approaches for adjusting treatment effects in the presence of covariate measurement error: errors-in-variables (EIV; Warren, White, & Fuller, 1974), Lord's (1960) method, Raaijmakers and Pieters's (1987) method (R&P), and structural equation modeling methods proposed by S?rbom (1978) and Hayduk (1996). Results show that EIV models are superior in terms of parameter accuracy, statistical power, and keeping Type I error close to the nominal value. Finally, we offer a program written in R that performs all needed computations for implementing EIV models so that ANCOVA can be used to obtain accurate results even when covariates are measured with error.     

When does measurement error in covariates impact causal effect estimates? Analytic derivations of different scenarios and an empirical illustration

The average causal treatment effect (ATE) can be estimated from observational data based on covariate adjustment. Even if all confounding covariates are observed, they might not necessarily be reliably measured and may fail to obtain an unbiased ATE estimate. Instead of fallible covariates, the respective latent covariates can be used for covariate adjustment. But is it always necessary to use latent covariates? How well do analysis of covariance (ANCOVA) or propensity score (PS) methods estimate the ATE when latent covariates are used? We first analytically delineate the conditions under which latent instead of fallible covariates are necessary to obtain the ATE. Then we empirically examine the difference between ATE estimates when adjusting for fallible or latent covariates in an applied example. We discuss the issue of fallible covariates within a stochastic theory of causal effects and analyse data of a within-study comparison with recently developed ANCOVA and PS procedures that allow for latent covariates. We show that fallible covariates do not necessarily bias ATE estimates, but point out different scenarios in which adjusting for latent covariates is required. In our empirical application, we demonstrate how latent covariates can be incorporated for ATE estimation in ANCOVA and in PS analysis.     

Using the analysis of covariance to increase the power of priming experiments.

Although priming paradigms are widely used in cognitive psychology, the statistical analyses typically applied to priming data may not be optimal. Conceiving of priming paradigms as change-from-baseline designs suggests that the analysis of covariance (ANCOVA), using baseline performance as the covariate, is a more efficient (i.e., powerful) analysis. Specifically, ANCOVA provides more powerful tests of 1) the presence of priming and 2) between-group differences in priming. In addition, for within-subject designs with multiple baseline conditions, ANCOVA may increase the power of within-subjects effects. Efficiency gains are demonstrated with a re-analysis of priming datasets from implicit memory research. It is suggested that similar gains may be realized in other areas of priming research. Important assumptions of this procedure, which must be evaluated for the appropriate application of ANCOVA, are discussed.     

Misunderstanding analysis of covariance

Despite numerous technical treatments in many venues, analysis of covariance (ANCOVA) remains a widely misused approach to dealing with substantive group differences on potential covariates, particularly in psychopathology research. Published articles reach unfounded conclusions, and some statistics texts neglect the issue. The problem with ANCOVA in such cases is reviewed. In many cases, there is no means of achieving the superficially appealing goal of "correcting" or "controlling for" real group differences on a potential covariate. In hopes of curtailing misuse of ANCOVA and promoting appropriate use, a nontechnical discussion is provided, emphasizing a substantive confound rarely articulated in textbooks and other general presentations, to complement the mathematical critiques already available. Some alternatives are discussed for contexts in which ANCOVA is inappropriate or questionable.     

Measurement error and ANCOVA: Functional and structural relationship approaches

This article discusses alternative procedures to the standardF-test for ANCOVA in case the covariate is measured with error. Both a functional and a structural relationship approach are described. Examples of both types of analysis are given for the simple two-group design. Several cases are discussed and special attention is given to issues of model identifiability. An approximate statistical test based on the functional relationship approach is described. On the basis of Monte Carlo simulation results it is concluded that this testing procedure is to be preferred to the conventionalF-test of the ANCOVA null hypothesis. It is shown how the standard null hypothesis may be tested in a structural relationship approach. It is concluded that some knowledge of the reliability of the covariate is necessary in order to obtain meaningful results.     

Attentional effects on concurrent psychophysical discriminations: Investigations of a sample-size model

In two experiments, a concurrent discrimination paradigm was used to study the effects of visual attention on psychophysical judgments and the consistency of these effects with a sample-size model in which attention influences the variance of the internal representation used to make psychophysical judgments. Two pairs of lines were presented simultaneously—one on each side of fixation—and subjects had to indicate for each pair separately whether or not the lines had the same length. Attention was manipulated by instructing subjects to pay 100%, 75%, 50%, 25%, or 0% of their attention to the discrimination on one side, with the complementary amount of attention to the other side. In the first experiment, the relationship between attention and discrimination accuracy was consistent with the sample-size model both when attentional allocation varied from trial to trial and when it varied between blocks, and the relationship held over more widely varying attentional allocations than had previously been studied. In addition, discriminations were more accurate overall with varied than with blocked attentional allocation, suggesting that the two types of allocation do not merely differ in the degree to which attention is focused. The second experiment examined the effects of attentional allocation and stimulus variance, the latter being manipulated by randomly incrementing or decrementing line lengths. These manipulations had additive effects on total Thurstonian variance, and a version of the samplesize model gave an excellent quantitative fit to the obtained results. Besides supporting the samplesize model, the results of Experiment 2 suggest that criterion variance is at least as large as sensory variance and that criterion but not sensory variance increases with stimulus variance.     

The effect of covariate mean differences on the standard error and confidence interval for the comparison of treatment means

The use of covariates is commonly believed to reduce the unexplained error variance and the standard error for the comparison of treatment means, but the reduction in the standard error is neither guaranteed nor uniform over different sample sizes. The covariate mean differences between the treatment conditions can inflate the standard error of the covariate‐adjusted mean difference and can actually produce a larger standard error for the adjusted mean difference than that for the unadjusted mean difference. When the covariate observations are conceived of as randomly varying from one study to another, the covariate mean differences can be related to a Hotelling's T². Using this Hotelling's T² statistic, one can always find a minimum sample size to achieve a high probability of reducing the standard error and confidence interval width for the adjusted mean difference.     

Power analysis for multivariate and repeated measures designs: A flexible approach using the SPSS MANOVA procedure

Although power analysis is an important component in the planning and implementation of research designs, it is often ignored. Computer programs for performing power analysis are available, but most have limitations, particularly for complex multivariate designs. An SPSS procedure is presented that can be used for calculating power for univariate, multivariate, and repeated measures models with and without time-varying and time-constant covariates. Three examples provide a framework for calculating power via this method: an ANCOVA, a MANOVA, and a repeated measures ANOVA with two or more groups. The benefits and limitations of this procedure are discussed.     

Exposed eye area (EEA) in the expression of various emotions.

Exposed eye area (EEA) was measured in photographs of Indian adults who modeled six emotions--happiness, sadness, fear, anger, surprise, and disgust--as well as a neutral expression. The data were analyzed with a 2 x 6 (Eyes x Emotions) factorial analysis of covariance (ANCOVA). EEA for neutral expression was used as the covariate measure. The EEAs of the two eyes did not differ significantly during the expression of emotion. The EEAs for fear and surprise were significantly larger, and the EEA for disgust was significantly smaller than those for either other emotions or neutral expression.     

STATISTICAL POWER OF TRAINING EVALUATION DESIGNS

Sample size requirements needed to achieve various levels of statistical power using posttest-only, gain-score, and analysis of covariance designs in evaluating training interventions have been developed. Results are presented which indicate that the power to detect true effects differs according to the type of design, the correlation between the pre- and posttest, and the size of the effect due to the training program. We show that the type of design and correlations between the pre- and posttest complexly determine the power curve. Finally, an estimate of typical sample sizes used in training evaluation design has been determined and reviewed to determine the power of the various designs to detect true effects, given this sample-size specification. Recommendations for type of design are provided based on sample size and projected correlations between pre- and posttest scores.     

R2 effect-size measures for mediation analysis

R ² effect-size measures are presented to assess variance accounted for in mediation models. The measures offer a means to evaluate both component paths and the overall mediated effect in mediation models. Statistical simulation results indicate acceptable bias across varying parameter and sample-size combinations. The measures are applied to a real-world example using data from a team-based health promotion program to improve the nutrition and exercise habits of firefighters. SAS and SPSS computer code are also provided for researchers to compute the measures in their own data.     

The effect of performer gender,performer skill level,and opponent gender on self-confidence in a competitive situation

An equal number of male and female subjects (N=48), ranging in age from 17 to 26, were randomly assigned to compete in three competitive video games against a male or female opponent. All subjects were given bogus feedback that they had lost two out of three video games by a standard margin. Initial performance expectancies, as well as postcompetition expectancies, of all subjects were recorded. Initial performance expectancy scores recorded prior to competition were analyzed in a 2 (subject gender)×2 (opponent gender) analysis of covariance (ANCOVA) design with initial skill level on a preliminary game as the covariate. No significant gender differences in initial expectancy scores were found. A 2 (subject gender)×2 (opponent gender) ANCOVA design was utilized to analyze the postcompetition expectancy scores with initial performance expectancy as the covariate. The analysis revealed no significant differences. These findings did not support Corbin's (1981) data suggesting that females express significantly less self-confidence than males for future performance after competing against and losing to a superior opponent on a video task. Initial performance expectancies in the present study were significantly correlated (p.05) to skill level, indicating that performance expectancies may be more related to skill level than to gender. Thus, a realistic perception about one's initial skill level on a particular task may be the most salient determinant of performance expectancies.     

The distribution matters: two types of sample-size tasks

Different studies on how well people take sample size into account have found a wide range of solution rates. In a recent review, Sedlmeier and Gigerenzer (1997) suggested that a substantial part of the variation in results can be explained by the fact that experimenters have used two different types of sample-size tasks, one involving frequency distributions and the other sampling distributions. This suggestion rested on an analysis of studies that, with one exception, did not systematically manipulate type of distribution. In the research reported in this paper, well-known sample-size tasks were used to examine the hypothesis that frequency distribution versions of sample-size tasks yield higher solution rates than corresponding sampling distribution versions. In Study 1, a substantial difference between solution rates for the two types of task was found. Study 2 replicated this finding and ruled out an alternative explanation for it, namely, that the solution rate for sampling distribution tasks was lower because the information they contained was harder to extract than that in frequency distribution tasks. Finally, in Study 3 an attempt was made to reduce the gap between the solution rates for the two types of tasks by giving participants as many hints as possible for solving a sampling distribution task. Even with hints, the gap in performance remained. A new computational model of statistical reasoning specifies cognitive processes that might explain why people are better at solving frequency than sampling distribution tasks. Copyright© 1998 John Wiley & Sons, Ltd.     

Robustness of parameter and standard error estimates against ignoring a contextual effect of a subject-level covariate in cluster-randomized trials

In experimental research, it is not uncommon to assign clusters to conditions. When analysing the data of such cluster-randomized trials, a multilevel analysis should be applied in order to take into account the dependency of first-level units (i.e., subjects) within a second-level unit (i.e., a cluster). Moreover, the multilevel analysis can handle covariates on both levels. If a first-level covariate is involved, usually the within-cluster effect of this covariate will be estimated, implicitly assuming the contextual effect to be equal. However, this assumption may be violated. The focus of the present simulation study is the effects of ignoring the inequality of the within-cluster and contextual covariate effects on parameter and standard error estimates of the treatment effect, which is the parameter of main interest in experimental research. We found that ignoring the inequality of the within-cluster and contextual effects does not affect the estimation of the treatment effect or its standard errors. However, estimates of the variance components, as well as standard errors of the constant, were found to be biased.     

Matching methods for selection of subjects for follow-up

This work examines ways to make the best use of limited resources when selecting individuals to follow up in a longitudinal study estimating causal effects. In the setting under consideration, covariate information is available for all individuals but outcomes have not yet been collected and may be expensive to gather, and thus only a subset of the comparison subjects will be followed. Expressions in Rubin and Thomas (1996, 2000) show the benefits that can be obtained, in terms of reduced bias and variance of the estimated treatment effect, of selecting comparison individuals well-matched to those in the treated group, as compared to a random sample of comparison individuals. We primarily consider non-experimental settings but also consider implications for randomized trials. The methods are illustrated using data from the Johns Hopkins University Baltimore Prevention Program, which included data collection from age 6 to young adulthood of participants in an evaluation of two early elementary-school based universal prevention programs.     

Establishing normative data for repeated cognitive assessment: A comparison of different statistical methods

Serial cognitive assessment is conducted to monitor changes in the cognitive abilities of patients over time. At present, mainly the regression-based change and the ANCOVA approaches are used to establish normative data for serial cognitive assessment. These methods are straightforward, but they have some severe drawbacks. For example, they can only consider the data of two measurement occasions. In this article, we propose three alternative normative methods that are not hampered by these problems—that is, multivariate regression, the standard linear mixed model (LMM), and the linear mixed model combined with multiple imputation (LMM with MI) approaches. The multivariate regression method is primarily useful when a small number of repeated measurements are taken at fixed time points. When the data are more unbalanced, the standard LMM and the LMM with MI methods are more appropriate because they allow for a more adequate modeling of the covariance structure. The standard LMM has the advantage that it is easier to conduct and that it does not require a Monte Carlo component. The LMM with MI, on the other hand, has the advantage that it can flexibly deal with missing responses and missing covariate values at the same time. The different normative methods are illustrated on the basis of the data of a large longitudinal study in which a cognitive test (the Stroop Color Word Test) was administered at four measurement occasions (i.e., at baseline and 3, 6, and 12 years later). The results are discussed and suggestions for future research are provided.