Common Misconceptions Concerning The Analysis Of Covariance

Four misconceptions about the requirements for proper use of analysis of covariance (ANCOVA) are examined by means of Monte Carlo simulation. Conclusions are that ANCOVA does not require covariates to be measured without error, that ANCOVA can be used effectively to adjust for initial group differences that result from nonrandom assignment which is dependent on observed covariate scores, that ANCOVA does not provide unbiased estimates of true treatment effects where initial group differences are due to nonrandom assignment which is dependent on the true latent covariable if the covariate contains measurement error, and that ANCOVA requires no assumption concerning the equality of within-groups and between-groups regression. Where treatments actually influence covariate scores, the hypothesis tested by ANCOVA concerns a weighted combination of effects on covariate and dependent variables.     

F-test bias for experimental designs in educational research

Reference is made to Neyman's study ofF-test bias for the randomized blocks and Latin square designs employed in agriculture, and some account is given of later statistical developments which sprang from his work—in particular, the classification of model-types and the technique of variance component analysis. It is claimed that there is a need to carry out an examination ofF-test bias for experimental designs in education and psychology which will utilize the method and, where appropriate, the known' results of this new branch of variance analysis. In the present paper, such an investigation is carried out for designs which may be regarded as derivatives of the agricultural randomized blocks design. In a paper to follow, a similar investigation will be carried out for experimental designs of the Latin square type.     

Power and Sample Size Calculations for Contrast Analysis in ANCOVA

Analysis of covariance (ANCOVA) is commonly used in behavioral and educational research to reduce the error variance and improve the power of analysis of variance by adjusting the covariate effects. For planning and evaluating randomized ANCOVA designs, a simple sample-size formula has been proposed to account for the variance deflation factor in the comparison of two treatment groups. The objective of this article is to highlight an overlooked and potential problem of the exiting approximation and to provide an alternative and exact solution of power and sample size assessments for testing treatment contrasts. Numerical investigations are conducted to reveal the relative performance of the two procedures as a reliable technique to accommodate the covariate features that make ANCOVA design particularly distinctive. The described approach has important advantages over the current method in general applicability, methodological justification, and overall accuracy. To enhance the practical usefulness, computer algorithms are presented to implement the recommended power calculations and sample-size determinations.     

F-test bias for experimental designs of the Latin square type

In an earlier paper, a method of analysis, due to Neyman and now known generally as variance component analysis, was used to examineF-test bias for experimental designs in education of the randomized block type. The same method is now applied to studyF-test bias for designs of the Latin square type. The results, in general, disprove the view that, for a valid application of Latin square techniques, it is necessary that all interactions are zero.     

ANCOVA Versus CHANGE From Baseline in Nonrandomized Studies: The Difference

The pretest-posttest control group design can be analyzed with the posttest as dependent variable and the pretest as covariate (ANCOVA) or with the difference between posttest and pretest as dependent variable (CHANGE). These 2 methods can give contradictory results if groups differ at pretest, a phenomenon that is known as Lord's paradox. Literature claims that ANCOVA is preferable if treatment assignment is based on randomization or on the pretest and questionable for preexisting groups. Some literature suggests that Lord's paradox has to do with measurement error in the pretest. This article shows two new things: First, the claims are confirmed by proving the mathematical equivalence of ANCOVA to a repeated measures model without group effect at pretest. Second, correction for measurement error in the pretest is shown to lead back to ANCOVA or to CHANGE, depending on the assumed absence or presence of a true group difference at pretest. These two new theoretical results are illustrated with multilevel (mixed) regression and structural equation modeling of data from two studies.     

The variable-criteria sequential stopping rule: Generality to unequal sample sizes,unequal variances,or to large ANOVAs

The variable-criteria sequential stopping rule (SSR) is a method for conducting planned experiments in stages after the addition of new subjects until the experiment is stopped because the p value is less than or equal to a lower criterion and the null hypothesis has been rejected, the p value is above an upper criterion, or a maximum sample size has been reached. Alpha is controlled at the expected level. The table of stopping criteria has been validated for a t test or ANOVA with four groups. New simulations in this article demonstrate that the SSR can be used with unequal sample sizes or heterogeneous variances in a t test. As with the usual t test, the use of a separate-variance term instead of a pooled-variance term prevents an inflation of alpha with heterogeneous variances. Simulations validate the original table of criteria for up to 20 groups without a drift of alpha. When used with a multigroup ANOVA, a planned contrast can be substituted for the global F as the focus for the stopping rule. The SSR is recommended when significance tests are appropriate and when the null hypothesis can be tested in stages. Because of its efficiency, the SSR should be used instead of the usual approach to the t test or ANOVA when subjects are expensive, rare, or limited by ethical considerations such as pain or distress.     

Comparing groups in a before–after design: When t test and ANCOVA produce different results

Background. Researchers often test people before and after some treatment and compare these scores with a control group. Sometimes it is not possible to allocate people into conditions randomly, which means the initial scores for the two groups may differ. There are two main approaches: t test on the gain scores and ANCOVA partialling out the initial scores. Lord (1967) showed that these can lead to different conclusions. This is an often‐discussed paradox in psychology and education. Aims. The reasons why these approaches can lead to different conclusions, the assumptions that each approach makes and how the approaches relate to group allocation, are discussed Methods. Three sets of simulations are reported that investigate the relationships among effect size, group allocation, measurement error and Lord's paradox. Conclusions. Recommendations are given that stress careful examination of the research questions, sampling and allocation of participants and graphing the data. ANCOVA is appropriate when allocation is based on the initial scores, t test can be appropriate if allocation is associated non‐causally with the initial scores, but often neither approach provides adequate results.     

A note on consistency of non-parametric rank tests and related rank transformations

The extent to which rank transformations result in the same statistical decisions as their non‐parametric counterparts is investigated. Simulations are presented using the Wilcoxon–Mann–Whitney test, the Wilcoxon signed‐rank test and the Kruskal–Wallis test, together with the rank transformations and t and F tests corresponding to each of those non‐parametric methods. In addition to Type I errors and power over all simulations, the study examines the consistency of the outcomes of the two methods on each individual sample. The results show how acceptance or rejection of the null hypothesis and differences in p‐values of the test statistics depend in a regular and predictable way on sample size, significance level, and differences between means, for normal and various non‐normal distributions.     

A comparison of subset selection and analysis of covariance for the adjustment of confounders

Two common methods for adjusting group comparisons for differences in the distribution of confounders, namely analysis of covariance (ANCOVA) and subset selection, are compared using real examples from neuropsychology, theory, and simulations. ANCOVA has potential pitfalls, but the blanket rejection of the method in some areas of empirical psychology is not justified. Assumptions of the methods are reviewed, with issues of selection bias, nonlinearity, and interaction emphasized. Advantages of ANCOVA include better power, improved ability to detect and estimate interactions, and the availability of extensions to deal with measurement error in the covariates. Forms of ANCOVA are advocated that relax the standard assumption of linearity between the outcome and covariates. Specifically, a version of ANCOVA that models the relationship between the covariate and the outcome through cubic spline with fixed knots outperforms other methods in simulations.     

The holophrastic hypothesis: Conceptual and empirical issues

A conceptual distinction is drawn between a structural and a functional version of the holophrastic hypothesis. The structural version of this hypothesis views the single-word utterances of children as implicit expressions of either syntactic or semantic structural relations, while the functional version views each of these utterances as consisting of a single lexical item which is used for a particular communicative function. The arguments which have been proposed in favour of these two versions of the hypothesis are critically examined in the light of the empirical evidence which is currently available. It is concluded that this evidence only supports the functional version of the holophrastic hypothesis, there being no evidence available to support the interpretation of children's single-word utterances as implicit expressions of either syntactic or semantic relations.     

The effect of performer gender,performer skill level,and opponent gender on self-confidence in a competitive situation

An equal number of male and female subjects (N=48), ranging in age from 17 to 26, were randomly assigned to compete in three competitive video games against a male or female opponent. All subjects were given bogus feedback that they had lost two out of three video games by a standard margin. Initial performance expectancies, as well as postcompetition expectancies, of all subjects were recorded. Initial performance expectancy scores recorded prior to competition were analyzed in a 2 (subject gender)×2 (opponent gender) analysis of covariance (ANCOVA) design with initial skill level on a preliminary game as the covariate. No significant gender differences in initial expectancy scores were found. A 2 (subject gender)×2 (opponent gender) ANCOVA design was utilized to analyze the postcompetition expectancy scores with initial performance expectancy as the covariate. The analysis revealed no significant differences. These findings did not support Corbin's (1981) data suggesting that females express significantly less self-confidence than males for future performance after competing against and losing to a superior opponent on a video task. Initial performance expectancies in the present study were significantly correlated (p.05) to skill level, indicating that performance expectancies may be more related to skill level than to gender. Thus, a realistic perception about one's initial skill level on a particular task may be the most salient determinant of performance expectancies.     

To what extent can communalities reduce rank?

The question is raised as to whether the null hypothesis concerning the number of common factors underlying a given set of correlations should be that this number is small. Psychological and algebraic evidence indicate that a more appropriate null hypothesis is that the number is relatively large, and that smallness should be but an alternative hypothesis. The question is also raised as to why approximation procedures should be aimed primarily at the observed correlation matrixR and not at, say,R ^–1. What may be best forR may be worst forR ^–1, and conversely, yetR ^–1 is directly involved in problems of multiple and partial regressions. It is shown that a widely accepted inequality for the possible rank to whichR can be reduced, when modified by communalities, is indeed false.This research was facilitated by a noncommitted grant-in-aid to the writer from the Ford Foundation.     

Power Analysis and Sample Size Planning in ANCOVA Designs

The analysis of covariance (ANCOVA) has notably proven to be an effective tool in a broad range of scientific applications. Despite the well-documented literature about its principal uses and statistical properties, the corresponding power analysis for the general linear hypothesis tests of treatment differences remains a less discussed issue. The frequently recommended procedure is a direct application of the ANOVA formula in combination with a reduced degrees of freedom and a correlation-adjusted variance. This article aims to explicate the conceptual problems and practical limitations of the common method. An exact approach is proposed for power and sample size calculations in ANCOVA with random assignment and multinormal covariates. Both theoretical examination and numerical simulation are presented to justify the advantages of the suggested technique over the current formula. The improved solution is illustrated with an example regarding the comparative effectiveness of interventions. In order to facilitate the application of the described power and sample size calculations, accompanying computer programs are also presented.

     

Misunderstanding analysis of covariance

Despite numerous technical treatments in many venues, analysis of covariance (ANCOVA) remains a widely misused approach to dealing with substantive group differences on potential covariates, particularly in psychopathology research. Published articles reach unfounded conclusions, and some statistics texts neglect the issue. The problem with ANCOVA in such cases is reviewed. In many cases, there is no means of achieving the superficially appealing goal of "correcting" or "controlling for" real group differences on a potential covariate. In hopes of curtailing misuse of ANCOVA and promoting appropriate use, a nontechnical discussion is provided, emphasizing a substantive confound rarely articulated in textbooks and other general presentations, to complement the mathematical critiques already available. Some alternatives are discussed for contexts in which ANCOVA is inappropriate or questionable.     

Using analysis of covariance (ANCOVA) with fallible covariates

Analysis of covariance (ANCOVA) is used widely in psychological research implementing nonexperimental designs. However, when covariates are fallible (i.e., measured with error), which is the norm, researchers must choose from among 3 inadequate courses of action: (a) know that the assumption that covariates are perfectly reliable is violated but use ANCOVA anyway (and, most likely, report misleading results); (b) attempt to employ 1 of several measurement error models with the understanding that no research has examined their relative performance and with the added practical difficulty that several of these models are not available in commonly used statistical software; or (c) not use ANCOVA at all. First, we discuss analytic evidence to explain why using ANCOVA with fallible covariates produces bias and a systematic inflation of Type I error rates that may lead to the incorrect conclusion that treatment effects exist. Second, to provide a solution for this problem, we conduct 2 Monte Carlo studies to compare 4 existing approaches for adjusting treatment effects in the presence of covariate measurement error: errors-in-variables (EIV; Warren, White, & Fuller, 1974), Lord's (1960) method, Raaijmakers and Pieters's (1987) method (R&P), and structural equation modeling methods proposed by S?rbom (1978) and Hayduk (1996). Results show that EIV models are superior in terms of parameter accuracy, statistical power, and keeping Type I error close to the nominal value. Finally, we offer a program written in R that performs all needed computations for implementing EIV models so that ANCOVA can be used to obtain accurate results even when covariates are measured with error.     

Paranormal belief and the conjunction fallacy: Controlling for temporal relatedness and potential surprise differentials in component events

Recent research suggests paranormal believers are especially prone to the ‘conjunction fallacy’. The current study extends this work by presenting believers and non‐believers with eight paranormal plus eight non‐paranormal scenarios. Participants were given either a paranormal or virtually identical non‐paranormal version of each scenario. Of these, half incorporated component events which were (virtually) co‐occurring with half including components which were temporally disjointed. Analysis of Covariance (ANCOVA; controlling for gender and maths/stats/psychology qualifications) found believers made more conjunction errors than non‐believers. Neither event type (paranormal vs. non‐paranormal) nor components' temporal relationship (co‐occurring vs. disjointed) had a significant effect on conjunction biases. Believers' tendency to produce larger conjunctive estimates was unrelated to group differences in component probability estimates (surprise values) and further, could not be attributed to group differences in the perceived functional relationship between component and conjunctive events. Possible explanations are discussed. Copyright © 2010 John Wiley & Sons, Ltd.     

Testing for negligible interaction: A coherent and robust approach

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap‐based intersection–union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a non‐significant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per‐contrast power of the proposed approach to the non‐centrality‐based negligible interaction test of Cheng and Shao (2007, Statistica Sinica, 17, 1441). For 2 × 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied; however, only our proposed bootstrapping approach was satisfactory under non‐normality and/or variance heterogeneity. In general a × b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection–union approach.     

Robust tests of equivalence for k independent groups

A common question of interest to researchers in psychology is the equivalence of two or more groups. Failure to reject the null hypothesis of traditional hypothesis tests such as the ANOVA F‐test (i.e., H₀: μ₁ = … = μ_k) does not imply the equivalence of the population means. Researchers interested in determining the equivalence of k independent groups should apply a one‐way test of equivalence (e.g., Wellek, 2003). The goals of this study were to investigate the robustness of the one‐way Wellek test of equivalence to violations of homogeneity of variance assumption, and compare the Type I error rates and power of the Wellek test with a heteroscedastic version which was based on the logic of the one‐way Welch (1951) F‐test. The results indicate that the proposed Wellek–Welch test was insensitive to violations of the homogeneity of variance assumption, whereas the original Wellek test was not appropriate when the population variances were not equal.     

等效性检验——结构方程模型评价和测量不变性分析的新视角

常用的结构方程模型拟合指数存在一定局限, 如χ ²以传统零假设为目标假设, 无法验证模型, 而RMSEA和CFI等描述性的拟合指数不具备推断统计性质, 等效性检验有效弥补了这些问题。首先说明等效性检验如何评价单个模型的拟合, 并解释其与零假设检验的不同, 然后介绍等效性检验如何分析测量不变性, 接着用实证数据展示了等效性检验在单个模型评价和测量不变性检验中的效果, 并与传统模型评价方法比较。