Robust techniques for testing heterogeneity of variance effects in factorial designs

Several ways of using the traditional analysis of variance to test heterogeneity of spread in factorial designs with equal or unequaln are compared using both theoretical and Monte Carlo results. Two types of spread variables, (1) the jackknife pseudovalues ofs ² and (2) the absolute deviations from the cell median, are shown to be robust and relatively powerful. These variables seem to be generally superior to the Z-variance and Box-Scheffé procedures.This research was sponsored by Public Health Service Training Grant MH-08258 from the National Institute of Mental Health. The author thanks Mark I. Appelbaum, Elliot M. Cramer, and Scott E. Maxwell for their helpful criticisms of this paper. An earlier version of this work was presented at the Annual Meeting of the Psychometric Society, Murray Hill, New Jersey, April, 1976.     

Comment on Levy's “an empirical comparison of the Z-variance and Box-Scheffé tests for homogeneity of variance”

An alternative selection of subsample sizes for the Box-Scheffé test is compared to the singlem used in Levy, An empirical comparison of the Z-variance and Box-Scheffé tests for homogeneity of variance. Use of the alternative subsample sizes is shown to suggest greater power for the Box-Scheffé test, although the test would still be less powerful than the Z-variance test. Because of the nonrobustness of the Z-variance test and the robustness of the Box-Scheffé test, the latter is recommended as a general technique unless an experimenter has assurance that all populations in the experiment are normal. Alternative techniques are also considered.The opinions expressed do not necessarily reflect the position or policy of the U. S. Civil Service Commission.     

Item performance in visual word recognition

Standard factorial designs in psycholinguistics have been complemented recently by large-scale databases providing empirical constraints at the level of item performance. At the same time, the development of precise computational architectures has led modelers to compare item-level performance with item-level predictions. It has been suggested, however, that item performance includes a large amount of undesirable error variance that should be quantified to determine the amount of reproducible variance that models should account for. In the present study, we provide a simple and tractable statistical analysis of this issue. We also report practical solutions for estimating the amount of reproducible variance for any database that conforms to the additive decomposition of the variance. A new empirical database consisting of the word identification times of 140 participants on 120 words is then used to test these practical solutions. Finally, we show that increases in the amount of reproducible variance are accompanied by the detection of new sources of variance.     

The High Cost of Complexity in Experimental Design and Data Analysis: Type I and Type II Error Rates in Multiway ANOVA

The availability of statistical software packages has led to a sharp increase in use of complex research designs and complex statistical analyses in communication research. An informal examination of studies from 2 leading communication journals suggests that the analysis of variance (ANOVA) is often the statistic of choice, and a substantial proportion of published research reports using ANOVA employ complex (k ≥ 3) factorial designs, often involving multiple dependent variables. This article reports a series of Monte Carlo simulations which demonstrate that this complexity may come at a heavier cost than many communication researchers realize. As frequently used, complex factorial ANOVA yield Type I and Type II error rates that many communication scholars would likely consider unacceptable. Consequently, quality of statistical inference in many studies is highly suspect. Communication researchers are warned about problems associated with design and statistical complexity and solutions are suggested.     

An empirical comparison of theZ-variance and Box-Scheffé tests for homogeneity of variance

This brief report provides a comparison of theZ-variance and Box-Scheffé tests for homogeneity of variance. Both procedures are relatively simple to perform and both may be readily utilized in complex, multifactor designs. TheZ-variance test is not robust against non-normality; the Box-Scheffé test is robust against non-normality but is not nearly as powerful as theZ-variance test.     

Applying the permutation test to factorial designs

The permutation test follows directly from the procedure in a comparative experiment, does not depend on a known distribution for error, and is sometimes more sensitive to real effects than are the corresponding parametric tests. Despite its advantages, the permutation test is seldom (if ever) applied to factorial designs because of the computational load that they impose. We propose two methods to limit the computation load. We show, first, that orthogonal contrasts limit the computational load and, second, that when combined with Gill’s (2007) algorithm, the factorial permutation test is both practical and efficient. For within-subjects designs, the factorial permutation test is equivalent to an ANOVA when the latter’s assumptions have been met. For between-subjects designs, the factorial test is conservative. Code to execute the routines described in this article may be downloaded from http://brm.psychonomic-journals.org/content/supplemental.     

Estimates of test size for several test procedures based on conventional variance ratios in the repeated measures design

Estimates of test size (probability of Type I error) were obtained for several specific repeated measures designs. Estimates were presented for configurations where the underlying covariance matrices exhibited varying degrees of heterogeneity. Conventional variance ratios were employed as basic statistics in order to produce estimates of size for a conventional test, an -adjusted test, and -adjusted test and a conservative test. Indices for empirical distributions of two estimators of , a measure of covariance heterogeneity, were also provided.     

Studentized maximum root procedures for coherent analyses of two-factor fixed-effects designs

The authors provide generalizations of R. J. Boik's (1993) studentized maximum root (SMR) procedure that allow for simultaneous inference on families of product contrasts including simple effect contrasts and differences among simple effect contrasts in coherent analyses of data from 2-factor fixed-effects designs. Unlike the F-based simultaneous test procedures (STPs) proposed by M. A. Betz and K. R. Gabriel (1978) for coherent analyses allowing for inferences on all factorial contrasts, SMR STPs are designed for analyses where each contrast of interest is a product contrast, which is usually the case in analyses of data from factorial experiments. When both factors have more than 2 levels, SMR STPs always provide more power and precision than F STPs for inferences on product contrasts.     

Robust step‐down tests for multivariate independent group designs

A composite step‐down procedure, in which a set of step‐down tests are summarized collectively with Fisher's combination statistic, was considered to test for multivariate mean equality in two‐group designs. An approximate degrees of freedom (ADF) composite procedure based on trimmed/Winsorized estimators and a non‐pooled estimate of error variance is proposed, and compared to a composite procedure based on trimmed/Winsorized estimators and a pooled estimate of error variance. The step‐down procedures were also compared to Hotelling's T² and Johansen's ADF global procedure based on trimmed estimators in a simulation study. Type I error rates of the pooled step‐down procedure were sensitive to covariance heterogeneity in unbalanced designs; error rates were similar to those of Hotelling's T² across all of the investigated conditions. Type I error rates of the ADF composite step‐down procedure were insensitive to covariance heterogeneity and less sensitive to the number of dependent variables when sample size was small than error rates of Johansen's test. The ADF composite step‐down procedure is recommended for testing hypotheses of mean equality in two‐group designs except when the data are sampled from populations with different degrees of multivariate skewness.     

Effects of Social Categorization (Religious and National) Among Guinean Workers in France

Abstract

Samples of graphic designs supposedly done by African elementary school children, Guinean or Cameroon, Catholic or Moslem, were presented to 240 Guineans of Catholic or Moslem religion working in Paris, France. Each subject was asked to evaluate the quality of the designs. In reference to social categorization (Tajfel, 1981), in an asymmetrical social situation (Doise, 1979; Turner, 1975), and with respect to the different religious situations in these two countries (Guinea: weak presence of Catholicism and strong presence of Islam; the Cameroon: the reverse situation), the present hypothesis was that the strategies used to evaluate the designs were not simply overevaluation of the ingroup and underevaluation of the outgroup. The results, collected according to a 2 × 2 × 2 factorial design and analyzed by an analysis of variance, confirmed this hypothesis.     

Testing for negligible interaction: A coherent and robust approach

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap‐based intersection–union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a non‐significant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per‐contrast power of the proposed approach to the non‐centrality‐based negligible interaction test of Cheng and Shao (2007, Statistica Sinica, 17, 1441). For 2 × 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied; however, only our proposed bootstrapping approach was satisfactory under non‐normality and/or variance heterogeneity. In general a × b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection–union approach.     

Effects of the heterogeneity of the variances on reliability generalization: an example with the Beck Depression Inventory

The studies of Reliability Generalization (RG) analyze estimates of the reliability of scores from a test provided by a set of studies. As their goals and designs are usually very varied, the sampling of individuals obeys very different schemas. Thus, the variances of the scores might be more heterogeneous than expected from random sampling. Two main problems associated with this potential source of heterogeneity should be taken into account. First, heterogeneity has been usually identified subjectively, not very rigorously. Second, once identified, it has not been taken into account in subsequent analyses. In previous papers, various ways to face both problems have been proposed. The procedures are summarized and applied to a set of 65 independent studies that report estimates of the internal consistency of the Beck Depression Inventory. The results show why any study of RG should take into account the heterogeneity of the variances. In addition to this, the only source that additionally accounts for significant variance in the coefficients is the version of the test employed: the second and third versions of the test involve significant increases in the internal consistency. The consequences of ignoring the heterogeneity of the variances are discussed.     

Factorial invariance in multilevel confirmatory factor analysis

This paper presents a procedure to test factorial invariance in multilevel confirmatory factor analysis. When the group membership is at level 2, multilevel factorial invariance can be tested by a simple extension of the standard procedure. However level‐1 group membership raises problems which cannot be appropriately handled by the standard procedure, because the dependency between members of different level‐1 groups is not appropriately taken into account. The procedure presented in this article provides a solution to this problem. This paper also shows Muthén's maximum likelihood (MUML) estimation for testing multilevel factorial invariance across level‐1 groups as a viable alternative to maximum likelihood estimation. Testing multilevel factorial invariance across level‐2 groups and testing multilevel factorial invariance across level‐1 groups are illustrated using empirical examples. SAS macro and Mplus syntax are provided.     

Generalization of the Brown-Forsythe approach to factorial designs

The current paper proposes a solution that generalizes ideas of Brown and Forsythe to the problem of comparing hypotheses in two-way classification designs with heteroscedastic error structure. Unlike the standard analysis of variance, the proposed approach does not require the homogeneity assumption. A comprehensive simulation study, in which sample size of the cells, relationship between the cell sizes and unequal variance, degree of variance heterogeneity, and population distribution shape were systematically manipulated, shows that the proposed approximation was generally robust when normality and heterogeneity were jointly violated.     

Multivariate multiple regression analyses: a permutation method for linear models

A multivariate extension of a univariate procedure for the analysis of experimental designs is presented. A Euclidean-distance permutation procedure is used to evaluate multivariate residuals obtained from a regression algorithm, also based on Euclidean distances. Applications include various completely randomized and randomized block experimental designs such as one-way, Latin square, factorial, nested, and split-plot designs, with and without covariates. Unlike parametric procedures, the only required assumption is the randomization of subjects to treatments.     

Fractional factorial designs for legal psychology

Researchers considering novel or exploratory psycholegal research are often able to easily generate a sizable list of independent variables (IVs) that might influence a measure of interest. Where the research question is novel and the literature is not developed, however, choosing from among a long list of potential variables those worthy of empirical investigation often presents a formidable task. Many researchers may feel compelled by legal psychology's heavy reliance on full-factorial designs to narrow the IVs under investigation to two or three in order to avoid an expensive and unwieldy design involving numerous high-order interactions. This article suggests that fractional factorial designs provide a reasonable alternative to full-factorial designs in such circumstances because they allow the psycholegal researcher to examine the main effects of a large number of factors while disregarding high-order interactions. An introduction to the logic of fractional factorial designs is provided and several examples from the social sciences are presented.     

Nominal analysis of “variance”

Nominal responses are the natural way for people to report actions or opinions. Because nominal responses do not generate numerical data, they have been underutilized in behavioral research. On those occasions in which nominal responses are elicited, the responses are customarily aggregated over people or trials so that large-sample statistics can be employed. A new analysis is proposed that directly associates differences among responses with particular sources in factorial designs. A pair of nominal responses either matches or does not; when responses do not match, they vary. That analogue to variance is incorporated in the nominal analysis of “variance” (Nanova ) procedure, wherein the proportions of matches associated with sources play the same role as do sums of squares in an anova . The Nanova table is structured like an ANOVA table. The significance levels of the N ratios formed by comparing proportions are determined by resampling. Fictitious behavioral examples featuring independent groups and repeated measures designs are presented. A Windows program for the analysis is available.     

LABSIM: A data-driven simulation program for instruction in research design and statistics

LABSIM is a general-purpose FORTRAN program that simulates the collection of data from laboratory experiments. Student-specified parameters are employed in a multiple-regression model that allows manipulation or control of up to 10 potential independent variables. Variables may be discrete (up to eight levels), continuous, or range (e.g., IQ). Raw scores may be generated for up to six groups, optionally accompanied by statistical analyses for simple between- or within-subjects designs or for factorial designs, including mixed factorial designs. Seventeen models (content areas) are currently available. As contrasted with other simulation programs, LABSIM models are exceptionally easy to design and implement; no computer expertise is required. Summary data automatically collected by LABSIM indicate a high success rate and extensive use of the program beyond course requirements. The current version of LABSIM is available in either batch or interactive form. A newer version that allows the simultaneous manipulation of up to three independent variables will be available soon.     

Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications

This article presents a d-statistic for single-case designs that is in the same metric as the d-statistic used in between-subjects designs such as randomized experiments and offers some reasons why such a statistic would be useful in SCD research. The d has a formal statistical development, is accompanied by appropriate power analyses, and can be estimated using user-friendly SPSS macros. We discuss both advantages and disadvantages of d compared to other approaches such as previous d-statistics, overlap statistics, and multilevel modeling. It requires at least three cases for computation and assumes normally distributed outcomes and stationarity, assumptions that are discussed in some detail. We also show how to test these assumptions. The core of the article then demonstrates in depth how to compute d for one study, including estimation of the autocorrelation and the ratio of between case variance to total variance (between case plus within case variance), how to compute power using a macro, and how to use the d to conduct a meta-analysis of studies using single-case designs in the free program R, including syntax in an appendix. This syntax includes how to read data, compute fixed and random effect average effect sizes, prepare a forest plot and a cumulative meta-analysis, estimate various influence statistics to identify studies contributing to heterogeneity and effect size, and do various kinds of publication bias analyses. This d may prove useful for both the analysis and meta-analysis of data from SCDs.