Applying the permutation test to factorial designs

The permutation test follows directly from the procedure in a comparative experiment, does not depend on a known distribution for error, and is sometimes more sensitive to real effects than are the corresponding parametric tests. Despite its advantages, the permutation test is seldom (if ever) applied to factorial designs because of the computational load that they impose. We propose two methods to limit the computation load. We show, first, that orthogonal contrasts limit the computational load and, second, that when combined with Gill’s (2007) algorithm, the factorial permutation test is both practical and efficient. For within-subjects designs, the factorial permutation test is equivalent to an ANOVA when the latter’s assumptions have been met. For between-subjects designs, the factorial test is conservative. Code to execute the routines described in this article may be downloaded from http://brm.psychonomic-journals.org/content/supplemental.     

A robust approach for analyzing unbalanced factorial designs with fixed levels

The goal of this study was to investigate the performance of Hall’s transformation of the Brunner-Dette-Munk (BDM) and Welch-James (WJ) test statistics and Box-Cox’s data transformation in factorial designs when normality and variance homogeneity assumptions were violated separately and jointly. On the basis of unweighted marginal means, we performed a simulation study to explore the operating characteristics of the methods proposed for a variety of distributions with small sample sizes. Monte Carlo simulation results showed that when data were sampled from symmetric distributions, the error rates of the original BDM and WJ tests were scarcely affected by the lack of normality and homogeneity of variance. In contrast, when data were sampled from skewed distributions, the original BDM and WJ rates were not well controlled. Under such circumstances, the results clearly revealed that Hall’s transformation of the BDM and WJ tests provided generally better control of Type I error rates than did the same tests based on Box-Cox’s data transformation. Among all the methods considered in this study, we also found that Hall’s transformation of the BDM test yielded the best control of Type I errors, although it was often less powerful than either of the WJ tests when both approaches reasonably controlled the error rates.     

Partial analysis of high-way factorial designs

Inclusion of several variables in a design is often desirable, but each added variable doubles the number of significance tests in the routine analysis of variance. This note discusses a procedure of partial analysis in which interactions, primarily those involving minor variables, are omitted from the analysis when prior evidence indicates that they are probably negligible. The rationale of this procedure is given, and its use illustrated in a model experiment. Specific recommendations for handling systematic sources and error terms are given.     

Fractional factorial designs for legal psychology

Researchers considering novel or exploratory psycholegal research are often able to easily generate a sizable list of independent variables (IVs) that might influence a measure of interest. Where the research question is novel and the literature is not developed, however, choosing from among a long list of potential variables those worthy of empirical investigation often presents a formidable task. Many researchers may feel compelled by legal psychology's heavy reliance on full-factorial designs to narrow the IVs under investigation to two or three in order to avoid an expensive and unwieldy design involving numerous high-order interactions. This article suggests that fractional factorial designs provide a reasonable alternative to full-factorial designs in such circumstances because they allow the psycholegal researcher to examine the main effects of a large number of factors while disregarding high-order interactions. An introduction to the logic of fractional factorial designs is provided and several examples from the social sciences are presented.     

A factorial approach to job families

This is an experimental study of the application of factor analysis to a new domain—the formation of job families. Correlations between jobs are computed from the formula based on the number of common elements between two variables and the job analyses provide the basic data on the presence or absence of the elements. A first-order general factor and four common factors are obtained in a small sample of twenty occupations. Tentative interpretations are made and implications for job analysis and the formation of job families are pointed out.The analysis the results of which are reported here was made possible by the Bureau of Psychological Services, Institute for Human Adjustment, Horace H. Rackham School of Graduate Studies, University of Michigan.     

Robust tests for multivariate factorial designs under heteroscedasticity

The question of how to analyze several multivariate normal mean vectors when normality and covariance homogeneity assumptions are violated is considered in this article. For the two-way MANOVA layout, we address this problem adapting results presented by Brunner, Dette, and Munk (BDM; 1997) and Vallejo and Ato (modified Brown-Forsythe [MBF]; 2006) in the context of univariate factorial and split-plot designs and a multivariate version of the linear model (MLM) to accommodate heterogeneous data. Furthermore, we compare these procedures with the Welch-James (WJ) approximate degrees of freedom multivariate statistics based on ordinary least squares via Monte Carlo simulation. Our numerical studies show that of the methods evaluated, only the modified versions of the BDM and MBF procedures were robust to violations of underlying assumptions. The MLM approach was only occasionally liberal, and then by only a small amount, whereas the WJ procedure was often liberal if the interactive effects were involved in the design, particularly when the number of dependent variables increased and total sample size was small. On the other hand, it was also found that the MLM procedure was uniformly more powerful than its most direct competitors. The overall success rate was 22.4% for the BDM, 36.3% for the MBF, and 45.0% for the MLM.     

A practical method for analyzing factorial designs with heteroscedastic data

The Type I error rates and powers of three recent tests for analyzing nonorthogonal factorial designs under departures from the assumptions of homogeneity and normality were evaluated using Monte Carlo simulation. Specifically, this work compared the performance of the modified Brown-Forsythe procedure, the generalization of Box's method proposed by Brunner, Dette, and Munk, and the mixed-model procedure adjusted by the Kenward-Roger solution available in the SAS statistical package. With regard to robustness, the three approaches adequately controlled Type I error when the data were generated from symmetric distributions; however, this study's results indicate that, when the data were extracted from asymmetric distributions, the modified Brown-Forsythe approach controlled the Type I error slightly better than the other procedures. With regard to sensitivity, the higher power rates were obtained when the analyses were done with the MIXED procedure of the SAS program. Furthermore, results also identified that, when the data were generated from symmetric distributions, little power was sacrificed by using the generalization of Box's method in place of the modified Brown-Forsythe procedure.     

Robust techniques for testing heterogeneity of variance effects in factorial designs

Several ways of using the traditional analysis of variance to test heterogeneity of spread in factorial designs with equal or unequaln are compared using both theoretical and Monte Carlo results. Two types of spread variables, (1) the jackknife pseudovalues ofs ² and (2) the absolute deviations from the cell median, are shown to be robust and relatively powerful. These variables seem to be generally superior to the Z-variance and Box-Scheffé procedures.This research was sponsored by Public Health Service Training Grant MH-08258 from the National Institute of Mental Health. The author thanks Mark I. Appelbaum, Elliot M. Cramer, and Scott E. Maxwell for their helpful criticisms of this paper. An earlier version of this work was presented at the Annual Meeting of the Psychometric Society, Murray Hill, New Jersey, April, 1976.     

A simple test for heterogeneity of variance in complex factorial designs

A simple procedure for testing heterogeneity of variance is developed which generalizes readily to complex, multi-factor experimental designs. Monte Carlo Studies indicate that the Z-variance test statistic presented here yields results equivalent to other familiar tests for heterogeneity of variance in simple one-way designs where comparisons are feasible. The primary advantage of the Z-variance test is in the analysis of factorial effects on sample variances in more complex designs. An example involving a three-way factorial design is presented.     

Rank‐based permutation approaches for non‐parametric factorial designs

Inference methods for null hypotheses formulated in terms of distribution functions in general non‐parametric factorial designs are studied. The methods can be applied to continuous, ordinal or even ordered categorical data in a unified way, and are based only on ranks. In this set‐up Wald‐type statistics and ANOVA‐type statistics are the current state of the art. The first method is asymptotically exact but a rather liberal statistical testing procedure for small to moderate sample size, while the latter is only an approximation which does not possess the correct asymptotic α level under the null. To bridge these gaps, a novel permutation approach is proposed which can be seen as a flexible generalization of the Kruskal–Wallis test to all kinds of factorial designs with independent observations. It is proven that the permutation principle is asymptotically correct while keeping its finite exactness property when data are exchangeable. The results of extensive simulation studies foster these theoretical findings. A real data set exemplifies its applicability.     

Rotation to achieve factorial invariance

Under certain conditions it is reasonable to assume that the same factor pattern matrix will describe the regression of observed on factor scores in different populations. However, ordinary factoring procedures will not reveal in general the existence of such a factor pattern matrix. Two procedures for rotating any number of factor pattern matrices based on different populations to conform to a single best fitting factor pattern matrix are developed in this paper. It is assumed that the same number of factors have been determined for each population. Both procedures will yield oblique results in the various populations. The procedures are illustrated with data taken from the 1939 Holzinger-Swineford monograph. Four groups of individuals are utilized.     

Alternative approaches to factorial invariance

Special cases of the factor analysis model are developed for four selection situations. Methods are suggested whereby parameters in each case can be estimated using a maximum likelihood procedure recently developed by Jöreskog. Also, a numerical illustration is presented for each case.This research was supported by a grant from the University Research Council, Vanderbilt University.     

Generalization of the negativity effect to self-attributions

In 1979 Reeder and Brewer reported that people make stronger attributions for negative behaviors than for positive ones, following Hastorf and Cantril's statement that people are motivated to maintain positive self-views. The present study tested whether 67 participants would make stronger self-attributions for negative behaviors than for positive ones by having them indicate the number of times that they had to engage in a specific behavior before it would become a trait they possess. Analysis confirmed that participants make stronger self-attributions for negative behaviors than for positive behaviors.     

Current misuse of the factorial methods

Papers involving factorial methods appearing in current literature often misuse factorial methods. These meaningless results are reached because of neglect of certain conditions basic to factor theory. The conditions are:

1. (1)
   The number of basic factors must be smaller than the number of tests.     
   
   

A semi-analytical method of factorial rotation to simple structure

A factorial rotational method is presented which represents a compromise between the use of subjective judgment characteristic of graphical methods and the routine application of analytical methods. At present the analytical methods seem to be inadequate for the discovery of a simple structure, while graphical methods require more subjective judgment. The method herein presented locates the axes for subgroups of tests by an analytical method. The judgments used in the selection of subgroups are based on graphic data concerning interrelation of the factors.