Rank‐based permutation approaches for non‐parametric factorial designs

Inference methods for null hypotheses formulated in terms of distribution functions in general non‐parametric factorial designs are studied. The methods can be applied to continuous, ordinal or even ordered categorical data in a unified way, and are based only on ranks. In this set‐up Wald‐type statistics and ANOVA‐type statistics are the current state of the art. The first method is asymptotically exact but a rather liberal statistical testing procedure for small to moderate sample size, while the latter is only an approximation which does not possess the correct asymptotic α level under the null. To bridge these gaps, a novel permutation approach is proposed which can be seen as a flexible generalization of the Kruskal–Wallis test to all kinds of factorial designs with independent observations. It is proven that the permutation principle is asymptotically correct while keeping its finite exactness property when data are exchangeable. The results of extensive simulation studies foster these theoretical findings. A real data set exemplifies its applicability.     

Generalization of the Brown-Forsythe approach to factorial designs

The current paper proposes a solution that generalizes ideas of Brown and Forsythe to the problem of comparing hypotheses in two-way classification designs with heteroscedastic error structure. Unlike the standard analysis of variance, the proposed approach does not require the homogeneity assumption. A comprehensive simulation study, in which sample size of the cells, relationship between the cell sizes and unequal variance, degree of variance heterogeneity, and population distribution shape were systematically manipulated, shows that the proposed approximation was generally robust when normality and heterogeneity were jointly violated.     

A simple test for heterogeneity of variance in complex factorial designs

A simple procedure for testing heterogeneity of variance is developed which generalizes readily to complex, multi-factor experimental designs. Monte Carlo Studies indicate that the Z-variance test statistic presented here yields results equivalent to other familiar tests for heterogeneity of variance in simple one-way designs where comparisons are feasible. The primary advantage of the Z-variance test is in the analysis of factorial effects on sample variances in more complex designs. An example involving a three-way factorial design is presented.     

Fractional factorial designs for legal psychology

Researchers considering novel or exploratory psycholegal research are often able to easily generate a sizable list of independent variables (IVs) that might influence a measure of interest. Where the research question is novel and the literature is not developed, however, choosing from among a long list of potential variables those worthy of empirical investigation often presents a formidable task. Many researchers may feel compelled by legal psychology's heavy reliance on full-factorial designs to narrow the IVs under investigation to two or three in order to avoid an expensive and unwieldy design involving numerous high-order interactions. This article suggests that fractional factorial designs provide a reasonable alternative to full-factorial designs in such circumstances because they allow the psycholegal researcher to examine the main effects of a large number of factors while disregarding high-order interactions. An introduction to the logic of fractional factorial designs is provided and several examples from the social sciences are presented.     

Partial analysis of high-way factorial designs

Inclusion of several variables in a design is often desirable, but each added variable doubles the number of significance tests in the routine analysis of variance. This note discusses a procedure of partial analysis in which interactions, primarily those involving minor variables, are omitted from the analysis when prior evidence indicates that they are probably negligible. The rationale of this procedure is given, and its use illustrated in a model experiment. Specific recommendations for handling systematic sources and error terms are given.     

The Fisher-Pitman permutation test: an attractive alternative to the F test

The Fisher-Pitman permutation test is shown to possess significant advantages over conventional alternatives when analyzing differences among independent samples with unequal variances.     

Robust tests for multivariate factorial designs under heteroscedasticity

The question of how to analyze several multivariate normal mean vectors when normality and covariance homogeneity assumptions are violated is considered in this article. For the two-way MANOVA layout, we address this problem adapting results presented by Brunner, Dette, and Munk (BDM; 1997) and Vallejo and Ato (modified Brown-Forsythe [MBF]; 2006) in the context of univariate factorial and split-plot designs and a multivariate version of the linear model (MLM) to accommodate heterogeneous data. Furthermore, we compare these procedures with the Welch-James (WJ) approximate degrees of freedom multivariate statistics based on ordinary least squares via Monte Carlo simulation. Our numerical studies show that of the methods evaluated, only the modified versions of the BDM and MBF procedures were robust to violations of underlying assumptions. The MLM approach was only occasionally liberal, and then by only a small amount, whereas the WJ procedure was often liberal if the interactive effects were involved in the design, particularly when the number of dependent variables increased and total sample size was small. On the other hand, it was also found that the MLM procedure was uniformly more powerful than its most direct competitors. The overall success rate was 22.4% for the BDM, 36.3% for the MBF, and 45.0% for the MLM.     

Analysis of trend: a permutation alternative to the F test

When the categories of the independent variable in an analysis of variance are quantitative, it is more informative to evaluate the trends in the treatment means than to simply compare differences among the treatment means. A permutation alternative to the conventional F test is shown to possess significant advantages when analyzing trend among quantitative treatments in a one-way analysis of variance. An example with and without an extreme data point illustrates the effectiveness of the permutation alternative for the analysis of trend when homogeneity of variance is compromised.     

A practical method for analyzing factorial designs with heteroscedastic data

The Type I error rates and powers of three recent tests for analyzing nonorthogonal factorial designs under departures from the assumptions of homogeneity and normality were evaluated using Monte Carlo simulation. Specifically, this work compared the performance of the modified Brown-Forsythe procedure, the generalization of Box's method proposed by Brunner, Dette, and Munk, and the mixed-model procedure adjusted by the Kenward-Roger solution available in the SAS statistical package. With regard to robustness, the three approaches adequately controlled Type I error when the data were generated from symmetric distributions; however, this study's results indicate that, when the data were extracted from asymmetric distributions, the modified Brown-Forsythe approach controlled the Type I error slightly better than the other procedures. With regard to sensitivity, the higher power rates were obtained when the analyses were done with the MIXED procedure of the SAS program. Furthermore, results also identified that, when the data were generated from symmetric distributions, little power was sacrificed by using the generalization of Box's method in place of the modified Brown-Forsythe procedure.     

The factorial interpretation of test difficulty

This paper discusses the influence of test difficulty on the correlation between test items and between tests. The greater the difference in difficulty between two test items or between two tests the smaller the maximum correlation between them. In general, the greater the number of degrees of difficulty among the items in a test or among the tests in a battery, the higher the rank of the matrix of intercorrelations; that is, differences in difficulty are represented in the factorial configuration as additional factors. The suggestion is made that if all tests included in a battery are roughly homogeneous with respect to difficulty existing hierarchies will be more clearly defined and meaningful psychological interpretation of factors more readily attained.     

Rethinking innovative designs to further test parasite-stress theory

Fincher & Thornhill's (F&T's) parasite-stress theory of sociality is supported largely by correlational evidence; its persuasiveness would increase significantly via lab and natural experiments and demonstrations of its mediating role. How the theory is linked to other approaches to group differences in psychological differences and to production and dissemination of cultural ideas and practices, need further clarification. So does the theory's view on the possible reduction of negative group interactions.     

Robust techniques for testing heterogeneity of variance effects in factorial designs

Several ways of using the traditional analysis of variance to test heterogeneity of spread in factorial designs with equal or unequaln are compared using both theoretical and Monte Carlo results. Two types of spread variables, (1) the jackknife pseudovalues ofs ² and (2) the absolute deviations from the cell median, are shown to be robust and relatively powerful. These variables seem to be generally superior to the Z-variance and Box-Scheffé procedures.This research was sponsored by Public Health Service Training Grant MH-08258 from the National Institute of Mental Health. The author thanks Mark I. Appelbaum, Elliot M. Cramer, and Scott E. Maxwell for their helpful criticisms of this paper. An earlier version of this work was presented at the Annual Meeting of the Psychometric Society, Murray Hill, New Jersey, April, 1976.     

A robust approach for analyzing unbalanced factorial designs with fixed levels

The goal of this study was to investigate the performance of Hall’s transformation of the Brunner-Dette-Munk (BDM) and Welch-James (WJ) test statistics and Box-Cox’s data transformation in factorial designs when normality and variance homogeneity assumptions were violated separately and jointly. On the basis of unweighted marginal means, we performed a simulation study to explore the operating characteristics of the methods proposed for a variety of distributions with small sample sizes. Monte Carlo simulation results showed that when data were sampled from symmetric distributions, the error rates of the original BDM and WJ tests were scarcely affected by the lack of normality and homogeneity of variance. In contrast, when data were sampled from skewed distributions, the original BDM and WJ rates were not well controlled. Under such circumstances, the results clearly revealed that Hall’s transformation of the BDM and WJ tests provided generally better control of Type I error rates than did the same tests based on Box-Cox’s data transformation. Among all the methods considered in this study, we also found that Hall’s transformation of the BDM test yielded the best control of Type I errors, although it was often less powerful than either of the WJ tests when both approaches reasonably controlled the error rates.     

The relationship between factorial composition of test items and measures of test reliability

For continuous distributions associated with dichotomous item scores, the proportion of common-factor variance in the test,H ², may be expressed as a function of intercorrelations among items.H ² is somewhat larger than the coefficienta except when the items have only one common factor and its loadings are restricted in value. The dichotomous item scores themselves are shown not to have a factor structure, precluding direct interpretation of the Kuder-Richardson coefficient,r _K-R, in terms of factorial properties. The value ofr _K-R is equal to that of a coefficient of equivalence,H ² , when the mean item variance associated with common factors equals the mean interitem covariance. An empirical study with synthetic test data from populations of varying factorial structure showed that the four parameters mentioned may be adequately estimated from dichotomous data.This study was supported in part by an Air Force project (Contract Number AF18(600-170), monitored by the Crew Research Laboratory, Air Force Personnel and Training Research Center, Randolph Air Force Base, Randolph Field, Texas. Permission is granted for reproduction, translation, publication, use and disposal in whole and in part by or for the United States Government. Further support was given by the Northwestern University Graduate School. The computational assistance of Mr. Norman Miller is acknowledged. Professor Meyer Dwass provided mathematical advice both directly and indirectly relevant to the paper.     

Applying mixed‐effects modeling to single‐subject designs: An introduction

Behavior analysis and statistical inference have shared a conflicted relationship for over fifty years. However, a significant portion of this conflict is directed toward statistical tests (e.g., t‐tests, ANOVA) that aggregate group and/or temporal variability into means and standard deviations and as a result remove much of the data important to behavior analysts. Mixed‐effects modeling, a more recently developed statistical test, addresses many of the limitations of more basic tests by incorporating random effects. Random effects quantify individual subject variability without eliminating it from the model, hence producing a model that can predict both group and individual behavior. We present the results of a generalized linear mixed‐effects model applied to single‐subject data taken from Ackerlund Brandt, Dozier, Juanico, Laudont, & Mick, 2015, in which children chose from one of three reinforcers for completing a task. Results of the mixed‐effects modeling are consistent with visual analyses and importantly provide a statistical framework to predict individual behavior without requiring aggregation. We conclude by discussing the implications of these results and provide recommendations for further integration of mixed‐effects models in the analyses of single‐subject designs.     

Rotation to achieve factorial invariance

Under certain conditions it is reasonable to assume that the same factor pattern matrix will describe the regression of observed on factor scores in different populations. However, ordinary factoring procedures will not reveal in general the existence of such a factor pattern matrix. Two procedures for rotating any number of factor pattern matrices based on different populations to conform to a single best fitting factor pattern matrix are developed in this paper. It is assumed that the same number of factors have been determined for each population. Both procedures will yield oblique results in the various populations. The procedures are illustrated with data taken from the 1939 Holzinger-Swineford monograph. Four groups of individuals are utilized.