Pairwise multiple comparisons: A model comparison approach versus stepwise procedures

Researchers in the behavioural sciences have been presented with a host of pairwise multiple comparison procedures that attempt to obtain an optimal combination of Type I error control, power, and ease of application. However, these procedures share one important limitation: intransitive decisions. Moreover, they can be characterized as a piecemeal approach to the problem rather than a holistic approach. Dayton has recently proposed a new approach to pairwise multiple comparisons testing that eliminates intransitivity through a model selection procedure. The present study compared the model selection approach (and a protected version) with three powerful and easy‐to‐use stepwise multiple comparison procedures in terms of the proportion of times that the procedure identified the true pattern of differences among a set of means across several one‐way layouts. The protected version of the model selection approach selected the true model a significantly greater proportion of times than the stepwise procedures and, in most cases, was not affected by variance heterogeneity and non‐normality.     

A general procedure for obtaining paired comparisons from multiple rank orders

From a theoretical point of view, paired comparisons and the law of comparative judgment provide an excellent approach to the problem of psychological measurement. However, if a reasonably large number of stimuli are to be investigated, paired comparisons become extremely time-consuming and fatiguing to the subjects. A balanced incomplete block design, requiring multiple rank order judgments for each subject, provides an efficient experimental method for obtaining paired comparisons judgments. Features of the analysis proposed for this design are discussed in detail. A program for the analysis is available for the IBM 650 electronic computer.Prepared in connection with research done under Office of Naval Research Contract Nonr 1858-(15), Project Designation NR 150-088, and National Science Foundation Grant G-3407. Reproduction of any part of this material is permitted for any purpose of the United States Government.     

A covariance analysis of multiple paired comparisons

In the method of multiple paired comparisons the dominance of objectj over objectk is observed uponp attributes. The present paper develops a covariance analysis for these paired comparisons in terms of a linear model which includes scale, bias, and interaction effects, along withs covariants upon which the comparisons are presumably dependent. The covariance model gives rise to adjusted parameter estimates and hypothesis tests for the residual pairwise layout from which the effects of thes covariants have been removed. These estimation and testing procedures are illustrated with an analysis of political judgment data, and their relevance to the general problem of residual scaling is discussed.This research was supported by National Institute of Mental Health grants MH 12972 and MH 15506. The author would like to express his appreciation to Jerry Solomon and Wei-Ching Chang of the Oregon Research Institute for the data analysis reported at the end of the paper. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by N.I.H. Grant FR-3.     

Mixed-model pairwise multiple comparisons of repeated measures means

One approach to the analysis of repeated measures data allows researchers to model the covariance structure of the data rather than presume a certain structure, as is the case with conventional univariate and multivariate test statistics. This mixed-model approach was evaluated for testing all possible pairwise differences among repeated measures marginal means in a Between-Subjects x Within-Subjects design. Specifically, the authors investigated Type I error and power rates for a number of simultaneous and stepwise multiple comparison procedures using SAS (1999) PROC MIXED in unbalanced designs when normality and covariance homogeneity assumptions did not hold. J. P. Shaffer's (1986) sequentially rejective step-down and Y. Hochberg's (1988) sequentially acceptive step-up Bonferroni procedures, based on an unstructured covariance structure, had superior Type I error control and power to detect true pairwise differences across the investigated conditions.     

A general method for computing hierarchical component structures by Goldberg’s Bass-Ackwards method

Goldberg [Goldberg, L. R.(in press) Doing it all Bass-Ackwards: The development of hierarchical factor structures from the top down. Journal of Research in Personality] has recently described a novel method for computing hierarchical component structures via a “top down” design. He calls his method the “Bass Ackwards” approach to distinguish it from more common “bottom up” methods for obtaining hierarchical solutions. Using simple matrix equations, in this commentary I demonstrate that the end result of a Bass Ackwards analysis—that is, the correlations between component scores from different levels of a hierarchy—can be calculated without calculating the actual component scores. By avoiding component-score calculation, Goldberg’s method (a) can be applied to any data set in which a correlation matrix is available, and (b) can be applied to factor scores rather than estimated factor scores. Goldberg suggests that the Bass Ackwards method represents a useful tool for elucidating the underlying dimensionality and latent structure (rotation in hyperspace) of a data set. Computer code is reported in the Appendix A of this article to help personality researchers more fully evaluate this claim in empirical and simulated data.     

A multiple change score comparison of traditional and behavioral college teaching procedures

Seventy-six students in a college-level course in human development were divided into an experimental and a control group of approximately equal size. Both groups were given a pretest composed of fill-in and multiple-choice items. The control group was exposed to conventional educational practices while the experimental group was treated in a manner similar to that described by Johnston and Pennypacker (1971), performing only on fill-in items. Post-test results showed significantly greater changes in the experimental group, regardless of the type of test item, although the difference was greater in the case of the fill-in items. The results are discussed in terms of their implications for both future research and tactics in the development of improved teaching technologies.     

Additional comparisons of randomization-test procedures for single-case multiple-baseline designs: Alternative effect types

A number of randomization statistical procedures have been developed to analyze the results from single-case multiple-baseline intervention investigations. In a previous simulation study, comparisons of the various procedures revealed distinct differences among them in their ability to detect immediate abrupt intervention effects of moderate size, with some procedures (typically those with randomized intervention start points) exhibiting power that was both respectable and superior to other procedures (typically those with single fixed intervention start points). In Investigation 1 of the present follow-up simulation study, we found that when the same randomization-test procedures were applied to either delayed abrupt or immediate gradual intervention effects: (1) the powers of all of the procedures were severely diminished; and (2) in contrast to the previous study's results, the single fixed intervention start-point procedures generally outperformed those with randomized intervention start points. In Investigation 2 we additionally demonstrated that if researchers are able to successfully anticipate the specific alternative effect types, it is possible for them to formulate adjusted versions of the original randomization-test procedures that can recapture substantial proportions of the lost powers.     

Basic statistics and the inconsistency of multiple comparison procedures.

This paper has two main themes. First, the various statistical measures used in this journal are summarized and their interrelationships described by way of a flow chart. These are the pooled standard deviation, the pooled variance or mean square error (MSE), the standard error of each treatment mean (SEM) and of the difference between two treatment means (SED), and the least difference between two means which is significant at (e.g.) the 5% level of significance (LSD(5%)). The last three measures can be displayed as vertical bars in graphs, and the relationship between the lengths of these bars is graphically illustrated. It is suggested that the LSD is the most useful of these three measures. Second, when the experimenter has no prior hypotheses to be tested using analysis of variance "contrasts," a multiple comparison procedure (MCP) that examines all pair-wise differences between treatment means, may be appropriate. In this paper a fictitious experimental data set is used to compare several well-known McPs by focussing on a particular operating characteristic, the consistency of the results between an overall analysis of all treatments and an analysis of a subset of the experimental treatments. The procedure that behaves best according to this criterion is the unrestricted least significant difference (LSD) procedure. The unrestricted LSD is therefore recommended with the proviso that it be used as a method of generating hypotheses to be tested in subsequent experimentation, not as a method that attempts to simultaneously formulate and test hypotheses.