Further evaluating the conditional decision rule for comparing two independent means

Many books on statistical methods advocate a ‘conditional decision rule’ when comparing two independent group means. This rule states that the decision as to whether to use a ‘pooled variance’ test that assumes equality of variance or a ‘separate variance’ Welch t test that does not should be based on the outcome of a variance equality test. In this paper, we empirically examine the Type I error rate of the conditional decision rule using four variance equality tests and compare this error rate to the unconditional use of either of the t tests (i.e. irrespective of the outcome of a variance homogeneity test) as well as several resampling‐based alternatives when sampling from 49 distributions varying in skewness and kurtosis. Several unconditional tests including the separate variance test performed as well as or better than the conditional decision rule across situations. These results extend and generalize the findings of previous researchers who have argued that the conditional decision rule should be abandoned.     

Power function charts for specification of sample size in analysis of variance

The specification of sample size is an important aspect of the planning of every experiment. When the investigator intends to use the techniques of analysis of variance in the study of treatments effects, he should, in specifying sample size, take into consideration the power of theF tests which will be made. The charts presented in this paper make possible a simple and direct estimate of the sample size required forF tests of specified power.     

Some Thoughts Regarding the Sensitivity of t to Error in Estimated Within-Group Variance

A. J. Riopelle (2003) has eloquently demonstrated that the null hypothesis assessed by the t test involves not only mean differences but also error in the estimation of the within-group standard deviation, s. He is correct in his conclusion that the precision of the interpretation of a significant t and the null hypothesis tested is complex, particularly when sample sizes are small. In this article, the author expands on Riopelle's thoughts by comparing t with some equivalent or closely related tests that make the reliance of t on the accurate estimation of error perhaps more salient and by providing a simulation that may address more directly the magnitude of the interpretational problem.     

Sample size determination for at test given at value from a previous study: A FORTRAN 77 program

When uncertain about the magnitude of an effect, researchers commonly substitute in the standard sample-size-determination formula an estimate of effect size derived from a previous experiment. A problem with this approach is that the traditional sample-size-determination formula was not designed to deal with the uncertainty inherent in an effect-size estimate. Consequently, estimate-substitution in the traditional sample-size-determination formula can lead to a substantial loss of power. A method of sample-size determination designed to handle uncertainty in effect-size estimates is described. The procedure uses thet value and sample size from a previous study, which might be a pilot study or a related study in the same area, to establish a distribution of probable effect sizes. The sample size to be employed in the new study is that which supplies an expected power of the desired amount over the distribution of probable effect sizes. A FORTRAN 77 program is presented that permits swift calculation of sample size for a variety oft tests, including independentt tests, relatedt tests,t tests of correlation coefficients, andt tests of multiple regressionb coefficients.     

Resisting buck-passing accounts of prudential value

This paper aims to cast doubt upon a certain way of analysing prudential value (or good for), namely in the manner of a ‘buck-passing’ analysis. It begins by explaining why we should be interested in analyses of good for and the nature of buck-passing analyses generally (§I). It moves on to considering and rejecting two sets of buck-passing analyses. The first are analyses that are likely to be suggested by those attracted to the idea of analysing good for in a buck-passing fashion (§II). The second are the buck-passing analyses of good for proposed by John Skorupski (§III), Henry Sidgwick (§IV), and Stephen Darwall (§V). Along the way the paper shows that Michael Smith’s and Peter Railton’s analyses of other concepts—analyses that could be (and have been) taken to be analyses of good for—are similarly unsuitable as analyses of it. The paper concludes by suggesting that the fact that none of the buck-passing accounts of good for considered here is satisfactory, coupled with an appreciation of the various problems that a buck-passing analysis of good for would have to avoid, suggests that we should be sceptical about the prospects of finding such an analysis and should look for one of a different type.     

Invariance of a two-factor model of Social Dominance Orientation across gender

This research examined the measurement and structural invariance of the 16-item Social Dominance Orientation (SDO) scale across gender. The Social Dominance Theory (SDT) invariance hypothesis proposes that men are higher than women in SDO. Tests of this hypothesis have compared summary scores with t-tests, but there is no psychometric work of measurement invariance across gender, which makes interpretation of t-test comparisons problematic. Some research suggests the scale captures two distinct factors: support for group-based dominance (GBD) and general opposition to equality (OEQ). However, gender differences in GBD and OEQ have mixed findings. Results replicated the two-factor structure. Multiple-group CFA indicated that loadings, intercepts, factor variances, and factor covariance are equivalent across gender. Males demonstrated higher latent means of GBD and OEQ, supporting the SDT invariance hypothesis.     

Goodness of fit of trend curves and significance of trend differences

Experimental studies of the successive changes (frequently represented by curves describing laws of learning and other similar functional relationships) in a criterion variable accompanying experimental variations in a given treatment, and experimental comparisons of such changes for different populations or for different treatments, constitute a large and important class of psychological experiments. In most such experiments, no attempt has been made to analyze or to make allowance for errors of sampling or of observation. In many others, the techniques of error analysis that have been employed have been inefficient, inexact, or inappropriate. This paper suggests tests, using the methods of analysis of variance, of certain hypotheses concerning trends and trend differences in sample means obtained in experiments of this general type. For means of successive independent samples, tests are provided of the hypotheses: (H ₁) that there is no trend, or that the trend is a horizontal straight line, (H ₃) that there is a linear trend, (H ₅) that the trend is as described by a line not derived from the observed means, and (H ₇) that the trend is as described by a line fitted to the observed means. Tests are also provided of similar hypotheses (H ₂,H ₄,H ₆, andH ₈, respectively) for means of successive measurements of the same sample. Finally, tests are provided of the null hypotheses that there is no difference in trend in two series of means: (H ₉) when each mean in each series is based on an independent sample, (H ₁₀) when each pair of corresponding means is based on an independent sample, (H ₁₁) when each series of means is based on an independent sample, and (H ₁₂) when both series are based on a single sample.     

On Nonequivalence of Several Procedures of Structural Equation Modeling

The normal theory based maximum likelihood procedure is widely used in structural equation modeling. Three alternatives are: the normal theory based generalized least squares, the normal theory based iteratively reweighted least squares, and the asymptotically distribution-free procedure. When data are normally distributed and the model structure is correctly specified, the four procedures are asymptotically equivalent. However, this equivalence is often used when models are not correctly specified. This short paper clarifies conditions under which these procedures are not asymptotically equivalent. Analytical results indicate that, when a model is not correct, two factors contribute to the nonequivalence of the different procedures. One is that the estimated covariance matrices by different procedures are different, the other is that they use different scales to measure the distance between the sample covariance matrix and the estimated covariance matrix. The results are illustrated using real as well as simulated data. Implication of the results to model fit indices is also discussed using the comparative fit index as an example. The work described in this paper was supported by a grant from the Research Grants Council of Hong Kong Special Administrative Region (Project No. CUHK 4170/99M) and by NSF grant DMS04-37167.     

Functional Anatomy of the Null Hypothesis and of Tests of It

The author compared simulations of the “true” null hypothesis (z) test, in which ò was known and fixed, with the t test, in which s, an estimate of ò, was calculated from the sample because the t test was used to emulate the “true” test. The true null hypothesis test bears exclusively on calculating the probability that a sample distance (mean) is larger than a specified value. The results showed that the value of t was sensitive to sampling fluctuations in both distance and standard error. Large values of t reflect small standard errors when n is small. The value of t achieves sensitivity primarily to distance only when the sample sizes are large. One cannot make a definitive statement about the probability or “significance” of a distance solely on the basis of the value of t.     

Internal Consistency and Power When Comparing Total Scores from Two Groups

Researchers now know that when theoretical reliability increases, power can increase, decrease, or stay the same. However, no analytic research has examined the relationship of power to the most commonly used type of reliability—internal consistency—and the most commonly used measures of internal consistency, coefficient alpha and ICC(A,k). We examine the relationship between the power of independent samples t tests and internal consistency. We explicate the mathematical model upon which researchers usually calculate internal consistency, one in which total scores are calculated as the sum of observed scores on K measures. Using this model, we derive a new formula for effect size to show that power and internal consistency are influenced by many of the same parameters but not always in the same direction. Changing an experiment in one way (e.g., lengthening the measure) is likely to influence multiple parameters simultaneously; thus, there are no simple relationships between such changes and internal consistency or power. If researchers revise measures to increase internal consistency, this might not increase power. To increase power, researchers should increase sample size, select measures that assess areas where group differences are largest, and use more powerful statistical procedures (e.g., ANCOVA).     

Mathematics beliefs and achievement of elementary school students in Japan and the United States: results from the Third International Mathematics and Science Study

Student self-beliefs are significantly related to several types of academic achievement. In addition, results from international assessments have indicated that students in Japan have typically scored above international averages (D. L. Kelly, I. V. S. Mullis, & M. O. Martin, 2000). In this study, the author examined relationships between mathematics beliefs and achievement of elementary school-aged students in the United States and Japan. The students had participated in the Third International Mathematics and Science Study (TIMSS; A. E. Beaton et al., 1996). The author examined several self-beliefs and used variance estimation techniques for complex sampling designs. The author identified a number of significant relationships between self-beliefs and mathematics achievement. Students who attributed success in mathematics to controllable factors (e.g., hard work, studying at home) showed higher test scores whereas students who attributed success in mathematics at school to external factors (e.g., good luck) tended to earn lower mathematics test scores. These results extend the findings of previous research results because the author examined large national samples of students in cross-cultural settings as part of a comprehensive international assessment.     

Structural analysis of covariance and correlation matrices

A general approach to the analysis of covariance structures is considered, in which the variances and covariances or correlations of the observed variables are directly expressed in terms of the parameters of interest. The statistical problems of identification, estimation and testing of such covariance or correlation structures are discussed.Several different types of covariance structures are considered as special cases of the general model. These include models for sets of congeneric tests, models for confirmatory and exploratory factor analysis, models for estimation of variance and covariance components, regression models with measurement errors, path analysis models, simplex and circumplex models. Many of the different types of covariance structures are illustrated by means of real data.1978 Psychometric Society Presidential Address.This research has been supported by the Bank of Sweden Tercentenary Foundation under the project entitledStructural Equation Models in the Social Sciences, Karl G. Jöreskog, project director.     

Factorial design and covariance in the study of individual educational development

This is the report of the application of the principles of factorial design to an investigation of individual educational development. The specific type of factorial design formulated was a 2 × 3 × 3 × 3 arrangement, that is, the effect of sex, grade location, scholastic standing, and individual order, singly and in all possible combinations was studied in relation to educational development as measured by theIowa Tests of Educational Development. An application of the covariance method was introduced which resulted in increased precision of this type of experimental design by significantly reducing experimental error. The two concomitant measures used to increase the sensitiveness of the experiment were initial status of individual development and mental age. Without these statistical controls all main effects and two first-order interactions would have been accepted as significant. With their use only sex (doubtful), scholastic standing, and individual order demonstrated significant effects. The chief beauty of the analysis of variance and covariance as an integral part of a self-contained experiment is demonstrated in the complete single analysis of the data. The statistical utilization of the experimental results has also been developed for purposes of estimation and prediction. The mathematical statistician is being continuously required to develop and analyze experimental designs of increasing complexity since the introduction of the analysis of variance and covariance. The mathematical formulation and solution of the problem of this investigation is carried out. The methods illustrated and explained in this study, and modifications and extensions of them are capable of very wide application. The general principles can be used to various degrees and in a number of ways.For the research grant to finance this study, grateful acknowledgment is given to the Graduate School, the University of Minnesota.     

Model selection in covariance structures analysis and the "problem" of sample size: a clarification

Complex models for covariance matrices are structures that specify many parameters, whereas simple models require only a few. When a set of models of differing complexity is evaluated by means of some goodness of fit indices, structures with many parameters are more likely to be selected when the number of observations is large, regardless of other utility considerations. This is known as the sample size problem in model selection decisions. This article argues that this influence of sample size is not necessarily undesirable. The rationale behind this point of view is described in terms of the relationships among the population covariance matrix and 2 model-based estimates of it. The implications of these relationships for practical use are discussed.     

F-test bias for experimental designs of the Latin square type

In an earlier paper, a method of analysis, due to Neyman and now known generally as variance component analysis, was used to examineF-test bias for experimental designs in education of the randomized block type. The same method is now applied to studyF-test bias for designs of the Latin square type. The results, in general, disprove the view that, for a valid application of Latin square techniques, it is necessary that all interactions are zero.     

The mean and standard deviation of the distribution of group assembly sums

An interesting problem in linear programming is the group assembly problem which is mathematically equivalent to the general transportation problem of economics. Computer programs designed for the determination of exact and approximate optimal group assemblies have been available for some time. This paper presents formulas for the mean and squared standard deviation of the distribution of all possible group assembly sums. Computational techniques are presented and the results are related to those of the analysis of variance of ak-factor problem withn levels of each factor.This research was supported in part by the United States Air Force under Project No. 7713, Task No. 77232, and Contract No. AF 18(600)-1050, monitored by the Crew Research Laboratory of the Air Force Personnel and Training Research Center. Permission is granted for reproduction, translation, publication, use, and disposal in whole and in part by or for the United States Government.     

A note on the Neyman-Johnson technique

A statistical problem which frequently arises in educational and psychological experimentation is that of testing the significance of the difference of the mean scores of two groups on some criterion variable, where the differential effects of one or more variables which are correlated with the criterion must be statistically eliminated. The usual analytical technique for this type of problem is the analysis of covariance (9). The Neyman-Johnson technique (7) provides another, and substantially different, approach. A computational procedure is suggested here which utilizes the advantages of both techniques without an undue increase in computational labor. In addition, the Neyman-Johnson technique is generalized to the case ofn predictor variables. Its application has heretofore been limited to a maximum of three predictor variables.This paper was written while the author was a Psychometric Fellow of the Educational Testing Service, Princeton, New Jersey.     

Information matrix estimation procedures for cognitive diagnostic models

Two new methods to estimate the asymptotic covariance matrix for marginal maximum likelihood estimation of cognitive diagnosis models (CDMs), the inverse of the observed information matrix and the sandwich-type estimator, are introduced. Unlike several previous covariance matrix estimators, the new methods take into account both the item and structural parameters. The relationships between the observed information matrix, the empirical cross-product information matrix, the sandwich-type covariance matrix and the two approaches proposed by de la Torre (2009, J. Educ. Behav. Stat., 34, 115) are discussed. Simulation results show that, for a correctly specified CDM and Q-matrix or with a slightly misspecified probability model, the observed information matrix and the sandwich-type covariance matrix exhibit good performance with respect to providing consistent standard errors of item parameter estimates. However, with substantial model misspecification only the sandwich-type covariance matrix exhibits robust performance.     

A dual scaling analysis for paired compositions

A paired composition is a response (upon a dependent variable) to the ordered pair <j, k> of stimuli, treatments, etc. The present paper develops an alternative analysis for the paired compositions layout previously treated by Bechtel's [1967] scaling model. The alternative model relaxes the previous one by including row and column scales that provide an expression of bias foreach pair of objects. The parameter estimation and hypothesis testing procedures for this model are illustrated by means of a small group analysis, which represents a new approach to pairwise sociometrics and personality assessment.This study was supported by Grant Nos. MH 12972, MH 10822, and MH 15506 from the National Institute of Mental Health, U. S. Public Health Service. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by N.I.H. special research resources Grant RR-3.The motivation of this effort has been enhanced by Gerald Patterson and his associates, who have kindly provided the illustrative data at the end of the paper. The author would also like to express his appreciation to Wei-Ching Chang of the Oregon Research Institute and the University of Oregon for his substantive contributions to the paper, and to William Chaplin and Mark Layman of the Oregon Research Institute for the programming of the data analytic method. FORTRAN IV program listings for this analysis are available upon request to the author.     

Repeated measurements analysis of variance when the correlations have a certain pattern

Consider a repeated measurements study in which a single observation is taken at each oft(t > 2) equally spaced time points for each ofn subjects. This paper uses a result of Box to find the appropriate multiplicative correction term for the degrees of freedom in the analysis of variance table for the case when there is equal variability per time point, and when the correlation between observationsu time units apart is equal to ^u . For large values oft, the multiplicative correction has a lower bound approximately equal to 2.5/(t – 1).Dr. Fleiss is also with the Biometrics Research Unit of the New York State Psychiatric Institute. This work was supported in part by grant DE 04068 from the National Institute of Dental Research.