F-test bias for experimental designs in educational research

Reference is made to Neyman's study ofF-test bias for the randomized blocks and Latin square designs employed in agriculture, and some account is given of later statistical developments which sprang from his work—in particular, the classification of model-types and the technique of variance component analysis. It is claimed that there is a need to carry out an examination ofF-test bias for experimental designs in education and psychology which will utilize the method and, where appropriate, the known' results of this new branch of variance analysis. In the present paper, such an investigation is carried out for designs which may be regarded as derivatives of the agricultural randomized blocks design. In a paper to follow, a similar investigation will be carried out for experimental designs of the Latin square type.     

The single latin square design in psychological research

The expected value of mean square concept is used to determine the effects of the presence of interactions in the single Latin square design onF tests. The results indicate that as the number of random effects included in the experiment increase, moreF tests are unbiased, and that some of these are validF tests. However, whenF test bias does occur it is almost always of a negative nature so that the conclusions stated are conservative ones. PositiveF test bias may occur when the triple interaction is extant and when zero or one random variate is included in the experiment.     

The effect of nonindependence on significance testing in dyadic research

Relationship researchers regularly gather data from both members of the dyad, and these two scores are likely to be correlated. This nonindependence of observations can bias p values in significance testing if person is the unit in the statistical analysis. A method for determining how much bias results from dyadic interdependence is presented. Correction factors based on the degree of interdependence, design type, and the number of dyads are used to adjust the F statistic and its degrees of freedom to produce a corrected p value. Bias depends on the type of design and the degree of nonindependence, while the number of dyads in the study ordinarily has only a small effect on bias. Various strategies for controlling for nonindependence are briefly reviewed.     

Measurement error and ANCOVA: Functional and structural relationship approaches

This article discusses alternative procedures to the standardF-test for ANCOVA in case the covariate is measured with error. Both a functional and a structural relationship approach are described. Examples of both types of analysis are given for the simple two-group design. Several cases are discussed and special attention is given to issues of model identifiability. An approximate statistical test based on the functional relationship approach is described. On the basis of Monte Carlo simulation results it is concluded that this testing procedure is to be preferred to the conventionalF-test of the ANCOVA null hypothesis. It is shown how the standard null hypothesis may be tested in a structural relationship approach. It is concluded that some knowledge of the reliability of the covariate is necessary in order to obtain meaningful results.     

A square root method of selecting a minimum set of variables in multiple regression: I. The method

An extension of Dwyer's “square root” method has been made to the problem of selecting a minimum set of variables in a multiple regression problem. The square root method of selection differs from the Wherry-Doolittle method primarily in that (1) the computations required are more compact, (2) anF ratio criterion is used which leads to the selection of fewer variables. The method provides solutions for the problems of test selection, item analysis, analysis of variance with disproportionate frequencies, and other problems requiring the rejection of superfluous variables. In a subsequent article a worked example will be given, and the square root and Wherry-Doolittle methods compared.     

Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit?

Inconsistencies in the research findings on F-test robustness to variance heterogeneity could be related to the lack of a standard criterion to assess robustness or to the different measures used to quantify heterogeneity. In the present paper we use Monte Carlo simulation to systematically examine the Type I error rate of F-test under heterogeneity. One-way, balanced, and unbalanced designs with monotonic patterns of variance were considered. Variance ratio (VR) was used as a measure of heterogeneity (1.5, 1.6, 1.7, 1.8, 2, 3, 5, and 9), the coefficient of sample size variation as a measure of inequality between group sizes (0.16, 0.33, and 0.50), and the correlation between variance and group size as an indicator of the pairing between them (1, .50, 0, ?.50, and ?1). Overall, the results suggest that in terms of Type I error a VR above 1.5 may be established as a rule of thumb for considering a potential threat to F-test robustness under heterogeneity with unequal sample sizes.     

A note on statistical power in multi‐site randomized trials with multiple treatments at each site

We derive the statistical power functions in multi‐site randomized trials with multiple treatments at each site, using multi‐level modelling. An F statistic is used to test multiple parameters in the multi‐level model instead of the Wald chi square test as suggested in the current literature. The F statistic is shown to be more conservative than the Wald statistic in testing any overall treatment effect among the multiple study conditions. In addition, we improvise an easy way to estimate the non‐centrality parameters for the means comparison t‐tests and the F test, using Helmert contrast coding in the multi‐level model. The variance of treatment means, which is difficult to fathom but necessary for power analysis, is decomposed into intuitive simple effect sizes in the contrast tests. The method is exemplified by a multi‐site evaluation study of the behavioural interventions for cannabis dependence.     

Multivariate multiple regression analyses: a permutation method for linear models

A multivariate extension of a univariate procedure for the analysis of experimental designs is presented. A Euclidean-distance permutation procedure is used to evaluate multivariate residuals obtained from a regression algorithm, also based on Euclidean distances. Applications include various completely randomized and randomized block experimental designs such as one-way, Latin square, factorial, nested, and split-plot designs, with and without covariates. Unlike parametric procedures, the only required assumption is the randomization of subjects to treatments.     

Alpha factor analysis of infallible variables

The relationship between the factor pattern,F, derived from fallible (containing measurement error) observations on variables and the factor pattern,F*, derived from infallible observations on variables is investigated. A widely believed relationship betweenF andF*, viz.,F*=AF whereA is a diagonal matrix containing the inverses of the square roots of the reliabilities of the variables, is shown to be false for several factor analytic techniques. Under suitable assumptions, it is shown that for Kaiser and Caffrey's alpha factor analysisF* andF are related byF*=AF. Empirical examples for which the corresponding elements ofF* andAF are equal to two decimal places are presented. The implications of the equality ofF* andAF for alpha factor analysis are discussed.I wish to acknowledge the generous assistance of Drs. Chester W. Harris and Henry F. Kaiser in the execution of the research reported in this paper.     

A special latin square for the use of each subject “as his own control”

When the purpose of the experiment is to compare treatments, the Sequences × Positions Latin Square has been employed to control unwanted effects attributable to individuals, position, and sequence. This particular Latin Square has been subjected to criticism on the grounds there is confounding due to structure, random variables, and subject interactions. Special Latin Square, a subclass of the Sequences × Positions Latin Square, is basically ap ×p factorial design in blocks of sizep. The two factors are treatments (T) and positions (P). Sequence is one component of theTP interaction, and square uniqueness is the sum of the remaining components. This completely replicated factorial design has no structural or random variable confounding; if subject interactions are present, square uniqueness may be used as the error term and the bias in the test of treatments will be conservative.     

Correcting an analysis of variance for clustering

A great deal of educational and social data arises from cluster sampling designs where clusters involve schools, classrooms, or communities. A mistake that is sometimes encountered in the analysis of such data is to ignore the effect of clustering and analyse the data as if it were based on a simple random sample. This typically leads to an overstatement of the precision of results and too liberal conclusions about precision and statistical significance of mean differences. This paper gives simple corrections to the test statistics that would be computed in an analysis of variance if clustering were (incorrectly) ignored. The corrections are multiplicative factors depending on the total sample size, the cluster size, and the intraclass correlation structure. For example, the corrected F statistic has Fisher's F distribution with reduced degrees of freedom. The corrected statistic reduces to the F statistic computed by ignoring clustering when the intraclass correlations are zero. It reduces to the F statistic computed using cluster means when the intraclass correlations are unity, and it is in between otherwise. A similar adjustment to the usual statistic for testing a linear contrast among group means is described.     

Equivalence of computerized and conventional versions of the Beck Depression Inventory-II (BDI-II)

This study examined the equivalence of the conventional and computerized versions of the Beck Depression Inventory-II (BDI-II), taking into account that computer aver-sion may negatively impact computer-administered BDI-II scores by elevating them. Participants were 180 psychology undergraduate students from a medium-sized midwestern university. Participants were divided into four experimental groups. Each group was administered the BDI-II twice in various combinations (conventional only, computerized only, conventional and computerized and vice versa). All participants completed measures of computer aversion and computer experience. Participants who received both versions of the BDI-II were also asked to specify their preference for method of administration. Independent samples t-test results indicated that the com-puterized and paper-and-pencil versions of the BDI-II may be considered equivalent in terms of measurement validity. Implications for future research are discussed. Portions of this article were presented at the 1999 meetings of the American Psychological Association, Boston. The authors thank Dr. Michael Granaas for statistical assistance with the Latin Squares analysis.     

Equivalence of computerized and conventional versions of the Beck Depression Inventory-II (BDI-II)

This study examined the equivalence of the conventional and computerized versions of the Beck Depression Inventory-II (BDI-II), taking into account that computer aver-sion may negatively impact computer-administered BDI-II scores by elevating them. Participants were 180 psychology undergraduate students from a medium-sized midwestern university. Participants were divided into four experimental groups. Each group was administered the BDI-II twice in various combinations (conventional only, computerized only, conventional and computerized and vice versa). All participants completed measures of computer aversion and computer experience. Participants who received both versions of the BDI-II were also asked to specify their preference for method of administration. Independent samples t-test results indicated that the com-puterized and paper-and-pencil versions of the BDI-II may be considered equivalent in terms of measurement validity. Implications for future research are discussed. Portions of this article were presented at the 1999 meetings of the American Psychological Association, Boston. The authors thank Dr. Michael Granaas for statistical assistance with the Latin Squares analysis.     

Statistical power in complex experimental designs

We show that power and sample size tables developed by Cohen (1988, pp. 289–354, 381–389) produce incorrect estimates for factorial designs: power is underestimated, and sample size is overestimated. The source of this bias is shrinkage in the implied value of the noncentrality parameter, λ, caused by using Cohen’s adjustment ton for factorial designs (pp. 365 and 396). The adjustment was intended to compensate for differences in the actual versus presumed (by the tables) error degrees of freedom; however, more accurate estimates are obtained if the tables are used without adjustment. The problems with Cohen’s procedure were discovered while testing subroutines in DATASIM 1.2 for computing power and sample size in completely randomized, randomized-blocks, and split-plot factorial designs. The subroutines give the user the ability to generate power and sample size tables that are as easy to use as Cohen’s, but that eliminate the conservative bias of his tables. We also implemented several improvements relative to “manual” use of Cohen’s tables: (1) Since the user can control the specific values of 1- β,n, andf used on the rows and columns of the table, interpolation is never required; (2) exact as opposed to approximate solutions for the noncentralF distribution are employed; (3) solutions for factorial designs, including those with repeated measures factors, take into account the actual error degrees of freedom for the effect being tested; and (4) provision is made for the computation of power for applications involving the doubly noncentralF distribution.     

The Search for Reliable Generalizations About Messages A Comparison of Research Strategies

Two ways of conducting the search for generalizations about messages are considered: Morley's (this volume) proposal that single-message research designs be used, with subsequent meta-analytic summaries, and Jackson and Jacobs's (1983) proposal that multiple-message designs be used, with messages treated as a random factor in the statistical analysis. Jackson and Jacobs's approach is shown to provide a more dependable, efficient, and practical means for gathering the requisite evidence for dependable generalizations. The charge that multiple-message designs suffer from irreparable problems of experimenter bias is refuted. The treatment of messages as a random factor is defended as statistically appropriate and as clearly preferable to the statistical alternatives.     

The Lutheran Confessions and Popular Religiosity in Latin America

Abstract: The early history of European missions in Latin America led to a close identity between Confessional Lutheranism and German culture. Following World War II that connection has been all but broken, and Lutherans now swim in the sea of Latin American pluralism, including popular religiosity or the people's religion. Constitutive of the latter is devotion to the saints; and the emphasis on “remembering the saints” in Martin Luther and Philip Melanchthon may become valuable resources in contextualization.     

Latin square analysis of variance using summary statistics

The method for computing a Latin square analysis of variance using summary statistics is presented.     

RETR_PWR: An SAS macro for retrospective statistical power analysis

In contrast to prospective power analysis, retrospective power analysis provides an estimate of the statistical power of a hypothesis test after an investigation has been conducted rather than before. In this article, three approaches to obtaining point estimates of power and an interval estimation algorithm are delineated. Previous research on the bias and sampling error of these estimates is briefly reviewed. Finally, an SAS macro that calculates the point and interval estimates is described. The macro was developed to estimate the power of anF test (obtained from analysis of variance, multiple regression analysis, or any of several multivariate analyses), but it may be easily adapted for use with other statistics, such as chi-square tests ort tests.     

Distributions of the Kullback–Leibler divergence with applications

The Kullback–Leibler divergence (KLD) is a widely used method for measuring the fit of two distributions. In general, the distribution of the KLD is unknown. Under reasonable assumptions, common in psychometrics, the distribution of the KLD is shown to be asymptotically distributed as a scaled (non‐central) chi‐square with one degree of freedom or a scaled (doubly non‐central) F. Applications of the KLD for detecting heterogeneous response data are discussed with particular emphasis on test security.     

Could the Aristotelian square of opposition be translated into Chinese?

Conclusion To translate the Aristotelian square of opposition into Chinese requires restructuring the Aristotelian system of genus-species into the Chinese way of classification and understanding of the focus-field relationship. The feature of the former is on a “tree” model, while that of the later is on the focusfield model. Difficulties arise when one tries to show contraries betweenA- type and E-type propositions in the Aristotelian square of opposition in Chinese, because there is no clear distinction between universal and particular in a focus-field structure of thinking. If there could be a chance to discuss the analytic identity between the two logical systems, then it might be only constituted during a face to face conversation in the present, or, in other words, in the translation of particular propositions (singular subjective,I-type, andO-type propositions) in a particular case. The best hope for a translator is that in the actual temporally situated practice,now he or she might find a temporary way to map the concepts of one to the other with relatively little loss of structure.