F-test bias for experimental designs in educational research

Reference is made to Neyman's study ofF-test bias for the randomized blocks and Latin square designs employed in agriculture, and some account is given of later statistical developments which sprang from his work—in particular, the classification of model-types and the technique of variance component analysis. It is claimed that there is a need to carry out an examination ofF-test bias for experimental designs in education and psychology which will utilize the method and, where appropriate, the known' results of this new branch of variance analysis. In the present paper, such an investigation is carried out for designs which may be regarded as derivatives of the agricultural randomized blocks design. In a paper to follow, a similar investigation will be carried out for experimental designs of the Latin square type.     

Latin square analysis of variance using summary statistics

The method for computing a Latin square analysis of variance using summary statistics is presented.     

A measure of effect size for experimental designs with heterogeneous variances

A recent trend in the psychological literature has been to include measures of effect size when reporting probability values. The several measures of effect size associated with the Student t test for two independent samples are appropriate only when the variances are homogeneous. In this paper, commonly used measures of effect size are considered and compared, using four data sets. A chance-corrected measure of effect size is provided for two or more treatment groups characterized by either homogeneous or heterogeneous variances.     

Pairs of Latin squares that produce digram-balanced Greco-Latin designs: A BASIC program

It is possible to create pairs of Latin squares that are digram balanced (in other words, that counterbalance immediate sequential effects) in a Greco-Latin design. A behavioral researcher can use these squares to design an efficient, well-balanced study without relying on chance. Researchers can apply these squares to any experiment in which they must pair conditions with different stimuli in a within-subject design. For experiments with a large number of conditions, however, the procedure is very time-consuming if done manually. A BASIC program is described that generates the correct pairs of squares for experiments with as many as 80 conditions.     

Publication bias and the failure of replication in experimental psychology

Replication of empirical findings plays a fundamental role in science. Among experimental psychologists, successful replication enhances belief in a finding, while a failure to replicate is often interpreted to mean that one of the experiments is flawed. This view is wrong. Because experimental psychology uses statistics, empirical findings should appear with predictable probabilities. In a misguided effort to demonstrate successful replication of empirical findings and avoid failures to replicate, experimental psychologists sometimes report too many positive results. Rather than strengthen confidence in an effect, too much successful replication actually indicates publication bias, which invalidates entire sets of experimental findings. Researchers cannot judge the validity of a set of biased experiments because the experiment set may consist entirely of type I errors. This article shows how an investigation of the effect sizes from reported experiments can test for publication bias by looking for too much successful replication. Simulated experiments demonstrate that the publication bias test is able to discriminate biased experiment sets from unbiased experiment sets, but it is conservative about reporting bias. The test is then applied to several studies of prominent phenomena that highlight how publication bias contaminates some findings in experimental psychology. Additional simulated experiments demonstrate that using Bayesian methods of data analysis can reduce (and in some cases, eliminate) the occurrence of publication bias. Such methods should be part of a systematic process to remove publication bias from experimental psychology and reinstate the important role of replication as a final arbiter of scientific findings.     

Multilevel meta-analysis of single-subject experimental designs: A simulation study

One way to combine data from single-subject experimental design studies is by performing a multilevel meta-analysis, with unstandardized or standardized regression coefficients as the effect size metrics. This study evaluates the performance of this approach. The results indicate that a multilevel meta-analysis of unstandardized effect sizes results in good estimates of the effect. The multilevel meta-analysis of standardized effect sizes, on the other hand, is suitable only when the number of measurement occasions for each subject is 20 or more. The effect of the treatment on the intercept is estimated with enough power when the studies are homogeneous or when the number of studies is large; the power of the effect on the slope is estimated with enough power only when the number of studies and the number of measurement occasions are large.     

A comparison of the precision of three experimental designs employing a concomitant variable

Three techniques are commonly employed to capitalize on a concomitant variate and improve the precision of treatment comparisons: (1) stratification of the experimental samples and use of a factorial design, (2) analysis of covariance, and (3) analysis of variance of difference scores. The purpose of this paper is to compare the effectiveness of these alternatives in improving experimental precision, to identify the most precise design and the conditions under which its advantage holds, and to derive, in the case of the factorial approach, recommendations as to the optimal numbers of levels.     

Looking happy: the experimental manipulation of a positive visual attention bias

Individuals with a positive visual attention bias may use their gaze to regulate their emotions while under stress. The current study experimentally trained differential biases in participants' (N = 55) attention toward positive or neutral information. In each training trial, one positive and one neutral word were presented and then a visual target appeared consistently in the location of the positive or neutral words. Participants were instructed to make a simple perceptual discrimination response to the target. Immediately before and after attentional training, participants were exposed to a stress task consisting of viewing a series of extremely negative images while having their eyes tracked. Visual fixation time to negative images, assessed with an eye tracker, served as an indicator of using gaze to successfully regulate emotion. Those participants experimentally trained to selectively attend to affectively positive information looked significantly less at the negative images in the visual stress task following the attentional training, thus demonstrating a learned aversion to negative stimuli. Participants trained toward neutral information did not show this biased gaze pattern.     

Observational fear conditioning in the acquisition and extinction of attentional bias for threat: an experimental evaluation

Anxious persons show automatic and strategic attentional biases for threatening information. Yet, the mechanisms and processes that underlie such biases remain unclear. The central aim of the present study was to elucidate the relation between observational threat learning and the acquisition and extinction of biased threat processing by integrating emotional Stroop color naming tasks within an observational differential fear conditioning procedure. Forty-three healthy female participants underwent several consecutive observational fear conditioning phases. During acquisition, participants watched a confederate displaying mock panic attacks (UCS) paired with a verbal stimulus (CS+), but not with a second nonreinforced verbal stimulus (CS-). As expected, participants showed greater magnitude electrodermal and verbal-evaluative (e.g., distress, fear) conditioned responses to the CS+ over the CS- word. Participants also demonstrated slower color-naming latencies to CS+ compared to the CS- word following acquisition and showed attenuation of this preferential processing bias for threat following extinction. Findings are discussed broadly in the context of the interplay between fear learning and processing biases for threat as observed in persons suffering from anxiety disorders.     

Implementation of experimental designs and storage of results in the PACER system of on-line control

PACER has been developed for use with assembly language programs running on a DEC PDP-11/05 machine. The system comprises an Executive and associated routines to perform many of the tasks common to experiment control programs. Attention has been given to the provision of routines to allow the implementation of experimental designs. Experimental variables are referred to asfactors, and the values of these variables are manipulated indirectly through the use of numerical labels, orlevels. Factors are described in tabular form, and means are provided for the independent or joint randomization of the associated levels. Experimental results can then be stored as functions of specified factors and output at the end of a session in a format corresponding to the structure of the experimental design.     

From a single-level analysis to a multilevel analysis of single-case experimental designs

Multilevel modeling provides one approach to synthesizing single-case experimental design data. In this study, we present the multilevel model (the two-level and the three-level models) for summarizing single-case results over cases, over studies, or both. In addition to the basic multilevel models, we elaborate on several plausible alternative models. We apply the proposed models to real datasets and investigate to what extent the estimated treatment effect is dependent on the modeling specifications and the underlying assumptions. By considering a range of plausible models and assumptions, researchers can determine the degree to which the effect estimates and conclusions are sensitive to the specific assumptions made. If the same conclusions are reached across a range of plausible assumptions, confidence in the conclusions can be enhanced. We advise researchers not to focus on one model but conduct multiple plausible multilevel analyses and investigate whether the results depend on the modeling options.     

Spoiled milk: an experimental examination of bias against mothers who breastfeed

Drawing from the objectification literature, three experiments tested the hypothesis that breastfeeding mothers are the victims of bias. In Study 1, participants rated a woman who had breastfed as incompetent. Study 2 replicated these effects and determined that the bias was specific to conditions that sexualized the breast. In Study 3, participants interacted with a confederate in which attention was drawn to her as a mother, as a mother who breastfeeds, as a woman with sexualized breasts, or in a neutral condition. Results showed the breastfeeding confederate was rated significantly less competent in general, in math and work specifically, and was less likely to be hired compared to all other conditions, except for the sexualized breast condition. Importantly, the breastfeeding mother emphasis and the sexualized breast emphasis resulted in equally negative evaluations. Results suggest that although breastfeeding may be economical and healthy, the social cost is potentially great.