The High Cost of Complexity in Experimental Design and Data Analysis: Type I and Type II Error Rates in Multiway ANOVA

The availability of statistical software packages has led to a sharp increase in use of complex research designs and complex statistical analyses in communication research. An informal examination of studies from 2 leading communication journals suggests that the analysis of variance (ANOVA) is often the statistic of choice, and a substantial proportion of published research reports using ANOVA employ complex (k ≥ 3) factorial designs, often involving multiple dependent variables. This article reports a series of Monte Carlo simulations which demonstrate that this complexity may come at a heavier cost than many communication researchers realize. As frequently used, complex factorial ANOVA yield Type I and Type II error rates that many communication scholars would likely consider unacceptable. Consequently, quality of statistical inference in many studies is highly suspect. Communication researchers are warned about problems associated with design and statistical complexity and solutions are suggested.     

Fractional Design Wizard: A computer program for cost-effective experimental research design

The computer program Fractional Design Wizard creates fractional factorial designs that are cost-effective and especially useful for discarding irrelevant factors from a large number of possible candidates. The program is intended for researchers who are relativelynew to the field of fractional design and who want to acquaint themselves with the use of fractions for the reduction of large experimental designs. Fractional designs allow estimation of main effects, and sometimes two-way interactions, without one’s having to examine all treatment conditions. The program needs Microsoft Windows 95 or better and 32 MB of memory. In a step-by-step fashion, the user can specify the required properties of the fractional design. When there are more valid designs, the user can generate these successively If necessary the user can go back to diminish the requirements. The output can be copied, printed, and saved. The program generates all the information that is needed for the use and interpretation of fractional designs. A help file explains the use of the program and also the purpose, the analysis, and the interpretation of fractional designs. The program, which is written in Object Pascal, is available as freeware on www.fss.uu.nl/ms/hl/fracdes.htm.     

Fractional designs: A simulation study of usefulness in the social sciences

Fractional designs can be extremely useful in social science research, especially when a large number of factors is involved. Reluctance for the use of fractional designs seems to be warranted for two reasons: (1) In the social sciences, the amount of measurement error is often large, which may decrease the power, and (2) higher order interactions are assumed to be nonsignificant, which is difficult to guarantee without sufficient research. This simulation study shows the effects of measurement error and assumption violations under various conditions. It is concluded that fractional designs handle measurement error gracefully and that they are as powerful as a full design when equal degrees of freedom are available. Significant interaction effects can cause serious problems, especially in situations with low or intermediate measurement error, and can lead to erroneous conclusions. Only when estimated confounded effects are clearly not significant, the chance of a wrong decision is reasonably small. Therefore, fractional designs are especially warranted for the exclusion of irrelevant factors. However, we note pitfalls in the use of Version 1.0 of the program Trail Run from SPSS, Inc., to implement the procedures.     

Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations

Factorial experimental designs have many potential advantages for behavioral scientists. For example, such designs may be useful in building more potent interventions by helping investigators to screen several candidate intervention components simultaneously and to decide which are likely to offer greater benefit before evaluating the intervention as a whole. However, sample size and power considerations may challenge investigators attempting to apply such designs, especially when the population of interest is multilevel (e.g., when students are nested within schools, or when employees are nested within organizations). In this article, we examine the feasibility of factorial experimental designs with multiple factors in a multilevel, clustered setting (i.e., of multilevel, multifactor experiments). We conduct Monte Carlo simulations to demonstrate how design elements-such as the number of clusters, the number of lower-level units, and the intraclass correlation-affect power. Our results suggest that multilevel, multifactor experiments are feasible for factor-screening purposes because of the economical properties of complete and fractional factorial experimental designs. We also discuss resources for sample size planning and power estimation for multilevel factorial experiments. These results are discussed from a resource management perspective, in which the goal is to choose a design that maximizes the scientific benefit using the resources available for an investigation.     

Psychology and capital litigation: Research contributions to courtroom consultation

After briefly reviewing the history of capital punishment in the United States, we describe the activities most often performed by behavioral science litigation consultants, and we review five areas of psycholegal research that courtroom consultants can call on to guide their interventions. The research literatures of most frequent use to psycholegal researchers and litigation consultants are (a) assessment of public opinion about crime and punishment, (b) the effects of death qualification during voir dire, (c) the process and effects of sentencing-phase evidence, (d) the possibility of discriminatory application of the death penalty, and (e) the effects of litigation consultation itself. Practical implications for consultants from each of these literatures are considered.     

LABSIM: A data-driven simulation program for instruction in research design and statistics

LABSIM is a general-purpose FORTRAN program that simulates the collection of data from laboratory experiments. Student-specified parameters are employed in a multiple-regression model that allows manipulation or control of up to 10 potential independent variables. Variables may be discrete (up to eight levels), continuous, or range (e.g., IQ). Raw scores may be generated for up to six groups, optionally accompanied by statistical analyses for simple between- or within-subjects designs or for factorial designs, including mixed factorial designs. Seventeen models (content areas) are currently available. As contrasted with other simulation programs, LABSIM models are exceptionally easy to design and implement; no computer expertise is required. Summary data automatically collected by LABSIM indicate a high success rate and extensive use of the program beyond course requirements. The current version of LABSIM is available in either batch or interactive form. A newer version that allows the simultaneous manipulation of up to three independent variables will be available soon.     

Selecting polychoric instrumental variables in confirmatory factor analysis: An alternative specification test and effects of instrumental variables

The polychoric instrumental variable (PIV) approach is a recently proposed method to fit a confirmatory factor analysis model with ordinal data. In this paper, we first examine the small-sample properties of the specification tests for testing the validity of instrumental variables (IVs). Second, we investigate the effects of using different numbers of IVs. Our results show that specification tests derived for continuous data are extremely oversized at all sample sizes when applied to ordinal variables. Possible modifications for ordinal data are proposed in the present study. Simulation results show that the modified specification tests with all available IVs are able to detect model misspecification. In terms of estimation accuracy, the PIV approach where the IVs outnumber the endogenous variables by one produces a lower bias but a higher variation than the PIV approach with more IVs for correctly specified factor loadings at small samples.     

Probing the effects of individual components in multiple component prevention programs

Assessing the contributions of individual components in multi-component interventions poses complex challenges for prevention researchers. We review the strengths and weaknesses of designs and analyses that may be useful in answering three questions: (1) Is each of the individual components contributing to the outcome? (2) Is the program optimal? and (3), Through what processes are the components of the program achieving their effects? Factorial and fractional factorial designs in which a systematically selected portion of all possible treatment combinations is implemented are used to address question 1. Response surface designs in which each component is quantitatively scaled are explored in relation to question 2. Mediational analysis, a hybrid of experimental and correlational approaches, is considered in relation to question 3. Design enhancements are offered that may further strengthen some of these techniques. These techniques offer promise of enhancing both the basic science and applied science contributions of prevention research.     

Psycholegal issues in sibling bone marrow donation

The only hope of survival for children with a number of life-threatening illnesses is a successful bone marrow transplant (BMT). Unlike the treatment source for most therapies, the raw material for transplant therapy comes from a human being. Although many BMTs are autologous, utilizing the patient's own bone marrow, a large percentage of childhood BMTs rely on bone marrow from children or adolescents who are biological siblings to the sick child. Medical and legal systems are confronted with a dilemma when healthy children are needed to undergo minimally risky, yet somewhat painful, procedures for the benefit of their critically ill siblings. This article reviews legal issues involved in sibling bone marrow donation and psychological research that is relevant to those issues. The article concludes with proposed directions for future psycholegal research and a discussion of ethical issues that are not amenable to empirical investigation.     

Psycholegal issues in Sibling Bone Marrow Donation

The only hope of survival for children with a number of life-threatening illnesses is a successful bone marrow transplant (BMT). Unlike the treatment source for most therapies, the raw material for transplant therapy comes from a human being. Although, many BMTs are autologous, utilizing the patient's own bone marrow, a large percentage of childhood BMTs rely on bone marrow from children or adolescents who are biological siblings to the sick child. Medical and legal systems are confronted with a dilemma when healthy children are needed to undergo minimally risky, yet somewhat painful, procedures for the benefit of their critically ill siblings. This article reviews legal issues involved in sibling bone marrow donation and psychological research that is relevant to those issues. The article concludes with proposed directions for future psycholegal research and a discussion of ethical issues that are not amenable to empirical investigation.     

Forward association, backward association, and the false-memory illusion

In the Deese-Roediger-McDermott false-memory illusion, forward associative strength (FAS) is unrelated to the strength of the illusion; this is puzzling, because high-FAS lists ought to share more semantic features with critical unpresented words than should low-FAS lists. The authors show that this null result is probably a truncated range artifact. When FAS and its complement, backward associative strength (BAS), were independently manipulated in factorial designs, they both affected illusion strength. Moreover, their effects did not interact and were of comparable magnitude. Conjoint-recognition analyses were used to pinpoint the influence of BAS and FAS on retrieval processes that support or suppress false-memory responses. Although the variables affected both types of processes, their effects on suppressive processes were larger and more consistent. This research also provided the first clear evidence that subjects often use suppressive processes inappropriately to reject studied items that preserve the gist of experience.     

Successful rural black adolescents: a psychosocial profile

The purpose of this study was to investigate important psychosocial variables associated with the educational development of black adolescents in a southern rural environment. The study explored factors related to academic and social success among rural black adolescents. The focus of the investigation was on identifying psychological and social variables which may successfully influence the transaction between black youth and the rural educational system. Such success variables are important for research because of the traditional emphasis in the social sciences on concepts of deficit in the person-environment interactions of black youth. This article presents a profile of successful black adolescents in a rural school system in the Southeast. The profile was developed from personal interviews with 68 black students identified by teachers as successful, often despite social or economic hardship. The implications for future research and rural educational policy and program from the profile are discussed.     

INTERACTION EFFECTS IN MULTIELEMENT DESIGNS: INEVITABLE,DESIRABLE, AND IGNORABLE

A single-subject design often used to compare the effectiveness of two or more independent variables (like treatment programs) is the multielement (alternating treatments or simultaneous treatments) design. Variants of this design approximate the concurrent comparison of the effects of two or more variables (or levels of variables) by programming the variables (or levels) in rapid alternation, typically across or within daily sessions. Properly combined with conventional reversal designs, these designs can also display a variety of interaction effects, some of them worrisome, others highly desirable for the future development of the field. A worrisome model is the possibility that when Treatment B alternates rapidly with Treatment C, the effects of each will not be the same as when each is the only treatment used. A desirable model is the use of the multielement design as a fast-paced component of an otherwise conventional reversal design examining contextual control of some relationship: the possibility that some behavior responds differently to Controlling Variables A and B in Context X than in Context Y. This second possibility opens single-subject designs to the more efficient examination of all interactive effects and is highly desirable, considering the prevalence and importance of interactions in determining the limits and the generality of currently understood behavioral phenomena.     

Cross-sectional,longitudinal and composite longitudinal data on the Career Maturity Inventory Attitude Scale

Longitudinal data collection designs are frequently used in the study of developmental variables but typically yield patterns of results that are different from those obtained using cross-sectional designs. Recent research has focused on ways to clarify and reconcile the methodological distinctions between these two designs by utilizing alternative designs that incorporate features of both. This investigation compared the results obtained by the two traditional designs on the Career Maturity Inventory-Attitude Scale with the effects of an alternative design: constructing a composite longitudinal gradient. The two traditional methodologies yielded different results from each other for males, but not for females. However, the alternative design failed to eliminate the obtained differences for males, and introduced differences for females, casting doubt on the efficacy of the “solution.” The implications of these results for developmental research and career maturity were discussed.     

Information Integration in Juror Decision Making

Research applying information integration theory to jury decision making has long assumed that people average informational scale values when making legal judgments. Although often consistent with research results, this hypothesis has never been tested in a legal context against a more general additive rule. The present paper describes two studies conducted as a critical test between these two models. Incriminating evidence and eyewitness confidence were varied in a full-factorial, within-subjects design involving a total of 131 subjects acting as mock jurors. Subject responses included eyewitness accuracy and defendant-guilt probability estimates, as well as final verdict decisions. Results strongly support an averaging model of legal decision making. Additional results concerning the influence of initial attitudes and the interrelationships between the variables considered are reported and their implications are discussed.     

Randomized controlled trial of an applied behavior analytic intervention for food selectivity in children with autism spectrum disorder

Food selectivity is a common problem for children with autism spectrum disorder (ASD; Schreck, Williams, & Smith, 2004). Behavior‐analytic interventions have the most empirical support for feeding disorders (Sharp, Jaquess, Morton, & Miles, 2011). However, there are no randomized controlled trials that have evaluated its effects with a well‐defined cohort of children with ASD. In the current investigation, we randomly assigned 6 young children with ASD and food selectivity to either an applied behavior analytic intervention or a wait‐list control. We used a crossover randomized controlled trial to evaluate the effects of a multicomponent applied behavior analytic intervention on independent acceptance and mouth clean of 16 novel foods. We subsequently exposed the wait‐list control group to the intervention. We also evaluated the effects of the intervention on individual participants with single‐case designs. The percentage of independent acceptance and mouth clean increased for the applied behavior analytic intervention group, but not for the wait‐list control group until we implemented the intervention.     

Information-processing architectures in multidimensional classification: a validation test of the systems factorial technology

A growing methodology, known as the systems factorial technology (SFT), is being developed to diagnose the types of information-processing architectures (serial, parallel, or coactive) and stopping rules (exhaustive or self-terminating) that operate in tasks of multidimensional perception. Whereas most previous applications of SFT have been in domains of simple detection and visual-memory search, this research extends the applications to foundational issues in multidimensional classification. Experiments are conducted in which subjects are required to classify objects into a conjunctive-rule category structure. In one case the stimuli vary along highly separable dimensions, whereas in another case they vary along integral dimensions. For the separable-dimension stimuli, the SFT methodology revealed a serial or parallel architecture with an exhaustive stopping rule. By contrast, for the integral-dimension stimuli, the SFT methodology provided clear evidence of coactivation. The research provides a validation of the SFT in the domain of classification and adds to the list of converging operations for distinguishing between separable-dimension and integral-dimension interactions.     

Robust techniques for testing heterogeneity of variance effects in factorial designs

Several ways of using the traditional analysis of variance to test heterogeneity of spread in factorial designs with equal or unequaln are compared using both theoretical and Monte Carlo results. Two types of spread variables, (1) the jackknife pseudovalues ofs ² and (2) the absolute deviations from the cell median, are shown to be robust and relatively powerful. These variables seem to be generally superior to the Z-variance and Box-Scheffé procedures.This research was sponsored by Public Health Service Training Grant MH-08258 from the National Institute of Mental Health. The author thanks Mark I. Appelbaum, Elliot M. Cramer, and Scott E. Maxwell for their helpful criticisms of this paper. An earlier version of this work was presented at the Annual Meeting of the Psychometric Society, Murray Hill, New Jersey, April, 1976.     

Rank‐based permutation approaches for non‐parametric factorial designs

Inference methods for null hypotheses formulated in terms of distribution functions in general non‐parametric factorial designs are studied. The methods can be applied to continuous, ordinal or even ordered categorical data in a unified way, and are based only on ranks. In this set‐up Wald‐type statistics and ANOVA‐type statistics are the current state of the art. The first method is asymptotically exact but a rather liberal statistical testing procedure for small to moderate sample size, while the latter is only an approximation which does not possess the correct asymptotic α level under the null. To bridge these gaps, a novel permutation approach is proposed which can be seen as a flexible generalization of the Kruskal–Wallis test to all kinds of factorial designs with independent observations. It is proven that the permutation principle is asymptotically correct while keeping its finite exactness property when data are exchangeable. The results of extensive simulation studies foster these theoretical findings. A real data set exemplifies its applicability.     

A 3-D Look at the Russian Personality Traits Structure

Attempts to account for most important differences in personality by applying factor analysis to long lists of personality trait variables led to extraction of 5 or 6 orthogonal factorial dimensions. The present paper addresses an issue of more parsimonious three-dimensional visualization of a set of 6 factorial dimensions. Russian speaking respondents used the lists of 496 and 296 personality-relevant nouns for judging 1242 and 447 peoples, respectively. To visualize 6 factorial dimensions of the personality traits structure, a three-dimensional rugby (ball) cake model was suggested and examined by comparison of model-predicted and observed patterns of correlation of each noun with 6 factorial scales. Each noun was linked to one of 43 narrow personality traits predicted by the rugby cake shape. A shorter list of nouns was selected for exemplifying each pole of these traits and all these nouns were evenly written on the surface of the rugby cake shape. Topological features of such three-dimensional shape provided an explanation of failure to reduce 5–6 factorial dimensions to just three spatial dimensions by means of a pure empirical analysis of either loadings of trait variables on these dimensions or their correlations with them.