Randomization tests as alternative analysis methods for behavior‐analytic data

Randomization statistics offer alternatives to many of the statistical methods commonly used in behavior analysis and the psychological sciences, more generally. These methods are more flexible than conventional parametric and nonparametric statistical techniques in that they make no assumptions about the underlying distribution of outcome variables, are relatively robust when applied to small‐n data sets, and are generally applicable to between‐groups, within‐subjects, mixed, and single‐case research designs. In the present article, we first will provide a historical overview of randomization methods. Next, we will discuss the properties of randomization statistics that may make them particularly well suited for analysis of behavior‐analytic data. We will introduce readers to the major assumptions that undergird randomization methods, as well as some practical and computational considerations for their application. Finally, we will demonstrate how randomization statistics may be calculated for mixed and single‐case research designs. Throughout, we will direct readers toward resources that they may find useful in developing randomization tests for their own data.     

Randomization tests for ABAB designs: comparing-data-division- specific and common distributions

Monte Carlo simulations were used to generate data for ABAB designs of different lengths. The points of change in phase are randomly determined before gathering behaviour measurements, which allows the use of a randomization test as an analytic technique. Data simulation and analysis can be based either on data-division-specific or on common distributions. Following one method or another affects the results obtained after the randomization test has been applied. Therefore, the goal of the study was to examine these effects in more detail. The discrepancies in these approaches are obvious when data with zero treatment effect are considered and such approaches have implications for statistical power studies. Data-division-specific distributions provide more detailed information about the performance of the statistical technique.     

Randomization tests for multiple-baseline designs: An extension of the SCRT-R package

Multiple-baseline designs are an extension of the basic single-case AB phase designs, in which several of those AB designs are implemented simultaneously to different persons, behaviors, or settings, and the intervention is introduced in a staggered way to the different units. These designs are well-suited for research in the behavioral sciences. We discuss the advantages and limitations for valid inferences, and suggest a statistical technique—randomization tests—for use with multiple-baseline data, to complement visual analysis. In addition, we provide an extension of our SCRT-R package (which already contained means for conducting randomization tests on single-case phase and alternation designs), for multiple-baseline AB data.     

Evaluating treatment effects in single-subject behavioral experiments using quality-control charts

Quality-control charts can be particularly useful in identifying treatment effects and patterns of behaviors in single-subject behavior-analytic experiments that cannot be determined by visual inspection of their graphs. Using an example from the behavior analysis literature the quality-control charts identified the presence of treatment effects across phases as well as the presence of trends within and between phases. The ease of their calculations suggest use of them by behavior analysts whenever the effects of particular interventions are questionable.     

An empirical investigation of the statistical properties of generalized single-case randomization tests

Adapting Edgington's [J. Psychol. 90 (1975) 57] randomly determined intervention start-point model, Levin and Wampold [Sch. Psychol. Quart. 14 (1999) 59] proposed a set of nonparametric randomization tests for analyzing the data from single-case designs. In the present study, the performance of Levin and Wampold's four basic tests (independent start-point general and comparative effectiveness, simultaneous start-point general and comparative effectiveness) was examined with respect to their Type I error rates and statistical power. Of Levin and Wampold's four tests, all except the independent start-point comparative effectiveness test maintained their empirical Type I error rates and had acceptable power at larger sample-size and effect-size combinations. The one-tailed comparative intervention effectiveness test for the independent start-point model was found to be too liberal, in that it did not maintain its Type I error rate. Although a two-tailed application of that test was found to be conservative at longer series lengths, it had acceptable power at larger sample-size and effect-size combinations. The results support the utility of a versatile new class of single-case designs that permit both within- and between-unit statistical assessments of intervention effectiveness.     

Randomization procedures in single‐case intervention research contexts: (Some of) “the rest of the story”

Following up on articles recently published in this journal, the present contribution tells (some of) “the rest of the story” about the value of randomization in single‐case intervention research investigations. Invoking principles of internal, statistical‐conclusion, and external validity, we begin by emphasizing the critical distinction between design randomization and analysis randomization, along with the necessary correspondence between the two. Four different types of single‐case design‐and‐analysis randomization are then discussed. The persistent negative influence of serially dependent single‐case outcome observations is highlighted, accompanied by examples of inappropriate applications of parametric and nonparametric tests that have appeared in the literature. We conclude by presenting valid applications of single‐case randomization procedures in various single‐case intervention contexts, with specific reference to a freely available Excel‐based software package that can be accessed to incorporate the present randomization schemes into a wide variety of single‐case intervention designs and analyses.     

临床医学类学术论文应正确使用“随机化”

"随机化"是科学研究的基本原则之一,是研究结果外推的重要保证。虚假"随机化"把偶然现象上升为规律,极大地玷污了科学研究的客观真实性,危害无穷。本文以具体的案例剖析了临床研究论文"随机化"的误用和乱用,并提出了应对的措施。     

Comparing instructor-led,video-model,and no-instruction control tutorials for creating single-subject graphs in Microsoft Excel: A systematic replication and extension

Visual inspection of single-subject data is the primary method for behavior analysts to interpret the effect of an independent variable on a dependent variable; however, there is no consensus on the most suitable method for teaching graph construction for single-subject designs. We systematically replicated and extended Tyner and Fienup (2015) using a repeated-measures between-subjects design to compare the effects of instructor-led, video-model, and no-instruction control tutorials on the graphing performance of 81 master's students with some reported Microsoft Excel experience. Our mixed-design analysis revealed a statistically significant main effect of pretest, tutorial, and posttest submissions for each tutorial group and a nonsignificant main effect of tutorial group. Tutorial group significantly interacted with submissions, suggesting that both instructor-led and video-model tutorials may be superior to providing graduate students with a written list of graphing conventions (i.e., control condition). Finally, training influenced performance on an untrained graph type (multielement) for all tutorial groups.     

Studies using single-subject designs in sport psychology: 30 years of research

A prominent feature of behavior-analytic research has been the use of single-subject designs. We examined sport psychology journals and behavioral journals published during the past 30 years, and located 40 studies using single-subject designs to assess interventions for enhancing the performance of athletes and coaches. In this paper, we summarize that body of research, discuss its strengths and limitations, and identify areas for future research.     

Both Sides of the Coin: Randomization from the Perspectives of Physician-Investigators and Patient-Subjects

Randomization is the “gold standard” design for clinical research trials and is accepted as the best way to reduce bias. Although some controversy remains over this matter, we believe equipoise is the fundamental ethical requirement for conducting a randomized clinical trial. Despite much attention to the ethics of randomization, the moral psychology of this study design has not been explored. This article analyzes the ethical tensions that arise from conducting these studies and examines the moral psychology of this design from the perspectives of physician-investigators and patient-subjects. We conclude with a discussion of the practical implications of this analysis.     

A comparison of severalk sample tests for ordered alternatives in completely randomized designs

The importance of appropriate test selection for a given research endeavor cannot be over-emphasized. Using samples drawn from eleven populations (differing in shape, peakedness, and density in the tails), this study investigates the small sample empirical powers of ninek-sample tests against ordered location alternatives under completely randomized designs. The results then are intended to aid the researcher in the selection of a particular procedure appropriate for a given endeavor. To highlight this an industrial psychology application involving work productivity is presented.Research was supported in part by the Scholastic Assistance Program, Baruch College. The author wishes to thank Professors Matthew Goldstein, Shulamith Gross, David Levine, and Edward Wolf for their helpful comments when writing this paper. In addition, the author wishes to thank the referees and editor for their useful suggestions for improving the paper.     

A priori tests in repeated measures designs: Effects of nonsphericity

The validity conditions for univariate repeated measures designs are described. Attention is focused on the sphericity requirement. For av degree of freedom family of comparisons among the repeated measures, sphericity exists when all contrasts contained in thev dimensional space have equal variances. Under nonsphericity, upper and lower bounds on test size and power of a priori, repeated measures,F tests are derived. The effects of nonsphericity are illustrated by means of a set of charts. The charts reveal that small departures from sphericity (.97 <1.00) can seriously affect test size and power. It is recommended that separate rather than pooled error term procedures be routinely used to test a priori hypotheses.Appreciation is extended to Milton Parnes for his insightful assistance.     

Desirable features in neuropsychological tests

Seven desirable features are presented as guidelines for the design and evaluation of neuropsychological tests. Such features are not systematically present in most commonly used testing procedures because the field's methods have been borrowed from disciplines whose theories and measurement goals are quite different from those of modern neuropsychology. In the course of the presentation, three meanings of the term deficit are differentiated.     

Transgenerational patterns of abusive parenting: Analog and clinical tests

Two studies were conducted to determine whether there was an association between punitive childhood histories and abusive parenting. In the first experiment, undergraduate students who reported more punitive childhood histories and undergraduates who reported mild or nonpunishing childhoods were tested with an Analog Parenting Task. The task used photographic stimuli to depict child behaviors and to elicit disciplinary choices in response to children's transgressions. Persons from the more punitive backgrounds were significantly more likely to endorse potentially injurious disciplinary responses. Additionally, in a post-hoc analysis, women with mild punishment backgrounds were more likely to indicate they were annoyed by the depicted child behaviors than women with the severe punishment backgrounds and men with both punishment histories. In the second experiment, childhood histories of parents of children seen at a psychiatric clinic were assessed. The data indicated an association between severely punitive backgrounds of parents and the probability that the referred child had been physically abused. Also, the punishment histories of the parents were associated with the presence of antisocial and aggressive presenting problems displayed by the referred child. The two experiments together were considered to provide support for a modified version of the trans-generational hypothesis of abuse.     

Combinatorial data analysis: Association and partial association

A combinatorial data analysis strategy is reviewed that is designed to compare two arbitrary measures of proximity defined between the objects from some set. Based on a particular cross-product definition of correspondence between these two numerically specified notions of proximity (typically represented in the form of matrices), extensions are then pursued to indices of partial association that relate the observed pattern of correspondence between the first two proximity measures to a third. The attendant issues of index normalization and significance testing are discussed; the latter is approached through a simple randomization model implemented either through a Monte Carlo procedure or distributional approximations based on the first three moments. Applications of the original comparison strategy and its extensions to partial association may be developed for a variety of methodological and substantive tasks. Besides rank correlation, we emphasize the topics of spatial autocorrelation for one variable and spatial association between two and mention the connection to the usual randomization approach for one-way analysis-of-variance.     

Cognitive and psychomotor tests as predictors of on-road driving ability in older primary care patients

This study examined the accuracy with which different cognitive and psychomotor assessment tools were able to predict driving ability among older primary care patients. A cross-sectional study of 50 older drivers (with an average age of 73.1 ± 7.0 years) was conducted. Participants who had been referred by their physicians for psychological assessment following a fitness-to-drive examination underwent both an on-road driving test and a cognitive assessment protocol that included the Senior Drivers Battery (SDB) that is currently administered at the Mobility and Land Transports Institute (MLTI) in Portugal, the Useful Field of View (UFOV) test, the Stroke Drivers Screening Assessment (SDSA), Addenbrooke’s Cognitive Examination Revised (ACE-R), the Trail Making Test, the Key Search test, and the Wechsler Adult Intelligence Scale (WAIS-III) Vocabulary and Block Design tests. Logistic regression analysis revealed that the performances of the participants on the SDSA, ACE-R, UFOV and SDB were the best predictors of on-road driving. Specific measures of processing speed and divided attention, visuospatial abilities, executive functions, psychomotor speed and global cognitive functioning may be useful for predicting unsafe driving. The practical implications of these findings are discussed with a view to developing new assessment models for determining driving fitness in older adults.     

Distinguishing genetic from nongenetic medical tests: Some implications for antidiscrimination legislation

Genetic discrimination is becoming an increasingly important problem in the United States. Information acquired from genetic tests has been used by insurance companies to reject applications for insurance policies and to refuse payment for the treatment of illnesses. Numerous states and the United States Congress have passed or are considering passage of laws that would forbid such use of genetic information by health insurance companies. Here we argue that much of this legislation is severely flawed because of the difficulty in distinguishing genetic from nongenetic tests. In addition, barring the use by insurance companies of a genetic test but not a nongenetic test (conceivably for the same multifactorial disease) raises issues of fairness in health insurance. These arguments suggest that ultimately the problems arising from genetic discrimination cannot be solved by narrowly focused legislation but only by a modification of the entire health care system.