Randomization procedures in single‐case intervention research contexts: (Some of) “the rest of the story”

Following up on articles recently published in this journal, the present contribution tells (some of) “the rest of the story” about the value of randomization in single‐case intervention research investigations. Invoking principles of internal, statistical‐conclusion, and external validity, we begin by emphasizing the critical distinction between design randomization and analysis randomization, along with the necessary correspondence between the two. Four different types of single‐case design‐and‐analysis randomization are then discussed. The persistent negative influence of serially dependent single‐case outcome observations is highlighted, accompanied by examples of inappropriate applications of parametric and nonparametric tests that have appeared in the literature. We conclude by presenting valid applications of single‐case randomization procedures in various single‐case intervention contexts, with specific reference to a freely available Excel‐based software package that can be accessed to incorporate the present randomization schemes into a wide variety of single‐case intervention designs and analyses.     

Dealing with Possible Baseline Inequalities Between Experimental Groups – The Case of Motor Learning

Abstract

One important concept of experimental design is the random assignment of participants to experimental groups. This randomization process is used to prevent selection bias, as well as to provide a strong basis for a cause-and-effect relationship between the independent variable/s and the dependent variable/s. In small sample sizes, simple randomization may not provide equal groups at baseline for one or more of the variables, and therefore more restricted types of randomization, such as the stratified permuted-block randomization, can be used. A code was written to calculate the probability that simple randomization will not lead to equality between groups at baseline, and then an example of stratified permuted-block randomization was examined. The findings suggest that for certain variables that are commonly measured in experiments in motor learning, there is a relatively high probability that groups will not be equal at baseline after simple randomization. This observation reflects the small sample sizes usually found in the literature on motor learning. However, stratified permuted-block randomization does lead to greater equality among groups. Implications for researchers are discussed, and a flowchart is proposed that will allow researchers to decide whether to use simple or stratified randomization.     

Randomization tests in single-subject designs: Illustrative examples

Randomization tests have recently been adapted for use in the analysis of single-subject data. The advantages of these tests lie in their ease of implementation and interpretation as well as their freedom from underlying distributions. Even though numerous articles and books have explicated randomization test procedures, due to the lack of appropriate examples, very little use of these procedures has been made by applied behavior analysts. Data sets reported in a prominent applied behavior journal are used to demonstrate the application of randomization tests to the following three single-subject design models: (a) two-phase random intervention point, (b) multiple phase, and (c) multiple phase with a predicted order of effect size.     

Both Sides of the Coin: Randomization from the Perspectives of Physician-Investigators and Patient-Subjects

Randomization is the “gold standard” design for clinical research trials and is accepted as the best way to reduce bias. Although some controversy remains over this matter, we believe equipoise is the fundamental ethical requirement for conducting a randomized clinical trial. Despite much attention to the ethics of randomization, the moral psychology of this study design has not been explored. This article analyzes the ethical tensions that arise from conducting these studies and examines the moral psychology of this design from the perspectives of physician-investigators and patient-subjects. We conclude with a discussion of the practical implications of this analysis.     

The Role of Randomization in Bayesian and Frequentist Design of Clinical Trial

A key role in inference is played by randomization, which has been extensively used in clinical trials designs. Randomization is primarily intended to prevent the source of bias in treatment allocation by producing comparable groups. In the frequentist framework of inference, randomization allows also for the use of probability theory to express the likelihood of chance as a source for the difference of end outcome. In the Bayesian framework, its role is more nuanced. The Bayesian analysis of clinical trials can afford a valid rationale for selective controls, pointing out a more limited role for randomization than it is generally accorded. This paper is aimed to offer a view of randomization from the perspective of both frequentist and Bayesian statistics and discussing the role of randomization also in theoretical decision models.

     

Meta- and statistical analysis of single-case intervention research data: Quantitative gifts and a wish list

In this commentary, we add to the spirit of the articles appearing in the special series devoted to meta- and statistical analysis of single-case intervention-design data. Following a brief discussion of historical factors leading to our initial involvement in statistical analysis of such data, we discuss: (a) the value added by including statistical-analysis recommendations in the What Works Clearinghouse Standards for single-case intervention designs; (b) the importance of visual analysis in single-case intervention research, along with the distinctive role that could be played by single-case effect-size measures; and (c) the elevated internal validity and statistical-conclusion validity afforded by the incorporation of various forms of randomization into basic single-case design structures. For the future, we envision more widespread application of quantitative analyses, as critical adjuncts to visual analysis, in both primary single-case intervention research studies and literature reviews in the behavioral, educational, and health sciences.     

Standards for Internet-based experimenting

This article summarizes expertise gleaned from the first years of Internet-based experimental research and presents recommendations on: (1) ideal circumstances for conducting a study on the Internet; (2) what precautions have to be undertaken in Web experimental design; (3) which techniques have proven useful in Web experimenting; (4) which frequent errors and misconceptions need to be avoided; and (5) what should be reported. Procedures and solutions for typical challenges in Web experimenting are discussed. Topics covered include randomization, recruitment of samples, generalizability, dropout, experimental control, identity checks, multiple submissions, configuration errors, control of motivational confounding, and pre-testing. Several techniques are explained, including "warm-up," "high hurdle," password methods, "multiple site entry," randomization, and the use of incentives. The article concludes by proposing sixteen standards for Internet-based experimenting.     

Both Sides of the Coin: Randomization from the Perspectives of Physician-Investigators and Patient-Subjects

Randomization is the "gold standard" design for clinical research trials, and is accepted as the best way to reduce bias. Although some controversy remains over this matter, we believe equipoise is the fundamental ethical requirement for conducting a randomized clinical trial. Despite much attention to the ethics of randomization, the moral psychology of this study design has not been explored. This paper analyzes the ethical tensions that arise from conducting these studies, and examines the moral psychology of this design from the perspectives of physician-investigators and patient-subjects. We conclude with a discussion of the practical implications of this analysis.     

An R package for single-case randomization tests

Randomization tests are nonparametric statistical tests that obtain their validity by computationally mimicking the random assignment procedure that was used in the design phase of a study. Because randomization tests do not rely on a random sampling assumption, they can provide a better alternative than parametric statistical tests for analyzing data from single-case designs. In this article, an R package is described for use in designing single-case phase (AB, ABA, and ABAB) and alternation (completely randomized, alternating treatments, and randomized block) experiments, as well as for conducting statistical analyses on data gathered by means of such designs. The R code is presented in a step-by-step way, which at the same time clarifies the rationale behind single-case randomization tests.     

Replicability and randomization test logic in behavior analysis

Randomization tests are a class of nonparametric statistics that determine the significance of treatment effects. Unlike parametric statistics, randomization tests do not assume a random sample, or make any of the distributional assumptions that often preclude statistical inferences about single‐case data. A feature that randomization tests share with parametric statistics, however, is the derivation of a p‐value. P‐values are notoriously misinterpreted and are partly responsible for the putative “replication crisis.” Behavior analysts might question the utility of adding such a controversial index of statistical significance to their methods, so it is the aim of this paper to describe the randomization test logic and its potentially beneficial consequences. In doing so, this paper will: (1) address the replication crisis as a behavior analyst views it, (2) differentiate the problematic p‐values of parametric statistics from the, arguably, more useful p‐values of randomization tests, and (3) review the logic of randomization tests and their unique fit within the behavior analytic tradition of studying behavioral processes that cut across species.     

Randomization tests as alternative analysis methods for behavior‐analytic data

Randomization statistics offer alternatives to many of the statistical methods commonly used in behavior analysis and the psychological sciences, more generally. These methods are more flexible than conventional parametric and nonparametric statistical techniques in that they make no assumptions about the underlying distribution of outcome variables, are relatively robust when applied to small‐n data sets, and are generally applicable to between‐groups, within‐subjects, mixed, and single‐case research designs. In the present article, we first will provide a historical overview of randomization methods. Next, we will discuss the properties of randomization statistics that may make them particularly well suited for analysis of behavior‐analytic data. We will introduce readers to the major assumptions that undergird randomization methods, as well as some practical and computational considerations for their application. Finally, we will demonstrate how randomization statistics may be calculated for mixed and single‐case research designs. Throughout, we will direct readers toward resources that they may find useful in developing randomization tests for their own data.     

Randomization tests for multiple-baseline designs: An extension of the SCRT-R package

Multiple-baseline designs are an extension of the basic single-case AB phase designs, in which several of those AB designs are implemented simultaneously to different persons, behaviors, or settings, and the intervention is introduced in a staggered way to the different units. These designs are well-suited for research in the behavioral sciences. We discuss the advantages and limitations for valid inferences, and suggest a statistical technique—randomization tests—for use with multiple-baseline data, to complement visual analysis. In addition, we provide an extension of our SCRT-R package (which already contained means for conducting randomization tests on single-case phase and alternation designs), for multiple-baseline AB data.     

Will patients accept randomization to psychoanalysis? A feasibility study

The feasibility of using a randomized design in a psychoanalytic outcome study was evaluated. Our hypothesis was that it would be feasible to randomize patients to psychoanalysis three or four times weekly on the couch for five years, supportive expressive therapy once or twice weekly for up to forty sessions, and cognitive behavior therapy once or twice weekly for up to forty sessions. Successful randomization was defined as a 30% recruitment rate among eligible patients. Recruitment began in September 2009 and closed in April 2010. A total of 132 subjects responded to study advertisements, 107 of whom (81%) were triaged out. The remaining 25 were scheduled for the first of two clinical interviews, and 21 of 25 (88%) completed the interview. Eleven of the 25 (44%) were determined to be eligible based on inclusion and exclusion criteria. Eight of the 11 accepted the idea of randomization and completed the diagnostic assessment phase. Calculated on the basis of 8 of 11 eligible patients accepting randomization, the 95% confidence interval was that 39% to 92% of eligible subjects would participate in a larger study of this design. Our findings support the feasibility of implementing an RCT comparing psychoanalysis as defined by the American Psychoanalytic Association (three or four times weekly on the couch for approximately five years) with shorter-term dynamic or cognitive behavioral therapy once or twice a week. Pre-treatment characteristics of these eight patients are presented, as are initial reliability data for the treatment adherence scales used in this trial.     

A comparison of the randomization test with the<Emphasis Type="Italic">F</Emphasis> test when error is skewed

I compared the randomization/permutation test and theF test for a two-cell comparative experiment. I varied (1) the number of observations per cell, (2) the size of the treatment effect, (3) the shape of the underlying distribution of error and, (4) for cases with skewed error, whether or not the skew was correlated with the treatment. With normal error, there was little difference between the tests. When error was skewed, by contrast, the randomization test was more sensitive than theF test, and if the amount of skew was correlated with the treatment, the advantage for the randomization test was both large and positively correlated with the treatment. I conclude that, because the randomization test was never less powerful than theF test, it should replace theF test in routine work.     

Addressing confounding errors when using non-experimental, observational data to make causal claims

In their recent book, Is Inequality Bad for Our Health?, Daniels, Kennedy, and Kawachi claim that to “act justly in health policy, we must have knowledge about the causal pathways through which socioeconomic (and other) inequalities work to produce differential health outcomes.” One of the central problems with this approach is its dependency on “knowledge about the causal pathways.” A widely held belief is that the randomized clinical trial (RCT) is, and ought to be the “gold standard” of evaluating the causal efficacy of interventions. However, often the only data available are non-experimental, observational data. For such data, the necessary randomization is missing. Because the randomization is missing, it seems to follow that it is not possible to make epistemically warranted claims about the causal pathways. Although we are not sanguine about the difficulty in using observational data to make warranted causal claims, we are not as pessimistic as those who believe that the only warranted causal claims are claims based on data from (idealized) RCTs. We argue that careful, thoughtful study design, informed by expert knowledge, that incorporates propensity score matching methods in conjunction with instrumental variable analyses, provides the possibility of warranted causal claims using observational data.     

Statistical testing and null distributions: what to do when samples are not random.

Selected literature related to statistical testing is reviewed to compare the theoretical models underlying parametric and nonparametric inference. Specifically, we show that these models evaluate different hypotheses, are based on different concepts of probability and resultant null distributions, and support different substantive conclusions. We suggest that cognitive scientists should be aware of both models, thus providing them with a better appreciation of the implications and consequences of their choices among potential methods of analysis. This is especially true when it is recognized that most cognitive science research employs design features that do not justify parametric procedures, but that do support nonparametric methods of analysis, particularly those based on the method of permutation/randomization.     

Estimating the causal effect of randomization versus treatment preference in a doubly randomized preference trial

Although randomized studies have high internal validity, generalizability of the estimated causal effect from randomized clinical trials to real-world clinical or educational practice may be limited. We consider the implication of randomized assignment to treatment, as compared with choice of preferred treatment as it occurs in real-world conditions. Compliance, engagement, or motivation may be better with a preferred treatment, and this can complicate the generalizability of results from randomized trials. The doubly randomized preference trial (DRPT) is a hybrid randomized and nonrandomized design that allows for estimation of the causal effect of randomization versus treatment preference. In the DRPT, individuals are first randomized to either randomized assignment or choice assignment. Those in the randomized assignment group are then randomized to treatment or control, and those in the choice group receive their preference of treatment versus control. Using the potential outcomes framework, we apply the algebra of conditional independence to show how the DRPT can be used to derive an unbiased estimate of the causal effect of randomization versus preference for each of the treatment and comparison conditions. Also, we show how these results can be implemented using full matching on the propensity score. The methodology is illustrated with a DRPT of introductory psychology students who were randomized to randomized assignment or preference of mathematics versus vocabulary training. We found a small to moderate benefit of preference versus randomization with respect to the mathematics outcome for those who received mathematics training.     

Randomization test for coupled data

Coupled data arise in perceptual research when subjects are contributing two scores to the data pool. These two scores, it can be reasonably argued, cannot be assumed to be independent of one another; therefore, special treatment is needed when performing statistical inference. This paper shows how the Type I error rate of randomization-based inference is affected by coupled data. It is demonstrated through Monte Carlo simulation that a randomization test behaves much like its parametric counterpart except that, for the randomization test, a negative correlation results in an inflation in the Type I error rate. A new randomization test, the couplet-referenced randomization test, is developed and shown to work for sample sizes of 8 or more observations. An example is presented to demonstrate the computation and interpretation of the new randomization test.     

Alternatives to Randomized Experiments

ABSTRACT— Randomized experiments are preferred for making inferences about causality when they can be implemented and their assumptions are met. Yet assumptions can fail (e.g., attrition, treatment noncompliance) or randomization may be unethical or infeasible. I describe alternative design and statistical approaches that permit testing causal hypotheses and present current empirical evidence related to alternative designs. Alternative designs permit a wider range of research questions to be answered and permit more direct generalization of causal effects; however, when using such designs, estimates of the magnitude of the causal effect may be more uncertain.     

临床均势原则辨析——对临床科研方法的伦理思考

随机对照试验作为一种评价新疗法的可靠设计方法,开始逐渐被世界范围内的临床医生所广泛接受和采用,成为目前最重要的试验方法之一。RCT的运用应该遵循一定的伦理准则,即临床均势原则。围绕临床均势原则这一主题,对随机对照试验的方法论进行较为深入的伦理学思考。并对临床均势的内涵、伦理意义,以及这一伦理要求在实践中所引发的各种伦理争论加以探讨。