Randomization tests for multiple-baseline designs: An extension of the SCRT-R package

Multiple-baseline designs are an extension of the basic single-case AB phase designs, in which several of those AB designs are implemented simultaneously to different persons, behaviors, or settings, and the intervention is introduced in a staggered way to the different units. These designs are well-suited for research in the behavioral sciences. We discuss the advantages and limitations for valid inferences, and suggest a statistical technique—randomization tests—for use with multiple-baseline data, to complement visual analysis. In addition, we provide an extension of our SCRT-R package (which already contained means for conducting randomization tests on single-case phase and alternation designs), for multiple-baseline AB data.     

Evaluating single-case research data for systematic review: A commentary for the special issue

The purpose of this commentary is to provide observation on the statistical procedures described throughout this special section from the perspective of researchers with experience in conducting systematic reviews and meta-analyses of single-case research to address issues of evidence-based practice. It is our position that both visual and statistical analyses are complimentary methods for evaluating single-case research data for these purposes. Given the recent developments regarding the use of single-case research to inform evidence-based practice and policy, the developments described in the present issue will be contextualized within the need for a widely accepted process for data evaluation to assist with extending the impact of single-case research. The commentary will, therefore, begin with providing an overview of the conceptual underpinnings of a systematic review of single-case research and will be followed by a discussion of several features that are essential to the development of a conceptually sound and widely used statistical procedure for single-case research. The commentary will conclude with recommendations and guidelines for the use of both visual and statistical analyses within primary research reports and recommendations for future research.     

The Analysis of Single-Case Research Data: Current Instructional Practices

Visual analysis is the predominant method of analysis in single-case research (SCR). However, most research suggests that agreement between visual analysts is poor, which may be due to a lack of clear guidelines and criteria for visual analysis, as well as variability in how individuals are trained. We developed a survey containing questions about the content and methods used to teach visual and statistical analysis of SCR data in verified course sequences (VCS) and distributed it via the VCS Coordinator Listserv. Thirty-seven instructors completed the survey. Results suggest that there is variability across instructors in some fundamental aspects of data analysis (e.g., number of effects required for a functional relation) but a great deal of consistency in others (e.g., emphasizing visual over statistical analysis). We discuss our results along with their implications both for teaching students to analyze SCR data and for conducting additional research on behavior-analytic training programs.

     

Single-case design standards: An update and proposed upgrades

In this paper, we provide a critique focused on the What Works Clearinghouse (WWC) Standards for Single-Case Research Design (Standards 4.1). Specifically, we (a) recommend the use of visual-analysis to verify a single-case intervention study's design standards and to examine the study's operational issues, (b) identify limitations of the design-comparable effect-size measure and discuss related statistical matters, (c) review the applicability and practicality of Standards 4.1 to single-case designs (SCDs), and (d) recommend inclusion of content pertaining to diversity, equity, and inclusion in future standards. Within the historical context of the WWC Pilot Standards for Single-Case Design (1.0), we suggest that Standards 4.1 may best serve as standards for meta-analyses of SCDs but will need to make clear distinctions among the various types of SCD studies that are included in any research synthesis. In this regard, we argue for transparency in SCD studies that meet design standards and those that do not meet design standards in any meta-analysis emanating from the WWC. The intent of these recommendations is to advance the science of SCD research both in research synthesis and in promoting evidence-based practices.     

Intervention-based school psychology services: Training for child-level accountability; preparing for program-level accountability

This research evaluated the outcomes of a school psychology training practicum by replicating intervention-based service delivery procedures established in prior research. The key components include describing a service delivery model, teaching the model, deriving practice guidelines that fit the model, supporting trainees in carrying out the steps, and evaluating the outcomes. Procedures to determine outcomes were based on single-case design facets including accountability design (A-B), visual analysis of graphic data, and social validity ratings. Meta-analysis techniques included calculation of effect sizes and percent of nonoverlapping data (PND). Goal attainment scaling (GAS) also was used to summarize outcomes. The analyses indicated that the interventions led to positive changes for most children. For example, the median effect size was 1.42 across cases. Social validity evidence showed that consumers judged the outcomes positively. Achieving ideal baseline and technical adequacy checks (e.g., observer agreement, intervention adherence) represented challenges for many consultations. The contributions of the study include describing methods for child- and program-level accountability in training and areas for improvement including further supporting the completion of technical checks for intervention services.     

Synthesizing Single-Case Research to Identify Evidence-Based Practices: Some Brief Reflections

The purposes of this paper are to (a) propose an operational standard for defining a ??practice,?? (b) encourage development of professional standards for visual and statistical analysis of single-case research, and (c) propose a standard for using single-case research results to identify practices that are ??evidence-based.?? These topics are not new for behavioral researchers, but they become critical for the larger conversations now occurring among researchers, policy makers, and educators working to implement effective practices in psychology and education.     

Extensions of a versatile randomization test for assessing single-case intervention effects

The purpose of the present study was to investigate the statistical properties of two extensions of the Levin-Wampold (1999) single-case simultaneous start-point model's comparative effectiveness randomization test. The two extensions were (a) adapting the test to situations where there are more than two different intervention conditions and (b) examining the test's performance in classroom-based intervention situations, where the number of time periods (and associated outcome observations) is much smaller than in the contexts for which the test was originally developed. Various Monte Carlo sampling situations were investigated, including from one to five participant blocks per condition and differing numbers of time periods, potential intervention start points, degrees of within-phase autocorrelation, and effect sizes. For all situations, it was found that the Type I error probability of the randomization test was maintained at an acceptable level. With a few notable exceptions, respectable power was observed only in situations where the numbers of observations and potential intervention start points were relatively large, effect sizes were large, and the degree of within-phase autocorrelation was relatively low. It was concluded that the comparative effectiveness randomization test, with its desirable internal validity and statistical-conclusion validity features, is a versatile analytic tool that can be incorporated into a variety of single-case school psychology intervention research situations.     

An Idiographic Single-Case Study Examining the Use of Rational Emotive Behavior Therapy (REBT) with Three Amateur Golfers to Alleviate Social Anxiety

Performance anxiety has been studied in relation to golf performance, but one phenomenon that has received scant attention is social anxiety. One potential intervention that could reduce social anxiety in golfers is rational emotive behavior therapy (REBT), a cognitive-behavioral approach for which research interest is growing. The current study used an idiographic single-case study design to assess the effects of REBT on the social anxiety of 3 male amateur golfers. REBT was employed both on and off the golf course to ensure integration of REBT into the golfers’ performance, offering a methodological advancement of past research. Data were collected prior to, during, and after the REBT intervention. Visual analysis following single-case guidelines revealed substantial reductions in irrational beliefs and social anxiety in all three golfers. Social validation data indicated the positive receipt of REBT by the golfers and supported the visual analysis findings. This current study supports the effectiveness of REBT and extends the research by applying REBT in a “real-world” performance setting, offering methodological advances and providing clear implications for future research and practice.     

An R package for single-case randomization tests

Randomization tests are nonparametric statistical tests that obtain their validity by computationally mimicking the random assignment procedure that was used in the design phase of a study. Because randomization tests do not rely on a random sampling assumption, they can provide a better alternative than parametric statistical tests for analyzing data from single-case designs. In this article, an R package is described for use in designing single-case phase (AB, ABA, and ABAB) and alternation (completely randomized, alternating treatments, and randomized block) experiments, as well as for conducting statistical analyses on data gathered by means of such designs. The R code is presented in a step-by-step way, which at the same time clarifies the rationale behind single-case randomization tests.     

Robust regression for single-case data analysis: How can it help?

This study examined the degree to which outliers were present in a convenience sample of published single-case research. Using a procedure for analyzing single-case data Allison &; Gorman (Behaviour Research and Therapy, 31, 621–631, 1993), this study compared the effect of outliers using ordinary least squares (OLS) regression to a robust regression method and attempted to answer four questions: (1) To what degree does outlier detection vary from OLS to robust regression? (2) How much do effect sizes differ from OLS to robust regression? (3) Are the differences produced by robust regression in more or less agreement with visual judgments of treatment effectiveness? (4) What is a typical range of effect sizes for robust regression versus OLS regression for data from “effective interventions”? Results suggest that outliers are common in single-case data. The effects of outliers in single-case data are explored, and the implications for researchers and practitioners using single-case designs are discussed.     

Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs

Because behavior analysis is a data-driven process, a critical skill for behavior analysts is accurate visual inspection and interpretation of single-case data. Study 1 was a basic study in which we increased the accuracy of visual inspection methods for A-B designs through two refinements of the split-middle (SM) method called the dual-criteria (DC) and conservative dual-criteria (CDC) methods. The accuracy of these visual inspection methods was compared with one another and with two statistical methods (Allison & Gorman, 1993; Gottman, 1981) using a computer-simulated Monte Carlo study. Results indicated that the DC and CDC methods controlled Type I error rates much better than the SM method and had considerably higher power (to detect real treatment effects) than the two statistical methods. In Study 2, brief verbal and written instructions with modeling were used to train 5 staff members to use the DC method, and in Study 3, these training methods were incorporated into a slide presentation and were used to rapidly (i.e., 15 min) train a large group of individuals (N = 87). Interpretation accuracy increased from a baseline mean of 55% to a treatment mean of 94% in Study 2 and from a baseline mean of 71% to a treatment mean of 95% in Study 3. Thus, Study 1 answered basic questions about the accuracy of several methods of interpreting A-B designs; Study 2 showed how that information could be used to increase the accuracy of human visual inspectors; and Study 3 showed how the training procedures from Study 2 could be modified into a format that would facilitate rapid training of large groups of individuals to interpret single-case designs.     

Single-case effect size calculation: Comparing regression and non-parametric approaches across previously published reading intervention data sets

Growing from demands for accountability and research-based practice in the field of education, there is recent focus on developing standards for the implementation and analysis of single-case designs. Effect size methods for single-case designs provide a useful way to discuss treatment magnitude in the context of individual intervention. Although a standard effect size methodology does not yet exist within single-case research, panel experts recently recommended pairing regression and non-parametric approaches when analyzing effect size data. This study compared two single-case effect size methods: the regression-based, Allison-MT method and the newer, non-parametric, Tau-U method. Using previously published research that measured the Words read Correct per Minute (WCPM) variable, these two methods were examined by comparing differences in overall effect size scores and rankings of intervention effect. Results indicated that the regression method produced significantly larger effect sizes than the non-parametric method, but the rankings of the effect size scores had a strong, positive relation. Implications of these findings for research and practice are discussed.     

A brief account of statistical tests for single-case research with persons with developmental disabilities

Education and rehabilitation research with persons with developmental disabilities is often based on single-case designs (with small numbers of ordinal data points collected at irregular intervals) and relies upon graphic display and visual inspection of the data. This paper (a) provides a brief account of some statistical tests, which may serve to supplement the visual inspection process and (b) underlines some of their strengths and limitations to help education and rehabilitation personnel make a reasonable choice among them.     

Assigning and combining probabilities in single-case studies: A second study

The present study builds on a previous proposal for assigning probabilities to the outcomes computed using different primary indicators in single-case studies. These probabilities are obtained by comparing the outcome to previously tabulated reference values, and they reflect the likelihood of the results in the case that no intervention effect is present. In the present study, we explored how well different metrics are translated into p values in the context of simulation data. Furthermore, two published multiple-baseline data sets were used to illustrate how well the probabilities might reflect the intervention effectiveness, as assessed by the original authors. Finally, the importance of which primary indicator would be used in each data set to be integrated was explored; two ways of combining probabilities were used: a weighted average and a binomial test. The results indicated that the translation into p values worked well for the two nonoverlap procedures, with the results for the regression-based procedure diverging due to some undesirable features of its performance. These p values, when either taken individually or combined, were well aligned with effectiveness for the real-life data. These results suggest that assigning probabilities can be useful for translating the primary measure into the same metric, using these probabilities as additional evidence of the importance of behavioral change, complementing visual analysis and professionals’ judgments.     

Visual analysis of single-case time series: Effects of variability, serial dependence, and magnitude of intervention effects

Visual analysis is the dominant method of analysis for single-case time series. The literature assumes that visual analysts will be conservative judges. We show that previous research into visual analysis has not adequately examined false alarm and miss rates or the effect of serial dependence. In order to measure false alarm and miss rates while varying serial dependence, amount of random variability, and effect size, 37 students undertaking a postgraduate course in single-case design and analysis were required to assess the presence of an intervention effect in each of 27 AB charts constructed using a first-order autoregressive model. Three levels of effect size and three levels of variability, representative of values found in published charts, were combined with autocorrelation coefficients of 0, 0.3 and 0.6 in a factorial design. False alarm rates were surprisingly high (16% to 84%). Positive autocorrelation and increased random variation both significantly increased the false alarm rates and interacted in a nonlinear fashion. Miss rates were relatively low (0% to 22%) and were not significantly affected by the design parameters. Thus, visual analysts were not conservative, and serial dependence did influence judgment.     

When time is of the essence: The use of rational emotive behavior therapy (REBT) informed single-session therapy (SST) to alleviate social and golf-specific anxiety,and improve wellbeing and performance,in amateur golfers

Anxieties associated with the social evaluative nature of golf are present in the majority of athletes and impact directly on their wellbeing and objective performance. One potential intervention that could reduce social anxiety in golfers is rational emotive behavior therapy (REBT). There is limited research into the effectiveness of REBT in reducing social anxiety and even less research to assess if REBT can be delivered in time-constricted situations when the expectations for immediate results are high, as is the case with many handicap golfers. The current study used an idiographic single-case study design to assess the effects of REBT-informed single-session therapy (SST) on the social anxiety of five amateur golfers. REBT was employed in a short telephone call to target the performance issue, followed up with a single face-to-face session. Data were collected prior to, immediately following, and four weeks after the REBT intervention. Visual analysis following single-case guidelines revealed substantial reductions in irrational beliefs and social anxiety as well as improvements in wellbeing, scoring average, competition placings and handicap reductions in four out of five golfers. Discussion with golfers at a telephone follow-up indicated the positive receipt of REBT by the golfers and supported the visual analysis findings. This current study supports the effectiveness of REBT and also extends the research by demonstrating that REBT-informed SST can be an effective intervention, in a time-restricted environment, for those golfers with a clear target problem and who are ready to take care of business, providing clear implications for future research and practice.     

Developing decision making in Rugby Union match officials using rational emotive behavior therapy (REBT)

There is a dearth of applied research to inform psychological provision for match-officials. The application of rational emotive behavior therapy (REBT; Ellis, 1957) is becoming established as an effective psychological intervention with elite performers. This paper examined the effects of four one-to-one REBT sessions on irrational beliefs, anxiety, decision rumination and reinvestment, and match officiating performance with two elite rugby match officials. Using a single-case, staggered multiple-baseline across participants research design, visual and statistical analyses showed that the intervention brought about significant, immediate and maintained (12 week follow-up) reductions in irrational beliefs. Furthermore, participants also reported reductions in anxiety, and decision making reinvestment, alongside increased match official performance for one participant. The current study lends some initial support for the application of REBT with match officials, and develops understanding into potential links between irrational beliefs and decision making performance. The results are discussed in terms of theory, applied implications, limitations and future research.     

RegRand: Statistical software for the multiple-baseline design

RegRand (Version 1.0) is Macintosh-based software that enables a researcher to conduct a nonparametric statistical analysis of the data from Koehler and Levin’s (1998) recently described regulated randomization single-case multiple-baseline design. Regulated randomization design and analysis principles are reviewed in relation to an educational research application and a step-by-step illustration of them in relation to the RegRand program is presented.     

A Gradual Effects Model for Single-Case Designs

Single-case designs are a class of repeated measures experiments used to evaluate the effects of interventions for small or specialized populations, such as individuals with low-incidence disabilities. There has been growing interest in systematic reviews and syntheses of evidence from single-case designs, but there remains a need to further develop appropriate statistical models and effect sizes for data from the designs. We propose a novel model for single-case data that exhibit nonlinear time trends created by an intervention that produces gradual effects, which build up and dissipate over time. The model expresses a structural relationship between a pattern of treatment assignment and an outcome variable, making it appropriate for both treatment reversal and multiple baseline designs. It is formulated as a generalized linear model so that it can be applied to outcomes measured as frequency counts or proportions, both of which are commonly used in single-case research, while providing readily interpretable effect size estimates such as log response ratios or log odds ratios. We demonstrate the gradual effects model by applying it to data from a single-case study and examine the performance of proposed estimation methods in a Monte Carlo simulation of frequency count data.