Using generalized additive (mixed) models to analyze single case designs

This article shows how to apply generalized additive models and generalized additive mixed models to single-case design data. These models excel at detecting the functional form between two variables (often called trend), that is, whether trend exists, and if it does, what its shape is (e.g., linear and nonlinear). In many respects, however, these models are also an ideal vehicle for analyzing single-case designs because they can consider level, trend, variability, overlap, immediacy of effect, and phase consistency that single-case design researchers examine when interpreting a functional relation. We show how these models can be implemented in a wide variety of ways to test whether treatment is effective, whether cases differ from each other, whether treatment effects vary over cases, and whether trend varies over cases. We illustrate diagnostic statistics and graphs, and we discuss overdispersion of data in detail, with examples of quasibinomial models for overdispersed data, including how to compute dispersion and quasi-AIC fit indices in generalized additive models. We show how generalized additive mixed models can be used to estimate autoregressive models and random effects and discuss the limitations of the mixed models compared to generalized additive models. We provide extensive annotated syntax for doing all these analyses in the free computer program R.     

Intervention-based school psychology services: Training for child-level accountability; preparing for program-level accountability

This research evaluated the outcomes of a school psychology training practicum by replicating intervention-based service delivery procedures established in prior research. The key components include describing a service delivery model, teaching the model, deriving practice guidelines that fit the model, supporting trainees in carrying out the steps, and evaluating the outcomes. Procedures to determine outcomes were based on single-case design facets including accountability design (A-B), visual analysis of graphic data, and social validity ratings. Meta-analysis techniques included calculation of effect sizes and percent of nonoverlapping data (PND). Goal attainment scaling (GAS) also was used to summarize outcomes. The analyses indicated that the interventions led to positive changes for most children. For example, the median effect size was 1.42 across cases. Social validity evidence showed that consumers judged the outcomes positively. Achieving ideal baseline and technical adequacy checks (e.g., observer agreement, intervention adherence) represented challenges for many consultations. The contributions of the study include describing methods for child- and program-level accountability in training and areas for improvement including further supporting the completion of technical checks for intervention services.     

Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model

Numerous ways to meta-analyze single-case data have been proposed in the literature; however, consensus has not been reached on the most appropriate method. One method that has been proposed involves multilevel modeling. For this study, we used Monte Carlo methods to examine the appropriateness of Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach for the meta-analysis of single-case data. Specifically, we examined the fixed effects (e.g., the overall average treatment effect) and the variance components (e.g., the between-person within-study variance in the treatment effect) in a three-level multilevel model (repeated observations nested within individuals, nested within studies). More specifically, bias of the point estimates, confidence interval coverage rates, and interval widths were examined as a function of the number of primary studies per meta-analysis, the modal number of participants per primary study, the modal series length per primary study, the level of autocorrelation, and the variances of the error terms. The degree to which the findings of this study are supportive of using Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach to meta-analyzing single-case data depends on the particular parameter of interest. Estimates of the average treatment effect tended to be unbiased and produced confidence intervals that tended to overcover, but did come close to the nominal level as Level-3 sample size increased. Conversely, estimates of the variance in the treatment effect tended to be biased, and the confidence intervals for those estimates were inaccurate.     

Trifactor Models for Multiple-Ratings Data

In this study we extend and assess the trifactor model for multiple-ratings data in which two different raters give independent scores for the same responses (e.g., in the GRE essay or to subset of PISA constructed-responses). The trifactor model was extended to incorporate a cross-classified data structure (e.g., items and raters) instead of a strictly hierarchical structure. we present a set of simulations to reflect the incompleteness and imbalance in real-world assessments. The effects of the rate of missingness in the data and of ignoring differences among raters are investigated using two sets of simulations. The use of the trifactor model is also illustrated with empirical data analysis using a well-known international large-scale assessment.     

A Cross-Classified CFA-MTMM Model for Structurally Different and Nonindependent Interchangeable Methods

Multirater (multimethod, multisource) studies are increasingly applied in psychology. Eid and colleagues (2008) proposed a multilevel confirmatory factor model for multitrait-multimethod (MTMM) data combining structurally different and multiple independent interchangeable methods (raters). In many studies, however, different interchangeable raters (e.g., peers, subordinates) are asked to rate different targets (students, supervisors), leading to violations of the independence assumption and to cross-classified data structures. In the present work, we extend the ML-CFA-MTMM model by Eid and colleagues (2008) to cross-classified multirater designs. The new C4 model (Cross-Classified CTC[M-1] Combination of Methods) accounts for nonindependent interchangeable raters and enables researchers to explicitly model the interaction between targets and raters as a latent variable. Using a real data application, it is shown how credibility intervals of model parameters and different variance components can be obtained using Bayesian estimation techniques.     

Standard errors of two-level scalability coefficients

For the construction of tests and questionnaires that require multiple raters (e.g., a child behaviour checklist completed by both parents) a novel ordinal scaling technique is currently being further developed, called two-level Mokken scale analysis. The technique uses within-rater and between-rater coefficients to assess the scalability of the test. These coefficients are generalizations of Mokken's scalability coefficients. In this paper we derived standard errors for the two-level coefficients and for their ratios. The coefficients, the estimates, the estimated standard errors and the software implementation are discussed and illustrated using a real-data example, and a small-scale simulation study demonstrates the accuracy of the estimates.     

A generalized least squares regression approach for computing effect sizes in single-case research: application examples

A new method for deriving effect sizes from single-case designs is proposed. The strategy is applicable to small-sample time-series data with autoregressive errors. The method uses Generalized Least Squares (GLS) to model the autocorrelation of the data and estimate regression parameters to produce an effect size that represents the magnitude of treatment effect from baseline to treatment phases in standard deviation units. In this paper, the method is applied to two published examples using common single case designs (i.e., withdrawal and multiple-baseline). The results from these studies are described, and the method is compared to ten desirable criteria for single-case effect sizes. Based on the results of this application, we conclude with observations about the use of GLS as a support to visual analysis, provide recommendations for future research, and describe implications for practice.     

The linear regression line as a judgmental aid in visual analysis of serially dependent A-B time-series data

The purpose of this study was to investigate the effects of different types and magnitudes of serial dependence (first-order moving average and autoregression) and of linear regression lines within experimental phases on the agreement between results of visual and results of statistical data analyses. The stimulus material consisted of computer-simulated A-B-design data graphs. The time series were generated with a constant variance, varying degrees of treatment effects (changes in level), five conditions of serial dependency, and with or without linear regression lines. The material was presented to three groups of student raters (n₁=52, n₂=14, n₃=17) who rated the treatment effect in the graphs on a five-point scale. These ratings were compared with statistical results (time-series analyses). Each group had to interpret 70 graphs, 35 of which had regression lines. Data were analyzed by means of two three-factor and one four-factor ANOVA and by graphic display. The linear regression lines generally enhanced the agreement between the raters' estimations and the statistical results. Serial dependency also increased the agreement between the two analysis methods. However, with strong autoregression processes in the data, the raters tended to overestimate treatment effects relative to time-series analysis.Parts of this study were presented at the World Congress on Behavior Therapy, Washington, DC, December 11, 1983. The authors wish to express their appreciation to Christoph Bonk and Willi Ecker for their extensive collaboration in data analysis and for their assistance in carrying out the study.     

Rater Personality,Rating Format,and Social Context: Implications for Performance Appraisal Ratings

This study investigates the effects of rater personality (Conscientiousness and Agreeableness), rating format (graphic rating scale vs. behavioral checklist), and the rating social context (face‐to‐face feedback vs. no face‐to‐face feedback) on rating elevation of performance ratings. As predicted, raters high on Agreeableness showed more elevated ratings than those low on Agreeableness when they expected to have the face‐to‐face feedback meeting. Furthermore, rating format moderated the relationship between Agreeableness and rating elevation, such that raters high on Agreeableness provided less elevated ratings when using the behavioral checklist than the graphic rating scale, whereas raters low on Agreeableness showed little difference in elevation across different rating formats. Results also suggest that the interactive effects of rater personality, rating format, and social context may depend on the performance level of the ratee. The implications of these findings will be discussed.     

A brief account of statistical tests for single-case research with persons with developmental disabilities

Education and rehabilitation research with persons with developmental disabilities is often based on single-case designs (with small numbers of ordinal data points collected at irregular intervals) and relies upon graphic display and visual inspection of the data. This paper (a) provides a brief account of some statistical tests, which may serve to supplement the visual inspection process and (b) underlines some of their strengths and limitations to help education and rehabilitation personnel make a reasonable choice among them.     

The Analysis of Single-Case Research Data: Current Instructional Practices

Visual analysis is the predominant method of analysis in single-case research (SCR). However, most research suggests that agreement between visual analysts is poor, which may be due to a lack of clear guidelines and criteria for visual analysis, as well as variability in how individuals are trained. We developed a survey containing questions about the content and methods used to teach visual and statistical analysis of SCR data in verified course sequences (VCS) and distributed it via the VCS Coordinator Listserv. Thirty-seven instructors completed the survey. Results suggest that there is variability across instructors in some fundamental aspects of data analysis (e.g., number of effects required for a functional relation) but a great deal of consistency in others (e.g., emphasizing visual over statistical analysis). We discuss our results along with their implications both for teaching students to analyze SCR data and for conducting additional research on behavior-analytic training programs.

     

Robust regression for single-case data analysis: How can it help?

This study examined the degree to which outliers were present in a convenience sample of published single-case research. Using a procedure for analyzing single-case data Allison &; Gorman (Behaviour Research and Therapy, 31, 621–631, 1993), this study compared the effect of outliers using ordinary least squares (OLS) regression to a robust regression method and attempted to answer four questions: (1) To what degree does outlier detection vary from OLS to robust regression? (2) How much do effect sizes differ from OLS to robust regression? (3) Are the differences produced by robust regression in more or less agreement with visual judgments of treatment effectiveness? (4) What is a typical range of effect sizes for robust regression versus OLS regression for data from “effective interventions”? Results suggest that outliers are common in single-case data. The effects of outliers in single-case data are explored, and the implications for researchers and practitioners using single-case designs are discussed.     

Meta-analysis for single-case consultation outcomes: Applications to research and practice

The application of meta-analysis holds much appeal for single-case consultation outcome research. We review a meta-analytic method for using within-study treatment effect sizes in reporting consultation outcomes. The strengths and limitations of traditional group design meta-analysis are examined. Various methods for analyzing single-case outcomes are discussed briefly, followed by an examination of the use of meta-analysis in single-case reviews across independent studies. Within-study meta-analytic results are presented that were derived from treatments implemented in consultations in natural settings. To conclude the article, an illustration is offered of a single-case data analysis display that incorporates meta-analytic results along with other indices of treatment outcome. Recommendations are provided for using meta-analytic methods to evaluate outcomes of single-case consultation treatment.     

Samples in applied psychology: over a decade of research in review

This study examines sample characteristics of articles published in Journal of Applied Psychology (JAP) from 1995 to 2008. At the individual level, the overall median sample size over the period examined was approximately 173, which is generally adequate for detecting the average magnitude of effects of primary interest to researchers who publish in JAP. Samples using higher units of analyses (e.g., teams, departments/work units, and organizations) had lower median sample sizes (Mdn ≈ 65), yet were arguably robust given typical multilevel design choices of JAP authors despite the practical constraints of collecting data at higher units of analysis. A substantial proportion of studies used student samples (~40%); surprisingly, median sample sizes for student samples were smaller than working adult samples. Samples were more commonly occupationally homogeneous (~70%) than occupationally heterogeneous. U.S. and English-speaking participants made up the vast majority of samples, whereas Middle Eastern, African, and Latin American samples were largely unrepresented. On the basis of study results, recommendations are provided for authors, editors, and readers, which converge on 3 themes: (a) appropriateness and match between sample characteristics and research questions, (b) careful consideration of statistical power, and (c) the increased popularity of quantitative synthesis. Implications are discussed in terms of theory building, generalizability of research findings, and statistical power to detect effects.     

Father participation in behavioral parent training for ADHD: review and recommendations for increasing inclusion and engagement.

Research on parenting has generally focused on mothers, with fathers' parenting approaches and interventions for fathers being relatively less studied. To investigate the involvement of fathers in behavioral parent training (BPT), the literature on BPT for attention-deficit/hyperactivity disorder (ADHD) was reviewed. A systematic review of this literature (N = 32) indicated that the majority of research studies are composed of mothers as participants in treatment and raters of outcome (87% of reviewed studies did not include information on father-related outcomes). Present barriers to father participation in BPT (e.g., content of classes, characteristics of fathers) are discussed. Strategies for increasing father participation are offered and include establishing the expectation that fathers will be involved in treatment at initial clinical contacts, collecting treatment-related information from both parents, conducting BPT classes that focus on issues of direct relevance to fathers, and integrating parent-child interactions in recreational settings into BPT programs.     

Integrating clients' moment‐to‐moment ratings into psychotherapy research: A novel approach

Counselling and psychotherapy researchers have considerably advanced the field's understanding of psychotherapy processes and how they relate to treatment outcomes. Despite these advances, little is known about the client's perspective of changes in psychotherapy processes that occur throughout a given session (i.e. micro‐processes). To address this gap, this article describes the novel application of methods that assess participants' moment‐to‐moment ratings to psychotherapy research. This method entails recording psychotherapy session content that clients and other potential raters (e.g. therapists, researchers) later review while simultaneously providing continuous ratings of psychotherapy processes (e.g. helpfulness, alliance). In addition, moment‐to‐moment ratings can facilitate significant events research by prompting researchers to elicit client feedback about the moments that are rated the most and least positively. However, few studies have used these methods in the context of psychotherapy research. Studies incorporating these methods may yield findings that advance psychotherapy research, training efforts and clinical practice. For example, studies may examine how the magnitude and timing of clients' moment‐to‐moment ratings of psychotherapy processes are associated with treatment outcomes, therapist ratings and physiological processes (e.g. heart rate variability). Trainee therapists and their supervisors may also use clients' moment‐to‐moment ratings to facilitate attunement to verbal and non‐verbal indicators of moments perceived more positively and negatively. Last, these methods can produce findings that are highly relevant to clinical practice, where therapists routinely navigate fluctuations in psychotherapy processes (e.g. alliance ruptures) that can be assessed using moment‐to‐moment ratings.     

Context effects,reliability, and internal consistency of intermodal joint scaling

Context effects, intraindividual variability, and internal consistency of intermodal joint scaling with magnitude estimation (“magnitude matching”) were studied by instructing 12 subjects to judge the three pairs of odor intensity, loudness, and brightness on a common scale of perceived intensity as well as to judge odor intensity separately (unimodal magnitude estimation). Significant context effects were found by comparing odor intensity judgments obtained by separate versus intermodal joint scaling as well as across different modalities (loudness vs. brightness) in joint scaling. But no such effects were found for loudness or brightness when compared across modality of joint scaling. Intraindividual variability in the estimates imply about equal reliability in intermodal joint scaling and separate scaling. Good internal consistency was found, indicating that subjects are successful in expressing perceived intensities of different modalities on a common scale.     

The Effects of Graphing a Second Observer’s Data on Judgments of Functional Relations in A–B–A–B Graphs

Collection of interobserver agreement data and reporting the results with summary statistics are standard practices in single-case research. An alternative to summary statistics is plotting the second observer’s data on the same graph as the primary observer. In this study, we evaluated whether plotting the second observer’s data differentially influenced the judgments about functional relations for A–B–A–B designs. Participants were graduate students and experts. Results suggested that (a) experts made more accurate judgments than graduate students, (b) raters made more accurate judgments for graphs with 1 rather than 2 primary data paths, and (c) raters were not influenced differentially in their judgments of functional relations by the presence or absence of the second observer’s plotted data. In addition to standard hypothesis testing, equivalence testing was performed and showed accuracy of judgments was equivalent for graphs with and without the second observers’ data being added.     

Automated classification of demographics from face images: A tutorial and validation

Examining disparities in social outcomes as a function of gender, age, or race has a long tradition in psychology and other social sciences. With an increasing availability of large naturalistic data sets, researchers are afforded the opportunity to study the effects of demographic characteristics with real‐world data and high statistical power. However, since traditional studies rely on human raters to asses demographic characteristics, limits in participant pools can hinder researchers from analyzing large data sets. Automated procedures offer a new solution to the classification of face images. Here, we present a tutorial on how to use two face classification algorithms, Face++ and Kairos. We also test and compare their accuracy under varying conditions and provide practical recommendations for their use. Drawing on two face databases (n = 2,805 images), we find that classification accuracy is (a) relatively high, with Kairos generally outperforming Face++ (b) similar for standardized and more variable images, and (c) dependent on target demographics. For example, accuracy was lower for Hispanic and Asian (vs. Black and White) targets. In sum, we propose that automated face classification can be a useful tool for researchers interested in studying the effects of demographic characteristics in large naturalistic data sets.