An exploration of the interrater agreement of visual analysis with and without context

Visual analysis is integral to the analysis of single-case experimental design (SCED) data. Previous studies have shown that many factors may influence the interrater agreement (IRA) of visual analysis. One factor that has received little direct attention is the impact of contextual information. In the current study, authors of recently published SCED studies were asked to make judgments regarding functional relations based on published datasets that met criteria for design quality. Respondents were randomly assigned to view graphs with or without contextual information and the degree of interrater agreement was compared. Results revealed that contextual information had no impact on IRA for decisions of a functional relation. IRA was high across both groups for 6 of the 7 datasets examined. Implications and recommendations based on these results are discussed.     

Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs

Because behavior analysis is a data-driven process, a critical skill for behavior analysts is accurate visual inspection and interpretation of single-case data. Study 1 was a basic study in which we increased the accuracy of visual inspection methods for A-B designs through two refinements of the split-middle (SM) method called the dual-criteria (DC) and conservative dual-criteria (CDC) methods. The accuracy of these visual inspection methods was compared with one another and with two statistical methods (Allison & Gorman, 1993; Gottman, 1981) using a computer-simulated Monte Carlo study. Results indicated that the DC and CDC methods controlled Type I error rates much better than the SM method and had considerably higher power (to detect real treatment effects) than the two statistical methods. In Study 2, brief verbal and written instructions with modeling were used to train 5 staff members to use the DC method, and in Study 3, these training methods were incorporated into a slide presentation and were used to rapidly (i.e., 15 min) train a large group of individuals (N = 87). Interpretation accuracy increased from a baseline mean of 55% to a treatment mean of 94% in Study 2 and from a baseline mean of 71% to a treatment mean of 95% in Study 3. Thus, Study 1 answered basic questions about the accuracy of several methods of interpreting A-B designs; Study 2 showed how that information could be used to increase the accuracy of human visual inspectors; and Study 3 showed how the training procedures from Study 2 could be modified into a format that would facilitate rapid training of large groups of individuals to interpret single-case designs.     

A comparison of sources of baseline data for treatments of problem behavior following a functional analysis

It is common practice in research on the treatment of problem behavior to compare levels of targeted behaviors during treatment to levels when treatment is not in place. Some researchers use data collected as part of a multielement functional analysis as the initial baseline, whereas others collect new baseline data following completion of the functional analysis. We evaluated whether the source of baseline data influences the reliability and efficiency of decision-making. Results suggest that similar decisions are made in regard to treatment efficacy using the different sources of baseline data, but using data from a multielement functional analysis as baseline may save time. Interrater agreement was adequate, but lower for some graphs than has been observed in past studies. Several potential explanations for this discrepancy are discussed.     

Data-based decision making: the impact of data variability, training, and context

The current study examines agreement among individuals with varying expertise in behavior analysis about the length of baseline when data were presented point by point. Participants were asked to respond to baseline data and to indicate when to terminate the baseline phase. When only minimal information was provided about the data set, experts and Board Certified Behavior Analyst participants generated baselines of similar lengths, whereas novices did not. Agreement was similar across participants when variability was low but deteriorated as variability in the data set increased. Participants generated shorter baselines when provided with information regarding the independent or dependent variable. Implications for training and the use of visual inspection are discussed.     

Intervention effects and relative variation as dimensions in experts' use of visual inference

Recent research indicates that when analyzing graphically presented single-subject data, subjects trained in visual inference appear to attend to large changes between phases regardless of relative variation and do not differentiate among common intervention effect patterns. In this follow-up study, experts in applied behavior analysis completed a free-sort task designed to assess the effects of these dimensions on their use of visual inference. The results indicate that they tended to differentiate among common intervention effect patterns but did not attend to relative variation in the data.     

Visual analysis of data in a multielement design

Ninety Board Certified Behavior Analysts (BCBAs) and 19 editorial board members evaluated hypothetical data presented in a multielement design. We manipulated the variability, trend, and mean shift of the data and asked the participants to determine if the data demonstrated experimental control. The results showed that variability, trend, and mean shift interacted to affect the participants’ ratings of experimental control. The level of agreement between participants was variable, but was generally lower than in previous research.     

A Monte Carlo evaluation of masked visual analysis in response‐guided versus fixed‐criteria multiple‐baseline designs

We developed masked visual analysis (MVA) as a structured complement to traditional visual analysis. The purpose of the present investigation was to compare the effects of computer‐simulated MVA of a four‐case multiple‐baseline (MB) design in which the phase lengths are determined by an ongoing visual analysis (i.e., response‐guided) versus those in which the phase lengths are established a priori (i.e., fixed criteria). We observed an acceptably low probability (less than .05) of false detection of treatment effects. The probability of correctly detecting a true effect frequently exceeded .80 and was higher when: (a) the masked visual analyst extended phases based on an ongoing visual analysis, (b) the effects were larger, (c) the effects were more immediate and abrupt, and (d) the effects of random and extraneous error factors were simpler. Our findings indicate that MVA is a valuable combined methodological and data‐analysis tool for single‐case intervention researchers.     

Extensions of a versatile randomization test for assessing single-case intervention effects

The purpose of the present study was to investigate the statistical properties of two extensions of the Levin-Wampold (1999) single-case simultaneous start-point model's comparative effectiveness randomization test. The two extensions were (a) adapting the test to situations where there are more than two different intervention conditions and (b) examining the test's performance in classroom-based intervention situations, where the number of time periods (and associated outcome observations) is much smaller than in the contexts for which the test was originally developed. Various Monte Carlo sampling situations were investigated, including from one to five participant blocks per condition and differing numbers of time periods, potential intervention start points, degrees of within-phase autocorrelation, and effect sizes. For all situations, it was found that the Type I error probability of the randomization test was maintained at an acceptable level. With a few notable exceptions, respectable power was observed only in situations where the numbers of observations and potential intervention start points were relatively large, effect sizes were large, and the degree of within-phase autocorrelation was relatively low. It was concluded that the comparative effectiveness randomization test, with its desirable internal validity and statistical-conclusion validity features, is a versatile analytic tool that can be incorporated into a variety of single-case school psychology intervention research situations.     

Effects of lines of progress and semilogarithmic charts on ratings of charted data

The extent to which interrater agreement and ratings of significance on both changes in level and trend are affected by lines of progress and semilogarithmic charts was investigated. Thirteen graduate students rated four sets of charts, each set containing 19 phase changes. Set I data were plotted on equal interval charts. In Set II a line of progress was drawn through each phase on each chart. In Set III data points were replotted on semilogarithmic charts. In Set IV a line of progress was drawn through each phase of each Set III chart. A significant main effect on interrater agreement was found for lines of progress as well as a significant 2-way interaction between lines of progress and change type. Three main effects (chart type, lines of progress, and type of change) and a significant 3-way interaction were found for ratings of significance. Implications of these data for visual analysis of charted data are discussed.     

From a single-level analysis to a multilevel analysis of single-case experimental designs

Multilevel modeling provides one approach to synthesizing single-case experimental design data. In this study, we present the multilevel model (the two-level and the three-level models) for summarizing single-case results over cases, over studies, or both. In addition to the basic multilevel models, we elaborate on several plausible alternative models. We apply the proposed models to real datasets and investigate to what extent the estimated treatment effect is dependent on the modeling specifications and the underlying assumptions. By considering a range of plausible models and assumptions, researchers can determine the degree to which the effect estimates and conclusions are sensitive to the specific assumptions made. If the same conclusions are reached across a range of plausible assumptions, confidence in the conclusions can be enhanced. We advise researchers not to focus on one model but conduct multiple plausible multilevel analyses and investigate whether the results depend on the modeling options.     

Incentive payment and nonmanagerial productivity: An interrupted time series analysis of magnitude and trend

Using an interrupted time series design, this paper examined long-term changes in the magnitude and trend of productivity following the introduction of nonmanagerial incentive payment. Analyses of three 114-month data series from a unionized iron foundry revealed a strong asymptotic increase in productivity without corresponding increases in labor costs or employee grievances. These findings substantiated a power-curve trend in productivity, implying that wage incentives might stimulate employees to learn efficacious task behaviors, and suggesting that incentive payment's long-term productive efficacy might be greater than many of the short-term gains reported in prior research     

Meta- and statistical analysis of single-case intervention research data: Quantitative gifts and a wish list

In this commentary, we add to the spirit of the articles appearing in the special series devoted to meta- and statistical analysis of single-case intervention-design data. Following a brief discussion of historical factors leading to our initial involvement in statistical analysis of such data, we discuss: (a) the value added by including statistical-analysis recommendations in the What Works Clearinghouse Standards for single-case intervention designs; (b) the importance of visual analysis in single-case intervention research, along with the distinctive role that could be played by single-case effect-size measures; and (c) the elevated internal validity and statistical-conclusion validity afforded by the incorporation of various forms of randomization into basic single-case design structures. For the future, we envision more widespread application of quantitative analyses, as critical adjuncts to visual analysis, in both primary single-case intervention research studies and literature reviews in the behavioral, educational, and health sciences.     

The Effects of Relaxation and Postural Training on External Perception: Improvement of Visual Acuity, Visual Field, and Hearing Acuity

This study examined the effects of muscular relaxation and postural training on external perception using a visual acuity test, a visual field test, and a hearing acuity test. Eighteen undergraduate students were randomly assigned to experimental and control groups. The experimental group underwent muscular relaxation and postural training. Each subject in this group was administered the tests before and after the training. Each subject in the control group carried out the tests before and after participating in a 30-min conversation with the experimenter. On all three tests, the experimental group improved significantly more than the control group.     

Crisis intervention and the prevention of institutionalization: An interrupted time series analysis

In the context of the community mental health movement in Illinois, the evolution and development of a crisis intervention program aimed at avoiding state hospitalization and bringing more appropriate and efficacious resources to bear on the difficulties of the individual and/or family is described. This intervention program is characterized by a more active-seeking style of delivering mental health services. With the novel application of interrupted time series analysis to both the targeted and matched nonequivalent control communities the efficacy of this more active intervention in reducing the number of state hospital administrations diagnosed as "mentally ill" is highlighted. The cost/benefit to the taxpayer is also discussed.     

Evaluating effects in short time series: alternative models of analysis

In the applied context, short time-series designs are suitable to evaluate a treatment effect. These designs present serious problems given autocorrelation among data and the small number of observations involved. This paper describes analytic procedures that have been applied to data from short time series, and an alternative which is a new version of the generalized least squares method to simplify estimation of the error covariance matrix. Using the results of a simulation study and assuming a stationary first-order autoregressive model, it is proposed that the original observations and the design matrix be transformed by means of the square root or Cholesky factor of the inverse of the covariance matrix. This provides a solution to the problem of estimating the parameters of the error covariance matrix. Finally, the results of the simulation study obtained using the proposed generalized least squares method are compared with those obtained by the ordinary least squares approach. The probability of Type I error associated with the proposed method is close to the nominal value for all values of rho1 and n investigated, especially for positive values of rho1. The proposed generalized least squares method corrects the effect of autocorrelation on the test's power.     

Progressing toward data intimacy: a review of within-session data analysis

Visual inspection of data is a common method for understanding, responding to, and communicating important behavior-environment relations in single-subject research. In a field that was once dominated by cumulative, moment-to-moment records of behavior, a number of graphic forms currently exist that aggregate data into larger units. In this paper, we describe the continuum of aggregation that ranges from distant to intimate displays of behavioral data. To aid in an understanding of the conditions under which a more intimate analysis is warranted (i.e., one that provides a richer analysis than that provided by condition or session aggregates), we review a sample of research articles for which within-session data depiction has enhanced the visual analysis of applied behavioral research.     

The effects of reinforcement magnitude on functional analysis outcomes

The duration or magnitude of reinforcement has varied and often appears to have been selected arbitrarily in functional analysis research. Few studies have evaluated the effects of reinforcement magnitude on problem behavior, even though basic findings indicate that this parameter may affect response rates during functional analyses. In the current study, 6 children with autism or developmental disabilities who engaged in severe problem behavior were exposed to three separate functional analyses, each of which varied in reinforcement magnitude. Results of these functional analyses were compared to determine if a particular reinforcement magnitude was associated with the most conclusive outcomes. In most cases, the same conclusion about the functions of problem behavior was drawn regardless of the reinforcement magnitude.     

Multiple time scales in serial production of force: a tutorial on power spectral analysis of motor variability

We present a tutorial on a power spectral approach to variability in serial motor performance, describing as a case study two experiments on the form of the variance in two force production tasks. In Experiment 1 we examine grip force and load force in repetitive unimanual pulling; in Experiment 2, we describe repetitive bimanual pressing. In both experiments log-log plots of power spectral density of peak force of the responses in each stream against frequency (i.e. periodicity or repetition cycle defined with respect to the ordered succession of responses) were approximately linear with negative slopes which varied systematically with test conditions. We propose the two response streams in each experiment are associated with different levels of sustained attention and state a simple model involving summation of moving average processes on multiple time scales as a qualitative account of the changes in 1/f slope.     

Effects of models and demonstrations of personal narratives on the use of lexical ties by writers with mental retardation

A single subject reversal design was used to investigate the effects of two methods for teaching writing to eight adolescents with mild or moderate mental retardation during a weekly process writing workshop. During each workshop, each student wrote a personal narrative while the teacher circulated among the group and provided positive verbal feedback and spelled words on request. Students' texts were later typed with corrected spelling and punctuation and returned to them with verbal praise before the next writing workshop. During the first experimental condition, the subjects listened to an exemplar text selected by the teacher. In the second phase, the teacher read a narrative she had written as the exemplar text. During the third experimental condition, the teacher orally self-instructed herself through steps in writing a narrative. The subjects' texts were analyzed for number of words written and number and type of lexical ties used to produce cohesion between sentences in a text. The model narrative produced the best gains in the number of words written by the poorer readers, while the demonstration improved the cohesiveness of their texts. The better readers and writers wrote more cohesively during the model narrative and produced longer text during the demonstration condition.     

Dynamic factor analysis of nonstationary multivariate time series

A dynamic factor model is proposed for the analysis of multivariate nonstationary time series in the time domain. The nonstationarity in the series is represented by a linear time dependent mean function. This mild form of nonstationarity is often relevant in analyzing socio-economic time series met in practice. Through the use of an extended version of Molenaar's stationary dynamic factor analysis method, the effect of nonstationarity on the latent factor series is incorporated in the dynamic nonstationary factor model (DNFM). It is shown that the estimation of the unknown parameters in this model can be easily carried out by reformulating the DNFM as a covariance structure model and adopting the ML algorithm proposed by Jöreskog. Furthermore, an empirical example is given to demonstrate the usefulness of the proposed DNFM and the analysis.