Predictive validity and efficiency of ongoing visual‐inspection criteria for interpreting functional analyses

Prior research has evaluated the reliability and validity of structured criteria for visually inspecting functional‐analysis (FA) results on a post‐hoc basis, after completion of the FA (i.e., post‐hoc visual inspection [PHVI]; e.g., Hagopian et al., 1997). However, most behavior analysts inspect FAs using ongoing visual inspection (OVI) as the FA is implemented, and the validity of applying structured criteria during OVI remains unknown. In this investigation, we evaluated the predictive validity and efficiency of applying structured criteria on an ongoing basis by comparing the interim interpretations produced through OVI with (a) the final interpretations produced by PHVI, (b) the authors’ post‐hoc interpretations (PHAI) reported in the research studies, and (c) the consensus interpretations of these two post‐hoc analyses. Ongoing visual inspection predicted the results of PHVI and the consensus interpretations with a very high degree of accuracy, and PHAI with a reasonably high degree of accuracy. Furthermore, the PHVI and PHAI results involved 32 FA sessions, on average, whereas the OVI required only 19 FA sessions to accurately identify the function(s) of destructive behavior (i.e., a 41% increase in efficiency). We discuss these findings relative to other methods designed to increase the accuracy and efficiency of FAs.     

Interpreting functional analysis outcomes using automated nonparametric statistical analysis

Current methods employed to interpret functional analysis data include visual analysis and post-hoc visual inspection (PHVI). However, these methods may be biased by dataset complexity, hand calculations, and rater experience. We examined whether an automated approach using nonparametric rank-based statistics could increase the accuracy and efficiency of functional analysis data interpretation. We applied Automated Nonparametric Statistical Analysis (ANSA) to a sample of 65 published functional analyses for which additional experimental evidence was available to verify behavior function. Results showed that exact behavior function agreement between ANSA and the publications authors was 83.1%, exact agreement between ANSA and PHVI was 75.4%, and exact agreement across all 3 methods was 64.6%. These preliminary findings suggest that ANSA has the potential to support the data interpretation process. A web application that incorporates the calculations and rules utilized by ANSA is accessible at https://ansa.shinyapps.io/ansa/     

Using equivalence‐based instruction to teach the visual analysis of graphs

Previous research has demonstrated low rater agreement when visually inspecting trends in single‐subject design graphs (see Wolfe, Seaman, & Drasgow, 2016). Didactic instruction of visual analysis of practicing behavior analysts has resulted in generally poor and unreliable performances (Danov & Symons, 2008; Diller, Barry, & Gelino, 2016). Therefore, a refined instructional technology to improve the reliability among behavior analysts is warranted. Developing research has focused on the application of equivalence‐based instruction (Brodsky & Fienup, 2018; Rehfeldt, 2011) for a variety of complex human behaviors. In the current study, equivalence‐based instruction was used to train four participants to identify functional relations displayed in five different classes of graphs. Training resulted in the formation of five equivalence classes by all participants consisting of three members (graph, functional relation rule, and functional relation statement). In addition, the skills were maintained for up to 2 weeks and generalized to novel graphs.     

How accurate are interpretations of curriculum-based measurement progress monitoring data? Visual analysis versus decision rules

Curriculum based measurement of oral reading (CBM-R) is used to monitor the effects of academic interventions for individual students. Decisions to continue, modify, or terminate these interventions are made by interpreting time series CBM-R data. Such interpretation is founded upon visual analysis or the application of decision rules. The purpose of this study was to compare the accuracy of visual analysis and decision rules. Visual analysts interpreted 108 CBM-R progress monitoring graphs one of three ways: (a) without graphic aids, (b) with a goal line, or (c) with a goal line and a trend line. Graphs differed along three dimensions, including trend magnitude, variability of observations, and duration of data collection. Automated trend line and data point decision rules were also applied to each graph. Inferential analyses permitted the estimation of the probability of a correct decision (i.e., the student is improving – continue the intervention, or the student is not improving – discontinue the intervention) for each evaluation method as a function of trend magnitude, variability of observations, and duration of data collection. All evaluation methods performed better when students made adequate progress. Visual analysis and decision rules performed similarly when observations were less variable. Results suggest that educators should collect data for more than six weeks, take steps to control measurement error, and visually analyze graphs when data are variable. Implications for practice and research are discussed.     

Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs

Because behavior analysis is a data-driven process, a critical skill for behavior analysts is accurate visual inspection and interpretation of single-case data. Study 1 was a basic study in which we increased the accuracy of visual inspection methods for A-B designs through two refinements of the split-middle (SM) method called the dual-criteria (DC) and conservative dual-criteria (CDC) methods. The accuracy of these visual inspection methods was compared with one another and with two statistical methods (Allison & Gorman, 1993; Gottman, 1981) using a computer-simulated Monte Carlo study. Results indicated that the DC and CDC methods controlled Type I error rates much better than the SM method and had considerably higher power (to detect real treatment effects) than the two statistical methods. In Study 2, brief verbal and written instructions with modeling were used to train 5 staff members to use the DC method, and in Study 3, these training methods were incorporated into a slide presentation and were used to rapidly (i.e., 15 min) train a large group of individuals (N = 87). Interpretation accuracy increased from a baseline mean of 55% to a treatment mean of 94% in Study 2 and from a baseline mean of 71% to a treatment mean of 95% in Study 3. Thus, Study 1 answered basic questions about the accuracy of several methods of interpreting A-B designs; Study 2 showed how that information could be used to increase the accuracy of human visual inspectors; and Study 3 showed how the training procedures from Study 2 could be modified into a format that would facilitate rapid training of large groups of individuals to interpret single-case designs.     

Values and the scientific culture of behavior analysis

As scientists and practitioners, behavior analysts must make frequent decisions that affect many lives. Scientific principles have been our guide as we work to promote effective action across a broad spectrum of cultural practices. Yet scientific principles alone may not be sufficient to guide our decision making in cases with potentially conflicting outcomes. In such cases, values function as guides to work through ethical conflicts. We will examine two ethical systems, radical behaviorism and functional contextualism, from which to consider the role of values in behavior analysis, and discuss potential concerns. Finally, we propose philosophical pragmatism, focusing on John Dewey's notions of community and dialogue, as a tradition that can help behavior analysts to integrate talk about values and scientific practices in ethical decision making.     

Data Smoothing: Prediction of Human Behavior, Detection of Behavioral Patterns, and Monitoring Treatment Effectiveness in Single-Subject Behavioral Studies

Data-smoothing can be particularly useful in predicting human behavior, detecting behavioral patterns, and monitoring treatment effectiveness in highly variable single-subject behavioral experiments that cannot be determined by only visual inspection of their graphs. Using an example from the applied behavior analytic literature, the use of moving-average and exponential data-smoothing aided the detection of the unique behavioral patterns of a child with autism across different treatments. Furthermore, the utility of the data-smoothing procedures to monitor and control the effectiveness of an intervention is discussed. The ease of their calculations suggest use of data-smoothing by behavior analysts whenever the effects of particular interventions are questionable.     

A Functional Analysis of Psychological Terms Redux

In his seminal paper, “An Operational Analysis of Psychological Terms,” Skinner (1945) offered the revolutionary suggestion that, rather than endlessly debating the meanings of psychological terms, psychologists should analyze the variables that control their occurrence as verbal behavior. Skinner''s suggestion reflected the essence of his 1957 book, Verbal Behavior, wherein he argued that the behaviors of which language is composed (i.e., speaking and listening) are controlled by variables found in the social environment (which he called the verbal community), and that analyzing those variables would lead to an understanding of the behaviors. Although Skinner formally introduced his radical approach to language in 1945, it has yet to be fully realized. The result is that psychologists, including behavior analysts, still debate the definitions of terms. In the present paper, I review Skinner''s functional approach to language and describe ways in which behavior analysts have already applied it to traditional psychological terms. I conclude by looking at other current terms in behavior analysis that engender some confusion and encourage behavior analysts to apply a functional analytic approach to their own verbal behavior.     

Functional analysis screening for multiple topographies of problem behavior

The current study evaluated a screening procedure for multiple topographies of problem behavior in the context of an ongoing functional analysis. Experimenters analyzed the function of a topography of primary concern while collecting data on topographies of secondary concern. We used visual analysis to predict the function of secondary topographies and a subsequent functional analysis to test those predictions. Results showed that a general function was accurately predicted for five of six (83%) secondary topographies. A specific function was predicted and supported for a subset of these topographies. The experimenters discuss the implication of these results for clinicians who have limited time for functional assessment.     

Reliability in Content Analysis

In a recent article in this journal, Lombard, Snyder‐Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests, compared the virtues and drawbacks of five popular reliability measures, and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This article proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, as well as two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data‐making efforts.     

The Analysis of Single-Case Research Data: Current Instructional Practices

Visual analysis is the predominant method of analysis in single-case research (SCR). However, most research suggests that agreement between visual analysts is poor, which may be due to a lack of clear guidelines and criteria for visual analysis, as well as variability in how individuals are trained. We developed a survey containing questions about the content and methods used to teach visual and statistical analysis of SCR data in verified course sequences (VCS) and distributed it via the VCS Coordinator Listserv. Thirty-seven instructors completed the survey. Results suggest that there is variability across instructors in some fundamental aspects of data analysis (e.g., number of effects required for a functional relation) but a great deal of consistency in others (e.g., emphasizing visual over statistical analysis). We discuss our results along with their implications both for teaching students to analyze SCR data and for conducting additional research on behavior-analytic training programs.

     

An Evaluation of Prompting and Reinforcement for Training Visual Analysis Skills

This study examined the use of an instructional package consisting of structured criteria, extrastimulus prompts, prompt delay, and reinforcement contingencies to improve decision making based on visual analysis of case-study (A/B) design data. Four participants without backgrounds in behavior analysis received training under two experimental conditions: first, a prompt-delay-plus-accuracy-reward condition and then a prompt-delay-plus-accuracy-and-latency-reward condition. A multiple-baseline-across-participants design was used to evaluate the impact of the instructional package on decision accuracy, response speed, and decision types. Results demonstrated that all participants made substantial improvements in decision accuracy as a function of the instructional package. Gains were highest for three participants who relied on prompts and developed mastery in the prompt-delay-plus-accuracy-reward condition. Generalization to unprompted conditions for all participants was immediate and maintained 1 week after training was concluded. Results are discussed in terms of behavioral mechanisms that appeared to control decision making during instruction, practical applications to applied settings, and needs for future research.     

An evaluation of the agreement between the conservative dual‐criterion method and expert visual analysis

The conservative dual‐criterion (CDC) method was developed to standardize the analysis of single‐subject experimental designs data, but to date its accuracy has been evaluated only by comparing results to the statistical parameters of graphs. Our study investigated agreement between expert visual analysts and the CDC method on 66 AB tiers from published multiple baseline graphs. We found strong agreement between the two methods for certain types of graphs and discuss implications of the findings and areas for future research.     

A mobile computing solution for collecting functional analysis data on a Pocket PC

The present paper provides a task analysis for creating a computerized data system using a Pocket PC and Microsoft Visual Basic. With Visual Basic software and any handheld device running the Windows Moble operating system, this task analysis will allow behavior analysts to program and customize their own functional analysis data-collection system. The program will allow the user to select the type of behavior to be recorded, choose between interval and frequency data collection, and summarize data for graphing and analysis. We also provide suggestions for customizing the data-collection system for idiosyncratic research and clinical needs.     

An Evaluation of the Effects of Graphic Aids in Improving Decision Accuracy in a Continuous Treatment Design

Educators, school psychologists, and other professionals must evaluate student progress and decide to continue, modify, or terminate instructional programs to ensure student success. For this purpose, progress-monitoring data are often collected, plotted graphically, and visually analyzed. The current study evaluated the impact of three common formats for visual analysis (scatter plots, scatter plots with trend lines, and scatter plots with trend and aim lines) on the decision accuracy of 52 novice analysts. All participants viewed 18 time-series graphs that depicted student growth on a continuous metric (e.g., oral reading fluency). Participants rated each graph as depicting substantial progress, minimal progress, or no progress. The magnitude of the true slope for each graph was fixed to 3.00 (substantial progress), 0.75 (minimal progress), or 0 (no progress). Inferential analyses were used to determine the probability of a participant correctly identifying different magnitudes of trend in the presence of different visual aids (no visual aid, trend line, and trend line with aim line). The odds of correctly identifying trend were influenced by visual aid (p < .01) and trend magnitude (p < .01). The addition of a trend line resulted in a sharp increase in the probability of making a correct decision. Graphs depicting minimal progress reduced the probability of a correct decision.     

Quantitative integration of single-subject studies: Methods and misinterpretations

Derenne and Baron (1999) criticized a quantitative literature review by Kollins, Newland, and Critchfield (1997) and raised several important issues with respect to the integration of single-subject data. In their criticism they argued that the quantitative integration of data across experiments conducted by Kollins et al. is a meta-analysis and, as such, is inappropriate. We reply that Kollins et al. offered behavior analysts a technique for integrating quantitative information in a way that draws from the strengths of behavior analysis. Although the quantitative technique is true to the original spirit of meta-analysis, it bears little resemblance to meta-analyses as currently conducted or defined and offers behavior analysts a potentially useful tool for comparing data from multiple sources. We also argue that other criticisms raised by Derenne and Baron were inaccurate or irrelevant to the original article. Our response highlights two main points: (a) There are meaningful quantitative techniques for examining single-subject data across studies without compromising the integrity of behavior analysis; and (b) the healthiest way to refute or question findings in any viable field of scientific inquiry is through empirical investigation.     

The Effects of Experience and a Decision Aid on the Slope,Scatter, and Bias of Earnings Forecasts

This study examines the joint effects of experience and a statistical decision aid on several dimensions of forecasting accuracy. Undergraduate business students, graduate business students, and experienced financial analysts made probabilistic earnings forecasts for 16 firms, either with or without access to a decision aid. Covariance decomposition was used to partition the overall accuracy score into the following components: bias, scatter, and slope. Consistent with prior research, the absolute accuracy of subjects was quite poor, even worse than a uniform forecaster. In contrast to previous results which documented an inverted experience–accuracy effect, experience had a positive impact on accuracy in the current study. This difference in results may be due, at least in part, to the different conditions under which subjects performed the earnings forecasting task. The covariance analysis showed that experienced financial analysts and MBA students had significantly better slope and lower scatter than undergraduate students. The financial analysts were also more biased than either of the student groups, yet still achieved the highest overall accuracy. The decision aid improved forecasting accuracy by increasing the slope of the analysts’ forecasts, while decreasing the scatter in the graduate and undergraduate students’ forecasts.     

Visual inspection of data revisited: Do the eyes still have it?

In behavior analysis, visual inspection of graphic information is the standard by which data are evaluated. Efforts to supplement visual inspection using inferential statistical procedures to assess intervention effects (e.g., analysis of variance or time-series analysis) have met with opposition. However, when serial dependence is present in the data, the use of visual inspection by itself may prove to be problematic. Previously published reports demonstrate that autocorrelated data influence trained observers' ability to identify level treatment effects and trends that occur in the intervention phase of experiments. In this report, four recent studies are presented in which autoregressive equations were used to produce point-to-point functions to simulate experimental data. In each study, various parameters were manipulated to assess trained observers' responses to changes in point-to-point functions from the baseline condition to intervention. Level shifts over baseline behavior (treatment effect), as well as no change from baseline (no treatment effect or trend), were most readily identified by observers, but trends were rarely recognized. Furthermore, other factors previously thought to augment and improve observers' responses had no impact. Results are discussed in terms of the use of visual inspection and the training of behavior analysts.