A COMPARISON OF INDIRECT MEASURES FOR LONG-DURATION BEHAVIORS1

A computer program was developed to simulate long-duration behavior. The program generated data by using two time-series models linked together by a covariance parameter. By varying the parameters of the program, the characteristics of the simulated behavior were altered, including the mean and variance of the durations of behavior, the mean and variance of the interresponse time, and the degree of relationship within and between durations of behavior and interresponse times. Four data-collection methods were applied to the simulated data: whole interval, partial interval, momentary time-sampling, and continuous measures. The accuracies of the first three recording measures were judged by comparing them to the continuous measure. The results indicated that only the momentary time-sampling approach yielded unbiased results. The degree of bias for the whole and partial interval measures was a function of the ratio of the interval recording length to the sum of the mean duration of behavior and the mean interresponse time. Unfortunately, it was concluded that researchers cannot estimate the magnitudes of these latter two parameters for most behaviors of interest and, consequently, are unable to choose an appropriate interval length. Therefore, it was recommended that the use of interval recording approaches be greatly restricted and that momentary time-sampling be substituted where possible.     

Quantifying false positives in simulated events using partial interval recording and momentary time sampling with dual‐criteria methods

The dual‐criteria (DC) and conservative dual‐criteria (CDC) methods allow clinicians and researchers to quantify the occurrence of false‐positive outcomes within single‐case experimental designs. The purpose of the current study was to use these DC and CDC methods to measure the incidence of false positives with simulated data collected via discontinuous interval methods (i.e., momentary time sampling, partial‐interval recording) as a function of data series length. We generated event data to create 316,800 unique simulated data series for analysis. In Experiment 1, we evaluated how changes in relevant parameters (i.e., interval sizes, event durations, IRT‐to‐event‐run ratios) produced false positives with momentary time sampling procedures. We also assessed the degree that the CDC method produced fewer false positives than the DC method with simulated interval data. In Experiment 2, we used similar procedures to quantify the occurrence of false positives with partial‐interval recording data. We successfully replicated outcomes from previous research in the current study, though such results only highlight the generality of the procedures relating to false positive (and not false negative) outcomes. That is, these results indicate MTS and PIR may adequately control for false positives, but our conclusions are limited by a lack of data on power.     

Application of a simple recording system to the analysis of free-play behavior in autistic children

An observational system, which has been developed to facilitate recording of the total behavioral repertoire of autistic children, involves time-sampling recording of behavior with the help of a common Stenograph machine. Categories which exhausted all behavior were defined. Each category corresponded with a designated key on the Stenograph machine. The observer depressed one key at each 1-sec interval. The observer was paced by audible beats from a metronome. A naive observer can be used with this method. The observer is not mechanically limited and a minimum of observer training is required to obtain reliable measures. The data sampled during a five-week observation period indicated the stability of a taxonomic instrument of behavior based upon direct, time-sampling observations and the stability of spontaneous autistic behavior. Results showed that the behavior of the subjects was largely nonrandom and unsocialized in character.     

A comparison of frequency, interval, and time-sampling methods of data collection

Data representing high, medium, and low response rates in constant and nonconstant patterns were generated by electromechanical equipment to determine whether the same data collected by time-sampling, interval recording, and frequency recording would be represented similarly by each method. Results indicated: (1) that time-sampling provided an extremely inaccurate estimate of responding, and (2) that interval recording accurately represented responding of low and medium rates, but grossly underestimated high-rate responding.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Methods of time sampling: A reappraisal of momentary time sampling and partial interval recording

We compared the accuracy of momentary time sampling (MTS) and partial interval recording (PIR) in estimating both absolute behavioral levels and relative change. A computer randomly generated runs of pseudobehavior varying in duration and rate and simulated MTS and PIR of each run. Results indicated that when estimating absolute behavioral levels, duration rather than rate should be used as the dependent measure, and MTS is more accurate than PIR. In contrast, PIR is the more sensitive method for detecting relative changes in behavioral levels, although, at high rates, PIR tends to underestimate the degree of change.     

Measurement error in direct observations: a comparison of common recording methods

Videotapes of three brief duration, three medium duration and three long duration types of stereotyped behaviour (of eight severely retarded children) were analysed to provide a criterion record of the true percentage duration of the behaviour. The criterion record was compared with the records produced by four time-sampling methods: a whole-interval method, two partial-interval methods and a momentary time-sampling method. As predicted, the whole-interval method grossly underestimated and the partial-interval methods grossly overestimated the true percentage duration of the behaviour, except when the duration of individual responses was much longer than the observation interval. Momentary time-sampling was not an errorless method but was consistently superior to the other methods.The implications of these findings for the detection of treatment effects by direct observations are discussed.     

ASSESSMENT OF THERAPISTS' PREFERENCES FOR DISCONTINUOUS MEASUREMENT SYSTEMS

Recent studies have evaluated the sensitivity of discontinuous recording methods for detecting behavior changes; however, relatively few have evaluated observers' preferences for such methods. To address this limitation, this study used a concurrent chains design to evaluate eight therapists' preferences for collecting data using momentary time sampling (MTS) and partial interval recording with interval sizes of 10 s and 1 min. Results of the preference assessment showed that six of the eight therapists preferred to collect data using 1‐min MTS; however, results of a follow‐up questionnaire suggest that most therapists preferred 1‐min MTS primarily because it was easy to use rather than that it was accurate. These results are discussed with respect to recent findings on the sensitivity of MTS and partial interval recording. Copyright © 2014 John Wiley & Sons, Ltd.     

The use of momentary time sampling to assess both frequency and duration of behavior

When the time-sampling interval size can be set so it is smaller than both the shortest behavior instance and the shortest interresponse time, momentary time sampling will yield an accurate count of the frequency of behavior occurrence. It will also yield an unbiased estimate of behavior duration.     

Detecting Changes in Simulated Events Using Partial‐Interval Recording and Momentary Time Sampling III: Evaluating Sensitivity as a Function of Session Length

In a series of two studies, we graphed simulated data representing continuous duration recording and continuous frequency recording into ABAB reversal designs depicting small, moderate, and large behavior changes during 10‐min, 30‐min, and 60‐min sessions. Data sets were re‐scored using partial‐interval recording and momentary time sampling with interval sizes set at 10 s, 20 s, 30 s, 1 min, and 2 min. In study 1, we visually inspected converted data for experimental control and compared the conclusion with those from the respective continuous duration recording or continuous frequency recording data to test for false negatives. In study 2, we evaluated the extent to which interval methods that were sensitive to changes in study 1 produced false positives. In part, the results show that momentary time sampling with interval sizes up to 30 s detected a wide range of changes in duration events and frequency events during lengthier observation periods. The practical implications of the findings are briefly discussed. Copyright © 2011 John Wiley & Sons, Ltd.     

Detecting changes in simulated events using partial‐interval recording and momentary time sampling

In a series of three studies, we evaluated simulated data with reversal designs to determine whether partial‐interval recording (PIR) and momentary time sampling (MTS) detected changes that were evident with continuous measures. The results from Study 1 showed that MTS with interval sizes up to 30 s detected most of the moderate and large changes in duration events and MTS with interval sizes up to 1 min detected most large changes in duration events. By comparison, PIR with 10‐s intervals detected approximately half of all changes in duration events. The results of Study 2 showed that only 10‐s PIR reliably detected most small, moderate, and large changes in frequency events. The results of Study 3 showed that PIR with 10‐s intervals generated a relatively high percentage of false positives for duration events, whereas MTS did not. As a whole, the results support previous findings, but also provide new guidelines for the use of PIR and MTS. Copyright © 2008 John Wiley & Sons, Ltd.     

A review of the observational data-collection and reliability procedures reported in The Journal of Applied Behavior Analysis

The research published in the Journal of Applied Behavior Analysis (1968 to 1975) was surveyed for three basic elements: data-collection methods, reliability procedures, and reliability scores. Three-quarters of the studies reported observational data. Most of these studies' observational methods were variations of event recording, trial scoring, interval recording, or time-sample recording. Almost all studies reported assessment of observer reliability, usually total or point-by-point percentage agreement scores. About half the agreement scores were consistently above 90%. Less than one-quarter of the studies reported that reliability was assessed at least once per condition.     

Detecting changes in simulated events II: Using variations of momentary time‐sampling to measure changes in duration events

The extent to which a greater proportion of small behavior changes could be detected with momentary time‐sampling (MTS) was evaluated by (a) combining various interval sizes of partial‐interval recording (PIR) with 20 s, 30 s, 1 min MTS and (b) using variable interval sizes of MTS that were based on means of 20 s and 1 min. For each targeted percentage, low, moderate, and high inter‐response times (IRTs) to event‐run ratios were compared with reversal designs to determine whether sensitivity increased with either variation of MTS. The results showed that (a) combinations of 30 s and 1 min MTS/PIR yielded increased sensitivity over MTS alone; however, the increased sensitivity was offset by an increased probability of generating false positives and (b) variable‐interval MTS produced comparable sensitivity to fixed‐interval MTS. Thus, none of the methods increased detection of small behavior changes (decreased false negatives) without also increasing false positives. Copyright © 2009 John Wiley & Sons, Ltd.     

Interval recording for duration events: a re‐evaluation

In two experiments, events that were recorded using continuous duration recording (CDR) were rescored using 10‐s partial interval (PIR), 10‐s momentary time sampling (MTS) and 20‐s MTS. Results of Experiment 1 showed that data paths generated by each interval method produced conclusions about functional control that were similar to those based on CDR when using reversal designs; however, for multielement designs, 10‐s PIR was prone to showing differentiation between data paths that was not evident with CDR. Results of Experiment 2 showed that both 10‐s and 20‐s MTS yielded data paths on behavior–behavior relations (e.g., covarying responses) that were consistent with CDR whereas 10‐s PIR produced some behavior–behavior patterns that were not. In both experiments, 10‐s MTS generated data paths that were nearly identical to the respective CDR data paths. The implications of these findings for researchers and clinicians are briefly discussed. Copyright © 2007 John Wiley & Sons, Ltd.     

Comparison of observational recordings in vivo, via mirror, and via television

Simultaneous observational recordings were made in vivo, via an observation mirror, and via closed circuit television. Three of nine observers had extensive experience recording behavior in vivo; three had extensive experience recording behavior via mirror; and three had extensive experience recording via television. Observers recorded nine categories of disruptive behavior for children in a special class setting. Frequencies of behavior recorded in vivo, via mirror, and via television differed significantly for only one category, vocalization. There were no significant main effects or interactions involving the observers' previous experience. Occurrence reliability coefficients computed within and between media demonstrated the similarity of observer agreement in all three media. Data collection procedures using an observational mirror or closed circuit television appear to be reasonable alternatives to in vivo observation in circumstances similar to those in the present study.     

THE BEHAVIOR OBSERVATION INSTRUMENT: A METHOD OF DIRECT OBSERVATION FOR PROGRAM EVALUATION1

The background and development of a multicategory direct observation system, the Behavior Observation Instrument (BOI), is described. This time-sampling procedure for recording the behavior of persons is demonstrated in several treatment settings and the results applied to issues of program evaluation. Elements that have prevented direct observation from being widely adopted, such as costs, manpower, and training requirements, are systematically analyzed. A basic psychometric analysis of the instrument is used to determine optimum frequency and duration of observation intervals as well as observer agreement. The results imply that direct observation methods, once assumed by some to belong to the special province of the single-subject design, can be used to assess the effects of programs on groups of psychiatric clients in an efficient and economic manner.     

Momentary time sampling as an estimate of percentage time: A field validation

This study examined the percentage time estimates of momentary time sampling against the real time obtained with handheld computers in a natural setting. Twenty-two concurrent observations were conducted in elementary schools by one observer who used 15-s momentary time sampling and a second who used a handheld computer. Results for the six behaviors showed a close correspondence between the momentary time sampling percentage observation intervals and the real time percentage observation time, although 15-s momentary time sampling tended not to sample low-frequency short-duration behaviors. The results confirmed laboratory findings that short-interval momentary time sampling estimates percentage time accurately for a wide range of behavior frequencies and durations, and suggested that observers using momentary time sampling in a natural setting are able to obtain accurate data.     

Four Methods for Analyzing Partial Interval Recording Data,with Application to Single-Case Research

Partial interval recording (PIR) is a procedure for collecting measurements during direct observation of behavior. It is used in several areas of educational and psychological research, particularly in connection with single-case research. Measurements collected using partial interval recording suffer from construct invalidity because they are not readily interpretable in terms of the underlying characteristics of the behavior. Using an alternating renewal process model for the behavior under observation, we demonstrate that ignoring the construct invalidity of PIR data can produce misleading inferences, such as inferring that an intervention reduces the prevalence of an undesirable behavior when in fact it has the opposite effect. We then propose four different methods for analyzing PIR summary measurements, each of which can be used to draw inferences about interpretable behavioral parameters. We demonstrate the methods by applying them to data from two single-case studies of problem behavior.     

Accuracy in parameter estimation for targeted effects in structural equation modeling: sample size planning for narrow confidence intervals

In addition to evaluating a structural equation model (SEM) as a whole, often the model parameters are of interest and confidence intervals for those parameters are formed. Given a model with a good overall fit, it is entirely possible for the targeted effects of interest to have very wide confidence intervals, thus giving little information about the magnitude of the population targeted effects. With the goal of obtaining sufficiently narrow confidence intervals for the model parameters of interest, sample size planning methods for SEM are developed from the accuracy in parameter estimation approach. One method plans for the sample size so that the expected confidence interval width is sufficiently narrow. An extended procedure ensures that the obtained confidence interval will be no wider than desired, with some specified degree of assurance. A Monte Carlo simulation study was conducted that verified the effectiveness of the procedures in realistic situations. The methods developed have been implemented in the MBESS package in R so that they can be easily applied by researchers.     

DETECTING CHANGES IN NON‐SIMULATED EVENTS USING PARTIAL INTERVAL RECORDING AND MOMENTARY TIME SAMPLING: EVALUATING FALSE POSITIVES,FALSE NEGATIVES,AND TRENDING

Several studies have evaluated false positives and false negatives produced with partial interval recording (PIR) and momentary time sampling (MTS) using simulated data. However, no study to date has evaluated false positives and negatives using a large sample of non‐simulated behaviors. In addition, few studies have evaluated whether interval methods of data collection alter trends that are evident in continuous records. We conducted three experiments to evaluate the extent to which various interval sizes of MTS and PIR produced false negatives (Experiment 1), false positives (Experiment 2), and trends that were inconsistent with the continuous records (Experiment 3). Collectively, the results show the following: (i) 10‐s PIR and 10‐s MTS produced few false negatives and few false positives (i.e., both were sensitive) to changes in duration events; (ii) 10‐s PIR produced very few false negatives, but an unexpected high percentage of false positives for frequency events; and (iii) each interval size of PIR and MTS produced a high percentage of changes in trending for duration events and frequency events. We briefly discuss the potential limitations and clinical implications of these findings. Copyright © 2012 John Wiley & Sons, Ltd.