A post hoc correction procedure for systematic errors in time-sampling duration estimates

Numerous previous studies have shown that partial-interval sampling in direct observation systematically overestimates duration and underestimates frequency. Whole-interval sampling systematically underestimates both duration and frequency. This paper presents a post hoc method through which the systematic errors in duration estimates in partial-interval sampling and whole-interval sampling can be minimized.     

Social and solipsistic observer training: Effects on agreement with a criterion

The need to train accurate, not necessarily agreeing, observers is discussed. Intraobserver consistency as an intermediate criterion in such training is proposed and contrasted with the more familiar criterion of interobserver agreement. Videotaped observations of social interactions between handicapped and nonhandicapped preschoolers provided the medium for examining the criterion agreement of four observers trained against each type of standard. Observers generally failed to show high levels of criterion agreement whether trained to a within- or to a between-observer agreement standard. The results varied somewhat with the frequency of behaviors, however. Correlations between interobserver agreement and intraobserver consistency were variable but somewhat higher when interobserver agreement was the training criterion than when intraobserver consistency was the criterion. Correlations between interobserver agreement and criterion agreement ranged from — .16 to .89 during interobserver agreement training. Correlations between intraobserver consistency and criterion agreement ranged from — .23 to .99 during intraobserver consistency training.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Frequency,complexity, and clarity as covariates of observer reliability

Factors affecting interobserver agreement (reliability) with a comprehensive coding system in the naturalistic observation of children were examined. Data from 117 pairs of observations on 35 children and their families were examined with respect to reliability and three possible covariates: response frequency, observation complexity, and code definition clarity. Analysis of results strongly supported response frequency as a positive covariate of interobserver agreement. Complexity was found to negatively covary with interobserver agreement. The relationship between code clarity and reliability was in the predicted direction but failed to obtain statistical significance. Implications for observer training and data collection in observational studies are discussed.An earlier version of this article was prepared for presentation at The Association for the Advancement of Behavior Therapy, Atlanta, Georgia, December 1977.     

Computer‐based training to teach observers to accurately score problem behavior using fast forwarding at 5x normal speed

The current study evaluated whether a computer‐based training program could improve observers' accuracy in scoring discrete instances of problem behavior at 5x normal speed using a multiple‐baseline design across subjects. During pretraining and posttraining, observers attempted to score multiple examples of problem behavior at 5.0x without feedback. During training, participants scored multiple examples of problem behavior at 5.0x with automated feedback. Researchers measured omission (missing problem behavior) and commission (scoring other behavior as problem behavior) errors and the total duration of scoring time to determine the observers' accuracy and efficiency, respectively. After training, all participants scored instances of problem behavior with less than 11% error using 5.0x. The time required to score the videos across 90‐min observations was reduced by 66%. Results extend previous evaluations of fast forwarding by demonstrating that the training program could be used to teach observers to accurately score problem behavior using a speed faster than 3.5x.     

OBSERVER TRAINING REVISITED: A COMPARISON OF IN VIVO AND VIDEO INSTRUCTION

We compared the effects of 2 observer‐training procedures. In vivo training involved practice during actual treatment sessions. Video training involved practice while watching progressively more complex simulations. Fifty‐nine undergraduate students entered 1 of the 2 training conditions sequentially according to an ABABAB design. Results showed that the 2 training methods produced almost identical scores on a posttraining observational test; however, the video method required fewer training sessions to complete.     

Do People Use Their Implicit Theories of Creativity as General Theories?

This study examines whether people use the general implicit theories of creativity or not when applying them to themselves and others. On the basis of the actor–observer asymmetry theory, the authors propose that conception of creativity would be differently constructed depending on the targets of attention: general, self, and other. Three studies attempted to examine this hypothesis. In the preliminary study, a measure was developed to assess the characteristics that describe general creativity. In Study 1, exploratory and confirmatory factor analyses found the conceptual factors of general creativity. In Study 2, the common and specific factors of general, self, and others’ creativity concepts were compared through invariance tests. As a result, it was revealed that the invariance test failed, which means that the general conception of creativity may not be applicable to monitor self and others’ creativity in a consistent way. Implications and limitations of this study are discussed at the end of the study.     

Shaping data collection congruent with experimental hypotheses

The influence of instructions and feedback from an experimenter on observational recordings of disruptive behavior was evaluated. Four subject-observers recorded four categories of disruptive behavior from videotapes of children in a classroom setting. Two sets of videotapes, labelled "baseline" and "treatment", were matched for rates of disruptive behavior in each category. The observers were told that two target behaviors were expected to decrease and the other two control behaviors were not expected to change during the treatment condition. During observational recording of treatment videotapes, the observers were given positive feedback when they reported decreases in the two target behaviors, and negative feedback when they reported either to change or increases in the two control behaviors. The target behaviors were recorded less frequently by observers, while recordings of control behaviors were unaffected during the treatment phase. These results suggest that contingent feedback to observers should be avoided as a possible source of bias in behavioral recordings.     

Evaluation of self-contained training procedures for the Time-Sample Behavioral Checklist

The comparative effectiveness of two time-limited modes of training observers to code the behavior of clients in residential treatment programs on the Time-Sample Behavioral Checklist (TSBC) was evaluated. The susceptibility of training procedures to consensual observer drift and the predictability of TSBC mastery from trainee characteristics were also examined. Two equated groups of undergraduate students (N=15 each) participated in full-time training on the TSBC and another instrument for 27 days, followed by criterion testingin vivo and on videotapes. One group was trained by experienced personnel using procedures known to be effective but potentially subject to consensual observer drift. The other group was trained using a previously untested set of written and videotape procedures that do not rely on experienced personnel. Comparative effectiveness and observer drift were evaluated by multivariate and univariate ANOVAs on mastery scores reflecting both pattern agreement and level differences between each trainee and criterion codings. The new, more efficient training procedures were found to be as effective as the original training procedures in the degree of mastery achieved by trainees. Original training procedures were found to be resistant to consensual observer drift, with such a phenomenon appearing in only 1 instance of 156 opportunities. The high degree of mastery achieved by trainees during the time-limited training period was comparable to that previously achieved with the original training procedures. No meaningful predictions of coding mastery were found, with only one trainee characteristic being significantly related to one of six mastery criteria. The results document procedures that are both efficient and resistant to invalidity for training observers in the use of multidimensional observational systems, as well as providing guidelines for the development of standardized procedures.This article is based on a thesis submitted to the Graduate College of the University of Illinois at Urbana-Champaign in partial fulfillment of the requirements for the Ph.D. degree in psychology by the first author and under direction of the second author. The third and fourth authors also participated as supervisors. Appreciation is extended to other members of the thesis committee, Fred Kanfer, Ross Parke, and Julian Rappaport, and to John Gottman and W. Robert Nay for their comments and recommendations. This study was partially supported by Public Health Service Grants MH-25464 and MH-14257 from the National Institute of Mental Health and by grants from the Joyce Foundation, the MacArthur Foundation, and the Illinois Department of Mental Health and Developmental Disabilities.