A review of the observational data-collection and reliability procedures reported in The Journal of Applied Behavior Analysis

The research published in the Journal of Applied Behavior Analysis (1968 to 1975) was surveyed for three basic elements: data-collection methods, reliability procedures, and reliability scores. Three-quarters of the studies reported observational data. Most of these studies' observational methods were variations of event recording, trial scoring, interval recording, or time-sample recording. Almost all studies reported assessment of observer reliability, usually total or point-by-point percentage agreement scores. About half the agreement scores were consistently above 90%. Less than one-quarter of the studies reported that reliability was assessed at least once per condition.     

The effects of instructions and calculation procedures on observers' accuracy, agreement, and calculation correctness

Although the quality of observational data is generally evaluated by observer agreement, measures of both observer agreement and accuracy were available in the present study. Videotapes with a criterion protocol were coded by 16 observers. All observers calculated agreement scores both on their own and their partner's data and on a contrived data set misrepresented as data collected by other observers. Compared with agreement scores calculated by the experimenter, observers erroneously inflated their own agreement scores and deflated the agreement scores on the contrived data. Half of the observers (n = 8) had been given instructions emphasizing the importance of accuracy during observation while the other half had been given instructions emphasizing interobserver agreement. Accuracy exceeded agreement for the former group, whereas agreement exceeded accuracy for the latter group. The implications are that agreement should be calculated by the experimenter and that the accuracy-agreement relationship can be altered by differential observer instructions.     

ASSESSING OBSERVER ACCURACY IN CONTINUOUS RECORDING OF RATE AND DURATION: THREE ALGORITHMS COMPARED

The three algorithms most frequently selected by behavior‐analytic researchers to compute interobserver agreement with continuous recording were used to assess the accuracy of data recorded from video samples on handheld computers by 12 observers. Rate and duration of responding were recorded for three samples each. Data files were compared with criterion records to determine observer accuracy. Block‐by‐block and exact agreement algorithms were susceptible to inflated agreement and accuracy estimates at lower rates and durations. The exact agreement method appeared to be overly stringent for recording responding at higher rates (23.5 responses per minute) and for higher relative duration (72% of session). Time‐window analysis appeared to inflate accuracy assessment at relatively high but not at low response rate and duration (4.8 responses per minute and 8% of session, respectively).     

An analysis of the relationship of teachers' reported use of classroom management strategies on types of classroom interactions

The results of an investigation to determine teachers' reported use of classroom management strategies and their relationship to teacher and student interactions are presented. We interviewed 20 teachers to determine how they developed and used classroom management strategies, and then observed the interactions of these teachers with children with serious behavior disorders (SBD). Results identified two groups of teachers: one that scored high on planned use of the strategies, and one that scored low. Comparisons between the two groups of the interaction patterns revealed small, but statistically significant differences in the mean length and total percent time involved in positive interactions. The group reporting higher use of the management strategies engaged in positive interactions which were longer in duration than the interactions of the group reporting lower use. The results are discussed in terms of further research needs and implications for teacher training programs.     

Reliability in the context of the experiment: A commentary on two articles by Birkimer and Brown

Two sources of variability must each be considered when examining change in level between two sets of data obtained by human observers; namely, variance within data sets (phases) and variability attributed to each data point (reliability). Birkimer and Brown (1979a, 1979b) have suggested that both chance levels and disagreement bands be considered in examining observer reliability and have made both methods more accessible to researchers. By clarifying and extending Birkimer and Brown's papers, a system is developed using observer agreement to determine the data point variability and thus to check the adequacy of obtained data within the experimental context.     

Frequency,complexity, and clarity as covariates of observer reliability

Factors affecting interobserver agreement (reliability) with a comprehensive coding system in the naturalistic observation of children were examined. Data from 117 pairs of observations on 35 children and their families were examined with respect to reliability and three possible covariates: response frequency, observation complexity, and code definition clarity. Analysis of results strongly supported response frequency as a positive covariate of interobserver agreement. Complexity was found to negatively covary with interobserver agreement. The relationship between code clarity and reliability was in the predicted direction but failed to obtain statistical significance. Implications for observer training and data collection in observational studies are discussed.An earlier version of this article was prepared for presentation at The Association for the Advancement of Behavior Therapy, Atlanta, Georgia, December 1977.     

A graphical judgmental aid which summarizes obtained and chance reliability data and helps assess the believability of experimental effects

Interval by interval reliability has been criticized for "inflating" observer agreement when target behavior rates are very low or very high. Scored interval reliability and its converse, unscored interval reliability, however, vary as target behavior rates vary when observer disagreement rates are constant. These problems, along with the existence of "chance" values of each reliability which also vary as a function of response rate, may cause researchers and consumers difficulty in interpreting observer agreement measures. Because each of these reliabilities essentially compares observer disagreements to a different base, it is suggested that the disagreement rate itself be the first measure of agreement examined, and its magnitude relative to occurrence and to nonoccurrence agreements then be considered. This is easily done via a graphic presentation of the disagreement range as a bandwidth around reported rates of target behavior. Such a graphic presentation summarizes all the information collected during reliability assessments and permits visual determination of each of the three reliabilities. In addition, graphing the "chance" disagreement range around the bandwidth permits easy determination of whether or not true observer agreement has likely been demonstrated. Finally, the limits of the disagreement bandwidth help assess the believability of claimed experimental effects: those leaving no overlap between disagreement ranges are probably believable, others are not.     

Marital satisfaction and family stress as predictors of classroom behaviors

The purpose of this research was to examine the relationship between parents' perceptions of marital satisfaction and family stress and their third grade children's classroom behavior. Twenty-one married couples completed questionnaires during home visits. Behavior observations were made from videotapes of children recorded in their classroom during lunch and group academic periods on each of three days. Frequencies of peer interactions, solitary behaviors, and teacher interactions were coded. Regression analyses showed that mothers' level of marital satisfaction (but not fathers') predicted their children's frequency of peer interactions during lunch. Fathers' level of perceived family stress (but not mothers') predicted their children's frequency of peer interactions during lunch. Neither measure for either parent was related to the frequency of peer interactions, solitary behaviors, or teacher interactions observed during the academic sessions. These findings highlight the importance of assessing the potential differential relation of parent variables to children's trans-situational behaviors, and the examination of these variables in relation to different social-environmental contexts in which children participate outside the home.     

Observer reliability as a function of circumstances of assessment

THREE FACTORS CHARACTERISTIC OF EXPERIMENTAL SETTINGS WERE HYPOTHESIZED TO INFLATE ARTIFACTUALLY THE RELIABILITY OF OBSERVATIONAL RECORDINGS: (a) knowledge by observers of when and by whom their reliability is being assessed, (b) the absence of the experimenter or a monitor to prevent cheating, and (c) computation of reliability within- (versus between-) observer group. Three groups of four observers used a standard nine-category observational code for disruptive behavior in recording from videotapes of a classroom for 22 days. Analyses revealed considerable increases in average occurrence reliability as a function of the main effects of each of the experimental factors. The specific increases in reliability associated with each of the 12 combinations of the experimental factors are presented for each category of behavior. The possible role of observer-training procedures and behavioral definitions as determiners of nonartifactual reliability is discussed.     

Contingency and choice: The implications of matching theory for classroom instruction

This paper describes a mathematical account of behavior known as matching theory. Matching theory evolved out of basic operant research and assumes that individuals can engage in a variety of behaviors at any moment, but they choose one to the exclusion of others. According to the matching equation, choices in behavior match the relative amount of reinforcement provided for each alternative. Although the principles of matching theory have proven useful in developing novel treatment strategies, few data exist validating the matching equation in natural human environments. Recent applications of matching theory to children's classroom behavior are described, and the implications of matching theory for classroom management and effective teaching are discussed.     

Effects of the good behavior game across classroom contexts

The Good Behavior Game (GBG), a well‐researched classroom group contingency, is typically played for brief periods of time, which raises questions about the effects on subsequent contexts. This study used a multiple baseline design and showed that when the GBG was implemented in one context, behavior improved in only that context. Behavior improved in the subsequent activity only when the GBG was implemented.     

Acquaintanceship and self/observer agreement in personality judgment

Self/observer agreement on HEXACO-PI-R scale scores was examined as a function of observers’ subjective ratings of acquaintanceship. For each participant (N = 2199), personality self-reports were obtained along with observer reports from a friend. Each factor-level scale displayed a different pattern of upward accuracy (agreement) trends in personality judgment. Self/observer agreement for Extraversion, Emotionality, and Openness was noticeably stronger at lower acquaintanceship than that for Conscientiousness, Agreeableness, and Honesty-Humility. Conscientiousness showed a steep upward accuracy trend across acquaintanceship levels, reaching a level of accuracy comparable to that of Extraversion and Emotionality. Self/observer agreement for Honesty-Humility and Agreeableness showed slower upward trends than that of Conscientiousness. In several cases, facet-level traits within the same broad factor differed in their accuracy trends.     

Observed classroom behavior of children with ADHD: relationship to gender and comorbidity

Examined hypothesized gender and comorbidity differences in the observed classroom behavior of children with attention deficit hyperactivity disorder (ADHD). The behavior of 403 boys and 99 girls with ADHD, ages 7–10, was compared (a) to observed, sex-specific classroom behavior norms, (b) by sex, and (c) by comorbid subgroups. Boys and girls with ADHD deviated significantly from classroom norms on 15/16 and 13/16 categories, respectively. Compared to comparison girls, girls with ADHD had relatively high rates of verbal aggression to children. Boys with ADHD engaged in more rule-breaking and externalizing behaviors than did girls with ADHD, but the sexes did not differ on more neutral, unobtrusive behaviors. The sex differences are consistent with notions of why girls with ADHD are identified and referred later than boys. Contrary to hypothesis, the presence of comorbid anxiety disorder (ANX) was not associated with behavioral suppression; yet, as hypothesized, children with a comorbid disruptive behavior disorder (DBD) had higher rates of rule-breaking, and impulsive and aggressive behavior, than did children with ADHD alone and those with ADHD+ANX. Elevated rates of ADHD behaviors were also observed in children with comorbid DBD, indicating that these behaviors are truly present and suggesting that reports of higher ADHD ratings in this subgroup are not simply a consequence of negative halo effects and rater biases.     

Role playing to train elementary teachers to use a classroom management "skill package"

Two teachers who led regular third-grade classrooms in a suburban elementary school were trained via role-playing to use a broad range of social skills in dealing with group behavioral management in the classroom. Teacher training reduced disruptive student behavior during both seat work and group discussions in both classrooms. A measure of student productivity during arithmetic period in one classroom showed significant gains in arithmetic problems correct per day for the middle and bottom thirds of the class, with the bottom third increasing by 76%.     

Pyramid training of elementary school teachers to use a classroom management "skill package"

Three regular elementary teachers were trained in the use of a classroom management "skill package". Subsequently, each of these three teachers (tier 1 of training) trained three more teachers to use the same skill package (tier 2 of training). Direct behavioral measures of student disruptiveness were taken in the three tier-1 classrooms and four tier-2 classrooms, and permanent product measures of student productivity in arithmetic were taken in the three tier-1 classrooms. Results indicated that student disruptiveness decreased at least as much in the tier-2 classrooms as in the tier-1 classrooms. Data also indicated that serving as trainers benefited two of the tier-1 teachers who profited least from the original training by producing further reductions in disruptiveness in their respective classrooms. Productivity data suggested that use of the "skill package" increased classroom academic output, especially for those students below the median in productivity during baseline. The investigators' time investment in training a tier-2 teacher was one-fourth that of training a tier-1 teacher.     

Free-time as a consequence for behavior in classroom settings: A critical review

Osborne (1969) presented adolescent hearing impaired students with five minutes of free- time contingent upon remaining seated for 15 minutes in the classroom. This procedure substantially reduced the frequency of out- of- seat behavior. Since the publication of this seminal work, 49 studies have used free- time as a consequence to change behavior. This review critically examines these 49 studies. The review shows that free-time has been used effectively with a variety of different behaviors and student populations, and with different methods of delivery. Based on a synthesis of information from the review, optimal methods for the use of free-time are suggested. Finally, it is concluded that additional research is needed, especially to answer the question of why free-time is effective.     

An analysis of daily report cards and parent-managed privileges in the improvement of adolescents' classroom performance

A daily report-card system, involving home privileges administered by parents, was developed for use with "problem" junior-high students. In Experiment I, when home privileges and parent praise were contingent on improved school conduct, classwork, daily grades, and teacher satisfaction, the school performance of three students improved considerably. In Experiment II, a similar report-card system was employed with two additional students, except only parent praise was contingent on improved school performance. Under these conditions, one of the students did not bring the report card home, and class performance did not improve until contingent home privileges were added. The school performance of the second student improved with the card and praise alone. However, there appeared to be a slow decline of classwork performance over time. For Experiment III, an instructional manual, describing the report-card program, was written for school guidance counsellors. Two guidance counsellors read the program, used it with one student each, and found that the school performance of both students improved. These results suggest that the daily report-card program with home consequences administered by parents can improve the school performance of, and teacher satisfaction about, students having considerable difficulty in school.     

A probability-based formula for calculating interobserver agreement

Estimates of observer agreement are necessary to assess the acceptability of interval data. A common method for assessing observer agreement, per cent agreement, includes several major weaknesses and varies as a function of the frequency of behavior recorded and the inclusion or exclusion of agreements on nonoccurrences. Also, agreements that might be expected to occur by chance are not taken into account. An alternative method for assessing observer agreement that determines the exact probability that the obtained number of agreements or better would have occurred by chance is presented and explained. Agreements on both occurrences and nonoccurrences of behavior are considered in the calculation of this probability.     

Comparison of components of the Good Behavior Game in a preschool classroom

The Good Behavior Game (GBG) is an effective intervention package for decreasing disruptive behavior in various populations and environments. There is, however, limited research evaluating the GBG with preschoolers. Furthermore, few studies have evaluated the effects of components of the GBG, and of those that have, most have done so only after exposure to the GBG package. We evaluated the effects (a) of the GBG on disruptive behavior of preschoolers during group instruction and (b) the major components of the GBG before and after implementation of the GBG package (c) at both the group and individual level. Results suggest that the GBG package was necessary for decreasing disruptive behavior. However, after exposure to the GBG, a response‐independent contingency was effective for maintaining low levels of disruptive behavior at both the group and individual level.