Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.     

The use of teachers as behavioral observers

The effect of using teachers as behavioral observers on both student and teacher behavior was examined with eight teachers and 32 elementary school children. The frequency of prompts (but not praise or criticism) to those students observed by the teacher increased significantly from nonobserver to teacher observed experimental phases. In addition, students observed by the teacher showed more change in appropriate behavior than students who were not observed. The significance of these findings for research and therapy is discussed.     

DOES USE OF TANGIBLE REWARDS WITH INDIVIDUAL CHILDREN AFFECT PEER OBSERVERS?1

The common assumption that employing tangible rewards with individual children will have adverse effects upon peer observers was studied in the preschool setting. Multiple-subject, multiple-baseline procedures were applied to two classes of children, aged 3.5 to 6 yr. In each group, three consecutive children with low base rates of in-seat behavior received a verbal contingency and food rewards for sitting, while peers (with either low or high rates of in-seat behavior) received neither food nor teacher attention for sitting. Peer reactions measured were in-seat behavior, aggression, nonaggressive disruptive behavior, and complaints. The procedures neither decreased the in-seat behavior of peer observers, nor increased their aggressive or disruptive behavior. On the contrary, peers with low base rates of sitting initially displayed an abrupt, but temporary, increase in sitting. Moreover, although no compensatory attention was delivered, all children exhibited improved sitting by the end of the study. Complaints, which consisted mainly of requests for rewards, decreased in frequency with successive program phases, and within each phase. It is suggested that the class improvement in sitting behavior and the absence of negative effects on observers may be partially due to the high frequency of attention the teacher maintained for other desired behavior and the lack of attention to children's complaints.     

A demonstration of the value of interchangeable observers

The remarkably high agreement between observers using the SRIC, the TSBC, and the CFRS makes the observers interchangeable. This agreement is a product of the intensive and extensive training and monitoring of full-time observers, the use of categories rather than continua, the low degree of interpretation required by the procedures, the use of lay language, the immediate recording after short observational periods, and the familiar and standard setting. Other important features include use of rates, emphasis on time, concern for positive behavior, and close linkages between concepts and categories so that concepts are explicitly specified. Taken together, these characteristics make these systems powerful instruments for the guidance of treatment and the prediction of outcomes. The key to their success seems to be the recording of simple actions and action sequences (the basic phenomena of psychology) as the basic data from which a variety of useful indexes can be readily formed.Preparation of this article was supported by Grant MH-30654 from the National Institute of Mental Health to S. Duncan and D. Fiske.Presented at the 87th Annual Meetings of the American Psychological Association, New York City, September 1979, as part of a symposium on New assessment systems for residential treatment, management, research, and evaluation.     

Observations of families in structured interactions: Parenting therapists provide reliable ratings of mothers' parenting

The reliability of observations of parenting by parenting therapists was assessed. An important predictor of externalizing behavior in children is quality of parenting. Data were videotapes of structured interactions in families with a child age 8–12 years referred to the evidence based Parent Management Training Oregon (PMTO) treatment program for child behavior problems. The therapists had clinical PMTO training but no training in systematic observation. PMTO observational coders with specific coder training were included as a reference for the therapists. Five therapists and two coders observed videotapes of 10 families and performed global evaluations of mothers' parenting skills. They used the coder's impression measure used in PMTO research. Scores were analyzed in a generalizability theory framework for the two groups of observers separately. Both observer types reliably rank‐ordered the mothers and assessed the level of parenting skills. PMTO therapists without coder training provided reliable ratings of parenting constructs relevant to the clinical PMTO program in a manner comparable to that of the trained reference coders.     

The Need for Power and Fear of Assertiveness in English-Speaking South African Female Graduates

The present study demonstrated that Winter's (1973) power arousal procedure and scoring system for the need for power, which were developed in the United States, are valid for English-speaking South African female graduates. Subjects in whom the need for power had been aroused by a prerecorded inspirational speech (n = 20) had significantly higher n Power scores than matched nonaroused subjects (n = 20), T ²(8, 31) = 56.17, p ≤ .01. Further, it was argued that the fear of assertiveness is a component of the need for power, and a scoring system for measuring this motive was suggested. Fear of assertiveness was shown to correlate significantly with various behavioral and dispositional measures of assertiveness, relating especially to a dependency pattern of behavior.     

Observers’ reactions to workplace incivility in the masculine domain: How does role congruency explain gender bias in future workers?

Based on Role Congruity Theory, we tested the hypothesis of gender bias by examining gender differences in observers’ evaluations of the awareness and acceptability of workplace incivility gender‐dyad interaction. Three hundred and ninety‐six Spanish high school students (55.3% female) read one scenario of overt incivility (publicly humiliates and openly doubts the employee's judgment) or covert (omits and pays little attention) from a leader (female vs. male) toward a subordinate (female vs. male) in engineering. Results indicated gender differences among observers. From the leader actor of incivility, males were more aware and accepted less the incivility when performed by a female leader in a male domain; whereas females were more aware and accepted less incivility than males in all cases. Regarding the subordinate target of incivility, only females were more aware and accepted less incivility, and both males and females were more aware and accepted less covert incivility. Our results reveal practical implications for interventions from a gender perspective.     

THE EFFECTS OF OBSERVER PRESENCE ON THE ACTIVITY LEVEL OF FAMILIES1

In the present study, the possibility that the act of observing behavior may result in changes in those behaviors was investigated. An investigation of reactivity is important to the validity of research findings because the result obtained when observers are present may not generalize to situations when observers are absent. The effects of observer presence were assessed with five families, each consisting of a mother and two children, in a laboratory setting designed to resemble a typical family living room, containing a table, couch, chairs, toys, books, and a bathroom and kitchen area. The families were each exposed to a sequence of four 30-minute conditions during which an observer was alternatively physically present or absent from the room. From behind an unobtrusive one-way mirror, family members' locations were recorded every 15 seconds on gridded floor-plan diagrams drawn to the scale of the room. After all data were collected, lines were drawn for each family member connecting the numbers sequentially and a map-reading device was then used to measure the distance covered during each minute of observation. The presence of observers markedly reduced the activity level of all families, with the average reduction approximating 50%. However, more extended observations in natural family settings may reveal that families habituate to observer presence in a relatively short period of time. If so, the solution to the reactivity problem would involve extending the observation sessions. However, if families do not habituate, new forms of data collection might be required to guarantee the validity and generalizability of data collected by obtrusive observers.     

Methodological problems in the use of participant observers

A multiple baseline design across observed students and teachers was used to investigate the relationship between observations by participant observers and changes in the behavior of those observed ("observee" reactivity) and the observers (observer-mediator reactivity). Two teachers recorded consecutively the appropriate student verbalizations of four students and two teachers recorded the inappropriate student verbalizations of four students. Independent observers simultaneously recorded student verbalizations (appropriate and inappropriate) as well as teacher behaviors (positive, negative, and instruction) throughout all phases of the study. The results substantiated the prediction of "observee" reactivity and observer-mediator reactivity in one of four classrooms. The results of the present study suggest that in some instances, observations by participant observers may result in changes in the behavior of those being observed ("observee" reactivity) and/or the observers (observer-mediator reactivity).