Observer agreement, credibility, and judgment: some considerations in presenting observer agreement data

Graphical and statistical indices employed to represent observer agreement in interval recording are described as "judgmental aids", stimuli to which the researcher and scientific community must respond when viewing observer agreement data. The advantages and limitations of plotting calibrating observer agreement data and reporting conventional statistical aids are discussed in the context of their utility for researchers and research consumers of applied behavior analysis. It is argued that plotting calibrating observer data is a useful supplement to statistical aids for researchers but is of only limited utility for research consumers. Alternatives to conventional per cent agreement statistics for research consumers include reporting special agreement estimates (e.g., per cent occurrence agreement and nonoccurrence agreement) and correlational statistics (e.g., Kappa and Phi).     

THE BEHAVIOR OBSERVATION INSTRUMENT: A METHOD OF DIRECT OBSERVATION FOR PROGRAM EVALUATION1

The background and development of a multicategory direct observation system, the Behavior Observation Instrument (BOI), is described. This time-sampling procedure for recording the behavior of persons is demonstrated in several treatment settings and the results applied to issues of program evaluation. Elements that have prevented direct observation from being widely adopted, such as costs, manpower, and training requirements, are systematically analyzed. A basic psychometric analysis of the instrument is used to determine optimum frequency and duration of observation intervals as well as observer agreement. The results imply that direct observation methods, once assumed by some to belong to the special province of the single-subject design, can be used to assess the effects of programs on groups of psychiatric clients in an efficient and economic manner.     

A SIMULTANEOUS DISCRIMINATION PROCEDURE FOR THE MEASUREMENT OF VISION IN NONVERBAL CHILDREN1

Few nonverbal developmentally disabled children ever receive adequate vision assessment because of their limited language skills. The present study details a simultaneous discrimination procedure for measuring subjective visual acuity in such children. A stimulus fading procedure was used to train a discrimination between Snellen Es differing in orientation, and a psychophysical tracking method was used to determine acuity thresholds. The procedure was tested with 11 nonverbal autistic and schizophrenic children and validated with four nonpsychotic children. Eight of the psychotic children were successfully examined in one to three sessions. Two of these children were identified as having significant acuity losses. The validity assessment showed that the experimental procedure resulted in thresholds equal to or slightly lower than those obtained with the Illiterate E chart.