A cautionary note on the use of probability values to evaluate interobserver agreement

Proposed methods of assessing the statistical significance of interobserver agreements provide erroneous probability values when conducted on serially correlated data. Investigators who wish to evaluate interobserver agreements by means of statistical significance can do so by limiting the analysis to every k(th) interval of data, or by using Markovian techniques which accommodate serial correlations.     

Measures of observer agreement when binomial data are collected in free operant situations

This paper examines a model and defines reasonable assumptions underlying different measures of observer agreement for categorical data collected in free operant situations. It is assumed that two or more observers classify operant behaviors of subjects into occurrences and nonoccurrences by recognition by validated response classes (categories) such that the rates of false positives and observer biases are acceptably low. Thus errors are mostly omissions, i.e., failing to observe events that occur. Four alternative cases are derived, together with formulas for calculating significance tests, variances, and standard errors, three of which do not depend on knowledge of the proportion of time points at which the event does not occur.We wish to acknowledge NICHD Grant HD-10570, The Neuropharmacology of Developmental Disorders, George Breese, Ph.D., and C. T. Gualtieri, M.D., Principal Investigators; NIEHS Grant ES-01104; USPHS Grant HD-03110; and MCH Project 916 to the Division for Disorders of Development and Learning.     

Youth-parent consistencies on ratings of difficulties and prosocial behavior: exploration of an at-risk sample

Low informant agreement is of particular concern in clinical settings, since diagnoses and interventions are largely based on reports from parents and/or youths. This study explores youth-parent consistencies in an at-risk sample of youths (n = 42), several from immigrant and refugee families, all with stressful background experiences. Data from a community sample (n = 165) was used to contrast the clinical results. Results showed that correlations between clinical youths and parents were within the same range as those in the community sample. Consistencies concerning absolute level of behavior were higher in the clinical sample. However, interesting differences in informant consistencies were found on difficulties and resources between the two samples. This study adds to the picture that there is no simple relationship between severity of behavior deviance and informant agreement. Asking about behavioral resources as well as difficulties may provide valuable additional information regarding differences in the perspectives of at-risk youths and parents.     

Reliability in the context of the experiment: A commentary on two articles by Birkimer and Brown

Two sources of variability must each be considered when examining change in level between two sets of data obtained by human observers; namely, variance within data sets (phases) and variability attributed to each data point (reliability). Birkimer and Brown (1979a, 1979b) have suggested that both chance levels and disagreement bands be considered in examining observer reliability and have made both methods more accessible to researchers. By clarifying and extending Birkimer and Brown's papers, a system is developed using observer agreement to determine the data point variability and thus to check the adequacy of obtained data within the experimental context.     

A graphical judgmental aid which summarizes obtained and chance reliability data and helps assess the believability of experimental effects

Interval by interval reliability has been criticized for "inflating" observer agreement when target behavior rates are very low or very high. Scored interval reliability and its converse, unscored interval reliability, however, vary as target behavior rates vary when observer disagreement rates are constant. These problems, along with the existence of "chance" values of each reliability which also vary as a function of response rate, may cause researchers and consumers difficulty in interpreting observer agreement measures. Because each of these reliabilities essentially compares observer disagreements to a different base, it is suggested that the disagreement rate itself be the first measure of agreement examined, and its magnitude relative to occurrence and to nonoccurrence agreements then be considered. This is easily done via a graphic presentation of the disagreement range as a bandwidth around reported rates of target behavior. Such a graphic presentation summarizes all the information collected during reliability assessments and permits visual determination of each of the three reliabilities. In addition, graphing the "chance" disagreement range around the bandwidth permits easy determination of whether or not true observer agreement has likely been demonstrated. Finally, the limits of the disagreement bandwidth help assess the believability of claimed experimental effects: those leaving no overlap between disagreement ranges are probably believable, others are not.     

A method for combining occurrence and nonoccurrence interobserver agreement scores

Various statistics have been proposed as standard methods for calculating and reporting interobserver agreement scores. The advantages and disadvantages of each have been discussed in this journal recently but without resolution. A formula is presented that combines separate measures of occurrence and nonoccurrence percentages of agreement, with weight assigned to each measure, varying according to the observed rate of behavior. This formula, which is a modification of a formula proposed by Clement (1976), appears to reduce distortions due to "chance" agreement encountered with very high or low observed rates of behavior while maintaining the mathematical and conceptual simplicity of the conventional method for calculating occurrence and nonoccurrence agreement.     

Test theory without true scores?

This paper traces the course of the consequences of viewing test responses as simply providing dichotomous data concerning ordinal relations. It begins by proposing that the score matrix is best considered to be items-plus-persons by items-plus-persons, and recording the wrongs as well as the rights. This shows how an underlying order is defined, and was used to provide the basis for a tailored testing procedure. It also was used to define a number of measures of test consistency. Test items provide person dominance relations, and the relations provided by one item can be in one of three relations with a second one: redundant, contradictory, or unique. Summary statistics concerning the number of relations of each kind are easy to get and provide useful information about the test, information which is related to but different from the usual statistics. These concepts can be extended to form the basis of a test theory which is based on ordinal statistics and frequency counts and which invokes the concept of true scores only in a limited sense.1979 Psychometric Society presidential address.I want to recognize the contributions which others have made to whatever I have accomplished. First to mention here are my teachers: principally Harold Gulliksen, Ledyard Tucker, and the late Edith Jay. Second, I would like to recognize the importance of my graduate students. Tom Reynolds has been especially important in developing the ideas that I will talk about here today, but at various times, the others have made major contributions in this and other topics. I would like to express also a debt to my family, primarily my wife, Rosemary, who herself has a longterm interest in the psychometric area. Finally, I must acknowledge the financial support of the NIMH some time ago, the Office of Naval Research, until about a year ago, and of the James McKeen Cattell Fund this past year.     

A demonstration of the value of interchangeable observers

The remarkably high agreement between observers using the SRIC, the TSBC, and the CFRS makes the observers interchangeable. This agreement is a product of the intensive and extensive training and monitoring of full-time observers, the use of categories rather than continua, the low degree of interpretation required by the procedures, the use of lay language, the immediate recording after short observational periods, and the familiar and standard setting. Other important features include use of rates, emphasis on time, concern for positive behavior, and close linkages between concepts and categories so that concepts are explicitly specified. Taken together, these characteristics make these systems powerful instruments for the guidance of treatment and the prediction of outcomes. The key to their success seems to be the recording of simple actions and action sequences (the basic phenomena of psychology) as the basic data from which a variety of useful indexes can be readily formed.Preparation of this article was supported by Grant MH-30654 from the National Institute of Mental Health to S. Duncan and D. Fiske.Presented at the 87th Annual Meetings of the American Psychological Association, New York City, September 1979, as part of a symposium on New assessment systems for residential treatment, management, research, and evaluation.     

A probability-based formula for calculating interobserver agreement

Estimates of observer agreement are necessary to assess the acceptability of interval data. A common method for assessing observer agreement, per cent agreement, includes several major weaknesses and varies as a function of the frequency of behavior recorded and the inclusion or exclusion of agreements on nonoccurrences. Also, agreements that might be expected to occur by chance are not taken into account. An alternative method for assessing observer agreement that determines the exact probability that the obtained number of agreements or better would have occurred by chance is presented and explained. Agreements on both occurrences and nonoccurrences of behavior are considered in the calculation of this probability.