Estimating the accuracy of dichotomous judgments

A reliability study is assumed to be carried out with each of a number of observers making a dichotomous judgment concerning each of a sample of subjects. A nonparametric model is proposed for the errors underlying such judgments, and conditions are given under which Cochran'sQ statistic is valid for testing the hypothesis of no systematic differences among the judgments of the different observers. Inferences concerning the probabilities of error are shown to be possible in terms of the intraclass correlation coefficient. A numerical example is given.This research was supported in part by NIMH grant MH-03546. I am indebted to Dr. E. I. Burdock, Associate Research Scientist, Biometrics Research, for his valuable suggestions and criticisms.