A rapid non-parametric estimate of multi-judge reliability

A technique is presented for obtaining a rapid estimate of reliability between judges, with special reference to qualitative judgments. It is shown that reliability and discrimination are independent and that estimates of both are needed. A method of obtaining an independent estimate of multi-judge discrimination is developed. It is shown that the size of item-samples is specified by the latter method. Tests of significance for both reliability and discrimination are described.This technique was developed in connection with research at the Counseling Center, University of Chicago. This investigation was supported by a research grant (PHS M 903) from the National Institute of Mental Health, of the National Institutes of Health, Public Health Service. Acknowledgment is made to Dr. Lyle V. Jones, University of Chicago, for valuable discussions relevant to the technique. Responsibility rests, however, entirely with the writer.