An approximation of theK outN reliability of a test,and a scoring procedure for determining which items an examinee knows

Consider any scoring procedure for determining whether an examinee knows the answer to a test item. Letx _i = 1 if a correct decision is made about whether the examinee knows the ith item; otherwisex _i = 0. Thek out ofn reliability of a test isρ _k = Pr (Σx _i ≥k). That is,ρ _k is the probability of making at leastk correct decisions for a typical (randomly sampled) examinee. This paper proposes an approximation ofρ _k that can be estimated with an answer-until-correct test. The paper also suggests a scoring procedure that might be used whenρ _k is judged to be too small under a conventional scoring rule where it is decided an examinee knows if and only if the correct response is given.