The effect of uncertainty of item parameter estimation on ability estimates

The conventional method of measuring ability, which is based on items with assumed true parameter values obtained from a pretest, is compared to a Bayesian method that deals with the uncertainties of such items. Computational expressions are presented for approximating the posterior mean and variance of ability under the three-parameter logistic (3PL) model. A 1987 American College Testing Program (ACT) math test is used to demonstrate that the standard practice of using maximum likelihood or empirical Bayes techniques may seriously underestimate the uncertainty in estimated ability when the pretest sample is only moderately large.This work was partially supported under contract No. N00014-85-K-0113, NR150-535, from the Cognitive Science Program, Office of Naval Research. The authors wish to thank Mark D. Reckase for providing the ACT data used in the illustration and two referees, Asociate Editor and Editor for helpful suggestions.