Estimation and visualization of confusability matrices from adaptive measurement data

We present a simple but effective method based on Luce’s choice axiom Luce, R.D. (1959). Individual choice behavior: A theoretical analysis. New York: John Wiley & Sons] for consistent estimation of the pairwise confusabilities of items in a multiple-choice recognition task with arbitrarily chosen choice-sets. The method combines the exact (non-asymptotic) Bayesian way of assessing uncertainty with the unbiasedness emphasized in the classical frequentist approach.We apply the method to data collected using an adaptive computer game designed for prevention of reading disability. A player’s estimated confusability of phonemes (or more accurately, phoneme-grapheme connections) and larger units of language is visualized in an easily understood way with color cues and explicit indication of the accuracy of the estimates. Visualization of learning-related changes in the player’s performance is considered.The empirical validity of the choice axiom is evaluated using the game data itself. The axiom appears to hold reasonably well although a small systematic violation is observable for the smallest choice-set sizes.