Polytomous multilevel testlet models for testlet‐based assessments with complex sampling designs

A pplications of standard item response theory models assume local independence of items and persons. This paper presents polytomous multilevel testlet models for dual dependence due to item and person clustering in testlet‐based assessments with clustered samples. Simulation and survey data were analysed with a multilevel partial credit testlet model. This model was compared with three alternative models – a testlet partial credit model (PCM), multilevel PCM, and PCM – in terms of model parameter estimation. The results indicated that the deviance information criterion was the fit index that always correctly identified the true multilevel testlet model based on the quantified evidence in model selection, while the Akaike and Bayesian information criteria could not identify the true model. In general, the estimation model and the magnitude of item and person clustering impacted the estimation accuracy of ability parameters, while only the estimation model and the magnitude of item clustering affected the item parameter estimation accuracy. Furthermore, ignoring item clustering effects produced higher total errors in item parameter estimates but did not have much impact on the accuracy of ability parameter estimates, while ignoring person clustering effects yielded higher total errors in ability parameter estimates but did not have much effect on the accuracy of item parameter estimates. When both clustering effects were ignored in the PCM, item and ability parameter estimation accuracy was reduced.     

A general proof of consistency of heuristic classification for cognitive diagnosis models

The Asymptotic Classification Theory of Cognitive Diagnosis (Chiu et al., 2009, Psychometrika, 74, 633–665) determined the conditions that cognitive diagnosis models must satisfy so that the correct assignment of examinees to proficiency classes is guaranteed when non‐parametric classification methods are used. These conditions have only been proven for the Deterministic Input Noisy Output AND gate model. For other cognitive diagnosis models, no theoretical legitimization exists for using non‐parametric classification techniques for assigning examinees to proficiency classes. The specific statistical properties of different cognitive diagnosis models require tailored proofs of the conditions of the Asymptotic Classification Theory of Cognitive Diagnosis for each individual model – a tedious undertaking in light of the numerous models presented in the literature. In this paper a different way is presented to address this task. The unified mathematical framework of general cognitive diagnosis models is used as a theoretical basis for a general proof that under mild regularity conditions any cognitive diagnosis model is covered by the Asymptotic Classification Theory of Cognitive Diagnosis.     

Evidence of a role for multiple memory systems in behavioral extinction

The acquisition of learned behavior involves multiple memory systems, and hippocampal system damage impairs cognitive learning while leaving stimulus-response habit learning intact. In view of evidence that extinction also involves new learning, the present experiments examined whether multiple memory systems theory may be applicable to the neural bases of extinction. Adult Long-Evans rats were trained to run in a straight-alley maze for food reward. Twenty-four hours later, rats matched for runway latencies during acquisition received extinction training. In a response extinction condition conducive to habit learning, rats performed a runway approach response to an empty food cup. In a latent extinction condition conducive to cognitive learning, rats were placed at an empty food cup without performing a runway approach response. Prior to daily extinction training, neural activity of the dorsal hippocampus was reversibly inactivated via infusion of bupivacaine (0.75%, 0.5 microl/side). Control rats receiving saline infusions displayed extinction behavior in both the response and latent training conditions. In contrast, rats receiving bupivacaine extinguished normally in the response condition, but did not display latent extinction. The findings (1) confirm that learning underlying extinction of the same overt behavior can occur with or without explicit performance of the previously acquired response, (2) indicate that extinction learning produced by response and latent training procedures can be neuroanatomically dissociated, and (3) suggest that similarly to initial task acquisition, the hippocampus may critically mediate extinction in situations requiring the use of cognitive learning, such as when performance of a previously acquired response habit is prevented.     

Latent class analysis of anxiety and depressive symptoms of the Youth Self-Report in a general population sample of young adolescents

This study examined whether distinct groups of young adolescents with mainly anxiety or mainly depression could be identified in a general population sample. Latent class analysis was used on self-report ratings of DSM-IV symptoms of anxiety and depressive disorders, because it was hypothesized that these ratings provide a bigger chance to identify distinct groups than parent ratings of symptoms that are poorly associated with DSM-IV. Results from exploratory and confirmatory latent class analysis showed that only very small numbers of young adolescents had mainly anxiety or mainly depressive symptoms. Instead, a five-group model fitted the data best. These five groups contained young adolescents who either had a high, intermediate, or low probability to have comorbid symptoms of anxiety and depression. It was concluded that symptoms of DSM-IV anxiety and depressive disorders co-occur in young adolescents, and that latent class analysis on items that capture also severe symptoms like suicidal thoughts are needed to derive groups with specific comorbidity patterns in a general population sample.     

Estimation and visualization of confusability matrices from adaptive measurement data

We present a simple but effective method based on Luce’s choice axiom [Luce, R.D. (1959). Individual choice behavior: A theoretical analysis. New York: John Wiley & Sons] for consistent estimation of the pairwise confusabilities of items in a multiple-choice recognition task with arbitrarily chosen choice-sets. The method combines the exact (non-asymptotic) Bayesian way of assessing uncertainty with the unbiasedness emphasized in the classical frequentist approach.We apply the method to data collected using an adaptive computer game designed for prevention of reading disability. A player’s estimated confusability of phonemes (or more accurately, phoneme-grapheme connections) and larger units of language is visualized in an easily understood way with color cues and explicit indication of the accuracy of the estimates. Visualization of learning-related changes in the player’s performance is considered.The empirical validity of the choice axiom is evaluated using the game data itself. The axiom appears to hold reasonably well although a small systematic violation is observable for the smallest choice-set sizes.     

School profiles of at-risk student concentration: Differential growth in oral reading fluency

The present study provides a data-driven approach to identifying groups of schools based on the concentration of at-risk students the school serves. The percentage of English language learners, minority students, and students eligible for free or reduced priced lunch were used as indicators in a latent profile analysis of 569 schools. The goal of the present study was to determine whether school-level average student reading performance varied as a function of the groups identified in the latent profile analysis. To do so, groups extracted by the latent profile analysis were used as school-level predictors of growth in oral reading fluency, which was modeled at the within-student level of a three-level hierarchical growth curve model. Oral reading fluency was measured at four points during the year in a large cross-sectional sample of first-, second-, and third-grade students. Results indicated that schools were able to be classified into four distinct groups based on their concentrations and types of at-risk students. Further, in all three grades, there were significant differences between the four identified groups observed in average reading fluency scores at the beginning of the year, the end of the year, and growth during the year indicating that groups based on school-concentration of at-risk students were significantly related to average student achievement in reading ability.     

Beta-MPT: Multinomial processing tree models for addressing individual differences

Traditionally, multinomial processing tree (MPT) models are applied to groups of homogeneous participants, where all participants within a group are assumed to have identical MPT model parameter values. This assumption is unreasonable when MPT models are used for clinical assessment, and it often may be suspect for applications to ordinary psychological experiments. One method for dealing with parameter variability is to incorporate random effects assumptions into a model. This is achieved by assuming that participants’ parameters are drawn independently from some specified multivariate hyperdistribution. In this paper we explore the assumption that the hyperdistribution consists of independent beta distributions, one for each MPT model parameter. These beta-MPT models are ‘hierarchical models’, and their statistical inference is different from the usual approaches based on data aggregated over participants. The paper provides both classical (frequentist) and hierarchical Bayesian approaches to statistical inference for beta-MPT models. In simple cases the likelihood function can be obtained analytically; however, for more complex cases, Markov Chain Monte Carlo algorithms are constructed to assist both approaches to inference. Examples based on clinical assessment studies are provided to demonstrate the advantages of hierarchical MPT models over aggregate analysis in the presence of individual differences.     

A Bayesian-optimal principle for learner-friendly adaptation in learning games

Adaptive learning games should provide opportunities for the student to learn as well as motivate playing until goals have been reached. In this paper, we give a mathematically rigorous treatment of the problem in the framework of Bayesian decision theory. To quantify the opportunities for learning, we assume that the learning tasks that yield the most information about the current skills of the student, while being desirable for measurement in their own right, would also be among those that are efficient for learning. Indeed, optimization of the expected information gain appears to naturally avoid tasks that are exceedingly demanding or exceedingly easy as their results are predictable and thus uninformative. Still, tasks that are efficient for learning may be experienced as too challenging, and the resulting failures can lower motivation. Therefore, in addition to quantifying the expected informational benefit for learning of any prospective task to be presented next, we also model the expected motivational cost of its presentation, measured simply as the estimated probability of failure in our example model. We propose a “learner-friendly” adaptation algorithm that chooses the learning tasks by optimizing the expected benefit divided by the expected cost. We apply this algorithm to a Rasch-like student model implemented within a real-world application and present initial results of a pilot experiment.