Reliability of test scores in nonparametric item response theory

Three methods for estimating reliability are studied within the context of nonparametric item response theory. Two were proposed originally by Mokken (1971) and a third is developed in this paper. Using a Monte Carlo strategy, these three estimation methods are compared with four classical lower bounds to reliability. Finally, recommendations are given concerning the use of these estimation methods.The authors are grateful for constructive comments from the reviewers and from Charles Lewis.     

The many null distributions of person fit indices

This paper deals with the situation of an investigator who has collected the scores ofn persons to a set ofk dichotomous items, and wants to investigate whether the answers of all respondents are compatible with the one parameter logistic test model of Rasch. Contrary to the standard analysis of the Rasch model, where all persons are kept in the analysis and badly fittingitems may be removed, this paper studies the alternative model in which a small minority ofpersons has an answer strategy not described by the Rasch model. Such persons are called anomalous or aberrant. From the response vectors consisting ofk symbols each equal to 0 or 1, it is desired to classify each respondent as either anomalous or as conforming to the model. As this model is probabilistic, such a classification will possibly involve false positives and false negatives. Both for the Rasch model and for other item response models, the literature contains several proposals for a person fit index, which expresses for each individual the plausibility that his/her behavior follows the model. The present paper argues that such indices can only provide a satisfactory solution to the classification problem if their statistical distribution is known under the null hypothesis that all persons answer according to the model. This distribution, however, turns out to be rather different for different values of the person's latent trait value. This value will be called ability parameter, although our results are equally valid for Rasch scales measuring other attributes.As the true ability parameter is unknown, one can only use its estimate in order to obtain an estimated person fit value and an estimated null hypothesis distribution. The paper describes three specifications for the latter: assuming that the true ability equals its estimate, integrating across the ability distribution assumed for the population, and conditioning on the total score, which is in the Rasch model the sufficient statistic for the ability parameter.Classification rules for aberrance will be worked out for each of the three specifications. Depending on test length, item parameters and desired accuracy, they are based on the exact distribution, its Monte Carlo estimate and a new and promising approximation based on the moments of the person fit statistic. Results for the likelihood person fit statistic are given in detail, the methods could also be applied to other fit statistics. A comparison of the three specifications results in the recommendation to condition on the total score, as this avoids some problems of interpretation that affect the other two specifications.The authors express their gratitude to the reviewers and to many colleagues for comments on an earlier version.     

Stochastic ordering using the latent trait and the sum score in polytomous IRT models

In a restricted class of item response theory (IRT) models for polytomous items the unweighted total score has monotone likelihood ratio (MLR) in the latent trait. MLR implies two stochastic ordering (SO) properties, denoted SOM and SOL, which are both weaker than MLR, but very useful for measurement with IRT models. Therefore, these SO properties are investigated for a broader class of IRT models for which the MLR property does not hold.In this study, first a taxonomy is given for nonparametric and parametric models for polytomous items based on the hierarchical relationship between the models. Next, it is investigated which models have the MLR property and which have the SO properties. It is shown that all models in the taxonomy possess the SOM property. However, counterexamples illustrate that many models do not, in general, possess the even more useful SOL property.Hemker's research was supported by the Netherlands Research Council, Grant 575-67-034. Junker's research was supported in part by the National Institutes of Health, Grant CA54852, and by the National Science Foundation, Grant DMS-94.04438.     

An extension of the partial credit model with an application to the measurement of change

The partial credit model is considered under the assumption of a certain linear decomposition of the item × category parameters _ih into basic parameters _j. This model is referred to as the linear partial credit model. A conditional maximum likelihood algorithm for estimation of the j is presented, based on (a) recurrences for the combinatorial functions involved, and (b) using a quasi-Newton approach, the so-called Broyden-Fletcher-Goldfarb-Shanno (BFGS) method; (a) guarantees numerically stable results, (b) avoids the direct computation of the Hesse matrix, yet produces a sequence of certain positive definite matricesB _k ,k=1, 2, ..., converging to the asymptotic variance-covariance matrix of the . The practicality of these numerical methods is demonstrated both by means of simulations and of an empirical application to the measurement of treatment effects in patients with psychosomatic disorders.The authors thank one anonymous reviewer for his constructive comments. Moreover, they thankfully acknowledge financial support by the Österreichische Nationalbank (Austrian National Bank) under Grant No. 3720.     

Treatment of obsessive-compulsive patients: The contribution of self-instructional training to the effectiveness of exposure

Fifteen obsessive-compulsive patients were given treatment consisting of ten sessions of gradual exposure in vivo. For half of the patients gradual exposure in vivo was preceded by self-instructional training.Treatment resulted in significant improvement on anxiety and avoidance scales, Leyton Obsessional Inventory, Self-rating Depression Scale and on ratings for anxious mood and depression. Neither the post-test nor the follow-ups 1 month and 6 months later indicated a difference between the effects of the two conditions. Self-instructional training did not enhance the effectiveness of gradual exposure in vivo.     

A person-fit index for polytomous rasch models,latent class models,and their mixture generalizations

A normally distributed person-fit index is proposed for detecting aberrant response patterns in latent class models and mixture distribution IRT models for dichotomous and polytomous data.This article extends previous work on the null distribution of person-fit indices for the dichotomous Rasch model to a number of models for categorical data. A comparison of two different approaches to handle the skewness of the person-fit index distribution is included.Major parts of this paper were written while the first author worked at the Institute for Science Education, Kiel, Germany. Any opinions expressed in this paper are those of the authors and not necessarily of Educational Testing Service. The results presented in this paper were improved by valuable comments from J. Rost, K. Yamamoto, N.D. Verhelst, E. Bedrick and two anonymous reviewers.     

Neurocognitive Performance in Children with ADHD and OCD

Attention-deficit/hyperactivity Disorder (ADHD) and obsessive–compulsive disorder (OCD) have both been linked to dysfunction in the cortico-striato-thalamo-cortical circuitry (CSTCC). However, the exact nature of neurocognitive deficits remains to be investigated in both disorders. We applied two neuropsychological tasks that tap into different functions associated with the CSTCC, namely a serial reaction time (SRT) task, developed to assess implicit sequence learning, and a delay aversion (DA) task in order to assess abnormal motivational processes. The performance data of boys with ADHD (n = 20), OCD (n = 20) and healthy controls (n = 25), all aged 10–18 years, were compared. Subjects with ADHD less frequently chose the larger, more delayed reward compared to those with OCD and controls, while subjects with OCD showed impaired implicit learning. In contrast, the ADHD group was unimpaired in their implicit learning behavior and the OCD group was not characterized by a DA style. Within the OCD-group, severity of obsessions was associated with implicit learning deficits and impulsive symptoms with DA in the ADHD-group. This double dissociation highlights the distinct cognitive dysfunctions associated with ADHD and OCD and might possibly point to different neural abnormalities in both disorders.     

A novel tool-use mode in animals: New Caledonian crows insert tools to transport objects

New Caledonian crows (Corvus moneduloides) rely heavily on a range of tools to extract prey. They manufacture novel tools, save tools for later use, and have morphological features that facilitate tool use. We report six observations, in two individuals, of a novel tool-use mode not previously reported in non-human animals. Insert-and-transport tool use involves inserting a stick into an object and then moving away, thereby transporting both object and tool. All transported objects were non-food objects. One subject used a stick to transport an object that was too large to be handled by beak, which suggests the tool facilitated object control. The function in the other cases is unclear but seems to be an expression of play or exploration. Further studies should investigate whether it is adaptive in the wild and to what extent crows can flexibly apply the behaviour in experimental settings when purposive transportation of objects is advantageous.