Reliability of test scores in nonparametric item response theory

Three methods for estimating reliability are studied within the context of nonparametric item response theory. Two were proposed originally by Mokken (1971) and a third is developed in this paper. Using a Monte Carlo strategy, these three estimation methods are compared with four classical lower bounds to reliability. Finally, recommendations are given concerning the use of these estimation methods.The authors are grateful for constructive comments from the reviewers and from Charles Lewis.     

The many null distributions of person fit indices

This paper deals with the situation of an investigator who has collected the scores ofn persons to a set ofk dichotomous items, and wants to investigate whether the answers of all respondents are compatible with the one parameter logistic test model of Rasch. Contrary to the standard analysis of the Rasch model, where all persons are kept in the analysis and badly fittingitems may be removed, this paper studies the alternative model in which a small minority ofpersons has an answer strategy not described by the Rasch model. Such persons are called anomalous or aberrant. From the response vectors consisting ofk symbols each equal to 0 or 1, it is desired to classify each respondent as either anomalous or as conforming to the model. As this model is probabilistic, such a classification will possibly involve false positives and false negatives. Both for the Rasch model and for other item response models, the literature contains several proposals for a person fit index, which expresses for each individual the plausibility that his/her behavior follows the model. The present paper argues that such indices can only provide a satisfactory solution to the classification problem if their statistical distribution is known under the null hypothesis that all persons answer according to the model. This distribution, however, turns out to be rather different for different values of the person's latent trait value. This value will be called ability parameter, although our results are equally valid for Rasch scales measuring other attributes.As the true ability parameter is unknown, one can only use its estimate in order to obtain an estimated person fit value and an estimated null hypothesis distribution. The paper describes three specifications for the latter: assuming that the true ability equals its estimate, integrating across the ability distribution assumed for the population, and conditioning on the total score, which is in the Rasch model the sufficient statistic for the ability parameter.Classification rules for aberrance will be worked out for each of the three specifications. Depending on test length, item parameters and desired accuracy, they are based on the exact distribution, its Monte Carlo estimate and a new and promising approximation based on the moments of the person fit statistic. Results for the likelihood person fit statistic are given in detail, the methods could also be applied to other fit statistics. A comparison of the three specifications results in the recommendation to condition on the total score, as this avoids some problems of interpretation that affect the other two specifications.The authors express their gratitude to the reviewers and to many colleagues for comments on an earlier version.     

Stochastic ordering using the latent trait and the sum score in polytomous IRT models

In a restricted class of item response theory (IRT) models for polytomous items the unweighted total score has monotone likelihood ratio (MLR) in the latent trait. MLR implies two stochastic ordering (SO) properties, denoted SOM and SOL, which are both weaker than MLR, but very useful for measurement with IRT models. Therefore, these SO properties are investigated for a broader class of IRT models for which the MLR property does not hold.In this study, first a taxonomy is given for nonparametric and parametric models for polytomous items based on the hierarchical relationship between the models. Next, it is investigated which models have the MLR property and which have the SO properties. It is shown that all models in the taxonomy possess the SOM property. However, counterexamples illustrate that many models do not, in general, possess the even more useful SOL property.Hemker's research was supported by the Netherlands Research Council, Grant 575-67-034. Junker's research was supported in part by the National Institutes of Health, Grant CA54852, and by the National Science Foundation, Grant DMS-94.04438.     

Dynamic models of development and the mechanistic-organismic controversy

Starting with a prototypical model of Newtonian mechanics, a sequence of dynamic models with increasing structural complexity is presented. The implicated smooth variation along a dimension of structural complexity yields dynamic models which display organismic properties such as stagewise development and emergent self-organization. Consequently, it is concluded that dynamic models of development are inherently neutral with respect to the mechanism-organicism controversy. Accordingly, a unitary research program is proposed in which these dynamic models serve as building blocks for the construction of a coherent explanatory system of development.     

Modeling Asymmetry in the Time–Distance Relation of Ordinal Personality Items

In analyzing responses and response times to personality questionnaire items, models have been proposed which include the so-called “inverted-U effect.” These models predict that response times to personality test items decrease as the latent trait value of a given person gets closer to the attractiveness of an item. Initial studies into these models have focused on dichotomous personality items, and more recently, models for Likert-type scale items have been proposed. In all these models, it is assumed that the inverted-U effect is symmetrical around 0, while, as will be explained in this article, there are substantive and statistical reasons to study this assumption. Therefore, in this article, a general inverted-U model is proposed which accommodates two sources of asymmetry between the response times and the attractiveness of the items. The viability of this model is demonstrated in a simulation study, and the model is applied to the responses and response times of the Temperament and Character Inventory–Revised, covering a broad range of personality dimensions.     

A person-fit index for polytomous rasch models,latent class models,and their mixture generalizations

A normally distributed person-fit index is proposed for detecting aberrant response patterns in latent class models and mixture distribution IRT models for dichotomous and polytomous data.This article extends previous work on the null distribution of person-fit indices for the dichotomous Rasch model to a number of models for categorical data. A comparison of two different approaches to handle the skewness of the person-fit index distribution is included.Major parts of this paper were written while the first author worked at the Institute for Science Education, Kiel, Germany. Any opinions expressed in this paper are those of the authors and not necessarily of Educational Testing Service. The results presented in this paper were improved by valuable comments from J. Rost, K. Yamamoto, N.D. Verhelst, E. Bedrick and two anonymous reviewers.     

Selecting a linear mixed model for longitudinal data: repeated measures analysis of variance, covariance pattern model, and growth curve approaches

With increasing popularity, growth curve modeling is more and more often considered as the 1st choice for analyzing longitudinal data. Although the growth curve approach is often a good choice, other modeling strategies may more directly answer questions of interest. It is common to see researchers fit growth curve models without considering alterative modeling strategies. In this article we compare 3 approaches for analyzing longitudinal data: repeated measures analysis of variance, covariance pattern models, and growth curve models. As all are members of the general linear mixed model family, they represent somewhat different assumptions about the way individuals change. These assumptions result in different patterns of covariation among the residuals around the fixed effects. In this article, we first indicate the kinds of data that are appropriately modeled by each and use real data examples to demonstrate possible problems associated with the blanket selection of the growth curve model. We then present a simulation that indicates the utility of Akaike information criterion and Bayesian information criterion in the selection of a proper residual covariance structure. The results cast doubt on the popular practice of automatically using growth curve modeling for longitudinal data without comparing the fit of different models. Finally, we provide some practical advice for assessing mean changes in the presence of correlated data.     

The emotional and cognitive effect of immersion in film viewing

This brief report presents an experiment testing the effect of immersion on emotional responses and cognitive genre categorisation of film viewers. Immersion of a film presentation was varied by presenting an animated movie either in a 3D-viewing condition (low immersive condition) or in a CAVE condition (high immersive condition, comparable to virtual reality experience). Viewers rated their emotions and categorised the movies into four basic film genres (action, drama, comedy, and non-fiction). Two distinct types of emotion were measured: Fictional World emotions (e.g., sadness) in response to the presented fictional events and Artefact emotions in response to the film as an artefact (e.g., fascination). Results showed that stronger immersion led to more intense emotions but did not influence genre categorisation. In line with expectations, both types of emotional response were intensified by high immersion. The results are explained by suggesting that highly immersive cinema has its impact on a basic dimension of emotion, namely arousal that underlies both types of emotions.     

On the Validity of Simulating Stagewise Development by Means of PDP Networks: Application of Catastrophe Analysis and an Experimental Test of Rule-Like Network Performance

This article addresses the ability of Parallel Distributed Processing (PDP) networks to generate stagewise cognitive development in accordance with Piaget's theory of cognitive epigenesis. We carried out a replication study of the simulation experiments by McClelland (1989) and McClelland and Jenkins (1991) in which a PDP network learns to solve balance scale problems. In objective tests motivated from catastrophe theory, a mathematical theory of transitions in epigenetical systems, no evidence for stage transitions in network performance was found. It is concluded that PDP networks lack the ability to recover the positive outcomes of analogous catastrophe analyses of real cognitive developmental data. In an attempt to further characterize the learning behaviour of PDP networks, we carried out a second simulation study using the discrimination-shift paradigm. The results thus obtained indicate that PDP learning is compatible with the learning of stimulus-response relationships, not with the acquisition of mediating rules such as conceived in (neo-)Piagetian theory. In closing, we speculate about the feasibility of simulating stagewise development with alternative network architectures.