Nonparametric polytomous IRT models for invariant item ordering, with results for parametric models

It is often considered desirable to have the same ordering of the items by difficulty across different levels of the trait or ability. Such an ordering is an invariant item ordering (IIO). An IIO facilitates the interpretation of test results. For dichotomously scored items, earlier research surveyed the theory and methods of an invariant ordering in a nonparametric IRT context. Here the focus is on polytomously scored items, and both nonparametric and parametric IRT models are considered.The absence of the IIO property in twononparametric polytomous IRT models is discussed, and two nonparametric models are discussed that imply an IIO. A method is proposed that can be used to investigate whether empirical data imply an IIO. Furthermore, only twoparametric polytomous IRT models are found to imply an IIO. These are the rating scale model (Andrich, 1978) and a restricted rating scale version of the graded response model (Muraki, 1990). Well-known models, such as the partial credit model (Masters, 1982) and the graded response model (Samejima, 1969), do no imply an IIO.     

Latent class models for nonmonotone dichotomous items

Starting from perfectly discriminating nonmonotone dichotomous items, a class of probabilistic models with or without response errors and with or without intrinsically unscalable respondents is described. All these models can be understood as simply restricted latent class analysis. Thus, the estimation and identifiability of the parameters (class sizes and item latent probabilities) as well as the chi-squared goodness-of-fit tests (Pearson and likelihood-ratio) are free of the problems. The applicability of the proposed variants of latent class models is demonstrated on real attitudinal data.This research was supported by the Kulturamt der Stadt Wien, Magistratsabteilung 7.The author wishes to thank the editor, Ivo W. Molenaar, as well as Clifford C. Clogg and the anonymous reviewers for their valuable comments on the earlier drafts of this paper.     

Bayesian estimation and model selection in ordered latent class models for polytomous items

In a latent class IRT model in which the latent classes are ordered on one dimension, the class specific response probabilities are subject to inequality constraints. The number of these inequality constraints increase dramatically with the number of response categories per item, if assumptions like monotonicity or double monotonicity of the cumulative category response functions are postulated. A Markov chain Monte Carlo method, the Gibbs sampler, can sample from the multivariate posterior distribution of the parameters under the constraints. Bayesian model selection can be done by posterior predictive checks and Bayes factors. A simulation study is done to evaluate results of the application of these methods to ordered latent class models in three realistic situations. Also, an example of the presented methods is given for existing data with polytomous items. It can be concluded that the Bayesian estimation procedure can handle the inequality constraints on the parameters very well. However, the application of Bayesian model selection methods requires more research.     

Application of four procedures for detecting differential item functioning in polytomous items

The authors describe and use four methods for detecting Differential Item Functioning in polytomous items: Mantel, Generalized Mantel-Haenszel (GMH), Ordinal Logistic Regression (RLO), and Discriminant Logistic Regression (RLD). For each procedure, the theoretical model and the measure of effect size are described. The data from the "Reading Comprehension Test" from the PISA2000 evaluation program were analyzed using a cross-validation design. Two booklets were independently evaluated in the American and Spanish samples. Adopting as decision rule the significance of the statistical test and the measurement of the effect size, agreement among the evaluated procedures was total for two of the analyzed items.     

A class of multidimensional IRT models for testing unidimensionality and clustering items

We illustrate a class of multidimensional item response theory models in which the items are allowed to have different discriminating power and the latent traits are represented through a vector having a discrete distribution. We also show how the hypothesis of unidimensionality may be tested against a specific bidimensional alternative by using a likelihood ratio statistic between two nested models in this class. For this aim, we also derive an asymptotically equivalent Wald test statistic which is faster to compute. Moreover, we propose a hierarchical clustering algorithm which can be used, when the dimensionality of the latent structure is completely unknown, for dividing items into groups referred to different latent traits. The approach is illustrated through a simulation study and an application to a dataset collected within the National Assessment of Educational Progress, 1996. The author would like to thank the Editor, an Associate Editor and three anonymous referees for stimulating comments. I also thank L. Scaccia, F. Pennoni and M. Lupparelli for having done part of the simulations.     

Essential independence and likelihood-based ability estimation for polytomous items

A definition ofessential independence is proposed for sequences of polytomous items. For items satisfying the reasonable assumption that the expected amount of credit awarded increases with examinee ability, we develop a theory ofessential unidimensionality which closely parallels that of Stout. Essentially unidimensional item sequences can be shown to have a unique (up to change-of-scale) dominant underlying trait, which can be consistently estimated by a monotone transformation of the sum of the item scores. In more general polytomous-response latent trait models (with or without ordered responses), anM-estimator based upon maximum likelihood may be shown to be consistent for under essentially unidimensional violations of local independence and a variety of monotonicity/identifiability conditions. A rigorous proof of this fact is given, and the standard error of the estimator is explored. These results suggest that ability estimation methods that rely on the summation form of the log likelihood under local independence should generally be robust under essential independence, but standard errors may vary greatly from what is usually expected, depending on the degree of departure from local independence. An index of departure from local independence is also proposed.This work was supported in part by Office of Naval Research Grant N00014-87-K-0277 and National Science Foundation Grant NSF-DMS-88-02556. The author is grateful to William F. Stout for many helpful comments, and to an anonymous reviewer for raising the questions addressed in section 2. A preliminary version of section 6 appeared in the author's Ph.D. thesis.     

Stochastic ordering using the latent trait and the sum score in polytomous IRT models

In a restricted class of item response theory (IRT) models for polytomous items the unweighted total score has monotone likelihood ratio (MLR) in the latent trait. MLR implies two stochastic ordering (SO) properties, denoted SOM and SOL, which are both weaker than MLR, but very useful for measurement with IRT models. Therefore, these SO properties are investigated for a broader class of IRT models for which the MLR property does not hold.In this study, first a taxonomy is given for nonparametric and parametric models for polytomous items based on the hierarchical relationship between the models. Next, it is investigated which models have the MLR property and which have the SO properties. It is shown that all models in the taxonomy possess the SOM property. However, counterexamples illustrate that many models do not, in general, possess the even more useful SOL property.Hemker's research was supported by the Netherlands Research Council, Grant 575-67-034. Junker's research was supported in part by the National Institutes of Health, Grant CA54852, and by the National Science Foundation, Grant DMS-94.04438.     

A person-fit index for polytomous rasch models,latent class models,and their mixture generalizations

A normally distributed person-fit index is proposed for detecting aberrant response patterns in latent class models and mixture distribution IRT models for dichotomous and polytomous data.This article extends previous work on the null distribution of person-fit indices for the dichotomous Rasch model to a number of models for categorical data. A comparison of two different approaches to handle the skewness of the person-fit index distribution is included.Major parts of this paper were written while the first author worked at the Institute for Science Education, Kiel, Germany. Any opinions expressed in this paper are those of the authors and not necessarily of Educational Testing Service. The results presented in this paper were improved by valuable comments from J. Rost, K. Yamamoto, N.D. Verhelst, E. Bedrick and two anonymous reviewers.     

Graphical regression models for polytomous variables

When modeling the relationship between two nominal categorical variables, it is often desirable to include covariates to understand how individuals differ in their response behavior. Typically, however, not all the relevant covariates are available, with the result that the measured variables cannot fully account for the associations between the nominal variables. Under the assumption that the observed and unobserved variables follow a homogeneous conditional Gaussian distribution, this paper proposesRC(M) regression models to decompose the residual associations between the polytomous variables. Based on Goodman's (1979, 1985)RC(M) association model, a distinctive feature ofRC(M) regression models is that they facilitate the joint estimation of effects due to manifest and omitted (continuous) variables without requiring numerical integration. TheRC(M) regression models are illustrated using data from the High School and Beyond study (Tatsuoka & Lohnes, 1988). This article was accepted for publication, when Willem J. Heiser was the Editor ofPsychometrika. This research was supported by grants from the National Science Foundation (#SBR96-17510 and #SBR94-09531) and the Bureau of Educational Research at the University of Illinois. We thank Jee-Seon Kim for comments and computational assistance.     

Generating items during testing: Psychometric issues and models

On-line item generation is becoming increasingly feasible for many cognitive tests. Item generation seemingly conflicts with the well established principle of measuring persons from items with known psychometric properties. This paper examines psychometric principles and models required for measurement from on-line item generation. Three psychometric issues are elaborated for item generation. First, design principles to generate items are considered. A cognitive design system approach is elaborated and then illustrated with an application to a test of abstract reasoning. Second, psychometric models for calibrating generating principles, rather than specific items, are required. Existing item response theory (IRT) models are reviewed and a new IRT model that includes the impact on item discrimination, as well as difficulty, is developed. Third, the impact of item parameter uncertainty on person estimates is considered. Results from both fixed content and adaptive testing are presented.This article is based on the Presidential Address Susan E. Embretson gave on June 26, 1999 at the 1999 Annual Meeting of the Psychometric Society held at the University of Kansas in Lawrence, Kansas. —Editor     

Two item response theory models for analysing normative forced‐choice personality items

This paper proposes two unidimensional item response theory (IRT) models for analysing normative forced‐choice personality items. Both models are derived from a common theoretical framework and arise as a result of different assumptions regarding the mechanism of choice. The simplest mechanism gives rise to the one‐parameter normal‐ogive model. The second mechanism gives rise to a new IRT model, which is closely related to the Coombs–Zinnes probabilistic unfolding model. The second model is compared theoretically to the normal‐ogive model in terms of item characteristic curves and amount of item information. Next, procedures for estimating the respondent and the item parameters in the second model are described. Finally, both models are empirically compared by using two well‐known personality measures.     

Structural equation models with continuous and polytomous variables

A two-stage procedure is developed for analyzing structural equation models with continuous and polytomous variables. At the first stage, the maximum likelihood estimates of the thresholds, polychoric covariances and variances, and polyserial covariances are simultaneously obtained with the help of an appropriate transformation that significantly simplifies the computation. An asymptotic covariance matrix of the estiates is also computed. At the second stage, the parameters in the structural covariance model are obtained via the generalized least squares approach. Basic statistical properties of the estimates are derived and some illustrative examples and a small simulation study are reported.This research was supported in part by a research grant DA01070 from the U. S. Public Health Service. We are indebted to several referees and the editor for very valuable comments and suggestions for improvement of this paper. The computing assistance of King-Hong Leung and Man-Lai Tang is also gratefully acknowledged.     

Rasch models for item bundles

This paper discusses the application of a class of Rasch models to situations where test items are grouped into subsets and the common attributes of items within these subsets brings into question the usual assumption of conditional independence. The models are all expressed as particular cases of the random coefficients multinomial logit model developed by Adams and Wilson. This formulation allows a very flexible approach to the specification of alternative models, and makes model testing particularly straightforward. The use of the models is illustrated using item bundles constructed in the framework of the SOLO taxonomy of Biggs and Collis.The work of both authors was supported by fellowships from the National Academy of Education Spencer Fellowship.     

Testing the conditional independence and monotonicity assumptions of item response theory

When item characteristic curves are nondecreasing functions of a latent variable, the conditional or local independence of item responses given the latent variable implies nonnegative conditional covariances between all monotone increasing functions of a set of item responses given any function of the remaining item responses. This general result provides a basis for testing the conditional independence assumption without first specifying a parametric form for the nondecreasing item characteristic curves. The proposed tests are simple, have known asymptotic null distributions, and possess certain optimal properties. In an example, the conditional independence hypothesis is rejected for all possible forms of monotone item characteristic curves.The author acknowledges Paul W. Holland for valuable conversations on the subject of this paper; Henry Braun and Fred Lord for comments at a presentation on this subject which led to improvements in the paper; Carl H. Haag for permission to use the data in §4; Bruce Kaplan for assistance with computing; and two referees for helpful suggestions. Requests for reprints should be sent to Paul R. Rosenbaum     

Bayesian item fit analysis for unidimensional item response theory models

Assessing item fit for unidimensional item response theory models for dichotomous items has always been an issue of enormous interest, but there exists no unanimously agreed item fit diagnostic for these models, and hence there is room for further investigation of the area. This paper employs the posterior predictive model‐checking method, a popular Bayesian model‐checking tool, to examine item fit for the above‐mentioned models. An item fit plot, comparing the observed and predicted proportion‐correct scores of examinees with different raw scores, is suggested. This paper also suggests how to obtain posterior predictive p‐values (which are natural Bayesian p‐values) for the item fit statistics of Orlando and Thissen that summarize numerically the information in the above‐mentioned item fit plots. A number of simulation studies and a real data application demonstrate the effectiveness of the suggested item fit diagnostics. The suggested techniques seem to have adequate power and reasonable Type I error rate, and psychometricians will find them promising.     

A three-stage estimation procedure for structural equation models with polytomous variables

This paper is concerned with the analysis of structural equation models with polytomous variables. A computationally efficient three-stage estimator of the thresholds and the covariance structure parameters, based on partition maximum likelihood and generalized least squares estimation, is proposed. An example is presented to illustrate the method.This research was supported in part by a research grant DA01070 from the U.S. Public Health Service. The production assistance of Julie Speckart is gratefully acknowledged.     

The asymptotic posterior normality of the latent trait for polytomous IRT models

Chang and Stout (1993) presented a derivation of the asymptotic posterior normality of the latent trait given examinee responses under nonrestrictive nonparametric assumptions for dichotomous IRT models. This paper presents an extention of their results to polytomous IRT models in a fairly straightforward manner. In addition, a global information function is defined, and the relationship between the global information function and the currently used information functions is discussed. An information index that combines both the global and local information is proposed for adaptive testing applications.This research was partially supported by Educational Testing Service Allocation Project No. 79424. The author wishes to thank Charles Davis, Xuming He, Frank Jenkins, Spence Swinton, William Stout, Ming-Mai Wang, and Zhiliang Ying for their helpful comments and discussions. The author particularly wishes to thank the Editor, Shizuhiko Nishisato, the Associate Editor, and three anonymous reviewers for their thoroughness and thoughtful suggestions.     

Bayesian item selection criteria for adaptive testing

Owen (1975) proposed an approximate empirical Bayes procedure for item selection in computerized adaptive testing (CAT). The procedure replaces the true posterior by a normal approximation with closed-form expressions for its first two moments. This approximation was necessary to minimize the computational complexity involved in a fully Bayesian approach but is no longer necessary given the computational power currently available for adaptive testing. This paper suggests several item selection criteria for adaptive testing which are all based on the use of the true posterior. Some of the statistical properties of the ability estimator produced by these criteria are discussed and empirically characterized.Portions of this paper were presented at the 60th annual meeting of the Psychometric Society, Minneapolis, Minnesota, June, 1995. The author is indebted to Wim M. M. Tielen for his computational support.