A flexible latent trait model for response times in tests

Latent trait models for response times in tests have become popular recently. One challenge for response time modeling is the fact that the distribution of response times can differ considerably even in similar tests. In order to reduce the need for tailor-made models, a model is proposed that unifies two popular approaches to response time modeling: Proportional hazard models and the accelerated failure time model with log–normally distributed response times. This is accomplished by resorting to discrete time. The categorization of response time allows the formulation of a response time model within the framework of generalized linear models by using a flexible link function. Item parameters of the proposed model can be estimated with marginal maximum likelihood estimation. Applicability of the proposed approach is demonstrated with a simulation study and an empirical application. Additionally, means for the evaluation of model fit are suggested.     

An accumulator model for responses and response times in tests based on the proportional hazards model

Latent trait models for responses and response times in tests often lack a substantial interpretation in terms of a cognitive process model. This is a drawback because process models are helpful in clarifying the meaning of the latent traits. In the present paper, a new model for responses and response times in tests is presented. The model is based on the proportional hazards model for competing risks. Two processes are assumed, one reflecting the increase in knowledge and the second the tendency to discontinue. The processes can be characterized by two proportional hazards models whose baseline hazard functions correspond to the temporary increase in knowledge and discouragement. The model can be calibrated with marginal maximum likelihood estimation and an application of the ECM algorithm. Two tests of model fit are proposed. The amenability of the proposed approaches to model calibration and model evaluation is demonstrated in a simulation study. Finally, the model is used for the analysis of two empirical data sets.     

Robust estimation of the hierarchical model for responses and response times

Van der Linden's (2007, Psychometrika, 72, 287) hierarchical model for responses and response times in tests has numerous applications in psychological assessment. The success of these applications requires the parameters of the model to have been estimated without bias. The data used for model fitting, however, are often contaminated, for example, by rapid guesses or lapses of attention. This distorts the parameter estimates. In the present paper, a novel estimation approach is proposed that is robust against contamination. The approach consists of two steps. In the first step, the response time model is fitted on the basis of a robust estimate of the covariance matrix. In the second step, the item response model is extended to a mixture model, which allows for a proportion of irregular responses in the data. The parameters of the mixture model are then estimated with a modified marginal maximum likelihood estimator. The modified marginal maximum likelihood estimator downweights responses of test-takers with unusual response time patterns. As a result, the estimator is resistant to several forms of data contamination. The robustness of the approach is investigated in a simulation study. An application of the estimator is demonstrated with real data.     

Shrinkage estimation of the three-parameter logistic model

The three-parameter logistic model is widely used to model the responses to a proficiency test when the examinees can guess the correct response, as is the case for multiple-choice items. However, the weak identifiability of the parameters of the model results in large variability of the estimates and in convergence difficulties in the numerical maximization of the likelihood function. To overcome these issues, in this paper we explore various shrinkage estimation methods, following two main approaches. First, a ridge-type penalty on the guessing parameters is introduced in the likelihood function. The tuning parameter is then selected through various approaches: cross-validation, information criteria or using an empirical Bayes method. The second approach explored is based on the methodology developed to reduce the bias of the maximum likelihood estimator through an adjusted score equation. The performance of the methods is investigated through simulation studies and a real data example.     

Modelling inter-individual differences in latent within-person variation: The confirmatory factor level variability model

Psychological theories often produce hypotheses that pertain to individual differences in within-person variability. To empirically test the predictions entailed by such hypotheses with longitudinal data, researchers often use multilevel approaches that allow them to model between-person differences in the mean level of a certain variable and the residual within-person variance. Currently, these approaches can be applied only when the data stem from a single variable. However, it is common practice in psychology to assess not just a single measure but rather several measures of a construct. In this paper we describe a model in which we combine the single-indicator model with confirmatory factor analysis. The new model allows individual differences in latent mean-level factors and latent within-person variability factors to be estimated. Furthermore, we show how the model's parameters can be estimated with a maximum likelihood estimator, and we illustrate the approach using an example that involves intensive longitudinal data.     

A latent trait model for dichotomous choice data

The PARELLA model is a probabilistic parallelogram model that can be used for the measurement of latent attitudes or latent preferences. The data analyzed are the dichotomous responses of persons to stimuli, with a one (zero) indicating agreement (disagreement) with the content of the stimulus. The model provides a unidimensional representation of persons and items. The response probabilities are a function of the distance between person and stimulus: the smaller the distance, the larger the probability that a person will agree with the content of the stimulus. An estimation procedure based on expectation maximization and marginal maximum likelihood is developed and the quality of the resulting parameter estimates evaluated.I gratefully acknowledge Ivo Molenaar and Wijbrandt van Schuur for their advice and encouragement during the course of the investigation, Derk-Jan Kiewiet who constructed the program for the ML estimator for the person parameter and Anne Boomsma, Wendy Post, Tom Snijders, and David Thissen for their comments on smaller aspects of the investigation.     

Testing for DIF in a model with single peaked item characteristic curves: The parella model

The PARELLA model is a probabilistic parallelogram model that can be used for the measurement of latent attitudes or latent preferences. The data analyzed are the dichotomous responses of persons to items, with a one (zero) indicating agreement (disagreement) with the content of the item. The model provides a unidimensional representation of persons and items. The response probabilities are a function of the distance between person and item: the smaller the distance, the larger the probability that a person will agree with the content of the item. This paper discusses how the approach to differential item functioning presented by Thissen, Steinberg, and Wainer can be implemented for the PARELLA model. Requests for the PARELLA software should be sent to Iec Progamma PO Box 841, 9700 AV Groningen, The Netherlands.     

Locally dependent latent trait model and the dutch identity revisited

In this paper, we propose a class of locally dependent latent trait models for responses to psychological and educational tests. Typically, item response models treat an individual's multiple response to stimuli as conditional independent given the individual's latent trait. In this paper, instead the focus is on models based on a family of conditional distributions, or kernel, that describes joint multiple item responses as a function of student latent trait, not assuming conditional independence. Specifically, we examine a hybrid kernel which comprises a component for one-way item response functions and a component for conditional associations between items given latent traits. The class of models allows the extension of item response theory to cover some new and innovative applications in psychological and educational research. An EM algorithm for marginal maximum likelihood of the hybrid kernel model is proposed. Furthermore, we delineate the relationship of the class of locally dependent models and the log-linear model by revisiting the Dutch identity (Holland, 1990). The work is supported by a research grant from the Marshall School of Business, University of Southern California. The author thanks the anonymous referees for their suggestions.     

A Note on the Hierarchical Model for Responses and Response Times in Tests of van der Linden (2007)

Findings suggest that in psychological tests not only the responses but also the times needed to give the responses are related to characteristics of the test taker. This observation has stimulated the development of latent trait models for the joint distribution of the responses and the response times. Such models are motivated by the hope to improve the estimation of the latent traits by additionally considering response time. In this article, the potential relevance of the response times for psychological assessment is explored for the model of van der Linden (Psychometrika 72:287–308, 2007) that seems to have become the standard approach to response time modeling in educational testing. It can be shown that the consideration of response times increases the information of the test. However, one also can prove that the contribution of the response times to the test information is bounded and has a simple limit.     

A joint modeling framework of responses and response times to assess learning outcomes

Abstract

A general modeling framework of response accuracy and response times is proposed to track skill acquisition and provide additional diagnostic information on the change of latent speed in a learning environment. This framework consists of two types of models: a dynamic response model that captures the response accuracy and the change of discrete latent attribute profile upon factors such as practice, intervention effects, and other latent and observable covariates, and a dynamic response time model that describes the change of the continuous response latency due to change of latent attribute profile. These two types of models are connected through a parameter, describing the change rate of the latent speed through the learning process, and a covariate defined as a function of the latent attribute profile. A Bayesian estimation procedure is developed to calibrate the model parameters and measure the latent variables. The estimation algorithm is evaluated through several simulation studies under various conditions. The proposed models are applied to a real data set collected through a spatial rotation diagnostic assessment paired with learning tools.     

Rasch's model for reading speed with manifest explanatory variables

In educational and psychological measurement we find the distinction between speed and power tests. Although most tests are partially speeded, the speed element is usually neglected. Here we consider a latent trait model developed by Rasch for the response time on a (set of) pure speed test(s), which is based on the assumption that the test response times are approximately gamma distributed, with known shape parameters and scale parameters depending on subject ability and test difficulty parameters. In our approach the subject parameters are treated as random variables having a common gamma distribution. From this, maximum marginal likelihood estimators are derived for the test difficulties and the parameters of the latent subject distribution. This basic model can be extended in a number of ways. Explanatory variables for the latent subject parameters and for the test parameters can be incorporated in the model. Our methods are illustrated by the analysis of a simulated and an empirical data set.     

Gaussian variational estimation for multidimensional item response theory

Multidimensional item response theory (MIRT) is widely used in assessment and evaluation of educational and psychological tests. It models the individual response patterns by specifying a functional relationship between individuals' multiple latent traits and their responses to test items. One major challenge in parameter estimation in MIRT is that the likelihood involves intractable multidimensional integrals due to the latent variable structure. Various methods have been proposed that involve either direct numerical approximations to the integrals or Monte Carlo simulations. However, these methods are known to be computationally demanding in high dimensions and rely on sampling data points from a posterior distribution. We propose a new Gaussian variational expectation--maximization (GVEM) algorithm which adopts variational inference to approximate the intractable marginal likelihood by a computationally feasible lower bound. In addition, the proposed algorithm can be applied to assess the dimensionality of the latent traits in an exploratory analysis. Simulation studies are conducted to demonstrate the computational efficiency and estimation precision of the new GVEM algorithm compared to the popular alternative Metropolis–Hastings Robbins–Monro algorithm. In addition, theoretical results are presented to establish the consistency of the estimator from the new GVEM algorithm.     

The bias function of the maximum likelihood estimate of ability for the dichotomous response level

Samejima has recently given an approximation for the bias function for the maximum likelihood estimate of the latent trait in the general case where item responses are discrete, generalizing Lord's bias function in the three-parameter logistic model for the dichotomous response level. In the present paper, observations are made about the behavior of this bias function for the dichotomous response level in general, and also with respect to several widely used mathematical models. Some empirical examples are given.     

A nonlinear mixed model framework for item response theory

Mixed models take the dependency between observations based on the same cluster into account by introducing 1 or more random effects. Common item response theory (IRT) models introduce latent person variables to model the dependence between responses of the same participant. Assuming a distribution for the latent variables, these IRT models are formally equivalent with nonlinear mixed models. It is shown how a variety of IRT models can be formulated as particular instances of nonlinear mixed models. The unifying framework offers the advantage that relations between different IRT models become explicit and that it is rather straightforward to see how existing IRT models can be adapted and extended. The approach is illustrated with a self-report study on anger.     

Analysis of multivariate mixed longitudinal data: A flexible latent process approach

Multivariate ordinal and quantitative longitudinal data measuring the same latent construct are frequently collected in psychology. We propose an approach to describe change over time of the latent process underlying multiple longitudinal outcomes of different types (binary, ordinal, quantitative). By relying on random‐effect models, this approach handles individually varying and outcome‐specific measurement times. A linear mixed model describes the latent process trajectory while equations of observation combine outcome‐specific threshold models for binary or ordinal outcomes and models based on flexible parameterized non‐linear families of transformations for Gaussian and non‐Gaussian quantitative outcomes. As models assuming continuous distributions may be also used with discrete outcomes, we propose likelihood and information criteria for discrete data to compare the goodness of fit of models assuming either a continuous or a discrete distribution for discrete data. Two analyses of the repeated measures of the Mini‐Mental State Examination, a 20‐item psychometric test, illustrate the method. First, we highlight the usefulness of parameterized non‐linear transformations by comparing different flexible families of transformation for modelling the test as a sum score. Then, change over time of the latent construct underlying directly the 20 items is described using two‐parameter longitudinal item response models that are specific cases of the approach.     

A hierarchical latent response model for inferences about examinee engagement in terms of guessing and item-level non-response

In low-stakes assessments, test performance has few or no consequences for examinees themselves, so that examinees may not be fully engaged when answering the items. Instead of engaging in solution behaviour, disengaged examinees might randomly guess or generate no response at all. When ignored, examinee disengagement poses a severe threat to the validity of results obtained from low-stakes assessments. Statistical modelling approaches in educational measurement have been proposed that account for non-response or for guessing, but do not consider both types of disengaged behaviour simultaneously. We bring together research on modelling examinee engagement and research on missing values and present a hierarchical latent response model for identifying and modelling the processes associated with examinee disengagement jointly with the processes associated with engaged responses. To that end, we employ a mixture model that identifies disengagement at the item-by-examinee level by assuming different data-generating processes underlying item responses and omissions, respectively, as well as response times associated with engaged and disengaged behaviour. By modelling examinee engagement with a latent response framework, the model allows assessing how examinee engagement relates to ability and speed as well as to identify items that are likely to evoke disengaged test-taking behaviour. An illustration of the model by means of an application to real data is presented.     

Stochastic EM for estimating the parameters of a multilevel IRT model

An item response theory (IRT) model is used as a measurement error model for the dependent variable of a multilevel model. The dependent variable is latent but can be measured indirectly by using tests or questionnaires. The advantage of using latent scores as dependent variables of a multilevel model is that it offers the possibility of modelling response variation and measurement error and separating the influence of item difficulty and ability level. The two‐parameter normal ogive model is used for the IRT model. It is shown that the stochastic EM algorithm can be used to estimate the parameters which are close to the maximum likelihood estimates. This algorithm is easily implemented. The estimation procedure will be compared to an implementation of the Gibbs sampler in a Bayesian framework. Examples using real data are given.     

A class of multidimensional IRT models for testing unidimensionality and clustering items

We illustrate a class of multidimensional item response theory models in which the items are allowed to have different discriminating power and the latent traits are represented through a vector having a discrete distribution. We also show how the hypothesis of unidimensionality may be tested against a specific bidimensional alternative by using a likelihood ratio statistic between two nested models in this class. For this aim, we also derive an asymptotically equivalent Wald test statistic which is faster to compute. Moreover, we propose a hierarchical clustering algorithm which can be used, when the dimensionality of the latent structure is completely unknown, for dividing items into groups referred to different latent traits. The approach is illustrated through a simulation study and an application to a dataset collected within the National Assessment of Educational Progress, 1996. The author would like to thank the Editor, an Associate Editor and three anonymous referees for stimulating comments. I also thank L. Scaccia, F. Pennoni and M. Lupparelli for having done part of the simulations.