Assessing Item Fit for Unidimensional Item Response Theory Models Using Residuals from Estimated Item Response Functions

Residual analysis (e.g. Hambleton & Swaminathan, Item response theory: principles and applications, Kluwer Academic, Boston, 1985; Hambleton, Swaminathan, & Rogers, Fundamentals of item response theory, Sage, Newbury Park, 1991) is a popular method to assess fit of item response theory (IRT) models. We suggest a form of residual analysis that may be applied to assess item fit for unidimensional IRT models. The residual analysis consists of a comparison of the maximum-likelihood estimate of the item characteristic curve with an alternative ratio estimate of the item characteristic curve. The large sample distribution of the residual is proved to be standardized normal when the IRT model fits the data. We compare the performance of our suggested residual to the standardized residual of Hambleton et al. (Fundamentals of item response theory, Sage, Newbury Park, 1991) in a detailed simulation study. We then calculate our suggested residuals using data from an operational test. The residuals appear to be useful in assessing the item fit for unidimensional IRT models.     

Signal Detection Models with Random Participant and Item Effects

The theory of signal detection is convenient for measuring mnemonic ability in recognition memory paradigms. In these paradigms, randomly selected participants are asked to study randomly selected items. In practice, researchers aggregate data across items or participants or both. The signal detection model is nonlinear; consequently, analysis with aggregated data is not consistent. In fact, mnemonic ability is underestimated, even in the large-sample limit. We present two hierarchical Bayesian models that simultaneously account for participant and item variability. We show how these models provide for accurate estimation of participants’ mnemonic ability as well as the memorability of items. The model is benchmarked with a simulation study and applied to a novel data set. This research is supported by NSF grants SES-0095919 and SES-0351523, NIH grant R01-MH071418, a University of Missouri Research Leave grant and fellowships from the Spanish Ministry of Education and the University of Leuven, Belgium.     

Incorporating Mobility in Growth Modeling for Multilevel and Longitudinal Item Response Data

Multilevel data often cannot be represented by the strict form of hierarchy typically assumed in multilevel modeling. A common example is the case in which subjects change their group membership in longitudinal studies (e.g., students transfer schools; employees transition between different departments). In this study, cross-classified and multiple membership models for multilevel and longitudinal item response data (CCMM-MLIRD) are developed to incorporate such mobility, focusing on students' school change in large-scale longitudinal studies. Furthermore, we investigate the effect of incorrectly modeling school membership in the analysis of multilevel and longitudinal item response data. Two types of school mobility are described, and corresponding models are specified. Results of the simulation studies suggested that appropriate modeling of the two types of school mobility using the CCMM-MLIRD yielded good recovery of the parameters and improvement over models that did not incorporate mobility properly. In addition, the consequences of incorrectly modeling the school effects on the variance estimates of the random effects and the standard errors of the fixed effects depended upon mobility patterns and model specifications. Two sets of large-scale longitudinal data are analyzed to illustrate applications of the CCMM-MLIRD for each type of school mobility.     

A Comparison of Multilevel Imputation Schemes for Random Coefficient Models: Fully Conditional Specification and Joint Model Imputation with Random Covariance Matrices

Abstract

Literature addressing missing data handling for random coefficient models is particularly scant, and the few studies to date have focused on the fully conditional specification framework and “reverse random coefficient” imputation. Although it has not received much attention in the literature, a joint modeling strategy that uses random within-cluster covariance matrices to preserve cluster-specific associations is a promising alternative for random coefficient analyses. This study is apparently the first to directly compare these procedures. Analytic results suggest that both imputation procedures can introduce bias-inducing incompatibilities with a random coefficient analysis model. Problems with fully conditional specification result from an incorrect distributional assumption, whereas joint imputation uses an underparameterized model that assumes uncorrelated intercepts and slopes. Monte Carlo simulations suggest that biases from these issues are tolerable if the missing data rate is 10% or lower and the sample is composed of at least 30 clusters with 15 observations per group. Furthermore, fully conditional specification tends to be superior with intraclass correlations that are typical of crosssectional data (e.g., ICC?=?.10), whereas the joint model is preferable with high values typical of longitudinal designs (e.g., ICC?=?.50).     

基于多层结构方程模型的情境效应分析—— 兼与多层线性模型比较

多层(嵌套)数据的变量关系研究, 必须借助多层模型来实现。两层模型中, 层一自变量Xij按组均值中心化, 并将组均值 置于层2截距方程式中, 可将Xij对因变量Yij的效应分解为组间和组内部分, 二者之差被称为情境效应, 称为情境变量。多层结构方程模型(MSEM)将多层线性模型(MLM)和结构方程模型(SEM)相结合, 通过设置潜变量和多指标的方法校正了MLM在情境效应分析中出现的抽样误差和测量误差, 同时解决了数据的多层(嵌套)结构和潜变量的估计问题。除了分析原理的说明, 还以班级平均竞争氛围对学生竞争表现的情境效应为例进行分析方法的示范, 并比较MSEM和MLM的异同, 随后展望了MSEM情境效应模型、情境效应无偏估计方法和情境变量研究的拓展方向。     

Modeling and Testing Differential Item Functioning in Unidimensional Binary Item Response Models with a Single Continuous Covariate: A Functional Data Analysis Approach

Differential item functioning (DIF), referring to between-group variation in item characteristics above and beyond the group-level disparity in the latent variable of interest, has long been regarded as an important item-level diagnostic. The presence of DIF impairs the fit of the single-group item response model being used, and calls for either model modification or item deletion in practice, depending on the mode of analysis. Methods for testing DIF with continuous covariates, rather than categorical grouping variables, have been developed; however, they are restrictive in parametric forms, and thus are not sufficiently flexible to describe complex interaction among latent variables and covariates. In the current study, we formulate the probability of endorsing each test item as a general bivariate function of a unidimensional latent trait and a single covariate, which is then approximated by a two-dimensional smoothing spline. The accuracy and precision of the proposed procedure is evaluated via Monte Carlo simulations. If anchor items are available, we proposed an extended model that simultaneously estimates item characteristic functions (ICFs) for anchor items, ICFs conditional on the covariate for non-anchor items, and the latent variable density conditional on the covariate—all using regression splines. A permutation DIF test is developed, and its performance is compared to the conventional parametric approach in a simulation study. We also illustrate the proposed semiparametric DIF testing procedure with an empirical example.     

A New Explanation and Proof of the Paradoxical Scoring Results in Multidimensional Item Response Models

In multidimensional item response models, paradoxical scoring effects can arise, wherein correct answers are penalized and incorrect answers are rewarded. For the most prominent class of IRT models, the class of linearly compensatory models, a general derivation of paradoxical scoring effects based on the geometry of item discrimination vectors is given, which furthermore corrects an error in an established theorem on paradoxical results. This approach highlights the very counterintuitive way in which item discrimination parameters (and also factor loadings) have to be interpreted in terms of their influence on the latent ability estimate. It is proven that, despite the error in the original proof, the key result concerning the existence of paradoxical effects remains true—although the actual relation to the item parameters is shown to be a more complicated function than previous results suggested. The new proof enables further insights into the actual mathematical causation of the paradox and generalizes the findings within the class of linearly compensatory models.     

代数应用题项目生成的结构分析方法

自动化项目生成是近年来兴起的测量领域, 是一种以项目认知加工理论为基础的原则性项目设计(principled item design)模式。其中, 如何在项目认知模型基础上, 通过任务结构分析的方式系统全面的鉴别和提取任务特征是一个关键环节。基于已有文献中代数应用题的命题分析法、网络语言分析法、关系-函数分析法、任务分析地图等四种结构分析方法, 研究探索了能够服务于自动化项目生成的代数应用题任务结构分析方法。该分析表明, 前三种方法分别对应于个体解题过程需要形成的三种中介表征, 即问题陈述背后的命题表征、事件时空关系的情境模型、以及变量间数量关系的问题模型, 第四种方法从过程角度分析了问题解决的认知需求。然而, 要实现项目生成的特征提取需求, 尚需对现有四种方法所揭示问题特征的心理现实性、特征提取的系统性和完备性、任务领域的适用范围、以及不同方法的整合等问题开展进一步研究。     

Explanatory Multidimensional Multilevel Random Item Response Model: An Application to Simultaneous Investigation of Word and Person Contributions to Multidimensional Lexical Representations

This paper presents an explanatory multidimensional multilevel random item response model and its application to reading data with multilevel item structure. The model includes multilevel random item parameters that allow consideration of variability in item parameters at both item and item group levels. Item-level random item parameters were included to model unexplained variance remaining when item related covariates were used to explain variation in item difficulties. Item group-level random item parameters were included to model dependency in item responses among items having the same item stem. Using the model, this study examined the dimensionality of a person’s word knowledge, termed lexical representation, and how aspects of morphological knowledge contributed to lexical representations for different persons, items, and item groups.     

Four Covariance Structure Models for Canonical Correlation Analysis: A COSAN Modeling Approach

The mathematical connection between canonical correlation analysis (CCA) and covariance structure analysis was first discussed through the Multiple Indicators and Multiple Causes (MIMIC) approach. However, the MIMIC approach has several technical and practical challenges. To address these challenges, a comprehensive COSAN modeling approach is proposed. Specifically, we define four COSAN-CCA models to correspond with four possible combinations of the data to be analyzed and the unique parameters to be estimated. In terms of the data, one can analyze either the unstandardized or standardized variables. In terms of the unique parameters, one can estimate either the weights or loadings. Besides the unique parameters of each COSAN-CCA model, all four COSAN-CCA models also estimate the canonical correlations as their common parameters. Taken together, the four COSAN-CCA models provide the correct point estimates and standard error estimates for all commonly used CCA parameters. Two numeric examples are used to compare the standard error estimates obtained from the MIMIC approach and the COSAN modeling approach. Moreover, the standard error estimates from the COSAN modeling approach are validated by a simulation study and the asymptotic theory. Finally, software implementation and future extensions are discussed.     

Bayesian Estimation for Item Factor Analysis Models with Sparse Categorical Indicators

Psychometric models for item-level data are broadly useful in psychology. A recurring issue for estimating item factor analysis (IFA) models is low-item endorsement (item sparseness), due to limited sample sizes or extreme items such as rare symptoms or behaviors. In this paper, I demonstrate that under conditions characterized by sparseness, currently available estimation methods, including maximum likelihood (ML), are likely to fail to converge or lead to extreme estimates and low empirical power. Bayesian estimation incorporating prior information is a promising alternative to ML estimation for IFA models with item sparseness. In this article, I use a simulation study to demonstrate that Bayesian estimation incorporating general prior information improves parameter estimate stability, overall variability in estimates, and power for IFA models with sparse, categorical indicators. Importantly, the priors proposed here can be generally applied to many research contexts in psychology, and they do not impact results compared to ML when indicators are not sparse. I then apply this method to examine the relationship between suicide ideation and insomnia in a sample of first-year college students. This provides an important alternative for researchers who may need to model items with sparse endorsement.     

Dimensionality of the Latent Structure and Item Selection Via Latent Class Multidimensional IRT Models

With reference to a questionnaire aimed at assessing the performance of Italian nursing homes on the basis of the health conditions of their patients, we investigate two relevant issues: dimensionality of the latent structure and discriminating power of the items composing the questionnaire. The approach is based on a multidimensional item response theory model, which assumes a two-parameter logistic parameterization for the response probabilities. This model represents the health status of a patient by latent variables having a discrete distribution and, therefore, it may be seen as a constrained version of the latent class model. On the basis of the adopted model, we implement a hierarchical clustering algorithm aimed at assessing the actual number of dimensions measured by the questionnaire. These dimensions correspond to disjoint groups of items. Once the number of dimensions is selected, we also study the discriminating power of every item, so that it is possible to select the subset of these items which is able to provide an amount of information close to that of the full set. We illustrate the proposed approach on the basis of the data collected on 1,051 elderly people hosted in a sample of Italian nursing homes.     

Item Response Modeling With BILOG-MG and MULTILOG for Windows

Item response theory (IRT) has become one of the most popular scoring frameworks for measurement data. IRT models are used frequently in computerized adaptive testing, cognitively diagnostic assessment, and test equating. This article reviews two of the most popular software packages for IRT model estimation, BILOG-MG (Zimowski, Muraki, Mislevy, & Bock, 1996) and MULTILOG (Thissen, 1991), which are for the first time available on a single CD-ROM with new features. Most prominently, the number of items to be calibrated and examinees to be scored is now limited only by memory capacities of the hardware, MULTILOG has an interactive Windows-oriented process for creating basic command file syntax, and both BILOG-MG and MULTILOG come with a new graphics interface that displays numerous curves relevant to IRT analyses in a professional format. This article reviews the models that are and are not estimable with these programs and describes the fundamental ideas of the underlying estimation algorithms without providing detailed derivations. Moreover, the user-friendliness of both programs is assessed with a user in mind who is interested in easy-to-use IRT estimation programs within a Windows point-and-click environment. Both programs fulfill such an expectation to a large degree; yet, this review also points out some obstacles that someone relatively unfamiliar to IRT or syntax programming might have to overcome to obtain meaningful results.     

Moving Beyond Causes: Optimality Models and Scientific Explanation

A prominent approach to scientific explanation and modeling claims that for a model to provide an explanation it must accurately represent at least some of the actual causes in the event's causal history. In this paper, I argue that many optimality explanations present a serious challenge to this causal approach. I contend that many optimality models provide highly idealized equilibrium explanations that do not accurately represent the causes of their target system(s). Furthermore, in many contexts, it is in virtue of their independence of causes that optimality models are able to provide a better explanation than competing causal models. Consequently, our account of explanation and modeling must expand beyond the causal approach.     

Abstract: Adequacy of Semiparametric Approximations for Growth Models with Nonnormal Random Effects

Abstract

Conventional growth models assume that the random effects describing individual trajectories are conditionally normal. In practice, this assumption may often be unrealistic. As an alternative, Nagin (2005) Nagin, D. 2005. Group-based modeling of development, Cambridge: Harvard University Press. [Crossref] , [Google Scholar] suggested a semiparametric group-based approach (SPGA) which approximates an unknown, continuous distribution of individual trajectories with a mixture of group trajectories.

Prior simulations (Brame, Nagin, &; Wasserman, 2006 Brame, R., Nagin, D. and Wasserman, L. 2006. Exploring some analytical characteristics of finite mixture models.. Journal of Quantitative Criminology, 22: 31–59. [Crossref], [Web of Science ®] , [Google Scholar]; Nagin, 2005 Nagin, D. 2005. Group-based modeling of development, Cambridge: Harvard University Press. [Crossref] , [Google Scholar]) indicated that SPGA could generate nearly-unbiased estimates of means and variances of a nonnormal distribution of individual trajectories, as functions of group-trajectory estimates. However, these studies used few random effects—usually only a random intercept. Based on the analytical relationship between SPGA and adaptive quadrature, we hypothesized that SPGA's ability to approximate (a) random effect variances/covariances and (b) effects of time-invariant predictors of growth should deteriorate as the dimensionality of the random effects distribution increases. We expected this problem to be mitigated by correlations among the random effects (highly correlated random effects functioning as fewer dimensions) and sample size (larger N supporting more groups).

We tested these hypotheses via simulation, varying the number of random effects (1, 2, or 3), correlation among the random effects (0 or .6), and N (250, 500). Results indicated that, as the number of random effects increased, SPGA approximations remained acceptable for fixed effects, but became increasingly negatively biased for random effect variances. Whereas correlated random effects and larger N reduced this underestimation, correlated random effects sometimes distorted recovery of predictor effects. To illustrate this underestimation, Figure 1 depicts SPGA's approximation of the intercept variance from a three correlated random effect generating model (N < eqid1 > 500). These results suggest SPGA approximations are inadequate for the nonnormal, high-dimensional distributions of individual trajectories often seen in practice.

Abstract: Adequacy of Semiparametric Approximations for Growth Models with Nonnormal Random Effects

All authors

Sonya K. Sterba, Ruth E. Mathiowetz &; Daniel J. Bauer

https://doi.org/10.1080/00273170802640509

Published online:

18 December 2008

FIGURE 1 SPGA-approximated intercept variance from a three correlated random effect generating model. Notes. The dashed horizontal lines denote + 10% bias. The solid horizontal line denotes the population-generating parameter value; * denotes the best-BIC selected number of groups. The vertical bars denote 90% confidence intervals.     

20.

Hidden Markov Item Response Theory Models for Responses and Response Times     

Dylan Molenaar  Daniel Oberski  Jeroen Vermunt  Paul De Boeck 《Multivariate behavioral research》2016,51(5):606-626

Current approaches to model responses and response times to psychometric tests solely focus on between-subject differences in speed and ability. Within subjects, speed and ability are assumed to be constants. Violations of this assumption are generally absorbed in the residual of the model. As a result, within-subject departures from the between-subject speed and ability level remain undetected. These departures may be of interest to the researcher as they reflect differences in the response processes adopted on the items of a test. In this article, we propose a dynamic approach for responses and response times based on hidden Markov modeling to account for within-subject differences in responses and response times. A simulation study is conducted to demonstrate acceptable parameter recovery and acceptable performance of various fit indices in distinguishing between different models. In addition, both a confirmatory and an exploratory application are presented to demonstrate the practical value of the modeling approach.     

Hidden Markov Item Response Theory Models for Responses and Response Times

Current approaches to model responses and response times to psychometric tests solely focus on between-subject differences in speed and ability. Within subjects, speed and ability are assumed to be constants. Violations of this assumption are generally absorbed in the residual of the model. As a result, within-subject departures from the between-subject speed and ability level remain undetected. These departures may be of interest to the researcher as they reflect differences in the response processes adopted on the items of a test. In this article, we propose a dynamic approach for responses and response times based on hidden Markov modeling to account for within-subject differences in responses and response times. A simulation study is conducted to demonstrate acceptable parameter recovery and acceptable performance of various fit indices in distinguishing between different models. In addition, both a confirmatory and an exploratory application are presented to demonstrate the practical value of the modeling approach.