Effects of Differential Item Functioning on Examinees' Test Performance and Reliability of Test

Simulations were conducted to examine the effect of differential item functioning (DIF) on measurement consequences such as total scores, item response theory (IRT) ability estimates, and test reliability in terms of the ratio of true-score variance to observed-score variance and the standard error of estimation for the IRT ability parameter. The objective was to provide bounds of the likely DIF effects on these measurement consequences. Five factors were manipulated: test length, percentage of DIF items per form, item type, sample size, and level of group ability difference. Results indicate that the greatest DIF effect was less than 2 points on the 0 to 60 total score scale and about 0.15 on the IRT ability scale. DIF had a limited effect on the ratio of true-score variance to observed-score variance, but its influence on the standard error of estimation for the IRT ability parameter was evident for certain ability values.     

高中英语阅读测验中题组模型的选择与应用

本研究通过高中英语阅读测验实测数据,对比分析双参数逻辑斯蒂克模型 (2PL-IRT)和加入不同数量题组的双参数逻辑斯蒂克模型 (2PL-TRT), 探究题组数量对参数估计及模型拟合的影响。结果表明：(1) 2PL-IRT模型对能力介于-1.50到0.50的被试,能力参数估计偏差较大;(2)将题组效应大于0.50的题组作为局部独立题目纳入模型,会导致部分题目区分度参数的低估和大部分题目难度参数的高估;(3)题组效应越大,将其当作局部独立题目纳入模型估计项目参数的偏差越大。     

Invariance Properties for General Diagnostic Classification Models

In item response theory (IRT), the invariance property states that item parameter estimates are independent of the examinee sample, and examinee ability estimates are independent of the test items. While this property has long been established and understood by the measurement community for IRT models, the same cannot be said for diagnostic classification models (DCMs). DCMs are a newer class of psychometric models that are designed to classify examinees according to levels of categorical latent traits. We examined the invariance property for general DCMs using the log-linear cognitive diagnosis model (LCDM) framework. We conducted a simulation study to examine the degree to which theoretical invariance of LCDM classifications and item parameter estimates can be observed under various sample and test characteristics. Results illustrated that LCDM classifications and item parameter estimates show clear invariance when adequate model data fit is present. To demonstrate the implications of this important property, we conducted additional analyses to show that using pre-calibrated tests to classify examinees provided consistent classifications across calibration samples with varying mastery profile distributions and across tests with varying difficulties.     

Latent variables should remain as such: Evidence from a Monte Carlo study

Use of subject scores as manifest variables to assess the relationship between latent variables produces attenuated estimates. This has been demonstrated for raw scores from classical test theory (CTT) and factor scores derived from factor analysis. Conclusions on scores have not been sufficiently extended to item response theory (IRT) theta estimates, which are still recommended for estimation of relationships between latent variables. This is because IRT estimates appear to have preferable properties compared to CTT, while structural equation modeling (SEM) is often advised as an alternative to scores for estimation of the relationship between latent variables. The present research evaluates the consequences of using subject scores as manifest variables in regression models to test the relationship between latent variables. Raw scores and three methods for obtaining theta estimates were used and compared to latent variable SEM modeling. A Monte Carlo study was designed by manipulating sample size, number of items, type of test, and magnitude of the correlation between latent variables. Results show that, despite the advantage of IRT models in other areas, estimates of the relationship between latent variables are always more accurate when SEM models are used. Recommendations are offered for applied researchers.     

多水平IRT的发展与应用述评

阶层线性模型是处理阶层结构数据的高级统计方法, 项目反应理论是精确测量被试能力的现代测量理论。多水平项目反应理论将阶层线性模型和项目反应理论相结合, 将项目反应模型嵌套在阶层线性模型内, 实现了项目参数和不同水平能力参数的估计, 对回归系数和误差项变异的估计也更加精确。作者概述了多水平项目反应理论的发展历程, 并从项目功能差异、测验等值、学校效能研究等方面评述了多水平项目反应理论在心理与教育测量中的应用, 总结了多水平项目反应理论的价值, 同时展望了今后的研究趋势。     

IRT与MIRT在测验垂直等值中的应用

测验垂直等值是指将测试同一心理特质的不同水平的测验转换到同一个分数量尺上的过程。IRT与MIRT是实现垂直等值的主要方法。IRT无需假设被试的能力分布, 参数估计不依赖于样本, 是构建垂直量表的有效方法, 但测验不满足单维假设时其应用受到限制。MIRT结合IRT和因素分析的特点对IRT进行了拓展, 可更有效估计多维测验的项目参数和被试能力参数, 在垂直等值中有重要应用。已有研究主要探讨IRT和MIRT在垂直等值应用中的适用性、标定方法和参数估计方法, 比较研究两种方法的特性。未来研究应纳入更多变量条件进行比较研究, 拓展方法的应用。     

项目特征曲线等值的抽样误差

现在,等值越来越受到各考试测验机构及测量学研究人员的重视,特别是项目反应理论等值的优越性更使他们有了信心。然而,很多人却没有注意到被试能力分布形态可能给等值结果带来的影响效果及程度。本研究以项目反应理论两级记分模型的项目参数等值在不同被试能力分布形态下的结果差异作为重点,探讨被试抽样偏差可能给项目特征曲线等值带来的误差问题。研究结果表明,被试能力分布形态会显著地影响项目参数等值的系数,特别地,能力分布的偏态系数与等值方程的截距存在显著的线性相关关系,但能力分布形态的变化对等值方程中斜率的影响并不明显     

自陈量表式测验应用IRT的可行性

对采用5级评分Likert式测题的情感能力量表的分析表明,各分量表项目都有较好的模型-数据拟合性,而且显示了参数估计的不变性,及与CTT参数的关联性。这些都表明Likert量表应用IRT模型的假设条件得到了满足,即IRT应用是可行的。研究还表明IRT能对测量精度进行更精确的估计。     

单维项目因素分析：CCFA与IRT估计方法的比较

当观测指标变量为二分分类数据时,传统的因素分析方法不再适用。作者简要回顾了SEM框架下的分类数据因素分析模型和IRT框架下的测验题目和潜在能力的关系模型,并对两种框架下主要采用的参数估计方法进行了总结。通过两个模拟研究,比较了SEM框架下GLSc和MGLSc估计方法与IRT框架下MML/EM估计方法的差异。研究结果表明：（1）三种方法中,GLSc得到参数估计的偏差最大,MGLSc和MML/EM估计方法相差不大;（2）随着样本量增大,各种项目参数估计的精度均提高;（3）项目因素载荷和难度估计的精度受测验长度的影响;（4）项目因素载荷和区分度估计的精度受总体因素载荷（区分度）高低的影响;（5）测验项目中阈值的分布会影响参数估计的精度,其中受影响最大的是项目区分度。（6）总体来看,SEM框架下的项目参数估计精度较IRT框架下项目参数估计的精度高。此外,文章还将两种方法在实际应用中应该注意的问题提供了一些建议。     

EDITOR'S NOTE

In the framework of a linear logistic testing model, Mislevy, Sheehan, and Wingersky (1993) showed how to incorporate collateral information in estimating item parameters required for test equating. The purpose of the study was to explore the feasibility of applying this method to equate tests constructed for college entrance examination by comparing its results with those of the item response theory (IRT) true-score equating. Overall, the equating results based on collateral information are relatively comparable with those of IRT equating. In terms of R2's, the prediction equations for item characteristics are good to excellent. The significant levels of correlation coefficients between IRT calibrated b (difficulty level) and predicted b parameters range from around .01 to .05. The goodness of fit of true-score test characteristic curves (TCCs) based on collateral information to IRT true-score TCCs are excellent. Results of the study are discussed in light of factors that may affect the validity of using collateral information in test equating.     

Reliability measures in item response theory: Manifest versus latent correlation functions

For item response theory (IRT) models, which belong to the class of generalized linear or non‐linear mixed models, reliability at the scale of observed scores (i.e., manifest correlation) is more difficult to calculate than latent correlation based reliability, but usually of greater scientific interest. This is not least because it cannot be calculated explicitly when the logit link is used in conjunction with normal random effects. As such, approximations such as Fisher's information coefficient, Cronbach's α, or the latent correlation are calculated, allegedly because it is easy to do so. Cronbach's α has well‐known and serious drawbacks, Fisher's information is not meaningful under certain circumstances, and there is an important but often overlooked difference between latent and manifest correlations. Here, manifest correlation refers to correlation between observed scores, while latent correlation refers to correlation between scores at the latent (e.g., logit or probit) scale. Thus, using one in place of the other can lead to erroneous conclusions. Taylor series based reliability measures, which are based on manifest correlation functions, are derived and a careful comparison of reliability measures based on latent correlations, Fisher's information, and exact reliability is carried out. The latent correlations are virtually always considerably higher than their manifest counterparts, Fisher's information measure shows no coherent behaviour (it is even negative in some cases), while the newly introduced Taylor series based approximations reflect the exact reliability very closely. Comparisons among the various types of correlations, for various IRT models, are made using algebraic expressions, Monte Carlo simulations, and data analysis. Given the light computational burden and the performance of Taylor series based reliability measures, their use is recommended.     

The Assessment of Dimensionality for Use in Item Response Theory

The application of item response theory (IRT) models requires the identification of the data's dimensionality. A popular method for determining the number of latent dimensions is the factor analysis of a correlation matrix. Unlike factor analysis, which is based on a linear model, IRT assumes a nonlinear relationship between item performance and ability. Because multidimensional scaling (MDS) assumes a monotonic relationship this method may be useful for the assessment of a data set's dimensionality for use with IRT models. This study compared MDS, exploratory and confirmatory factor analysis (EFA and CFA, respectively) in the assessment of the dimensionality of data sets which had been generated to be either one- or two-dimensional. In addition, the data sets differed in the degree of interdimensional correlation and in the number of items defining a dimension. Results showed that MDS and CFA were able to correctly identify the number of latent dimensions for all data sets. In general, EFA was able to correctly identify the data's dimensionality, except for data whose interdimensional correlation was high.     

Simple imputation methods versus direct likelihood analysis for missing item scores in multilevel educational data

Missing data, such as item responses in multilevel data, are ubiquitous in educational research settings. Researchers in the item response theory (IRT) context have shown that ignoring such missing data can create problems in the estimation of the IRT model parameters. Consequently, several imputation methods for dealing with missing item data have been proposed and shown to be effective when applied with traditional IRT models. Additionally, a nonimputation direct likelihood analysis has been shown to be an effective tool for handling missing observations in clustered data settings. This study investigates the performance of six simple imputation methods, which have been found to be useful in other IRT contexts, versus a direct likelihood analysis, in multilevel data from educational settings. Multilevel item response data were simulated on the basis of two empirical data sets, and some of the item scores were deleted, such that they were missing either completely at random or simply at random. An explanatory IRT model was used for modeling the complete, incomplete, and imputed data sets. We showed that direct likelihood analysis of the incomplete data sets produced unbiased parameter estimates that were comparable to those from a complete data analysis. Multiple-imputation approaches of the two-way mean and corrected item mean substitution methods displayed varying degrees of effectiveness in imputing data that in turn could produce unbiased parameter estimates. The simple random imputation, adjusted random imputation, item means substitution, and regression imputation methods seemed to be less effective in imputing missing item scores in multilevel data settings.     

在GLMM框架下统一GT与IRT

本文首次提出使用广义线性混合模型(Generalized Linear Mixed Model, GLMM)对概化理论(GT)和项目反应理论(IRT)进行统合,即在一次统计中就能同时获得GT和IRT所需要的估计结果。模拟研究结果显示：相比于传统的GT方差分量估计方法——期望均值平方(Expected Mean Squares, EMS),GLMM可以获得更准确的方差分量、G系数和Φ系数,而且GLMM获得的题目难度参数估计精度优于传统Rasch模型。实证研究展示GLMM在实际心理测量数据分析中的应用。     

二级项目题组混合IRT模型及其稳健权重估计

在心理测量和教育测量中,二级项目和题组项目是两类常见的项目类型。由这两种项目混合构成的测试在实践中有着重要的应用。被试在答题时,由于个人的潜在能力和项目难度不匹配,常常会产生异常反应,这些异常反应会影响IRT中潜在特质估计的准确性。仿真实验证明,二级项目题组混合IRT模型的稳健估计方法在出现异常值的情况下,能够比极大似然估计对被试的潜在特质做出更加准确的估计,能够满足实际测试的需求。     

混合IRT潜在模型及其应用轨迹

项目反应理论是测量被试潜在特质的现代测量理论, 潜在类别分析是基于模型的潜在特质分类技术。混合项目反应理论将项目反应理论与潜在类别分析相结合, 能够同时对被试分类并量化其潜在特质。在阐述混合项目反应理论概念、原理的基础上, 介绍了MRM、mNRM和mPCM等几种常见混合模型及其参数估计方法, 并从心理与行为特征分类、项目功能差异检测、测验效度评价等方面评述了其在心理测验中的应用发展轨迹。     

基于联结主义的连续记分IRT模型的项目参数和被试能力估计

运用联结主义中的级连相关模型对于小样本条件下的连续记分项目反应理论 (IRT)模型的项目参数和被试能力进行了估计。一组被试对于一组项目的反应矩阵作为级连相关模型的输入 ,这组被试的能力θ或该组项目的参数a、b和c作为该模型的输出 ,对神经网络进行训练使之具备了估计θ,a ,b或c的能力。计算机模拟的实验表明 ,如果测验中有少量项目取自于题库 ,就可以运用联结主义方法对IRT参数和被试能力进行较好的估计     

Minimax d-optimal designs for item response theory models

Various different item response theory (IRT) models can be used in educational and psychological measurement to analyze test data. One of the major drawbacks of these models is that efficient parameter estimation can only be achieved with very large data sets. Therefore, it is often worthwhile to search for designs of the test data that in some way will optimize the parameter estimates. The results from the statistical theory on optimal design can be applied for efficient estimation of the parameters.A major problem in finding an optimal design for IRT models is that the designs are only optimal for a given set of parameters, that is, they are locally optimal. Locally optimal designs can be constructed with a sequential design procedure. In this paper minimax designs are proposed for IRT models to overcome the problem of local optimality. Minimax designs are compared to sequentially constructed designs for the two parameter logistic model and the results show that minimax design can be nearly as efficient as sequentially constructed designs.     

计算机适应性测验条件下认知设计项目预测参数的影响

自动化项目生成(Automatic Item Generation)中的项目参数是基于认知项目设计的刺激特征集预测的, 在不确定性来源上较之用经验数据标定的参数更为复杂。文章通过实证研究分析了在计算机适应性测验条件下基于认知设计系统法生成的抽象推理测验(ART)项目预测参数对能力参数估计的精确性。研究表明, 项目预测参数比相应标定参数分布更为趋中。这种回归效应既影响到能力参数估计误差大小, 也导致适应性测验过程中项目选择的差异。在控制了项目选择差异之后, 能力参数估计误差较之基于项目标定参数的能力估计误差大, 但差别并不明显。两者相应的能力估计值相关很高, 对应能力值之间的差异很小, 且几乎贯彻整个能力分布区间。