Item difficulty parameter estimation using the idea of the graded response model and computerized adaptive testing

Abstract:  In test operations using IRT (item response theory), items are included in a test before being used to rate subjects and the response data is used to estimate their item parameters. However, this method of test operation may lead to item content leakage and an adequate test operation can become difficult. To address this problem, Ozaki and Toyoda (2005, 2006 ) developed item difficulty parameter estimation methods that use paired comparison data from the perspective of the difficulty of items as judged by raters familiar with the field. In the present paper, an improved method of item difficulty parameter estimation is developed. In this new method, an item for which the difficulty parameter is to be estimated is compared with multiple items simultaneously, from the perspective of their difficulty. This is not a one-to-one comparison but a one-to-many comparison. In the comparisons, raters are informed that items selected from an item pool are ordered according to difficulty. The order will provide insight to improve the accuracy of judgment.     

Identifying Variables Responsible for Data not Missing at Random

When data are not missing at random (NMAR), maximum likelihood (ML) procedure will not generate consistent parameter estimates unless the missing data mechanism is correctly modeled. Understanding NMAR mechanism in a data set would allow one to better use the ML methodology. A survey or questionnaire may contain many items; certain items may be responsible for NMAR values in other items. The paper develops statistical procedures to identify the responsible items. By comparing ML estimates (MLE), statistics are developed to test whether the MLEs are changed when excluding items. The items that cause a significant change of the MLEs are responsible for the NMAR mechanism. Normal distribution is used for obtaining the MLEs; a sandwich-type covariance matrix is used to account for distribution violations. The class of nonnormal distributions within which the procedure is valid is provided. Both saturated and structural models are considered. Effect sizes are also defined and studied. The results indicate that more missing data in a sample does not necessarily imply more significant test statistics due to smaller effect sizes. Knowing the true population means and covariances or the parameter values in structural equation models may not make things easier either. The research was supported by NSF grant DMS04-37167, the James McKeen Cattell Fund.     

Power of the likelihood ratio test in covariance structure analysis

A procedure for computing the power of the likelihood ratio test used in the context of covariance structure analysis is derived. The procedure uses statistics associated with the standard output of the computer programs commonly used and assumes that a specific alternative value of the parameter vector is specified. Using the noncentral Chi-square distribution, the power of the test is approximated by the asymptotic one for a sequence of local alternatives. The procedure is illustrated by an example. A Monte Carlo experiment also shows how good the approximation is for a specific case.This research was made possible by a grant from the Dutch Organization for Advancement of Pure Research (ZWO). The authors also like to acknowledge the helpful comments and suggestions from the editor and anonymous reviewers.     

Data, model, conclusion, doing it again

This paper explores the robustness of conclusions from a statistical model against variations in model choice (rather than variations in random sampling and random assignment to treatments, which are the usual variations covered by inferential statistics). After the problem formulation in section 1, section 2 presents an example from Box and Tiao in which variation in parent distribution is modeled for a one sample location problem. An adaptive Bayesian procedure permits to use a weighted mixture of parent distributions rather than choosing just one, such as a normal or a uniform distribution.In section 3 similar considerations are applied to an event history model for the influence of education and gender on age at first marriage, but here the conclusions are hardly influenced by the choice of the duration distribution. In section 4 a brief discussion of model choice in factor analysis and structural equation models is followed by a more elaborate treatment of the choice of integer valued slopes for item response functions in the OPLM model which is an extension of the Rasch model. A modest simulation study suggests that Adaptive Bayesian Modeling with a mixture of sets of slopes works better than fixing one set of postulated slopes.The paper concludes with some remarks on the roles and sources of prior distributions followed by a short epilogue which argues that simultaneous consideration of a class of models for the same data is sometimes superior to exclusively analyzing the data under one specific model chosen from such a class.This article is based on the Presidential Address Ivo W. Molenaar gave on June 20, 1998 at the 1998 Annual Meeting of the Psychometric Society held at the University of Illinois in Champaign, Illinois. Thanks is given to John Wiley & Sons and George C. Tiao for granting permission to reprint three figures from the book George C. Tiao wrote with George E. P. Box titled Bayesian Inference in Statistical Analysis.—EditorThanks are due to Anne Boomsma, Jeffrey Hoogland, Mark Huisman, Tom Snijders, Marijtje Van Duijn, and Norman Verhelst for suggesting improvements and/or assisting with data analyses.     

On the loss of information in conditional maximum likelihood estimation of item parameters

In item response models of the Rasch type (Fischer & Molenaar, 1995), item parameters are often estimated by the conditional maximum likelihood (CML) method. This paper addresses the loss of information in CML estimation by using the information concept of F-information (Liang, 1983). This concept makes it possible to specify the conditions for no loss of information and to define a quantification of information loss. For the dichotomous Rasch model, the derivations will be given in detail to show the use of the F-information concept for making comparisons for different estimation methods. It is shown that by using CML for item parameter estimation, some information is almost always lost. But compared to JML (joint maximum likelihood) as well as to MML (marginal maximum likelihood) the loss is very small. The reported efficiency in the use of information of CML to JML and to MML in several comparisons is always larger than 93%, and in tests with a length of 20 items or more, larger than 99%.     

题组反应理论及其在中学英语考试中的应用研究

题组作为众多测验中的一种常见题型,由于项目间存在一定程度的依赖性而违背了局部独立性假设,若用项目反应模型进行参数估计将会出现较大的偏差.题组反应理论将被试与题组的交互作用纳入到模型中,解决了项目间相依性的问题.笔者对题组反应理论的发展、基本原理及其相关研究进行了综述,并将其应用在中学英语考试中.与项目反应理论相对比,结果发现：（1）题组反应模型与项目反应模型在各参数估计值的相关系数较强,尤其是能力参数和难度参数;（2）在置信区间宽度的比较上,题组反应模型在各个参数上均窄于项目反应模型,即题组反应模型的估计精度优于项目反应模型.     

Item Calibration Samples and the Stability of Achievement Estimates and System Rankings: Another Look at the PISA Model

Using an empirically-based simulation study, we show that typically used methods of choosing an item calibration sample have significant impacts on achievement bias and system rankings. We examine whether recent PISA accommodations, especially for lower performing participants, can mitigate some of this bias. Our findings indicate that standard operational methods, while not ideal, recover underlying proficiency reasonably well and generally outperform methods that more completely include all participants. Translating results onto the PISA scale, the calibration sample can induce bias of up to 12.49 points, which is important given that standard errors are around three points. Although ranking correlations are at least.95, we note the policy implications of slight ranking changes. Our findings indicate that limited accommodations targeted at low achieving educational systems do not outperform either of the other methods considered. Research that further explores accommodations for heterogeneous populations is recommended.     

An experimental analysis of psychometric functions in a threshold discrimination task with four response categories1

An experiment on a threshold discrimination task with four response categories was conducted to check the implicit assumptions of a threshold discrimination task with three response categories. Consideration of a task with three response categories, for example, “stronger,”“don't know,” and “weaker,” leads us to anticipate a relation between position parameters and that between the sizes of scale parameters of a model for a task with four response categories, such as, “stronger,”“probably stronger,”“probably weaker,” and “weaker.” That is, position parameters will be equidistant between adjacent categories and the middle scale parameter will be the largest. The results of the experiment on a four‐category task did not support these anticipated relation, and a possibility of biases in the estimation of point of subjective equality (PSE), using a three‐category task is pointed out. As an alternative to a three‐category task, a four‐category task is suggested when more than two categories are required.     

4PLRT模型与3PLRT模型在瑞文标准推理测验中的比较研究

研究目的：考察反应时与智力的关系以及通过模型对比考察4PLRT模型特性。方法：以瑞文标准推理测验为例,对4PLRT模型与3PLRT模型作比较研究。结论：（1）项目时间参数与难度参数正相关,可以依据时间参数将瑞文标准测验划分成三大部分。（2）能力估计结果比较：由于项目参数和被试速度参数的作用,CTT条件下能力相等的不同被试在计时与非计时条件下存在差异。（3）关于被试的答题策略：被试会基于自身能力而充分权衡时间与准确性的关系。这两种不同模型下被试的能力值之间的比较也说明了时间是一个重要的智力评估因子。     

Dielectric and thermodynamic behaviour of binary mixtures of anisole with o-chlorophenol and with o-cresol

The molecular interactions in binary mixtures of anisole with o-chlorophenol and o-cresol have been studied at three different temperatures using dielectric measurements. The static permittivity and the permittivity at optical frequency have also been determined and used to obtain the Kirkwood correlation factors, the Bruggeman parameter, the excess permittivity and the excess free energy for different concentrations of two binary mixtures, namely (1) anisole + o-chlorophenol and (2) anisole + o-cresol. Using the effective Kirkwood correlation factor, parallel and anti-parallel alignments among the dipoles are identified. The corrective Kirkwood correlation factor, the excess permittivity and the Bruggeman parameter are used to explain the interaction between unlike molecules. The temperature dependences of the above parameters are also reported. The investigation shows systematic changes in the dielectric behaviour with variations in concentration and temperature.