Optimal Online Calibration Designs for Item Replenishment in Adaptive Testing

The maintenance of item bank is essential for continuously implementing adaptive tests. Calibration of new items online provides an opportunity to efficiently replenish items for the operational item bank. In this study, a new optimal design for online calibration (referred to as D-c) is proposed by incorporating the idea of original D-optimal design into the reformed D-optimal design proposed by van der Linden and Ren (Psychometrika 80:263–288, 2015) (denoted as D-VR design). To deal with the dependence of design criteria on the unknown item parameters of new items, Bayesian versions of the locally optimal designs (e.g., D-c and D-VR) are put forward by adding prior information to the new items. In the simulation implementation of the locally optimal designs, five calibration sample sizes were used to obtain different levels of estimation precision for the initial item parameters, and two approaches were used to obtain the prior distributions in Bayesian optimal designs. Results showed that the D-c design performed well and retired smaller number of new items than the D-VR design at almost all levels of examinee sample size; the Bayesian version of D-c using the prior obtained from the operational items worked better than that using the default priors in BILOG-MG and PARSCALE; and Bayesian optimal designs generally outperformed locally optimal designs when the initial item parameters of the new items were poorly estimated.

     

题目位置效应的概念及检测

题目位置效应(Item Position Effect, IPE)是指在剔除随机误差的影响之后, 同一道题目在不同测验间因题目位置的变化而导致题目参数的变化。IPE的存在会严重威胁依赖于项目反应理论参数不变性特征的相关应用, 比如测验等值和计算机化自适应测验。目前关于这一领域的研究主要集中于对IPE的检测, 而对所检测到的效应进行进一步的解释, 则是今后的研究重点。另外, 在不同的研究情境下深入探讨IPE, 对于基础研究领域和实践领域都具有重要意义。     

Maximum information per time unit designs for continuous online item calibration

Previous designs for online calibration have only considered examinees’ responses to items. However, the use of response time, a useful metric that can easily be collected by a computer, has not yet been embedded in calibration designs. In this article we utilize response time to optimize the assignment of new items online, and accordingly propose two new adaptive designs. These are the D-optimal per expectation time unit design (D-ET) and the D-optimal per time unit design (D-T). The former method uses the conditional maximum likelihood estimation (CMLE) method to estimate the expected response times, while the latter employs the nonparametric k-nearest-neighbour method to predict the response times. Simulations were conducted to compare the two new designs with the D-optimal online calibration design (D design) in the context of continuous online calibration. In addition, a preliminary study was carried out to evaluate the performance of CMLE prior to its application in D-ET. The results showed that, compared to the D design, the D-ET and D-T designs saved response time and accrued larger calibration information per time unit, without sacrificing item calibration precision.