测验相对拟合检验方法CVLL法在认知诊断中的拓展及应用

本文将IRT中表现较好的CVLL法引入到认知诊断领域,同时比较并分析CVLL及认知诊断领域已有的测验相对拟合检验统计量的表现,为实际工作者在认知诊断模型选用上提供方法学支持和借鉴。结果表明:CVLL的表现比其它传统测验相对拟合统计量要好;且当对Q矩阵进行误设时,该统计量也能选择较优的Q矩阵,说明CVLL在Q矩阵侦查上有较好的应用前景。

Expansion and application of CVLL in Cognitive Diagnosis

The validity in all applications of CDT (cognitive diagnosis theory) depends on the extent to which the selected model accurately reflects the real data. Only when the CDMs (cognitive diagnosis models) fit the data well, can the advantages and functions of CDT emerge. Selection of the wrong model would lead to relatively large error in parameter estimation, test equating, the analysis of differential item functioning and so on, which would result in adverse effect. Therefore, it is required to evaluate model-data fit before applying CDT. Common fit statistics in the field of CDT consist of test fit statistics and item fit statistics; the former evaluate fit from test angle, the latter evaluate fit from item angle, which can guide the selection of items. Test fit statistics usually are relative fit statistics and item fit statistics usually are absolute fit statistics. In fact, it is very tedious and blind to use item fit statistic to assess model-data fit for all items, and absolute fit assessment is quite difficult. Thus, many researchers only consider relative fit assessment, which can give a direction to model selection. Relative fit statistics can be expounded and compared from model misspecification and Q-matrix misspecification. Traditional relative fit statistics such as, -2LL, AIC, AICc, BIC and DIC4 have been used to test the validity of CD models. However, CVLL has never been applied in the real application. Given that CVLL is a reliable index in item response theory, it is worthy to investigate how well it performed in detecting the validity in CD models. The aim of this study is to evaluate these different statistics in terms of the proportion of times each fitted model was selected out of the 30 iterations. Four conditions were considered in the simulation study, including sample sizes, test lengths, numbers of attributes, and Q-matrix misspecifications. Three models were used to generate the data: the DINA model, the NIDA model, and R-RUM. An empirical example involving real data was used to illustrate how the different fit statistics can be employed to identify misspecifications. This research discovered that : a) CVLL performed best, b) AIC and BIC performed better than AICc, c) AICc and BIC tended to select the reduced model than the saturated model in small sample size and long test length, d) When the number of attributes became 9, the correct rates of these statistics decreased, meanwhile, CVLL performed best, e) If the true model was R-RUM, the selection rates of correct Q-matrix of these statistics decreased except CVLL, f) The empirical example proved that DINA, NIDA and R-RUM neither fit the fraction data adequately, but R-RUM is the best-fitting model among a set of competing models. Although the results of this work are encouraging, additional work is needed to further understand model-data fit evaluation under the CDM context and to broaden the generalizability of the current findings. First, this study only covered three models. Other types of CDMs – such as multiple-strategies models or high-order models – need to be investigated. Second, due to the important role the Q-matrix played in cognitive diagnosis modeling, future research should systematically examine the impact of not only the degree but also the type of Q-matrix misspecifications on the different fit statistics.