多级评分聚类诊断法的影响因素

从测验和被试两个层面探讨了属性数目、属性层级关系、被试知识状态分布、属性层级误设和Q矩阵误设等因素对GRCDM的影响, 以进一步考察GRCDM的特性。研究发现：(1)GRCDM对属性数目无依赖, 随属性数目的增多判准率反而增高; (2)被试知识状态分布对GRCDM判准率高低无影响; (3)属性层级误设对GRCDM的影响与属性层级类型有关, 当属性层级为无结构型和发散型时, “属性层级关系错乱”的判准率降幅最大; (4)Q矩阵误设对GRCDM的影响因层级关系而异, 收敛型和发散型受影响较小, 无结构型和线型的判准率在属性既冗余又缺失时降幅最大。     

基于作答数据的模型参数和Q矩阵联合估计

Q矩阵在认知诊断的模型参数估计和诊断分类中起着重要作用。本文通过研究Liu等人的方法, 设计了同时估计项目参数和Q矩阵的联合估计算法。在DINA模型下, 对项目参数未知时开展模拟研究。研究假设项目为20个, 考察的属性个数分别是3、4和5, 初始Q矩阵中分别存在3、4和5个属性界定错误的项目。结果表明, 联合估计算法能在错误的初始Q矩阵基础上以很高的概率得到正确的Q矩阵。另外, 当专家认定测验的属性个数存在错误时, 该方法推导的Q矩阵和模型参数能提供很好的鉴别Q矩阵错误的信息。     

基于可达阵的一种Q矩阵标定方法

Q矩阵标定是实施认知诊断评估的前提,已有Q矩阵修正方法并不太适合测验中已知属性向量的题目数较少的情形。根据拓展Q矩阵理论中可达阵R列与简化Q阵列存在布尔“或”关系,在一定认知假设下,率先提出可达阵R与简化Q阵的潜在反应列存在布尔“与”关系,并由此提出基于可达阵的Q矩阵标定方法。研究显示：在已知一个可达阵下,当可达阵项目的猜测或失误参数在.20以下且待标定项目的项目参数约在.30以下时,新方法所得Q矩阵元素返真率基本在.90以上,并且真实Q矩阵与估计Q矩阵下被试分类准确率差异很小;对于含5个属性的独立结构,新方法要求的随机样本的样本量较小;实证研究也印证了模拟研究的结论。新方法只需专家标定少量题目的Q矩阵,即已经标定的Q矩阵对应属性层级结构的可达阵。     

一种简单有效的Q矩阵估计方法开发：基于非参数化方法视角

摘要：Q矩阵是认知诊断的基础,错误的Q矩阵会影响参数估计和被试诊断正确率,开发一种简单而有效的Q矩阵估计方法有助于Q矩阵的正确界定。相对于参数化的Q矩阵估计方法,本研究将海明距离（Hamming Distance,HD）用于Q矩阵估计,开发出一种简单有效的非参数化的Q矩阵估计方法。采用Monte Carlo模拟方法与实证研究相结合的研究范式,对该方法的科学性与合理性及其效果进行研究,研究结果发现（1）基于海明距离的Q矩阵估计法具有较高的估计正确率,并且该方法不受被试样本容量影响。（2）该方法简单易懂,运算时间短,是一种简单而有效的Q矩阵估计方法。（3）新方法对于Tatsuka（1990）分数减法测验的Q矩阵的估计准确率尚可,说明新方法在实践中具有较好的潜在应用前景与应用价值。     

一种简单有效的Q矩阵修正新方法

Q矩阵的正确性是影响题目参数估计和被试分类准确性的重要因素。针对Q矩阵修正问题, 首先提出了一种简单有效的新方法(ORDP)。然后, 模拟研究通过改变被试知识状态的分布、样本容量(N)、测验长度(L)、Q矩阵错误率(M)、项目质量(Iq)和属性层级结构, 比较了ORDP与已有方法(R、RMSEA和HD)的表现。研究表明：(1) 当知识状态服从均匀分布时, ORDP方法在所有层级结构下最优; 当知识状态服从多元正态分布时, RMSEA和ORDP表现没有明显差异, 除独立结构外, RMSEA方法均稍优于ORDP方法; (2) 各方法在多元正态分布下的修正效果不及均匀分布时的修正结果; (3) N、L、M、Iq和属性层级结构对4种方法的表现均有明显影响; (4) 基于Tatsuoka (1984)分数减法数据的修正结果表明, 采用ORDP方法修正的Q矩阵与数据拟合最优。     

认知诊断Q矩阵估计(修正)方法

Q矩阵代表着项目考察的属性, 反映了项目的重要特征, 其正确性是影响认知诊断分类准确性的关键因素。研究Q矩阵估计(修正)方法具有重要价值。首先, 研究从是否采用认知诊断模型将Q矩阵估计(修正)分为基于认知诊断模型视角下的参数化方法和基于统计视角下的非参数方法。然后, 分别从最优项目质量、最优模型数据拟合和参数估计视角对它们进行分类介绍, 评析不同方法的特征和表现、区别与联系、优势与不足。最后, 提出几个未来研究问题：在复杂测验条件下系统比较各种方法; 校准知识状态和参数估计误差、结合多种思路和方法等多角度提出Q矩阵估计(修正)方法; 研究多级评分项目、混合测验模型、属性多级、属性个数未知甚至Q矩阵元素为连续变量等条件下的Q矩阵估计(修正)方法。     

Evaluation of Model Fit in Cognitive Diagnosis Models

Cognitive diagnosis models (CDMs) estimate student ability profiles using latent attributes. Model fit to the data needs to be ascertained in order to determine whether inferences from CDMs are valid. This study investigated the usefulness of some popular model fit statistics to detect CDM fit including relative fit indices (AIC, BIC, and CAIC), and absolute fit indices (RMSEA2, ABS(fcor) and MAX(χ²_jj′)). These fit indices were assessed under different CDM settings with respect to Q-matrix misspecification and CDM misspecification. Results showed that relative fit indices selected the correct DINA model most of the times and selected the correct G-DINA model well across most conditions. Absolute fit indices rejected the true DINA model if the Q-matrix was misspecified in any way. Absolute fit indices rejected the true G-DINA model whenever the Q-matrix was under-specified. RMSEA2 could be artificially low when the Q-matrix was over-specified.     

Data-driven Q-matrix validation using a residual-based statistic in cognitive diagnostic assessment

In a cognitive diagnostic assessment (CDA), attributes refer to fine-grained knowledge points or skills. The Q -matrix is a central component of CDA, which specifies the relationship between items and attributes. Oftentimes, attributes and Q -matrix are defined by subject-matter experts, and assumed to be appropriate without any misspecifications. However, this assumption does not always hold in real applications. To address this concern, this paper proposes a residual-based statistic for validating the Q -matrix. Its performance is evaluated in a simulation study and compared against that of an existing method proposed in Liu, Xu and Ying (2012, Applied Psychological Measurement, 36, 548). Simulation results indicate that the proposed method leads to a higher recovery rate of the Q -matrix and is computationally more efficient. The advantage in computational efficiency is particularly pronounced when the number of attributes measured by the test reaches five or more. Results also suggest that the two methods have different tendencies in estimating the attribute vector for each item. In cases where the methods fail to recover the correct Q -matrix, the method in Liu et al. (2012, Applied Psychological Measurement, 36, 548) tends to overestimate the number of attributes measured by the items, whereas our method does not show that bias.     

基于类别水平的多级计分认知诊断Q矩阵修正：相对拟合统计量视角

多级计分认知诊断模型的开发对认知诊断的发展具有重要作用,但对于多级计分模型下的Q矩阵修正还有待研究。本研究尝试对多级计分认知诊断Q矩阵修正进行研究,并聚焦更具诊断价值的基于项目类别水平的Q矩阵修正。将相对拟合统计量应用于多级计分认知诊断Q矩阵修正,并与已有方法 Stepwise方法(Ma&de la Torre,2019)进行比较。研究表明:BIC方法对多级计分认知诊断模型的Q矩阵修正具有较高的模式判准率和属性判准率,其对Q矩阵的恢复率也高于Stepwise方法, BIC方法修正后的Q矩阵与数据更加拟合;在复杂模型中,相对拟合指标BIC比AIC和-2LL表现更好,在实践中,使用者可以选择BIC法进行测验Q矩阵修正; Q矩阵修正效果受到被试人数的影响,增加被试人数可以提高Q矩阵修正的正确率。总之,本研究为多级计分认知诊断Q矩阵修正提供了重要的方法支持。     

Balancing fit and parsimony to improve Q-matrix validation

The Q-matrix identifies the subset of attributes measured by each item in the cognitive diagnosis modelling framework. Usually constructed by domain experts, the Q-matrix might contain some misspecifications, disrupting classification accuracy. Empirical Q-matrix validation methods such as the general discrimination index (GDI) and Wald have shown promising results in addressing this problem. However, a cut-off point is used in both methods, which might be suboptimal. To address this limitation, the Hull method is proposed and evaluated in the present study. This method aims to find the optimal balance between fit and parsimony, and it is flexible enough to be used either with a measure of item discrimination (the proportion of variance accounted for, PVAF) or a coefficient of determination (pseudo-R²). Results from a simulation study showed that the Hull method consistently showed the best performance and shortest computation time, especially when used with the PVAF. The Wald method also performed very well overall, while the GDI method obtained poor results when the number of attributes was high. The absence of a cut-off point provides greater flexibility to the Hull method, and it places it as a comprehensive solution to the Q-matrix specification problem in applied settings. This proposal is illustrated using real data.