Consistency of Cluster Analysis for Cognitive Diagnosis: The Reduced Reparameterized Unified Model and the General Diagnostic Model

The asymptotic classification theory of cognitive diagnosis (ACTCD) provided the theoretical foundation for using clustering methods that do not rely on a parametric statistical model for assigning examinees to proficiency classes. Like general diagnostic classification models, clustering methods can be useful in situations where the true diagnostic classification model (DCM) underlying the data is unknown and possibly misspecified, or the items of a test conform to a mix of multiple DCMs. Clustering methods can also be an option when fitting advanced and complex DCMs encounters computational difficulties. These can range from the use of excessive CPU times to plain computational infeasibility. However, the propositions of the ACTCD have only been proven for the Deterministic Input Noisy Output “AND” gate (DINA) model and the Deterministic Input Noisy Output “OR” gate (DINO) model. For other DCMs, there does not exist a theoretical justification to use clustering for assigning examinees to proficiency classes. But if clustering is to be used legitimately, then the ACTCD must cover a larger number of DCMs than just the DINA model and the DINO model. Thus, the purpose of this article is to prove the theoretical propositions of the ACTCD for two other important DCMs, the Reduced Reparameterized Unified Model and the General Diagnostic Model.     

重参数化的多分属性诊断分类模型及其判准率影响因素

多分属性比传统的二分属性提供更多更详细的诊断反馈信息, 符合对知识技能的多水平要求, 具有较好的应用前景。本文首先介绍了多分属性和多分Q矩阵的概念; 之后重参数化了3个分别满足连接、分离和补偿缩合规则的多分属性诊断分类模型并研究了其判准率影响因素, 结果发现它们的判准率(1)均随多分属性数量的增加而降低, 建议实际使用中不宜高于5个; (2)均随多分属性的最高水平数增加而降低, 建议实际使用中不宜高于4水平; (3)均随多分属性间统计相关性增加而增加, 但影响不大; (4)受多分属性层级结构的影响较大; (4)受被试量影响不大; (5)均随题目数量增加而增加且影响较大。最后, 针对“多分属性与多级评分的关系”和“多分属性与二分属性之间的关系”这两个问题进行了讨论。以期为实证研究者提供相关的理论支持和使用建议。     

非参数认知诊断方法下诊断结果的概率化表征

非参数认知诊断分类方法非常适合课堂评估,其诊断结果采用0-1形式而缺乏概率化表征,不能精细地区分被试属性掌握程度的差异或变化,还缺乏可用于评价真实测验分类结果的信度和效度指标。要刻画被试属性掌握程度的差异,首要的问题是要为非参数认知诊断方法提供一种可以量化属性掌握概率的方法。针对此问题,基于二项分布和玻尔兹曼分布提出非参数认知诊断方法下诊断结果的概率化表征方法,并用于构建分类准确性和分类一致性指标。模拟研究与实测数据分析结果显示：概率化表征方法与非参数认知诊断方法的分类结果高度一致;概率化表征方法与认知诊断模型所得的属性掌握概率十分接近;概率化表征方法所得的属性（模式）掌握概率可用于计算属性（模式）分类准确性和分类一致性指标,在实际测验情景下可作为信度和效度指标,评价诊断结果的重测一致率和判准率。     

认知诊断模型的比较及其应用研究：饱和模型、简化模型还是混合方法？

GDINA是一个饱和认知诊断模型（Cognitive Diagnosis Models, CDM）,Wald检验被用于在题目水平上检验GDINA是否可以被简化模型（如DINA, DINO, ACDM和RRUM）替代,并为测验的每一个题目选择一个最恰当的CDM（简称混合CDM）。选择合适的CDM是进行诊断评估的一个关键步骤,通过Monte Carlo 模拟实验,比较了不同的测验情境下,GDINA、简化CDM和混合CDM在测验整体拟合指标、模式判准率和项目参数估计的返真性等效果,研究发现混合模型的整体表现是最好的,其次是GDINA,最后是简化CDM。     

多维题组效应认知诊断模型

当前认知诊断领域还缺少对包含题组的测验进行诊断分析的研究, 即已开发的认知诊断模型无法合理有效地处理含有题组效应的测验数据, 且已开发的题组反应模型也不具有对被试知识结构或认知过程进行诊断的功能。针对该问题, 本文尝试性地将多维题组效应向量参数引入线性Logistic模型中, 同时开发了属性间具有补偿作用的和属性间具有非补偿作用的多维题组效应认知诊断模型。模拟研究结果显示新模型合理有效, 与线性Logistic模型和DINA模型对比研究后表明：(1)作答数据含有题组效应时, 忽略题组效应会导致项目参数的偏差估计并降低对目标属性的判准率; (2)新模型更具普适性, 即便当作答数据不存在题组效应时, 采用新模型进行测验分析亦能得到很好的项目参数估计结果且不影响对目标属性的判准率。整体来看, 新模型既具有认知诊断功能又可有效处理题组效应。     

基于混合模型（Mixed-CDMs）视角的CD-CAT及其应用研究

传统CD-CAT通常选择一个认知诊断模型（cognitive diagnosis model, CDM）标定题库参数,但在实际应用中一个CDM很难完全拟合题库中所有的题目。G-DINA模型是一般化的饱和模型,可以通过Wald统计量检验在题目水平上,比较简约模型（DINA、DINO、ACDM、LLM和RRUM）是否能够代替饱和模型（G-DINA）,并为每个题目选择一个相对最优的CDM,从而充分发挥各个CDM的优势,从而在一个题库中有的题目采用简约CDM,而有的题目采用饱和CDM,本文把这种思路称为混合模型（Mixed-CDMs）思路。基于此,本文探讨了基于混合模型的CD-CAT,并通过两个模拟研究及其应用研究验证了该方法的效果。研究结果表明基于混合模型建立的CD-CAT具有理想的效果,从而为CD-CAT在实际使用中提供了新思路和新方法。     

基于Gini指数的认知诊断计算机化自适应选题策略

Gini指数可用来描述分布的不均匀性,已广泛应用于决策树算法,本文开发了基于Gini指数的认知诊断计算机化自适应选题策略,并在饱和模型和缩减模型下与SHE, MPWKL,GDI,PWKL选题策略进行比较。模拟研究表明,基于Gini指数的选题策略与SHE,MPWKL,GDI相比,分类精度相近并提高了题库的利用率;与PWKL相比,提高了分类的精度和选题速度,综合来看,基于Gini指数的选题策略能够兼顾分类精度和题库使用均匀性。     

Invariance Properties for General Diagnostic Classification Models

In item response theory (IRT), the invariance property states that item parameter estimates are independent of the examinee sample, and examinee ability estimates are independent of the test items. While this property has long been established and understood by the measurement community for IRT models, the same cannot be said for diagnostic classification models (DCMs). DCMs are a newer class of psychometric models that are designed to classify examinees according to levels of categorical latent traits. We examined the invariance property for general DCMs using the log-linear cognitive diagnosis model (LCDM) framework. We conducted a simulation study to examine the degree to which theoretical invariance of LCDM classifications and item parameter estimates can be observed under various sample and test characteristics. Results illustrated that LCDM classifications and item parameter estimates show clear invariance when adequate model data fit is present. To demonstrate the implications of this important property, we conducted additional analyses to show that using pre-calibrated tests to classify examinees provided consistent classifications across calibration samples with varying mastery profile distributions and across tests with varying difficulties.     

Q矩阵包含错误的诊断测验分类准确性比较

Q矩阵是认知诊断测验的重要组成部分之一,围绕Q矩阵构建的诊断模型对Q矩阵中包含的错误较敏感。贝叶斯网分类模型是基于网络结点之间的关系构建的模型,将朴素贝叶斯网作为诊断模型,与DINA模型进行比较。模拟实验结果表明：Q矩阵中是否包含可达矩阵和错误界定的项目数量对DINA模型影响较大,对贝叶斯网模型影响较小;项目数量对DINA和贝叶斯网模型影响都较大;样本大小对贝叶斯网模型影响较大,对DINA模型影响较小。模拟研究结果显示,当Q矩阵中不包含可达阵、包含5个以上错误项目或样本数较大时,贝叶斯网分类模型优于DINA模型;而当Q矩阵中包含可达阵和5个(以下)错误项目时,DINA模型优于贝叶斯分类模型。     

认知诊断模型中项目水平模型比较统计量的健壮性

使用模拟研究方法比较了以往研究中提出的基于观察信息矩阵、三明治矩阵的Wald（分别表示为W_Obs、W_Sw）、似然比（Likelihood Ratio）统计量以及新提出的基于经验交叉相乘信息矩阵的Wald统计量（W_XPD）在模型——数据失拟条件下进行项目水平上模型比较时的表现。结果显示：（1）W_Sw的一类错误控制率有很强的健壮性。（2）W_XPD在Q矩阵错误设定的大多数条件下的表现优于W_Sw。结论：模型—数据拟合良好时可以使用W_Sw进行项目水平上的模型比较,当模型与数据失拟时W_XPD可能是更好的选择。     

The DINA model as a constrained general diagnostic model: Two variants of a model equivalency

The ‘deterministic‐input noisy‐AND’ (DINA) model is one of the more frequently applied diagnostic classification models for binary observed responses and binary latent variables. The purpose of this paper is to show that the model is equivalent to a special case of a more general compensatory family of diagnostic models. Two equivalencies are presented. Both project the original DINA skill space and design Q ‐matrix using mappings into a transformed skill space as well as a transformed Q ‐matrix space. Both variants of the equivalency produce a compensatory model that is mathematically equivalent to the (conjunctive) DINA model. This equivalency holds for all DINA models with any type of Q ‐matrix, not only for trivial (simple‐structure) cases. The two versions of the equivalency presented in this paper are not implied by the recently suggested log‐linear cognitive diagnosis model or the generalized DINA approach. The equivalencies presented here exist independent of these recently derived models since they solely require a linear – compensatory – general diagnostic model without any skill interaction terms. Whenever it can be shown that one model can be viewed as a special case of another more general one, conclusions derived from any particular model‐based estimates are drawn into question. It is widely known that multidimensional models can often be specified in multiple ways while the model‐based probabilities of observed variables stay the same. This paper goes beyond this type of equivalency by showing that a conjunctive diagnostic classification model can be expressed as a constrained special case of a general compensatory diagnostic modelling framework.     

Diagnostic Classification Models: Recent Developments,Practical Issues,and Prospects

More than three decades after their introduction, diagnostic classification models (DCM) do not seem to have been implemented in educational systems for the purposes they were devised. Most DCM research is either methodological for model development and refinement or retrofitting to existing nondiagnostic tests and, in the latter case, basically for model demonstration or constructs identification. DCMs have rarely been used to develop diagnostic assessment right from the start with the purpose of identifying individuals’ strengths and weaknesses (referred to as true applications in this study). In this article, we give an introduction to DCMs and their latest developments along with guidelines on how to proceed to employ DCMs to develop a diagnostic test or retrofit to a nondiagnostic assessment. Finally, we enumerate the reasons why we believe DCMs have not become fully operational in educational systems and suggest some advice to make their advent smooth and quick.     

认知诊断模型中项目水平模型比较统计量的健壮性

使用模拟研究方法比较了以往研究中提出的基于观察信息矩阵、三明治矩阵的Wald（分别表示为W_Obs、W_Sw）、似然比（Likelihood Ratio）统计量以及新提出的基于经验交叉相乘信息矩阵的Wald统计量（W_XPD）在模型——数据失拟条件下进行项目水平上模型比较时的表现。结果显示：（1）W_Sw的一类错误控制率有很强的健壮性。（2）W_XPD在Q矩阵错误设定的大多数条件下的表现优于W_Sw。结论：模型—数据拟合良好时可以使用W_Sw进行项目水平上的模型比较,当模型与数据失拟时W_XPD可能是更好的选择。     

Joint Maximum Likelihood Estimation for Diagnostic Classification Models

Joint maximum likelihood estimation (JMLE) is developed for diagnostic classification models (DCMs). JMLE has been barely used in Psychometrics because JMLE parameter estimators typically lack statistical consistency. The JMLE procedure presented here resolves the consistency issue by incorporating an external, statistically consistent estimator of examinees’ proficiency class membership into the joint likelihood function, which subsequently allows for the construction of item parameter estimators that also have the consistency property. Consistency of the JMLE parameter estimators is established within the framework of general DCMs: The JMLE parameter estimators are derived for the Loglinear Cognitive Diagnosis Model (LCDM). Two consistency theorems are proven for the LCDM. Using the framework of general DCMs makes the results and proofs also applicable to DCMs that can be expressed as submodels of the LCDM. Simulation studies are reported for evaluating the performance of JMLE when used with tests of varying length and different numbers of attributes. As a practical application, JMLE is also used with “real world” educational data collected with a language proficiency test.     

认知诊断中多分属性与二分属性的对比研究

摘 要：Karelitz（2004）和詹沛达等（2016）认为1个多分属性内部(Lk+1)个水平的关系相当于Lk个部分满足线型层级关系的二分属性。本研究的目的是通过比较多分属性模型和二分属性模型的判准率,从而验证多分属性和二分属性间是否存在以上关系。结果表明：当属性个数较少时,两个模型的模式判准率相当,随着属性个数增加,多分属性模型的模式判准率高于二分属性模型的模式判准率。结论：在一定程度上,多分属性和二分属性之间确实存在以上关系,但两者并非完全等价,二者间的差异随着属性个数增加更加明显。     

基于认知诊断测评的个性化补救教学效果分析：以“一元一次方程”为例

基于“为学习而测评”理念,以促进学生学习为目的,本研究进行了基于认知诊断测评的个性化补救教学效果分析。首先,以“一元一次方程”章节为例,编制两份平行的认知诊断测评试卷。然后,通过对不同地区（城市和农村）七年级学生的施测与分析,发现城市学生对属性的掌握情况优于农村学生对属性的掌握情况。之后,选择农村学生为补救对象,通过对比基于认知诊断测评和传统教学两种个性化补救教学的效果,发现两种补救教学方法均能提高学习成绩,但前者的补救效果显著优于后者的。总之,本研究结果表明采用基于认知诊断测评的个性化补救教学能够有效促进学生学习,为实践者应用认知诊断测评促进学生学习提供了实践依据。     

概率性输入,噪音“与”门(PINA)模型

当前认知诊断测验的主要目的是对被试进行合理分类,进而采用类别变量去描述被试对某技能或知识(即认知属性)的掌握情况,但该粗糙的分类方法不能精细地区分不同被试之间的差异。对此,采用掌握概率这一连续变量去描述被试对某认知属性的掌握情况是一种值得尝试的做法。本文首先基于高阶潜在特质(简称"潜质")模型给出了认知属性掌握概率的量化定义,之后与多成分潜质模型相结合提出了概率性输入,噪音"与"门(PINA)模型;其次,采用MCMC算法实现了对PINA的参数估计,结果表明参数估计程序对各参数的估计返真性均较好;最后,以ECPE数据为例来说明PINA在实际测验分析中具有可行性。     

几种常用非补偿型认知诊断模型的比较与选用：基于属性层级关系的考量

当前国际上开发了60多种认知诊断计量模型(Fu &; Li, 2007), 各种模型各具特点, 实际应用者应根据实际情况选用恰当的模型。本研究以属性层级关系为切入点, 采用Monte Carlo模拟的研究方法, 比较了属性层级关系正确及有误两种情况下, 当前国际上常用的五种认知诊断模型的性能, 以充分考察不同认知诊断模型对属性层级关系的依赖程度, 及属性层级关系的错误界定对各认知诊断模型诊断正确率的影响, 从而为实际运用者在认知诊断模型选用上提供借鉴和参考。     

Bayesian Estimation of the DINA Q matrix

Cognitive diagnosis models are partially ordered latent class models and are used to classify students into skill mastery profiles. The deterministic inputs, noisy “and” gate model (DINA) is a popular psychometric model for cognitive diagnosis. Application of the DINA model requires content expert knowledge of a Q matrix, which maps the attributes or skills needed to master a collection of items. Misspecification of Q has been shown to yield biased diagnostic classifications. We propose a Bayesian framework for estimating the DINA Q matrix. The developed algorithm builds upon prior research (Chen, Liu, Xu, & Ying, in J Am Stat Assoc 110(510):850–866, 2015) and ensures the estimated Q matrix is identified. Monte Carlo evidence is presented to support the accuracy of parameter recovery. The developed methodology is applied to Tatsuoka’s fraction-subtraction dataset.