使用验证性补偿多维IRT模型进行认知诊断评估

随着人们对测验反馈结果精细化的需求逐渐提高, 具有认知诊断功能的测量方法逐渐受到人们的关注。在认知诊断模型(CDMs)闪耀着光芒的同时, 另一类能够在连续量尺上提供精细反馈的多维IRT模型(MIRTMs)似乎受到些许冷落。为探究MIRTMs潜在的认知诊断功能, 本文以补偿模型为视角, 聚焦于分别属于MIRTMs的多维两参数logistic模型(M2PLM)和属于CDMs的线性logistic模型(LLM); 之后为使两者具有可比性, 可对补偿M2PLM引入验证性矩阵(Q矩阵)来界定题目与维度之间的关系, 进而得到验证性的补偿M2PLM (CC-M2PLM), 并通过把潜在特质按切点划分为跨界属性, 以期使CC-M2PLM展现出其本应具有的认知诊断功能; 预研究表明logistic量尺上的0点可作为相对合理的切点; 然后, 通过模拟研究对比探究CC-M2PLM和LLM的认知诊断功能, 结果表明CC-M2PLM可用于分析诊断测验数据, 且认知诊断功能与直接使用LLM的效果相当; 最后, 以两则实证数据为例来说明CC-M2PLM在实际诊断测验分析中的可行性。     

A diagnostic tree model for polytomous responses with multiple strategies

Constructed-response items have been shown to be appropriate for cognitively diagnostic assessments because students’ problem-solving procedures can be observed, providing direct evidence for making inferences about their proficiency. However, multiple strategies used by students make item scoring and psychometric analyses challenging. This study introduces the so-called two-digit scoring scheme into diagnostic assessments to record both students’ partial credits and their strategies. This study also proposes a diagnostic tree model (DTM) by integrating the cognitive diagnosis models with the tree model to analyse the items scored using the two-digit rubrics. Both convergent and divergent tree structures are considered to accommodate various scoring rules. The MMLE/EM algorithm is used for item parameter estimation of the DTM, and has been shown to provide good parameter recovery under varied conditions in a simulation study. A set of data from TIMSS 2007 mathematics assessment is analysed to illustrate the use of the two-digit scoring scheme and the DTM.     

Modelling multiple problem-solving strategies and strategy shift in cognitive diagnosis for growth

Problem-solving strategies, defined as actions people select intentionally to achieve desired objectives, are distinguished from skills that are implemented unintentionally. In education, strategy-oriented instructions that guide students to form problem-solving strategies are found to be more effective for low-achieving students than the skill-oriented instructions designed for enhancing their skill implementation ability. Although the existing longitudinal cognitive diagnosis models (CDMs) can model the change in students' dynamic skill mastery status over time, they are not designed to model the shift in students' problem-solving strategies. This study proposes a longitudinal CDM that considers both between-person multiple strategies and within-person strategy shift. The model, separating the strategy choice process from the skill implementation process, is intended to provide diagnostic information on strategy choice as well as skill mastery status. A simulation study is conducted to evaluate the parameter recovery of the proposed model and investigate the consequences of ignoring the presence of multiple strategies or strategy shift. Further, an empirical data analysis is conducted to illustrate the use of the proposed model to measure strategy shift, growth in skill implementation ability and skill mastery status.     

Higher-order latent trait models for cognitive diagnosis

Higher-order latent traits are proposed for specifying the joint distribution of binary attributes in models for cognitive diagnosis. This approach results in a parsimonious model for the joint distribution of a high-dimensional attribute vector that is natural in many situations when specific cognitive information is sought but a less informative item response model would be a reasonable alternative. This approach stems from viewing the attributes as the specific knowledge required for examination performance, and modeling these attributes as arising from a broadly-defined latent trait resembling theϑ of item response models. In this way a relatively simple model for the joint distribution of the attributes results, which is based on a plausible model for the relationship between general aptitude and specific knowledge. Markov chain Monte Carlo algorithms for parameter estimation are given for selected response distributions, and simulation results are presented to examine the performance of the algorithm as well as the sensitivity of classification to model misspecification. An analysis of fraction subtraction data is provided as an example. This research was funded by National Institute of Health grant R01 CA81068. We would like to thank William Stout and Sarah Hartz for many useful discussions, three anonymous reviewers for helpful comments and suggestions, and Kikumi Tatsuoka and Curtis Tatsuoka for generously sharing data.     

A general proof of consistency of heuristic classification for cognitive diagnosis models

The Asymptotic Classification Theory of Cognitive Diagnosis (Chiu et al., 2009, Psychometrika, 74, 633–665) determined the conditions that cognitive diagnosis models must satisfy so that the correct assignment of examinees to proficiency classes is guaranteed when non‐parametric classification methods are used. These conditions have only been proven for the Deterministic Input Noisy Output AND gate model. For other cognitive diagnosis models, no theoretical legitimization exists for using non‐parametric classification techniques for assigning examinees to proficiency classes. The specific statistical properties of different cognitive diagnosis models require tailored proofs of the conditions of the Asymptotic Classification Theory of Cognitive Diagnosis for each individual model – a tedious undertaking in light of the numerous models presented in the literature. In this paper a different way is presented to address this task. The unified mathematical framework of general cognitive diagnosis models is used as a theoretical basis for a general proof that under mild regularity conditions any cognitive diagnosis model is covered by the Asymptotic Classification Theory of Cognitive Diagnosis.     

A Polytomous Model of Cognitive Diagnostic Assessment for Graded Data

Pursuing the line of the difference models in IRT (Thissen &; Steinberg, 1986 Thissen, D., &; Steinberg, L. (1986). A taxonomy of item response models. Psychometrika, 51:567–577. doi:10.1007/BF02295596.[Crossref], [Web of Science ®] , [Google Scholar]), this article proposed a new cognitive diagnostic model for graded/polytomous data based on the deterministic input, noisy, and gate (Haertel, 1989 Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352. doi:10.1111/j.1745-3984.1989.tb00336.x.[Crossref], [Web of Science ®] , [Google Scholar]; Junker &; Sijtsma, 2001 Junker, B. W., &; Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. doi:10.1177/01466210122032064.[Crossref], [Web of Science ®] , [Google Scholar]), which is named the DINA model for graded data (DINA-GD). We investigated the performance of a full Bayesian estimation of the proposed model. In the simulation, the classification accuracy and item recovery for the DINA-GD model were investigated. The results indicated that the proposed model had acceptable examinees' correct attribute classification rate and item parameter recovery. In addition, a real-data example was used to illustrate the application of this new model with the graded data or polytomously scored items.     

教育认知诊断测验与认知模型一致性的评估

构建正确的认知模型是成功进行认知诊断的关键之一,如果认知诊断测验不能完整准确地代表这个认知模型,这个测验的效度就存在问题.属性及其层级可以表示一个认知模型.在认知模型正确基础上,给出了一个计量公式以衡量认知诊断测验能够多大程度上代表认知模型;对于不止包含一个知识状态的等价类及其形成原因进行了分析,对Cui等人的属性层级相合性指标(HCI)提出修改建议,以更好地探查数据与专家给出的认知模型的一致性.     

Reliability coefficients for multiple group item response theory models

Reliability of scores from psychological or educational assessments provides important information regarding the precision of measurement. The reliability of scores is however population dependent and may vary across groups. In item response theory, this population dependence can be attributed to differential item functioning or to differences in the latent distributions between groups and needs to be accounted for when estimating the reliability of scores for different groups. Here, we introduce group-specific and overall reliability coefficients for sum scores and maximum likelihood ability estimates defined by a multiple group item response theory model. We derive confidence intervals using asymptotic theory and evaluate the empirical properties of estimators and the confidence intervals in a simulation study. The results show that the estimators are largely unbiased and that the confidence intervals are accurate with moderately large sample sizes. We exemplify the approach with the Montreal Cognitive Assessment (MoCA) in two groups defined by education level and give recommendations for applied work.     

多项选择题认知诊断潜能的最大化

多项选择题是认知诊断中常用的测验项目形式, 其正确答案选项和经过特殊编制的干扰项都能提供诊断信息。为了提取干扰项的信息, 需要采用不同于传统认知诊断模型的心理计量模型。分类介绍能使干扰项提供诊断信息的多项选择题编制方法, 分析这类多项选择题的认知诊断模型, 认为未来应加强干扰项编制方法和能提取干扰项信息的认知诊断模型的选择和开发等方面的研究。     

认知诊断计算机自适应测验中平衡属性收敛的新方法

提出两种认知诊断计算机自适应测验下平衡属性收敛的新方法（MABI、RTA）,模拟研究系统探讨和比较了此二者与已有方法（ABI、IABI和RABI）的表现。结果发现：（1）新方法较不考虑属性收敛的方法有更高的准确率以及更均衡的题目使用率;（2）新方法较ABI和RABI有稍低的准确性,但有更平衡的题目使用率;（3）新方法与IABI的准确性和题目使用率在不同选题策略下各有合优势。总之,两种新方法较好地兼顾测量准确性、题目使用率以及题库曝光情况。     

认知诊断计算机自适应测验中平衡属性收敛的新方法

提出两种认知诊断计算机自适应测验下平衡属性收敛的新方法（MABI、RTA）,模拟研究系统探讨和比较了此二者与已有方法（ABI、IABI和RABI）的表现。结果发现：（1）新方法较不考虑属性收敛的方法有更高的准确率以及更均衡的题目使用率;（2）新方法较ABI和RABI有稍低的准确性,但有更平衡的题目使用率;（3）新方法与IABI的准确性和题目使用率在不同选题策略下各有合优势。总之,两种新方法较好地兼顾测量准确性、题目使用率以及题库曝光情况。     

测验项目反应机制与心理测量模型假设的对应性分析

从测验项目解决的认知过程的视角分析了在不同测验理论框架下的测量模型中的基本假设, 指出测量模型是测验开发者有关测验项目反应机制的理论假设的具体表征, 是系统检验测量假设和过程的统计框架。然而, 不管是经典测验理论、概化理论, 还是早期的项目反应理论模型, 相关假设都过于简化, 缺少相应实质理论的支持。与之相比, 认知测量模型强调与个体在测验项目反应过程中的认知过程、认知策略和知识结构的对应性, 提供了在实质理论基础上界定测量建构、设计测验项目、进行建模分析和解释的可能性, 为日益边缘化的心理测量学和主流心理学研究的融合奠定了基础。     

Nested diagnostic classification models for multiple-choice items

This study proposes and evaluates a diagnostic classification model framework for multiple-choice items. Models in the proposed framework have a two-level nested structure which allows for binary scoring (for correctness) and polytomous scoring (for distractors) at the same time. One advantage of these models is that they can provide distractor information while maintaining the statistical properties of the correct response option. We evaluated parameter recovery through a simulation study using Hamiltonian Monte Carlo algorithms in Stan. We also discussed three approaches to implementing the proposed modelling framework for different purposes and testing scenarios. We illustrated those approaches and compared them with a binary model and a traditional nominal model through an operational study.     

认知诊断中基于条件期望的距离判别方法

分类是认知诊断评估的一个核心问题。基于观察反应模式与理想反应模式之间的距离的判别方法, 以确定性的理想反应模式为类中心, 而这没有考虑误差, 故未充分利用总体分布信息。为了更充分地利用总体分布信息、提高诊断分类效果和拓展诊断评估的适用性, 本研究提出给定知识状态条件下项目反应模式的条件期望向量为类中心的欧氏距离判别方法, 同时提出认知诊断模型下项目反应函数估计方法以获得这个条件期望向量。模拟研究表明：认知诊断模型下的项目反应函数估计方法得到的条件期望向量返真性较高, 获得的分布信息较准确; 在观察反应模式与理想反应模式差异大的情形下, 基于条件期望向量为类中心的欧氏距离判别方法优于基于理想反应模式为类中心的分类方法(广义距离方法和非参数方法）。研究可为认知诊断分类和等值方法提供一个参考。     

定长CD-CAT中两种新的题目曝光控制方法

提出了两种适用于定长CD-CAT的题目曝光控制方法（HIRP、HIRT）,这些方法在保证较高分类准确率的同时还有较合理的题目曝光率,新方法由二分化方法和RP及RT方法进行结合并适当调整而得到。模拟研究比较了其与RP、RT、SM、SMIE、RHA和SDBS的表现,结果表明: （1）HIRP的分类准确率和题目曝光率均好于SM、SMIE和SDBS;（2）HIRT的题目曝光率较RP、SM、SMIE、RHA和SDBS稍差,但分类准确率更高;（3）HIRP的分类准确率低于RT和RP,但题目曝光控制要更好。     

On Interim Cognitive Diagnostic Computerized Adaptive Testing in Learning Context

Interim assessment occurs throughout instruction to provide feedback about what students know and have achieved. Different from the current available cognitive diagnostic computerized adaptive testing (CD-CAT) design that focuses on assessment at a single time point, the authors discuss several designs of interim CD-CAT that are suitable in the learning context. The interim CD-CAT differs from the current available CD-CAT designs primarily because students’ mastery profile (i.e., skills mastery) changes due to learning, and new attributes are added periodically. Moreover, hierarchies exist among attributes taught sequentially and such information could be used during item selection. Two specific designs are considered: The first one is when new attributes are taught in Stage II, but the student mastery status of the previously taught attributes stays the same. The second design is when both new attributes are taught, and previously taught attributes can be further learned or forgotten in Stage II. For both designs, the authors propose an individual prior, which considers a person’s learning history and population learning model, to start an interim CD-CAT. Simulation results show that the Stage II CD-CAT using individual prior outperforms the methods using population priors. The GDINA (generalized deterministic inputs, noisy, “and” gate) diagnostic index (GDI) is extended to accommodate item hierarchies, and analytic results are provided to further illustrate the types of items that are most popular during item selection. As the first study that focuses on the application of CD-CAT in a learning context, the methods and results present herein showed the great promise of using CD-CAT to monitor learning.     

认知诊断CAT中项目曝光控制方法的比较

项目曝光率关系到题库建设和测验安全,是计算机化自适应测验(Computerized Adaptive Testing, CAT)需要考虑的重要问题。在认知诊断 CAT 情形下,首先基于传统 CAT 中 a-分层方法的思想提出按项目信息量对题库分层的分层多阶段(Stratified Multistage, SM)选题方法;然后将 SM 方法与项目合格(Item Eligibility, IE)方法相结合得到SMIE方法。在此基础上,开展模拟研究比较SM、IE、SMIE、最大修正优先指标(Maximum Modified Priority Index, MMPI)方法、限制阈值(Restrictive Threshold, RT)方法和限制进度(Restrictive Progressive, RPG)方法的选题表现。总体上,它们的测量精度从高到低依次为IE、SM、SMIE、RT、RPG和MMPI方法;项目曝光分布均匀性的优劣次序为MMPI、RPG、SMIE、RT、SM和IE方法;SMIE和RT方法能较好地平衡测量精度和项目曝光均匀性要求。     

选项具有诊断信息的多选题认知诊断测验编制

编制选项具有诊断信息的多选题是提高多选题认知诊断测验诊断效果的有效方法.研究从认知诊断的目标出发,根据认知诊断测验质量的评价标准,结合多选题的特点,探讨选项具有诊断信息的多选题认知诊断测验编制的原则.同时,结合多选题的特点和多选题采用称名计分方式的需要,对编制选项具有诊断信息的多选题提出两点要求.根据多选题认知诊断测验编制的原则和要求,给出具有可操作性的多选题认知诊断测验编制的步骤.模拟研究结果表明：根据所提出的原则和要求编制的多选题认知诊断测验具有较好的诊断效果,说明这些原则和要求合理、可行.由于这些原则、要求和步骤具有较强的可操作性,因此它对于编制多选题认知诊断测验具有一定的指导意义.     

认知诊断计算机化自适应测验中在线标定方法的开发

项目增补对认知诊断计算机化自适应测验(CD-CAT)中的题库维护至关重要。在传统CAT中, 在线标定方法经常用于估计新题的项目参数。然而直到现在, 在CD-CAT领域还没有任何关于在线标定的论文公开发表。为将传统CAT中3种有代表性的在线标定方法(Method A、OEM和 MEM)推广至CD-CAT (CD-Method A、CD-OEM和CD-MEM)建立分析基础, 并采用模拟方法对这3种方法进行比较。研究表明：CD-Method A方法在项目参数的返真性方面优于其它两种方法; 自适应标定设计较随机标定设计可以提高项目参数的返真质量。     

基于项目自动生成的认知诊断测验开发

认知诊断、项目自动生成是现代心理测量领域的重要发展领域,二者的结合更是心理测量领域亟待开展的重要课题。本研究以小学数学问题解决认知诊断项目自动生成为例,探讨认知诊断领域的项目生成技术及算法。研究发现：(1)计算机自生成的项目参数与原模板参数具有较高的一致性。(2)同一项目模板下生成的不同试题的测量学特征基本不变。(3)同一批被试在自动生成的两份试卷的前、后测的能力( )值高度相关(r=0.811),前、后两次对被试诊断结果的一致性高达86.5%。这表明本文所设计的认知诊断测验项目的自动生成技术及其算法基本可行,小学数学问题解决认知诊断项目的自动生成效果较好。这也为其它认知诊断领域的项目自动生成提供了技术借鉴和支持。