Identifiability of Diagnostic Classification Models

Diagnostic classification models (DCMs) are important statistical tools in cognitive diagnosis. In this paper, we consider the issue of their identifiability. In particular, we focus on one basic and popular model, the DINA model. We propose sufficient and necessary conditions under which the model parameters are identifiable from the data. The consequences, in terms of the consistency of parameter estimates, of fulfilling or failing to fulfill these conditions are illustrated via simulation. The results can be easily extended to the DINO model through the duality of the DINA and DINO models. Moreover, the proposed theoretical framework could be applied to study the identifiability issue of other DCMs.     

Invariance Properties for General Diagnostic Classification Models

In item response theory (IRT), the invariance property states that item parameter estimates are independent of the examinee sample, and examinee ability estimates are independent of the test items. While this property has long been established and understood by the measurement community for IRT models, the same cannot be said for diagnostic classification models (DCMs). DCMs are a newer class of psychometric models that are designed to classify examinees according to levels of categorical latent traits. We examined the invariance property for general DCMs using the log-linear cognitive diagnosis model (LCDM) framework. We conducted a simulation study to examine the degree to which theoretical invariance of LCDM classifications and item parameter estimates can be observed under various sample and test characteristics. Results illustrated that LCDM classifications and item parameter estimates show clear invariance when adequate model data fit is present. To demonstrate the implications of this important property, we conducted additional analyses to show that using pre-calibrated tests to classify examinees provided consistent classifications across calibration samples with varying mastery profile distributions and across tests with varying difficulties.     

Joint Maximum Likelihood Estimation for Diagnostic Classification Models

Joint maximum likelihood estimation (JMLE) is developed for diagnostic classification models (DCMs). JMLE has been barely used in Psychometrics because JMLE parameter estimators typically lack statistical consistency. The JMLE procedure presented here resolves the consistency issue by incorporating an external, statistically consistent estimator of examinees’ proficiency class membership into the joint likelihood function, which subsequently allows for the construction of item parameter estimators that also have the consistency property. Consistency of the JMLE parameter estimators is established within the framework of general DCMs: The JMLE parameter estimators are derived for the Loglinear Cognitive Diagnosis Model (LCDM). Two consistency theorems are proven for the LCDM. Using the framework of general DCMs makes the results and proofs also applicable to DCMs that can be expressed as submodels of the LCDM. Simulation studies are reported for evaluating the performance of JMLE when used with tests of varying length and different numbers of attributes. As a practical application, JMLE is also used with “real world” educational data collected with a language proficiency test.     

On the Identifiability of Diagnostic Classification Models

Psychometrika - This paper establishes fundamental results for statistical analysis based on diagnostic classification models (DCMs). The results are developed at a high level of generality and are...     

谱聚类算法在不同属性层级结构诊断评估中的应用

聚类分析已成功用于认知诊断评估(CDA)中,使用广泛的聚类分析方法为K-means算法,有研究已证明K-means在CDA中具有较好的聚类效果。而谱聚类算法通常比K-means分类效果更佳,本研究将谱聚类算法引进CDA,探讨了属性层级结构、属性个数、样本量和失误率对该方法的影响。研究发现：（1）谱聚类算法要比K-means提供更好的聚类结果,尤其在实验条件较苛刻时,谱聚类算法更加稳健;（2）线型结构聚类效果最好,收敛型和发散型相近,独立型结构表现较差;（3）属性个数和失误率增加后,聚类效果会下降;（4）样本量增加后,聚类效果有所提升,但K-means方法有时会有反向结果出现。     

认知诊断计算机自适应测验中平衡属性收敛的新方法

提出两种认知诊断计算机自适应测验下平衡属性收敛的新方法（MABI、RTA）,模拟研究系统探讨和比较了此二者与已有方法（ABI、IABI和RABI）的表现。结果发现：（1）新方法较不考虑属性收敛的方法有更高的准确率以及更均衡的题目使用率;（2）新方法较ABI和RABI有稍低的准确性,但有更平衡的题目使用率;（3）新方法与IABI的准确性和题目使用率在不同选题策略下各有合优势。总之,两种新方法较好地兼顾测量准确性、题目使用率以及题库曝光情况。     

认知诊断计算机自适应测验中平衡属性收敛的新方法

提出两种认知诊断计算机自适应测验下平衡属性收敛的新方法（MABI、RTA）,模拟研究系统探讨和比较了此二者与已有方法（ABI、IABI和RABI）的表现。结果发现：（1）新方法较不考虑属性收敛的方法有更高的准确率以及更均衡的题目使用率;（2）新方法较ABI和RABI有稍低的准确性,但有更平衡的题目使用率;（3）新方法与IABI的准确性和题目使用率在不同选题策略下各有合优势。总之,两种新方法较好地兼顾测量准确性、题目使用率以及题库曝光情况。     

Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions

Traditional testing procedures typically utilize unidimensional item response theory (IRT) models to provide a single, continuous estimate of a student’s overall ability. Advances in psychometrics have focused on measuring multiple dimensions of ability to provide more detailed feedback for students, teachers, and other stakeholders. Diagnostic classification models (DCMs) provide multidimensional feedback by using categorical latent variables that represent distinct skills underlying a test that students may or may not have mastered. The Scaling Individuals and Classifying Misconceptions (SICM) model is presented as a combination of a unidimensional IRT model and a DCM where the categorical latent variables represent misconceptions instead of skills. In addition to an estimate of ability along a latent continuum, the SICM model provides multidimensional, diagnostic feedback in the form of statistical estimates of probabilities that students have certain misconceptions. Through an empirical data analysis, we show how this additional feedback can be used by stakeholders to tailor instruction for students’ needs. We also provide results from a simulation study that demonstrate that the SICM MCMC estimation algorithm yields reasonably accurate estimates under large-scale testing conditions.     

Diagnostic Classification Models: Recent Developments,Practical Issues,and Prospects

More than three decades after their introduction, diagnostic classification models (DCM) do not seem to have been implemented in educational systems for the purposes they were devised. Most DCM research is either methodological for model development and refinement or retrofitting to existing nondiagnostic tests and, in the latter case, basically for model demonstration or constructs identification. DCMs have rarely been used to develop diagnostic assessment right from the start with the purpose of identifying individuals’ strengths and weaknesses (referred to as true applications in this study). In this article, we give an introduction to DCMs and their latest developments along with guidelines on how to proceed to employ DCMs to develop a diagnostic test or retrofit to a nondiagnostic assessment. Finally, we enumerate the reasons why we believe DCMs have not become fully operational in educational systems and suggest some advice to make their advent smooth and quick.     

认知诊断中多分属性与二分属性的对比研究

摘 要：Karelitz（2004）和詹沛达等（2016）认为1个多分属性内部(Lk+1)个水平的关系相当于Lk个部分满足线型层级关系的二分属性。本研究的目的是通过比较多分属性模型和二分属性模型的判准率,从而验证多分属性和二分属性间是否存在以上关系。结果表明：当属性个数较少时,两个模型的模式判准率相当,随着属性个数增加,多分属性模型的模式判准率高于二分属性模型的模式判准率。结论：在一定程度上,多分属性和二分属性之间确实存在以上关系,但两者并非完全等价,二者间的差异随着属性个数增加更加明显。     

Hierarchical Multinomial Processing Tree Models: A Latent-Class Approach

Multinomial processing tree models are widely used in many areas of psychology. Their application relies on the assumption of parameter homogeneity, that is, on the assumption that participants do not differ in their parameter values. Tests for parameter homogeneity are proposed that can be routinely used as part of multinomial model analyses to defend the assumption. If parameter homogeneity is found to be violated, a new family of models, termed latent-class multinomial processing tree models, can be applied that accommodates parameter heterogeneity and correlated parameters, yet preserves most of the advantages of the traditional multinomial method. Estimation, goodness-of-fit tests, and tests of other hypotheses of interest are considered for the new family of models. The author thanks Bill Batchelder, Edgar Erdfelder, Thorsten Meiser, and Christoph Stahl for helpful comments on a previous version of this paper. The author is also grateful to Edgar Erdfelder for making available the data set analyzed in this paper.     

Diagnostic Classification Models for a Mixture of Ordered and Non-ordered Response Options in Rating Scales

When developing ordinal rating scales, we may include potentially unordered response options such as “Neither Agree nor Disagree,” “Neutral,” “Don’t Know,” “No Opinion,” or “Hard to Say.” To handle responses to a mixture of ordered and unordered options, Huggins-Manley et al. (2018) proposed a class of semi-ordered models under the unidimensional item response theory framework. This study extends the concept of semi-ordered models into the area of diagnostic classification models. Specifically, we propose a flexible framework of semi-ordered DCMs that accommodates most earlier DCMs and allows for analyzing the relationship between those potentially unordered responses and the measured traits. Results from an operational study and two simulation studies show that the proposed framework can incorporate both ordered and non-ordered responses into the estimation of the latent traits and thus provide useful information about both the items and the respondents.     

Hierarchical Multinomial Processing Tree Models: A Latent-Trait Approach

Multinomial processing tree models are widely used in many areas of psychology. A hierarchical extension of the model class is proposed, using a multivariate normal distribution of person-level parameters with the mean and covariance matrix to be estimated from the data. The hierarchical model allows one to take variability between persons into account and to assess parameter correlations. The model is estimated using Bayesian methods with weakly informative hyperprior distribution and a Gibbs sampler based on two steps of data augmentation. Estimation, model checks, and hypotheses tests are discussed. The new method is illustrated using a real data set, and its performance is evaluated in a simulation study.     

有中介的调节模型检验方法：甄别和整合*

在心理和教育研究中,经常遇到调节和中介效应。模型包含不止3个变量时,可能同时包含调节和中介变量,一种常见的模型是有中介的调节模型。本文评介了文献上可以找到的五种检验有中介的调节模型的方法,逐一甄别,指出其中一些不合理之处,并推荐较好的方法。整合其中主要方法,总结出检验有中介的调节模型的流程,用偏差校正的百分位 Bootstrap 法或马尔科夫链蒙特卡罗法检验其中的中介效应,并用一个实际例子演示如何用此流程检验有中介的调节模型。     

A Simple Method for Comparing Complex Models: Bayesian Model Comparison for Hierarchical Multinomial Processing Tree Models Using Warp-III Bridge Sampling

Psychometrika - Multinomial processing trees (MPTs) are a popular class of cognitive models for categorical data. Typically, researchers compare several MPTs, each equipped with many parameters,...     

Family Demands and Job Attribute Preferences: A 4-Year Longitudinal Study of Women and Men

A longitudinal data set was used to examine the relationships between family demands and job attribute preferences. Study participants were 207 students who responded to surveys upon entering the MBA program of a large university and to follow-up surveys 1, 2, and 3 years later. Hierarchical regression analyses indicated that preferences for short, flexible work hours at earlier time periods positively predicted hours of household labor at later times, which supports a rational action model. Significant interaction effects indicated that the relationship between the importance of work hours and household labor was stronger for women than for men, which indicates that women were more likely than men to develop plans for combining work and family. Higher levels of household labor were associated with increased preferences for short, flexible work hours, and a comfortable work environment, which supports an accommodation model, but MBA students performing more household labor did not show a reduction in the desire for high salaries, good benefits, and intrinsically rewarding work.     

Erratum to: A Simple Method for Comparing Complex Models: Bayesian Model Comparison for Hierarchical Multinomial Processing Tree Models Using Warp-III Bridge Sampling

Psychometrika - The article A SIMPLE METHOD FOR COMPARING COMPLEX MODELS:     

A Theory-Based Empirical Classification of Family Problem-Solving Behavior

A method for classifying family interaction behavior is presented that underlies several related studies of variation in families' modes of interpreting and interacting with the social environment. Evidence is presented for the existence of three dimensions of family problem-solving behavior that: (a) are consistent with theory of variation in families' orientations to the environment; (b) are useful for interpreting differences within nonclinical as well as clinical samples of families; and (c) have been shown, within current samples' ranges, to be essentially independent of surface markers of family variation and accumulated skills of family members.