Limited‐information goodness‐of‐fit testing of diagnostic classification item response models

Despite the growing popularity of diagnostic classification models (e.g., Rupp et al., 2010, Diagnostic measurement: theory, methods, and applications, Guilford Press, New York, NY) in educational and psychological measurement, methods for testing their absolute goodness of fit to real data remain relatively underdeveloped. For tests of reasonable length and for realistic sample size, full‐information test statistics such as Pearson's X² and the likelihood ratio statistic G² suffer from sparseness in the underlying contingency table from which they are computed. Recently, limited‐information fit statistics such as Maydeu‐Olivares and Joe's (2006, Psychometrika, 71, 713) M₂ have been found to be quite useful in testing the overall goodness of fit of item response theory models. In this study, we applied Maydeu‐Olivares and Joe's (2006, Psychometrika, 71, 713) M₂ statistic to diagnostic classification models. Through a series of simulation studies, we found that M₂ is well calibrated across a wide range of diagnostic model structures and was sensitive to certain misspecifications of the item model (e.g., fitting disjunctive models to data generated according to a conjunctive model), errors in the Q‐matrix (adding or omitting paths, omitting a latent variable), and violations of local item independence due to unmodelled testlet effects. On the other hand, M₂ was largely insensitive to misspecifications in the distribution of higher‐order latent dimensions and to the specification of an extraneous attribute. To complement the analyses of the overall model goodness of fit using M₂, we investigated the utility of the Chen and Thissen (1997, J. Educ. Behav. Stat., 22, 265) local dependence statistic $X_{\mathrm{LD}}^2$ for characterizing sources of misfit, an important aspect of model appraisal often overlooked in favour of overall statements. The $X_{\mathrm{LD}}^2$ statistic was found to be slightly conservative (with Type I error rates consistently below the nominal level) but still useful in pinpointing the sources of misfit. Patterns of local dependence arising due to specific model misspecifications are illustrated. Finally, we used the M₂ and $X_{\mathrm{LD}}^2$ statistics to evaluate a diagnostic model fit to data from the Trends in Mathematics and Science Study, drawing upon analyses previously conducted by Lee et al., (2011, IJT, 11, 144).     

On the asymptotic standard error of a class of robust estimators of ability in dichotomous item response models

In item response theory, the classical estimators of ability are highly sensitive to response disturbances and can return strongly biased estimates of the true underlying ability level. Robust methods were introduced to lessen the impact of such aberrant responses on the estimation process. The computation of asymptotic (i.e., large‐sample) standard errors (ASE) for these robust estimators, however, has not yet been fully considered. This paper focuses on a broad class of robust ability estimators, defined by an appropriate selection of the weight function and the residual measure, for which the ASE is derived from the theory of estimating equations. The maximum likelihood (ML) and the robust estimators, together with their estimated ASEs, are then compared in a simulation study by generating random guessing disturbances. It is concluded that both the estimators and their ASE perform similarly in the absence of random guessing, while the robust estimator and its estimated ASE are less biased and outperform their ML counterparts in the presence of random guessing with large impact on the item response process.     

义务教育阶段学业质量监测题库建设的理论与实践

义务教育阶段学业质量监测是学生学业质量评价的重要方式,在当前国内外教育质量评价领域中发挥着极为重要的作用.加强学业质量监测的题库建设,则是促进学业质量监测工具的命制与实施从经验型向科学化迈进的必由之路.本文在对义务教育阶段学业质量监测题库的性质和功能进行阐述的基础上,重点探讨了学业质量监测题库建设的理论基础、基本流程和关键环节,并进一步分析指出了当前题库建设中有待深入研究的问题.     

Age-related differences in effectiveness of item restudy choices: the role of value

We examined the effect of value on effectiveness of restudy choices in younger and older adults by using an honor or dishonor procedure. In the honor condition, participants restudied the half of the items that they selected for restudy. In the dishonor condition, participants restudied the half of the items that they did not select for restudy. The results showed that younger adults restudied more low-value (1-point) items that they had rated as least-well learned, while older adults restudied more high-value (5-point) items. For younger adults, 1-point-value item performance in the honor condition was reliably higher than it was in the dishonor condition. For older adults, 5-point-value items performance in the honor condition was reliably higher than it was in the dishonor condition. These results suggest that older adults can exert effective restudy choices like younger adults when item value is manipulated.     

Validation of the trust relationship audit (TRA): An item parcelling approach

This study sought to apply an item parcelling approach to confirm the factor structure of trust in the direct supervisor as measured by the trust relationship audit (TRA). The researchers analysed an existing data set on the TRA from 9 060 South African employees. For the analysis, the researchers utilised structural equation modelling, using item parcelling to confirm the factor structure. The results confirm that, in essence, the large sample structural model replicates the original small sample model, consisting of separate personality and managerial practices factors as antecedents of trust in supervisors. Two items measuring personality traits loaded differently in the small and the combined sample. The results suggest item parcelling to be a value-add in measure validation when data mining.     

Extreme Response Style and the Measurement of Intra-Individual Variability in Affect

Extreme response style (ERS) has the potential to bias the measurement of intra-individual variability in psychological constructs. This paper explores such bias through a multilevel extension of a latent trait model for modeling response styles applied to repeated measures rating scale data. Modeling responses to multi-item scales of positive and negative affect collected from smokers at clinic visits following a smoking cessation attempt revealed considerable ERS bias in the intra-individual sum score variances. In addition, simulation studies suggest the magnitude and direction of bias due to ERS is heavily dependent on the mean affect level, supporting a model-based approach to the study and control of ERS effects. Application of the proposed model-based adjustment is found to improve intra-individual variability as a predictor of smoking cessation.     

可以兼顾策略、认知状态和能力的CD-CAT选题方法

当CD-CAT测验需要同时诊断被试的解题策略、认知状态并评估被试的宏观能力时,就需要在选题过程中兼顾这三个测量目标。用两种不同方式将多策略香农熵(MSSHE)指标与Fisher信息量相结合,提出多策略情境中的DWI指标MSDWI)选题法与“先用MSSHE后用Fisher信息量”的两步选题法。基于多策略RRUM模型(MS-RRUM),将这两种方法与随机选题法在不同属性数量条件下进行模拟比较,结果表明：当属性数量为4个或6个时,两步选题法在策略判准率、认知状态判准率和能力估计三个方面都有最佳的效果。     

Detecting Curvilinear Relationships: A Comparison of Scoring Approaches Based on Different Item Response Models

There is a growing use of noncognitive assessments around the world, and recent research has posited an ideal point response process underlying such measures. A critical issue is whether the typical use of dominance approaches (e.g., average scores, factor analysis, and the Samejima's graded response model) in scoring such measures is adequate. This study examined the performance of an ideal point scoring approach (e.g., the generalized graded unfolding model) as compared to the typical dominance scoring approaches in detecting curvilinear relationships between scored trait and external variable. Simulation results showed that when data followed the ideal point model, the ideal point approach generally exhibited more power and provided more accurate estimates of curvilinear effects than the dominance approaches. No substantial difference was found between ideal point and dominance scoring approaches in terms of Type I error rate and bias across different sample sizes and scale lengths, although skewness in the distribution of trait and external variable can potentially reduce statistical power. For dominance data, the ideal point scoring approach exhibited convergence problems in most conditions and failed to perform as well as the dominance scoring approaches. Practical implications for scoring responses to Likert-type surveys to examine curvilinear effects are discussed.     

四种新的基于属性平衡的CD-CAT选题策略开发研究

基于属性平衡的CD-CAT选题策略能够保证每个认知属性被相当数量的题目测量,从而提高被试属性判准率,传统的基于属性平衡的选题策略包括MMGDI法和MGCDI法。本文针对传统的基于属性测量次数平衡选题策略进行改进,提出4种新的基于属性平衡的选题策略：RMGDI、RMCDI、SE-RMGDI、SE-RMCDI,前两种为基于属性测量次数平衡,后两种为基于属性测量精度平衡的选题策略。模拟研究表明：（1）定长CD-CAT条件下,短测验中,MMGDI表现最好,而长测验中,SE-RMGDI和SE-RMCDI的表现优于传统的属性平衡选题策略。（2）不定长CD-CAT条件下,RMGDI在判准率指标上表现优于传统的属性平衡选题策略,4种新的属性平衡策略在测量效率和综合指标上的表现均优于传统的选题策略。