Detecting Curvilinear Relationships: A Comparison of Scoring Approaches Based on Different Item Response Models

There is a growing use of noncognitive assessments around the world, and recent research has posited an ideal point response process underlying such measures. A critical issue is whether the typical use of dominance approaches (e.g., average scores, factor analysis, and the Samejima's graded response model) in scoring such measures is adequate. This study examined the performance of an ideal point scoring approach (e.g., the generalized graded unfolding model) as compared to the typical dominance scoring approaches in detecting curvilinear relationships between scored trait and external variable. Simulation results showed that when data followed the ideal point model, the ideal point approach generally exhibited more power and provided more accurate estimates of curvilinear effects than the dominance approaches. No substantial difference was found between ideal point and dominance scoring approaches in terms of Type I error rate and bias across different sample sizes and scale lengths, although skewness in the distribution of trait and external variable can potentially reduce statistical power. For dominance data, the ideal point scoring approach exhibited convergence problems in most conditions and failed to perform as well as the dominance scoring approaches. Practical implications for scoring responses to Likert-type surveys to examine curvilinear effects are discussed.     

四种新的基于属性平衡的CD-CAT选题策略开发研究

基于属性平衡的CD-CAT选题策略能够保证每个认知属性被相当数量的题目测量,从而提高被试属性判准率,传统的基于属性平衡的选题策略包括MMGDI法和MGCDI法。本文针对传统的基于属性测量次数平衡选题策略进行改进,提出4种新的基于属性平衡的选题策略：RMGDI、RMCDI、SE-RMGDI、SE-RMCDI,前两种为基于属性测量次数平衡,后两种为基于属性测量精度平衡的选题策略。模拟研究表明：（1）定长CD-CAT条件下,短测验中,MMGDI表现最好,而长测验中,SE-RMGDI和SE-RMCDI的表现优于传统的属性平衡选题策略。（2）不定长CD-CAT条件下,RMGDI在判准率指标上表现优于传统的属性平衡选题策略,4种新的属性平衡策略在测量效率和综合指标上的表现均优于传统的选题策略。     

Dissecting the expanded cognitive reflection test: an item response theory analysis

This study presents a psychometric evaluation of the Expanded Cognitive Reflection Test (CRT7) based on item response theory. The participants (N?=?1204) completed the CRT7 and provided self-reported information about their cognitive styles through the Preference for Intuition and Deliberation Scale (PID). A two-parameter logistic model was fitted to the data to obtain the item difficulty and discrimination parameters of the CRT7. The results showed that the items had good discriminatory power (αs?=?.80 ? 2.92), but the range of difficulty was restricted (βs ranged from ?.60 to .32). Moreover, the CRT7 showed a pattern of correlations with the PID which was similar to that of the original CRT. When taken together, these results are evidence of the adequacy of the CRT7 as an expanded tool for measuring cognitive reflection; however, one of the newer items (the pig item) was consistently problematic across analyses, and so it is recommended that in future studies it should be removed from the CRT7.     

应用Rasch模型分析Rosenberg自尊量表

采用Rosenberg自尊量表(RSES)对425名在校大学生进行施测,应用项目反应理论的Rasch模型对项目指标进行分析及DIF检验。结果表明,Rosenberg自尊量表具有单维性,量表的信度为0.84; 除项目8以外,其他项目拟合指标良好,较适用来区分中等及偏低自尊水平的个体,项目功能差异检验发现在项目1和项目5上存在DIF,表现为男生自尊水平要高于女生。相对于经典测量理论,应用Rasch模型分析Rosenberg自尊量表具有优势,为进一步的完善和使用该自尊量表提供依据。     

四参数Logistic模型和传统模型对被试作答拟合能力的比较研究

针对测验中高能力被试答错容易试题的睡眠现象,可使用四参数Logistic模型分析数据。研究选取了来自心理测验和成就测验的实际数据,分别采用传统模型和四参数Logistic模型进行拟合,对不同模型的拟合指标及参数估计结果进行比较。结果表明,四参数Logistic模型能够提高拟合程度,增强估计结果的准确性,有效纠正高能力被试能力被低估的现象。建议在必要时使用四参数Logistic模型进行数据分析。     

考虑题目选项信息的非参数认知诊断计算机自适应测验

选择题中的作答选项能提供额外诊断信息, 为充分利用选项信息, 研究提出认知诊断计算机自适应测验(CD-CAT)中两种处理选择题选项信息的非参数选题策略和变长终止规则。模拟研究的结果发现：(1)定长条件下两种非参数选题策略的分类准确性整体要高于参数选题策略; (2)两种非参数选题策略较参数选题策略具有更加均衡的题库使用情况; (3)非参数选题策略在两种新的变长终止规则下具有更高的分类准确率; (4)两种非参数选题策略均适用于选择题CD-CAT情境, 使用者可任选其一进行测验分析。     

认知诊断框架下多级评分题目的DIF检测方法及其应用

本文对多级计分认知诊断测验的DIF概念进行了界定,并通过模拟实验以及实证研究对四种常见的多级计分DIF检验方法的适用性进行理论以及实践性的探索。研究结果表明：四种方法均能对多级计分认知诊断中的DIF进行有效的检验,且各方法的表现受模型的影响不大;相较于以总分为匹配变量,以KS为匹配变量时更利于DIF的检测;以KS为匹配变量的LDFA方法以及以KS为匹配变量的曼特尔检验方法在检测DIF题目时有着最高的检验力。     

The Mental Vulnerability Questionnaire: A psychometric evaluation

Eplov, L.F., Petersen, J., Jørgensen, T., Johansen, C., Birket‐Smith, M., Lyngberg, A. C. & Mortensen, E. L. (2010). The Mental Vulnerability Questionnaire: A psychometric evaluation. Scandinavian Journal of Psychology 51, 548–554. The Mental Vulnerability Questionnaire was originally a 22 item scale, later reduced to a 12 item scale. In population studies the 12 item scale has been a significant predictor of health and illness. The scale has not been psychometrically evaluated for more than 30 years, and the aim of the present study was both to evaluate the psychometric properties of the 22 and 12 item scales and of three new scales. The main study sample was a community sample comprising more than 6,000 men and women. In this sample the coefficients of homogeneity were all over 0.30 for the three new scales, but below 0.30 for the 12 and the 22 item scales. All five Mental Vulnerability scales had positively skewed score distributions which were associated significantly with both SCL‐90‐R symptom scores and NEO‐PI‐R personality scales (primarily Neuroticism and Extraversion). Coefficient alpha was highest for the 22 and 12 item scales, and the two scales also showed the highest long‐term stability. The three new scales reflect relatively independent dimensions of Psychosomatic Symptoms, Mental Symptoms, and Interpersonal Problems, but because of reliability problems it remains an open question whether they will prove useful as predictors of health and morbidity.     

Comparison of classical and modern methods for measuring and correcting for acquiescence

Likert-type self-report scales are frequently used in large-scale educational assessment of social-emotional skills. Self-report scales rely on the assumption that their items elicit information only about the trait they are supposed to measure. However, different response biases may threaten this assumption. Specifically, in children, the response style of acquiescence is an important source of systematic error. Balanced scales, including an equal number of positively and negatively keyed items, have been proposed as a solution to control for acquiescence, but the reasons why this design feature worked from the perspective of modern psychometric models have been underexplored. Three methods for controlling for acquiescence are compared: classical method by partialling out the mean; an item response theory method to measure differential person functioning (DPF); and multidimensional item response theory (MIRT) with random intercept. Comparative analyses are conducted on simulated ratings and on self-ratings provided by 40,649 students (aged 11–18) on a fully balanced 30-item scale assessing conscientious self-management. Acquiescence bias was explained as DPF and it was demonstrated that: the acquiescence index is highly related to DPF; balanced scales produce scores controlled for DPF; and MIRT factor scores are highly related to scores controlled for DPF and the random intercept is highly related to DPF.     

Cognitive diagnosis models for multiple strategies

Cognitive diagnosis models (CDMs) have been used as psychometric tools in educational assessments to estimate students’ proficiency profiles. However, most CDMs assume that all students adopt the same strategy when approaching problems in an assessment, which may not be the case in practice. This study develops a generalized multiple-strategy CDM for dichotomous response data. The proposed model provides a unified framework to accommodate various condensation rules (e.g., conjunctive, disjunctive, and additive) and different strategy selection approaches (i.e., probability-matching, over-matching, and maximizing). Model parameters are estimated using the marginal maximum likelihood estimation via expectation-maximization algorithm. Simulation studies showed that the parameters of the proposed model can be adequately recovered and that the proposed model was relatively robust to some types of model misspecifications. A set of real data was analysed as well to illustrate the use of the proposed model in practice.