基于项目节点的多级评分模型的统一

多级评分项目由于可以提供更多关于被试的信息而被广泛的使用。本文首先通过引用一个多级评分的数学试题, 给出了项目节点这一概念。假设被试在项目节点上的正确反应概率为二参数逻辑斯蒂模型之下, 本文通过分析三种不同类型的多级评分项目, 得出了三个评分模型, 其中一个和等级反应模型在形式上是一样的。鉴于我国目前考试测量所使用的多级评分项目的形式, 可以运用本文所述的项目节点的方法将项目评分模型统一提出。     

物理概念图试题的评分方法

详细介绍了运用知识编码对物理概念图试题进行评分的具体方法,并对物理概念图试题的难度、区分度、信度和效度进行了检验,计算了物理概念图试题与传统物理题型共同组成的期末物理测验的信度,说明依据知识编码评定物理概念图试题分数,是一种行之有效的评分方法,按照知识编码评定物理概念图试题的分数使物理概念图试题成为一道高质量的试题。     

Optimal correction for guessing in multiple-choice tests

Subjects’ decisions in multiple-choice tests are an interesting domain for the analysis of decision making under uncertainty. When the test is graded using a rule that penalizes wrong answers, each item can be viewed as a lottery where a rational examinee would choose whether to omit (sure reward) or answer (take the lottery) depending on risk aversion and level of knowledge. We formalize students as heterogeneous decision makers with different risk attitudes and levels of knowledge. Building on IRT, we compute the optimal penalty given students’ optimal behavior and the trade-off between bias and measurement error. Although MCQ examinations are frequently used, there is no consensus as to whether a penalty for wrong answers should be used or not. For example, examinations for medical licensing in some countries include MCQ sections with penalty while in others there is no penalty for wrong answers. We contribute to this discussion with a formal analysis of the effects of penalties; our simulations indicate that the optimal penalty is relatively high for perfectly rational students but also when they are not fully rational: even though penalty discriminates against risk averse students, this effect is small compared with the measurement error that it prevents.     

创造力测评中的评分者效应

创造力测评中的评分者效应(rater effects)是指在创造性测评过程中, 由于评分者参与而对测评结果造成的影响.评分者效应本质上源于评分者内在认知加工的不同, 具体体现在其评分结果的差异.本文首先概述了评分者认知的相关研究, 以及评分者,创作者,社会文化因素对测评的影响.其次在评分结果层面梳理了评分者一致性信度的指标及其局限, 以及测验概化理论和多面Rasch模型在量化,控制该效应中的应用.最后基于当前研究仍存在的问题, 指出了未来可能的研究方向, 包括深化评分者认知研究,整合不同层面评分者效应的研究, 以及拓展创造力测评方法和技术等.     

The counterintuitive impact of responses and response times on parameter estimates in the drift diffusion model

Given a drift diffusion model with unknown drift and boundary parameters, we analyse the behaviour of maximum likelihood estimates with respect to changes of responses and response times. It is shown analytically that a single fast response time can dominate the estimation in that no matter how many correct answers a test taker provides, the estimate of the drift (ability) parameter decreases to zero. In addition, it is shown that although higher drift rates imply shorter response times, the reverse implication does not hold for the estimates: shorter response times can decrease the drift rate estimate. In the light of these analytical results, we illustrate the actual impact of the findings in a small simulation for a mental rotation test. The method of analysis outlined is applicable to a broader range of models, and we emphasize the need to further check currently used reaction time models within this framework.     

A diagnostic tree model for polytomous responses with multiple strategies

Constructed-response items have been shown to be appropriate for cognitively diagnostic assessments because students’ problem-solving procedures can be observed, providing direct evidence for making inferences about their proficiency. However, multiple strategies used by students make item scoring and psychometric analyses challenging. This study introduces the so-called two-digit scoring scheme into diagnostic assessments to record both students’ partial credits and their strategies. This study also proposes a diagnostic tree model (DTM) by integrating the cognitive diagnosis models with the tree model to analyse the items scored using the two-digit rubrics. Both convergent and divergent tree structures are considered to accommodate various scoring rules. The MMLE/EM algorithm is used for item parameter estimation of the DTM, and has been shown to provide good parameter recovery under varied conditions in a simulation study. A set of data from TIMSS 2007 mathematics assessment is analysed to illustrate the use of the two-digit scoring scheme and the DTM.     

Assessment of generalised Bayesian structural equation models for continuous and binary data

The paper proposes a novel model assessment paradigm aiming to address shortcoming of posterior predictive $p$-values, which provide the default metric of fit for Bayesian structural equation modelling (BSEM). The model framework presented in the paper focuses on the approximate zero approach (Psychological Methods, 17 , 2012, 313), which involves formulating certain parameters (such as factor loadings) to be approximately zero through the use of informative priors, instead of explicitly setting them to zero. The introduced model assessment procedure monitors the out-of-sample predictive performance of the fitted model, and together with a list of guidelines we provide, one can investigate whether the hypothesised model is supported by the data. We incorporate scoring rules and cross-validation to supplement existing model assessment metrics for BSEM. The proposed tools can be applied to models for both continuous and binary data. The modelling of categorical and non-normally distributed continuous data is facilitated with the introduction of an item-individual random effect. We study the performance of the proposed methodology via simulation experiments as well as real data on the ‘Big-5’ personality scale and the Fagerstrom test for nicotine dependence.