多维计算机化自适应测验中项目曝光控制选题策略的比较

在MCAT中考查四种项目选择指标在有无曝光控制条件下的选题表现。项目选择指标分别是:(1)贝叶斯的D优化方法(D-optimality)、后验期望Kullback-Leibler方法(KLP)、基于等权重复合分数的最小误差方差方法(the minimized error variance of the linear combination score with equal weight,V1)和基于最优权重复合分数的最小误差方差方法(the minimized error variance of the composite score with optimized weight,V2)。将针对认知诊断CAT项目曝光控制的的限制阈值方法(Restrictive Threshold,RT)和限制进度(Restrictive Progressive,RPG)方法、单维CAT中的最大优先指标方法(Maximum Priority Index,MPI)推广到MCAT。模拟研究表明:(1)KLP,D-优化和V1对领域分数估计准确,能力返真性比V2更好。(2)尽管V1和V2方法相比KLP和D-优化方法提高了题库利用率,但这四种选题指标都产生不均匀的项目曝光率分布。(2)三种曝光控制策略都极大地提高项目曝光均匀性,且不明显降低测量精度。(3)MPI与RPG方法在曝光控制方面表现类似,且比RT的方法表现更好。     

Eliciting Subjective Probabilities through Pair‐wise Comparisons

We propose and test a novel approach for eliciting subjective joint probabilities. In the proposed approach, judges compare pairs of possible outcomes and identify which of the two is more likely and by how much. These pair‐wise comparative judgments create a matrix of ratio judgments from which the target probabilities are extracted using the rows' (or columns') geometric means. In Study 1, subjects provided direct assessments of the likelihood of joint events (e.g., sunny days and stock market gains) and also made pair‐wise comparisons of the same joint events. Subjects in Study 2 learnt the distribution of hypothetical events pairs and provided direct and ratio estimates. In both studies, the ratio estimates were significantly more accurate than the direct estimates. The results suggest that it is possible to elicit probabilistic estimates without explictly asking for probabilities and that the pair‐wise approach is a candidate for complementing or replacing traditional elicitation approaches. Copyright © 2016 John Wiley & Sons, Ltd.     

多维计算机化自适应测验选题策略的开发及比较

本研究提出一种新的多维计算机化自适应测验(MCAT)选题策略,并将现有及本研究新提出的MCAT选题策略归纳为三类进行了较为系统全面的Monte Carlo模拟实验比较。结果表明:A-优化方法、后验期望KL信息方法 (KB)和本文提出的修正的连续熵方法 (MCEM)在各自类别中估计精度最高;结合题库安全性来看,MCEM综合表现最好。     

One or two dimensions in spontaneous classification: a simplicity approach

When participants are asked to spontaneously categorize a set of items, they typically produce unidimensional classifications, i.e., categorize the items on the basis of only one of their dimensions of variation. We examine whether it is possible to predict unidimensional vs. two-dimensional classification on the basis of the abstract stimulus structure, by employing Pothos and Chater's simplicity model of spontaneous categorization [Pothos, E. M., & Chater, N. (2002). A simplicity principle in unsupervised human categorization. Cognitive Science, 26, 303-343]. The simplicity model provides a quantitative measure of how intuitive a particular classification is. With objects represented in two dimensions, we propose that a unidimensional classification will be preferred if it is more intuitive than all possible two-dimensional ones, and vice versa. Empirical results supporting this proposal are reported. Implications for Goodman's paradox are discussed.     

The underlying structure of the Beck Depression Inventory II: A multidimensional scaling approach

The purpose of the present study was to confirm Beck’s six symptom categorization of depression through the most recent version of Beck Depression Inventory (BDI-II) by employing a confirmatory multidimensional scaling (MDS). This analysis indicates two independent dimensions in the BDI-II. The disturbance domain dimension relates to the domains wherein the disturbances occur, and validates Beck’s six symptom categorization dividing the items into disturbances in basic need-satisfaction, energy regulation, focused attention, regulation of emotion, motivation, and cognitive distortion in self-evaluation. The level of arousal dimension relates to how the disturbances are expressed, namely in an under-active, regular, or over-active manifestation. The MDS solution suggests 18 (6 × 3) symptom areas of depression, and may serve as a guide for constructing additional items.     

Item format and the structure of the buss-durkee hostility inventory

The Buss-Durkee Hostility Inventory is a widely employed multidimensional measure of aggression. Two studies, each involving the administration of both two-choice and seven-choice response format versions of the instrument, were conducted to determine if (1) the theoretical scales could be reproduced empirically, (2) the change in response format either changes or improves the structure, and (3) the structure of either response format replicates across administrations. The two-choice version provided some support for the present theoretical scoring but was not very stable across administrations. The seven-choice version resulted in a structure that was different from both the two-choice structure and theoretical scoring but was more stable across administrations.     

A taxonomic study of aggression in institutionalized delinquents

The taxonomy of aggressive behavior has been studied by many social scientists. In the present study this topic is approached empirically. A group of convicts were asked to quantify pairwise dissimilarity among aggressive social situations. The data obtained were processed by Multidimensional Scaling. The results show the necessity of taking into account 3 dimensions (justifiability, instrumentality and degree of involvement with the victim) in distinguishing among aggressive episodes. This finding is discussed in terms of cultural differences, and the usefulness of the perception of aggressive episodes as a basis for a taxonomic system is proposed.     

分类数据测量等价性检验方法及其比较：项目阈值(难度)参数的组间差异性检验

测量工具满足等价性是进行多组比较的前提, 测量等价性的检验方法主要有基于CFA的多组比较法和基于IRT的DIF检验两类方法。文章比较了单维测验情境下基于CCFA的DIFFTEST检验方法和基于IRT模型的IRT-LR检验方法, 以及多维测验情境下DIFFTEST和基于MIRT的卡方检验方法的差异。通过模拟研究的方法, 比较了几种方法的检验力和第一类错误, 并考虑了样本总量、样本量的组间均衡性、测验长度、阈值差异大小以及维度间相关程度的影响。研究结果表明：(1)在单维测验下, IRT-LR是比DIFFTEST更为严格的检验方法; 多维测验下, 在测验较长、测验维度之间相关较高时, MIRT-MG比DIFFTEST更容易检验出项目阈值的差异, 而在测验长度较短、维度之间相关较小时, DIFFTEST的检验力反而略高于MIRT-MG方法。(2)随着阈值差值增加, DIFFTEST、IRT-LR和MIRT-MG三种方法的检验力均在增加, 当阈值差异达到中等或较大时, 三种方法都可以有效检验出测验阈值的不等价性。(3)随着样本总量增加, DIFFTEST、IRT-LR和MIRT-MG方法的检验力均在增加; 在总样本量不变, 两组样本均衡情况下三种方法的检验力均高于不均衡的情况。(4)违背等价性题目个数不变时, 测验越长DIFFTEST的检验力会下降, 而IRT-LR和MIRT-MG检验力则上升。(5) DIFFTEST方法的一类错误率平均值接近名义值0.05; 而IRT-LR和MIRT-MG方法的一类错误率平均值远低于0.05。     

使用验证性补偿多维IRT模型进行认知诊断评估

随着人们对测验反馈结果精细化的需求逐渐提高, 具有认知诊断功能的测量方法逐渐受到人们的关注。在认知诊断模型(CDMs)闪耀着光芒的同时, 另一类能够在连续量尺上提供精细反馈的多维IRT模型(MIRTMs)似乎受到些许冷落。为探究MIRTMs潜在的认知诊断功能, 本文以补偿模型为视角, 聚焦于分别属于MIRTMs的多维两参数logistic模型(M2PLM)和属于CDMs的线性logistic模型(LLM); 之后为使两者具有可比性, 可对补偿M2PLM引入验证性矩阵(Q矩阵)来界定题目与维度之间的关系, 进而得到验证性的补偿M2PLM (CC-M2PLM), 并通过把潜在特质按切点划分为跨界属性, 以期使CC-M2PLM展现出其本应具有的认知诊断功能; 预研究表明logistic量尺上的0点可作为相对合理的切点; 然后, 通过模拟研究对比探究CC-M2PLM和LLM的认知诊断功能, 结果表明CC-M2PLM可用于分析诊断测验数据, 且认知诊断功能与直接使用LLM的效果相当; 最后, 以两则实证数据为例来说明CC-M2PLM在实际诊断测验分析中的可行性。     

Conditional Covariance Theory and Detect for Polytomous Items

This paper extends the theory of conditional covariances to polytomous items. It has been proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, given an appropriately chosen composite is positive if, and only if, the two items measure similar constructs besides the composite. The theory provides a theoretical foundation for dimensionality assessment procedures based on conditional covariances or correlations, such as DETECT and DIMTEST, so that the performance of these procedures is theoretically justified when applied to response data with polytomous items. Various estimators of conditional covariances are constructed, and special attention is paid to the case of complex sampling data, such as those from the National Assessment of Educational Progress (NAEP). As such, the new version of DETECT can be applied to response data sets not only with polytomous items but also with missing values, either by design or at random. DETECT is then applied to analyze the dimensional structure of the 2002 NAEP reading samples of grades 4 and 8. The DETECT results show that the substantive test structure based on the purposes for reading is consistent with the statistical dimensional structure for either grade. This research was supported by the Educational Testing Service and the National Assessment of Educational Progress (Grant R902F980001), US Department of Education. The opinions expressed herein are solely those of the author and do not necessarily represent those of the Educational Testing Service. The author would like to thank Ting Lu, Paul Holland, Shelby Haberman, and Feng Yu for their comments and suggestions. Requests for reprints should be sent to Jinming Zhang, Educational Testing Service, MS 02-T, Rosedale Road, Princeton, NJ 08541, USA. E-mail: jzhang@ets.org