Stopping rules for multi-category computerized classification testing

Computerized classification testing (CCT) aims to classify persons into one of two or more possible categories to make decisions such as mastery/non-mastery or meet most/meet all/exceed. A defining feature of CCT is its stopping criterion: the test terminates when there is enough confidence to make a decision. There is abundant research on CCT with a single cut-off, and two common stopping criteria are the sequential probability ratio test (SPRT) statistic and the generalized likelihood ratio statistic (GLR). However, there is a relative scarcity of research extending the SPRT to the multi-hypothesis case for when there is more than one cut-off. In this paper, we propose a new multi-category GLR (mGLR) statistic as well as a stochastically curtailed version of the CCT with three or more categories. A simulation study was conducted to show that the mGLR statistic outperformed the existing stopping rules by generating shorter average test length without sacrificing classification accuracy. Results also revealed that the stochastically curtailed mGLR successfully increased test efficiency in certain testing conditions.     

On the existence and uniqueness of JML estimates for the partial credit model

A necessary and sufficient condition is given in this paper for the existence and uniqueness of the maximum likelihood (the so-called joint maximum likelihood) estimate of the parameters of the Partial Credit Model. This condition is stated in terms of a structural property of the pattern of the data matrix that can be easily verified on the basis of a simple iterative procedure. The result is proved by using an argument of Haberman (1977). The author wishes to thank the Editor and the anonymous reviewers for their comments that helped to substantially improve the final version of this paper. This research was supported in part by a MURST grant (ex 60%).     

Likelihood inference on the underlying structure of IRT models

The assumptions underlying item response theory (IRT) models may be expressed as a set of equality and inequality constraints on the parameters of a latent class model. It is well known that the same assumptions imply that the parameters of the manifest distribution have to satisfy a more complicated set of inequality constraints which, however, are necessary but not sufficient. In this paper, we describe how the theory for likelihood-based inference under equality and inequality constraints may be used to test the underlying assumptions of IRT models. It turns out that the analysis based directly on the latent structure is simpler and more flexible. In particular, we indicate how several interesting extensions of the Rasch model may be obtained by partial relaxation of the basic constraints. An application to a data set provided by Educational Testing Service is used to illustrate the approach.We thank Dr. Gorman and Dr. Rogers of the Educational Testing Service for providing the data analyzed in Section 4. We also thank three reviewers for comments and suggestions.This revised article was published online in August 2005 with the PDF paginated correctly.     

Selective influence through conditional independence

Let each of several (generally interdependent) random vectors, taken separately, be influenced by a particular set of external factors. Under what kind of the joint dependence of these vectors on the union of these factor sets can one say that each vector is selectively influenced by “its own” factor set? The answer proposed and elaborated in this paper is: One can say this if and only if one can find a factor-independent random vector given whose value the vectors in question are conditionally independent, with their conditional distributions selectively influenced by the corresponding factor sets. Equivalently, the random vectors should be representable as deterministic functions of “their” factor sets and of some mutually independent and factor-independent random variables, some of which may be shared by several of the functions.     

EFFECTS OF PREFERENCE AND REINFORCER VARIATION ON WITHIN-SESSION PATTERNS OF RESPONDING

We examined correspondence between preference assessment outcome and within‐session patterns of responding in one subject with autism. Responding maintained by a single highly preferred item resulted in a greater total number of responses, a slower decline in within‐session response rates, and a greater proportion of short interresponse times compared to responding maintained by varied moderately preferred (MP) stimuli. Presenting varied MP stimuli within the same session produced greater levels and more sustained responding than presenting those same stimuli individually.     

Multilevel Reliability Measures of Latent Scores Within an Item Response Theory Framework

Abstract

This paper evaluated multilevel reliability measures in two-level nested designs (e.g., students nested within teachers) within an item response theory framework. A simulation study was implemented to investigate the behavior of the multilevel reliability measures and the uncertainty associated with the measures in various multilevel designs regarding the number of clusters, cluster sizes, and intraclass correlations (ICCs), and in different test lengths, for two parameterizations of multilevel item response models with separate item discriminations or the same item discrimination over levels. Marginal maximum likelihood estimation (MMLE)-multiple imputation and Bayesian analysis were employed to evaluate the accuracy of the multilevel reliability measures and the empirical coverage rates of Monte Carlo (MC) confidence or credible intervals. Considering the accuracy of the multilevel reliability measures and the empirical coverage rate of the intervals, the results lead us to generally recommend MMLE-multiple imputation. In the model with separate item discriminations over levels, marginally acceptable accuracy of the multilevel reliability measures and empirical coverage rate of the MC confidence intervals were found in a limited condition, 200 clusters, 30 cluster size, .2 ICC, and 40 items, in MMLE-multiple imputation. In the model with the same item discrimination over levels, the accuracy of the multilevel reliability measures and the empirical coverage rate of the MC confidence intervals were acceptable in all multilevel designs we considered with 40 items under MMLE-multiple imputation. We discuss these findings and provide guidelines for reporting multilevel reliability measures.     

公众政治参与和地方政府信任的关系:一个有调节的中介模型

研究聚焦政府宣传教育,考察公众政治参与和地方政府信任的关系。对2061名有效被试进行问卷调查并分析。结果发现:(1)政治参与显著正向预测地方政府信任;(2)活动有用性评价在政治参与和地方政府信任间起部分中介作用;(3)宣传教育频率负向调节政治参与和有用性评价的关系以及有用性评价的中介效应。上述结果表明,多渠道提升公众政治参与意识,合理把握政府宣传教育频率是促进地方政府信任提高的可选策略。     

认知诊断缺失数据处理方法的比较：零替换、多重插补与极大似然估计法

数据缺失在测验中经常发生, 认知诊断评估也不例外, 数据缺失会导致诊断结果的偏差。首先, 通过模拟研究在多种实验条件下比较了常用的缺失数据处理方法。结果表明：(1)缺失数据导致估计精确性下降, 随着人数与题目数量减少、缺失率增大、题目质量降低, 所有方法的PCCR均下降, Bias绝对值和RMSE均上升。(2)估计题目参数时, EM法表现最好, 其次是MI, FIML和ZR法表现不稳定。(3)估计被试知识状态时, EM和FIML表现最好, MI和ZR表现不稳定。其次, 在PISA2015实证数据中进一步探索了不同方法的表现。综合模拟和实证研究结果, 推荐选用EM或FIML法进行缺失数据处理。     

计算机化分类测验终止规则的类别、特点及应用

计算机化分类测验(Computerized Classification Testing, CCT)能够高效地对被试进行分类, 已广泛应用于合格性测验及临床心理学中。作为CCT的重要组成部分, 终止规则决定测验何时停止以及将被试最终划分到何种类别, 因此直接影响测验效率及分类准确率。已有的三大类终止规则(似然比规则、贝叶斯决策理论规则及置信区间规则)的核心思想分别为构造假设检验、设计损失函数和比较置信区间相对位置。同时, 在不同测验情境下, CCT的终止规则发展出不同的具体形式。未来研究可以继续开发贝叶斯规则、考虑多维多类别情境以及结合作答时间和机器学习算法。针对测验实际需求, 三类终止规则在合格性测验上均有应用潜力, 而临床问卷则倾向应用贝叶斯规则。     

Two-Stage maximum likelihood estimation in the misspecified restricted latent class model

The maximum likelihood classification rule is a standard method to classify examinee attribute profiles in cognitive diagnosis models (CDMs). Its asymptotic behaviour is well understood when the model is assumed to be correct, but has not been explored in the case of misspecified latent class models. This paper investigates the asymptotic behaviour of a two-stage maximum likelihood classifier under a misspecified CDM. The analysis is conducted in a general restricted latent class model framework addressing all types of CDMs. Sufficient conditions are proposed under which a consistent classification can be obtained by using a misspecified model. Discussions are also provided on the inconsistency of classification under certain model misspecification scenarios. Simulation studies and a real data application are conducted to illustrate these results. Our findings can provide some guidelines as to when a misspecified simple model or a general model can be used to provide a good classification result.