Developing new online calibration methods for multidimensional computerized adaptive testing

Multidimensional computerized adaptive testing (MCAT) has received increasing attention over the past few years in educational measurement. Like all other formats of CAT, item replenishment is an essential part of MCAT for its item bank maintenance and management, which governs retiring overexposed or obsolete items over time and replacing them with new ones. Moreover, calibration precision of the new items will directly affect the estimation accuracy of examinees’ ability vectors. In unidimensional CAT (UCAT) and cognitive diagnostic CAT, online calibration techniques have been developed to effectively calibrate new items. However, there has been very little discussion of online calibration in MCAT in the literature. Thus, this paper proposes new online calibration methods for MCAT based upon some popular methods used in UCAT. Three representative methods, Method A, the ‘one EM cycle’ method and the ‘multiple EM cycles’ method, are generalized to MCAT. Three simulation studies were conducted to compare the three new methods by manipulating three factors (test length, item bank design, and level of correlation between coordinate dimensions). The results showed that all the new methods were able to recover the item parameters accurately, and the adaptive online calibration designs showed some improvements compared to the random design under most conditions.     

Non‐ignorable missingness item response theory models for choice effects in examinee‐selected items

Examinee‐selected item (ESI) design, in which examinees are required to respond to a fixed number of items in a given set, always yields incomplete data (i.e., when only the selected items are answered, data are missing for the others) that are likely non‐ignorable in likelihood inference. Standard item response theory (IRT) models become infeasible when ESI data are missing not at random (MNAR). To solve this problem, the authors propose a two‐dimensional IRT model that posits one unidimensional IRT model for observed data and another for nominal selection patterns. The two latent variables are assumed to follow a bivariate normal distribution. In this study, the mirt freeware package was adopted to estimate parameters. The authors conduct an experiment to demonstrate that ESI data are often non‐ignorable and to determine how to apply the new model to the data collected. Two follow‐up simulation studies are conducted to assess the parameter recovery of the new model and the consequences for parameter estimation of ignoring MNAR data. The results of the two simulation studies indicate good parameter recovery of the new model and poor parameter recovery when non‐ignorable missing data were mistakenly treated as ignorable.     

Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis

Subscores are of increasing interest in educational and psychological testing due to their diagnostic function for evaluating examinees' strengths and weaknesses within particular domains of knowledge. Previous studies about the utility of subscores have mostly focused on the overall reliability of individual subscores and ignored the fact that subscores should be distinct and have added value over the total score. This study introduces a profile reliability approach that partitions the overall subscore reliability into within-person and between-person subscore reliability. The estimation of between-person reliability and within-person reliability coefficients is demonstrated using subscores from number-correct scoring, unidimensional and multidimensional item response theory scoring, and augmented scoring approaches via a simulation study and a real data study. The effects of various testing conditions, such as subtest length, correlations among subscores, and the number of subtests, are examined. Results indicate that there is a substantial trade-off between within-person and between-person reliability of subscores. Profile reliability coefficients can be useful in determining the extent to which subscores provide distinct and reliable information under various testing conditions.     

大五人格测验在中国应用的信度概化分析

对过去20年(1994~2013年)间国内有关大五人格测验的研究文献进行信度概化分析。结果表明:(1)检索到的文献中约68.15%存在"信度引入"现象;(2)未加权估计中,A和O的均值最低,N和C的均值最高,国内所得结果均略低于国外(O除外),而后者的变异性略大(E除外);采用α系数效果量方法,在随机效应模型中,N的估计值最高,O和A的估计值最低;(3)回归分析显示,分数均值、量表来源和南北地域差异是N维度信度的预测变量;量表来源、文章专业类型、测验版本和测验记分对E维度信度具有预测作用;样本量、文章专业类型和量表来源是O维度信度的预测变量;量表来源、文章专业类型、项目数和样本类型对A维度信度具有预测作用;量表来源、项目数、文章专业类型和测验记分是C维度信度的预测变量。     

Normal Personality Correlates of Chronic Pain Subgroups

A heterogeneous sample of chronic pain patients was first classified into one of three previously identified subgroups (i.e., dysfunctional, interpersonally distressed, adaptive) using the Multidimensional Pain Inventory (R. D. Kerns, D. C. Turk, & T. E. Rudy, 1985). Multiple differences at the domain and facet levels of the Revised NEO PI-R Personality Inventory (P. T. Costa, Jr. & R. R. McCrae, 1992) were found among the three subgroups of patients. In particular, the interpersonally distressed patients showed a greater degree of Neuroticism and were more introverted. Further analyses revealed that this group had tendencies toward feeling more vulnerable, being less trustful, and experiencing less positive emotions than the other groups. The dysfunctional group of patients was notable in terms of their low activity level and lower Openness (to new experience). Adaptive copers reported being emotionally stable, having a moderate degree of Extraversion, and being open to a wider variety of experiences. These differences may act as mediating influences in differing patient responsiveness to pain management programs.     

TILLICHIAN TELEODYNAMICS: AN EXAMINATION OF THE MULTIDIMENSIONAL UNITY OF EMERGENT LIFE

Abstract Emergence theory has generated many significant new questions for dialogue between theology and science. My work will examine the models of one emergence theorist, Terrence Deacon, and consider the constructive potential of Tillich's multidimensional unity of life for responding to the theological ramifications of this account of emergence theory. Such a Tillich‐inspired constructive process will rely upon Robert Russell's method of “Creative Mutual Interaction.” Building on the interactive quality of Russell's method, I will also begin to offer suggestions for how Tillich's theological themes might influence scientific research programs using Deacon's emergence theory by contributing to the process of defining life. Finally, I will conclude by identifying three facets of continued research that stem from this analysis, focusing primarily on its implications for theological anthropology and what it means to be in the image of God.     

Is Originality in the Eye of the Beholder? Comparison of Scoring Techniques in the Assessment of Divergent Thinking

Divergent thinking (DT) tests are among the most popular techniques for measuring creativity. However, the validity evidence for DT tests, as applied in educational settings, is inconsistent partly due to different scoring methods. This study explored the reliability and validity issues of various techniques for administering and scoring two DT tests. Results show distinct differences among several methods for scoring these DT tests and suggest that the percentage scoring method (i.e., dividing originality scores by fluency scores) may be the most appropriate scoring strategy. The potential impact on educational research and practice is discussed in detail.     

Hewitt多维完美主义量表在中学生群体中的修订

本研究以中学生群体为样本,对Hewitt多维完美主义量表(HMPS)进行修订。首先根据我国文化背景和青少年发展的具体状况建立了理论构想,接着对原量表进行了四次测查和检验修订,我们首先把指向他人和社会期望两个维度上的反向题改成正向题,并对所有项目的表述做了必要的修改;然后根据随后的测查结果进行了各维度和项目上的修改;最后删除了指向自我维度上的1个项目、指向他人维度上的2个项目和社会期望维度上的6个项目,修订后的HMPS量表的内部一致性信度α系数为0.91,指向自我、指向他人、社会期望的α系数分别为0.89,0.80,0.78,重测信度分别为0.80,0.79,0.67,与Frost多维完美主义量表中文修订版的相关分别为0.58,0.38,0.57。