Developing new online calibration methods for multidimensional computerized adaptive testing

Multidimensional computerized adaptive testing (MCAT) has received increasing attention over the past few years in educational measurement. Like all other formats of CAT, item replenishment is an essential part of MCAT for its item bank maintenance and management, which governs retiring overexposed or obsolete items over time and replacing them with new ones. Moreover, calibration precision of the new items will directly affect the estimation accuracy of examinees’ ability vectors. In unidimensional CAT (UCAT) and cognitive diagnostic CAT, online calibration techniques have been developed to effectively calibrate new items. However, there has been very little discussion of online calibration in MCAT in the literature. Thus, this paper proposes new online calibration methods for MCAT based upon some popular methods used in UCAT. Three representative methods, Method A, the ‘one EM cycle’ method and the ‘multiple EM cycles’ method, are generalized to MCAT. Three simulation studies were conducted to compare the three new methods by manipulating three factors (test length, item bank design, and level of correlation between coordinate dimensions). The results showed that all the new methods were able to recover the item parameters accurately, and the adaptive online calibration designs showed some improvements compared to the random design under most conditions.     

Non‐ignorable missingness item response theory models for choice effects in examinee‐selected items

Examinee‐selected item (ESI) design, in which examinees are required to respond to a fixed number of items in a given set, always yields incomplete data (i.e., when only the selected items are answered, data are missing for the others) that are likely non‐ignorable in likelihood inference. Standard item response theory (IRT) models become infeasible when ESI data are missing not at random (MNAR). To solve this problem, the authors propose a two‐dimensional IRT model that posits one unidimensional IRT model for observed data and another for nominal selection patterns. The two latent variables are assumed to follow a bivariate normal distribution. In this study, the mirt freeware package was adopted to estimate parameters. The authors conduct an experiment to demonstrate that ESI data are often non‐ignorable and to determine how to apply the new model to the data collected. Two follow‐up simulation studies are conducted to assess the parameter recovery of the new model and the consequences for parameter estimation of ignoring MNAR data. The results of the two simulation studies indicate good parameter recovery of the new model and poor parameter recovery when non‐ignorable missing data were mistakenly treated as ignorable.     

Assessment of Differential Rater Functioning in Latent Classes with New Mixture Facets Models

Multifaceted data are very common in the human sciences. For example, test takers' responses to essay items are marked by raters. If multifaceted data are analyzed with standard facets models, it is assumed there is no interaction between facets. In reality, an interaction between facets can occur, referred to as differential facet functioning. A special case of differential facet functioning is the interaction between ratees and raters, referred to as differential rater functioning (DRF). In existing DRF studies, the group membership of ratees is known, such as gender or ethnicity. However, DRF may occur when the group membership is unknown (latent) and thus has to be estimated from data. To solve this problem, in this study, we developed a new mixture facets model to assess DRF when the group membership is latent and we provided two empirical examples to demonstrate its applications. A series of simulations were also conducted to evaluate the performance of the new model in the DRF assessment in the Bayesian framework. Results supported the use of the mixture facets model because all parameters were recovered fairly well, and the more data there were, the better the parameter recovery.     

A Hierarchical Rater Model for Longitudinal Data

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM.     

“To‐be‐forgotten” statements become less true: Memory processes involved in selection and forgetting lead to truthfulness changes of ambiguous sentences

What happens when people try to forget something? What are the consequences of instructing people to intentionally forget a sentence? Recent studies employing the item‐method directed forgetting paradigm have shown that to‐be‐forgotten (TBF) items are, in a subsequent task, emotionally devaluated relative to to‐be‐remembered (TBR) items, an aftereffect of memory selection (Vivas, Marful, Panagiotidou & Bajo, 2016). As such, distractor devaluation by attentional selection generalizes to memory selection. In this study, we use the item‐method directed forgetting paradigm to test the effects of memory selection and inhibition on truth judgments of ambiguous sentences. We expected the relative standing of an item in the task (i.e., whether it was instructed to be remembered or forgotten) to affect the truthfulness value of that item, making TBF items less valid/truthful than TBR items. As predicted, ambiguous sentences associated with a “Forget” cue were subsequently judged as less true than sentences associated with a “Remember” cue, suggesting that instructions to intentionally forget a statement can produce changes in the validity/truthfulness of that statement. To our knowledge, this is the first study to show an influence of memory processes involved in selection and forgetting on the perceived truthfulness of sentences.     

工作记忆成分的年龄相关差异对算术策略运用的预测效应

采用选择/无选范式,借助工作记忆成套测验,在两位数乘法估算问题中探讨了工作记忆系统各成分对不同年龄段个体算术策略运用的预测效应。结果显示:(1)工作记忆的不同成分与年龄之间存在明显的相关。表现为,除视空模板成分外,其他各成分得分随着年龄增长而呈现上升趋势;(2)估算策略运用中,年龄与策略选择显著相关,表现为随着年龄增长,策略选择表现明显提高;(3)估算策略运用中,不同年龄个体的工作记忆不同成分和策略选择表现出不同的联系,中央执行均显示出显著的预测效应,语音环路和视空模板的预测效应均不显著。不同年龄个体的工作记忆不同成分对策略执行的预测效应均不显著。上述发现对于深刻理解工作记忆系统在算术认知策略运用中的作用机制具有重要理论含义。     

Holistic Information Integration in Child Decision Making

Weighted‐additive (WADD) strategies require decision makers to integrate multiple values weighted by their relevance. From what age can children make choices in line with such a WADD‐strategy? We compare multi‐attribute decisions of children (6–7; 8–10; 11–12‐year‐olds) with adults in an open information‐board environment without pre‐decisional information search. In two experiments, we classify decision strategies based on individual choice patterns and find that in all age groups the majority of participants are users of a WADD‐strategy. Simple decision heuristics such as lexicographic strategies were applied rarely by children and not at all by adults. In two additional follow‐up studies, we further investigate the underlying process of WADD‐application by analysing decision latencies in combination with a retrospective think‐aloud study. Results suggest that children did not apply WADD‐strategies in a deliberate fashion in our experiments. Overall, our findings demonstrate that the ability to make good and quick decisions by holistically integrating information is already present in young children. Copyright © 2017 John Wiley & Sons, Ltd.     

The Influence of Time Estimation and Time‐Saving Preferences on Learning to Make Temporally Dependent Decisions from Experience

In this paper, we merge research related to experiential learning, temporal perception, and the value of time and money by examining decisions where the timing of action (response) determines the outcome received. We predicted that time‐saving preferences and impatience would decrease maximization (i.e., taking action when it returned the largest reward), and that the constraints of temporal perception would compound their effects. Across three studies, participants undershot on average (i.e., responded earlier than the period of time during which a response would return the maximal reward) showed a preference for shorter‐delay options and often did not find the maximal reward. In addition, participants' reliance on temporal perception increased undershooting, increased preferences for shorter‐delay options, and reduced maximization. Nevertheless, participants who found the maximal reward continued to maximize at a high rate rather than opting for shorter delays and smaller rewards. Thus, while most participants appeared to have a preference for saving time, most behaved as reward maximizers rather than temporal discounters. Copyright © 2017 John Wiley & Sons, Ltd.     

What did Popper learn from Lakatos?

The canonical version of the history of twentieth century philosophy of science tells us that Lakatos was Popper’s disciple, but it is rarely mentioned that Popper would have learned anything from Lakatos. The aim of this paper is to examine Lakatos’ influence on Popper’s philosophical system and to argue that Lakatos did have an important, yet somewhat unexpected, impact on Popper’s thinking: he influenced Popper’s evolutionary model for ‘progress’ in science. And Lakatos’ influence sheds new light on why and how Popper continually revised one of the central claims of his philosophy of science: the evolutionary account of scientific theory change.