Measurement of Ability in Adaptive Learning and Assessment Systems when Learners Use On-Demand Hints

Adaptive learning and assessment systems support learners in acquiring knowledge and skills in a particular domain. The learners’ progress is monitored through them solving items matching their level and aiming at specific learning goals. Scaffolding and providing learners with hints are powerful tools in helping the learning process. One way of introducing hints is to make hint use the choice of the student. When the learner is certain of their response, they answer without hints, but if the learner is not certain or does not know how to approach the item they can request a hint. We develop measurement models for applications where such on-demand hints are available. Such models take into account that hint use may be informative of ability, but at the same time may be influenced by other individual characteristics. Two modeling strategies are considered: (1) The measurement model is based on a scoring rule for ability which includes both response accuracy and hint use. (2) The choice to use hints and response accuracy conditional on this choice are modeled jointly using Item Response Tree models. The properties of different models and their implications are discussed. An application to data from Duolingo, an adaptive language learning system, is presented. Here, the best model is the scoring-rule-based model with full credit for correct responses without hints, partial credit for correct responses with hints, and no credit for all incorrect responses. The second dimension in the model accounts for the individual differences in the tendency to use hints.     

Criteria instability and the isolated option effect

In two-stage choices, decision makers often compare a new (isolated) option with the winner from the first stage. Previous research has identified a choice advantage for an isolated option, ostensibly due to loss aversion. We propose an alternative mechanism suggesting that instability of the criteria used in each choice stage is the main driver of the isolated option effect. Results from a series of experiments support the criteria instability account and not loss aversion as the explanation for the isolated option effect.     

Recognition memory: a review of the critical findings and an integrated theory for relating them

The development of formal models has aided theoretical progress in recognition memory research. Here, I review the findings that are critical for testing them, including behavioral and brain imaging results of single-item recognition, plurality discrimination, and associative recognition experiments under a variety of testing conditions. I also review the major approaches to measurement and process modeling of recognition. The review indicates that several extant dual-process measures of recollection are unreliable, and thus they are unsuitable as a basis for forming strong conclusions. At the process level, however, the retrieval dynamics of recognition memory and the effect of strengthening operations suggest that a recall-to-reject process plays an important role in plurality discrimination and associative recognition, but not necessarily in single-item recognition. A new theoretical framework proposes that the contribution of recollection to recognition depends on whether the retrieval of episodic details improves accuracy, and it organizes the models around the construct of efficiency. Accordingly, subjects adopt strategies that they believe will produce a desired level of accuracy in the shortest amount of time. Several models derived from this framework are shown to account the accuracy, latency, and confidence with which the various recognition tasks are performed.     

Outcomes as affirmation of membership value: Material compensation as an administrative response to procedural injustice

The current line of research suggests that the provision of compensation by group representatives may be an effective way to address the identity concerns resulting from procedural violations because compensation serves to reaffirm the victim’s membership value, protecting his or her identity. A series of five studies is presented, demonstrating that compensation can function symbolically as a legitimate act of concern for the injustice victim. Results showed that offers of compensation by group representatives resulted in more favorable evaluations of the group and higher identification than when no compensation was offered, but only when the compensation was construed as a benevolent gesture and only when the injustice was identity relevant. Even unsuccessful attempts to compensate the victim resulted in positive reactions towards the group. Consistent with relational models of procedural justice, these effects were mediated by perceptions of membership value.     

Children's knowledge of the earth: a new methodological and statistical approach

In the field of children’s knowledge of the earth, much debate has concerned the question of whether children’s naive knowledge—that is, their knowledge before they acquire the standard scientific theory—is coherent (i.e., theory-like) or fragmented. We conducted two studies with large samples (N = 328 and N = 381) using a new paper-and-pencil test, denoted the EARTH (EArth Representation Test for cHildren), to discriminate between these two alternatives. We performed latent class analyses on the responses to the EARTH to test mental models associated with these alternatives. The naive mental models, as formulated by Vosniadou and Brewer, were not supported by the results. The results indicated that children’s knowledge of the earth becomes more consistent as children grow older. These findings support the view that children’s naive knowledge is fragmented.     

MODELING RELATIONS

In the ordinary way of representing relations, the order of the relata plays a structural role, but in the states themselves such an order often does not seem to be intrinsically present. An alternative way to represent relations makes use of positions for the arguments. This is no problem for the love relation, but for relations like the adjacency relation and cyclic relations, different assignments of objects to the positions can give exactly the same states. This is a puzzling situation. The question is what is the internal structure of relations? Is the use of positions still justified, and if so, what is their ontological status? In this paper mathematical models for relations are developed that provide more insight into the structure of relations “out there” in the real world.     

Latent Class Models for Diary Method Data: Parameter Estimation by Local Computations

The increasing use of diary methods calls for the development of appropriate statistical methods. For the resulting panel data, latent Markov models can be used to model both individual differences and temporal dynamics. The computational burden associated with these models can be overcome by exploiting the conditional independence relations implied by the model. This is done by associating a probabilistic model with a directed acyclic graph, and applying transformations to the graph. The structure of the transformed graph provides a factorization of the joint probability function of the manifest and latent variables, which is the basis of a modified and more efficient E-step of the EM algorithm. The usefulness of the approach is illustrated by estimating a latent Markov model involving a large number of measurement occasions and, subsequently, a hierarchical extension of the latent Markov model that allows for transitions at different levels. Furthermore, logistic regression techniques are used to incorporate restrictions on the conditional probabilities and to account for the effect of covariates. Throughout, models are illustrated with an experience sampling methodology study on the course of emotions among anorectic patients. Frank Rijmen was partly supported by the Fund for Scientific Research Flanders (FWO).     

Computer simulation through an error-statistical lens

After showing how Deborah Mayo’s error-statistical philosophy of science might be applied to address important questions about the evidential status of computer simulation results, I argue that an error-statistical perspective offers an interesting new way of thinking about computer simulation models and has the potential to significantly improve the practice of simulation model evaluation. Though intended primarily as a contribution to the epistemology of simulation, the analysis also serves to fill in details of Mayo’s epistemology of experiment.     

Contributions of Neuroscience to Our Understanding of Cognitive Development

ABSTRACT— One major contribution of neuroscience to understanding cognitive development has been in demonstrating that biology is not destiny—that is, demonstrating the remarkable role of experience in shaping the mind, brain, and body. Only rarely has neuroscience provided wholly new insights into cognitive development, but often it has provided evidence of mechanisms by which observations of developmental psychologists could be explained. Behavioral findings have often remained controversial until an underlying biological mechanism for them was offered. Neuroscience has demonstrated promise for detecting cognitive problems before they are behaviorally observable—and, hence, promise for early intervention. In this article, we discuss examples drawn from imitation and mirror neurons, phenylketonuria (PKU) and prefrontal dopamine, maternal touch and stress reactivity, and nongenetic (behavioral) intergenerational transmission of biological characteristics.     

Optimal Scoring of the Multidimensional Pain Inventory in a Chronic Pain Sample

The Multidimensional Pain Inventory (MPI) is one of the most commonly used self-report instruments in pain settings. The MPI can be used to classify patients into three clusters or its nine scales can be treated as dimensions in efforts to understand patient heterogeneity. Previous research suggests the existence of a fourth cluster, whose members have been labeled ‘repressors,’ that emerges with the addition of a defensiveness scale to the MPI. The current paper compared the abilities of MPI cluster and dimensional models with and without a measure of defensiveness to capture variability in validating variables related to personality, psychopathology, physical functioning, and treatment outcome in a chronic pain sample. Results suggest that dimensional models consistently outperform cluster models in explaining variance in outcome variables, and that the addition of a measure of defensiveness increments the validity offered by the MPI scales. Implications for the assessment of pain patients are discussed.