Multilevel Reliability Measures of Latent Scores Within an Item Response Theory Framework

Abstract

This paper evaluated multilevel reliability measures in two-level nested designs (e.g., students nested within teachers) within an item response theory framework. A simulation study was implemented to investigate the behavior of the multilevel reliability measures and the uncertainty associated with the measures in various multilevel designs regarding the number of clusters, cluster sizes, and intraclass correlations (ICCs), and in different test lengths, for two parameterizations of multilevel item response models with separate item discriminations or the same item discrimination over levels. Marginal maximum likelihood estimation (MMLE)-multiple imputation and Bayesian analysis were employed to evaluate the accuracy of the multilevel reliability measures and the empirical coverage rates of Monte Carlo (MC) confidence or credible intervals. Considering the accuracy of the multilevel reliability measures and the empirical coverage rate of the intervals, the results lead us to generally recommend MMLE-multiple imputation. In the model with separate item discriminations over levels, marginally acceptable accuracy of the multilevel reliability measures and empirical coverage rate of the MC confidence intervals were found in a limited condition, 200 clusters, 30 cluster size, .2 ICC, and 40 items, in MMLE-multiple imputation. In the model with the same item discrimination over levels, the accuracy of the multilevel reliability measures and the empirical coverage rate of the MC confidence intervals were acceptable in all multilevel designs we considered with 40 items under MMLE-multiple imputation. We discuss these findings and provide guidelines for reporting multilevel reliability measures.     

Study for now,but judge for later: Delayed judgments of learning promote long‐term retention

Larsson Sundqvist, M., Todorov, I., Kubik, V. & Jönsson, F.U. (2012) Study for now, but judge for later: Delayed judgments of learning promote long‐term retention. Scandinavian Journal of Psychology 53, 450–454. Delayed judgments of learning (JOL) are assumed to be based on covert retrieval attempts. A common finding is that testing memory during learning improves later retention (i.e., the testing effect), and even more so than an equivalent amount of study, but only after a longer retention interval. To test the assertion that also delayed JOLs improve memory, the participants either studied Swahili‐Swedish word pairs four times, or they both studied (two times) and performed delayed JOLs (two times) alternately. Final cued recall test were given after either five minutes or one week. Results showed a reliable learning‐group by retention‐interval interaction, with less forgetting in the group that alternated between studying and making JOLs. The results are discussed in relation to the self‐fulfilling prophecy account of Spellman and Bjork (1992) , and in terms of study advice, the results further underscore the importance of delaying JOLs when studying and evaluating one’s ongoing learning.     

Stopping speech suppresses the task-irrelevant hand

Some situations require one to quickly stop an initiated response. Recent evidence suggests that rapid stopping engages a mechanism that has diffuse effects on the motor system. For example, stopping the hand dampens the excitability of the task-irrelevant leg. However, it is unclear whether this ‘global suppression’ could apply across wider motor modalities. Here we tested whether stopping speech leads to suppression of the task-irrelevant hand. We used Transcranial Magnetic Stimulation over the primary motor cortex with concurrent electromyography from the hand. We found that when speech was successfully stopped the motor evoked potential from the task-irrelevant hand was significantly reduced compared to when the participant failed to stop speaking, or responded on non stop signal trials, or compared to baseline. This shows that when speech is quickly stopped, there is a broad suppression across the motor system. This has implications for the neural basis of speech control and stuttering.     

Dissociable stages of problem solving (II): first evidence for process-contingent temporal order of activation in dorsolateral prefrontal cortex

In a companion study, eye-movement analyses in the Tower of London task (TOL) revealed independent indicators of functionally separable cognitive processes during problem solving, with processes of building up an internal representation of the problem preceding actual planning processes. These results imply that processes of internalization and planning should also be distinguishable in time and space with respect to concomitant brain activation patterns. To investigate this possibility, here we conducted analyses of fMRI data for left and right dorsolateral prefrontal cortex (dlPFC) during problem solving in the TOL task by accounting for the trial-by-trial variability of onsets and durations of the different cognitive processing stages. Comparisons between stimulus-locked and response-locked modeling approaches affirmed that activation in left dlPFC was elicited particularly during early processes of internalization, comprising the extraction of goal information and the generation of an internal problem representation, whereas activation in right dlPFC was predominantly attributable to later processes of mental transformations on this representation, that is planning proper. Thus, present data corroborate the proposal that often observed bilateral dlPFC activation patterns during complex cognitive tasks such as problem solving may reflect functionally and, to some extent, even temporally separable processes with opposing lateralizations.     

Dissociable stages of problem solving (I): temporal characteristics revealed by eye-movement analyses

Understanding the functional neuroanatomy of planning and problem solving may substantially benefit from better insight into the chronology of the cognitive processes involved. Based on the assumption that regularities in cognitive processing are reflected in overtly observable eye-movement patterns, here we recorded eye movements while participants worked on Tower of London (TOL) problems that comprised an experimental manipulation of different task demands. Single-trial saccade-locked analyses revealed that higher demands on forming an internal problem representation were associated with an increased number of gaze alternations between start state and goal state, but did not show any effect on the durations of these inspections of the states. In contrast, higher demands on actual planning in terms of mental manipulations of working memory contents coincided with a prolonged duration of the very last inspection of the start state (i.e., immediately preceding movement execution) but did not show any effect on the number of gaze alterations. Differential task demands on internalization and planning processes during problem solving hence selectively affect different eye-movement parameters. Moreover, these findings complement previous neuroimaging data on dissociable contributions of left and right dorsolateral prefrontal cortex in problem solving with novel evidence for a corresponding dissociation in the eye-movement patterns reflecting the associated cognitive processes.     

RACE/A: an architectural account of the interactions between learning, task control, and retrieval dynamics

This article discusses how sequential sampling models can be integrated in a cognitive architecture. The new theory Retrieval by Accumulating Evidence in an Architecture (RACE/A) combines the level of detail typically provided by sequential sampling models with the level of task complexity typically provided by cognitive architectures. We will use RACE/A to model data from two variants of a picture-word interference task in a psychological refractory period design. These models will demonstrate how RACE/A enables interactions between sequential sampling and long-term declarative learning, and between sequential sampling and task control. In a traditional sequential sampling model, the onset of the process within the task is unclear, as is the number of sampling processes. RACE/A provides a theoretical basis for estimating the onset of sequential sampling processes during task execution and allows for easy modeling of multiple sequential sampling processes within a task.     

The effect of food toppings on calorie estimation and consumption

In this research we examine the effect of adding a food topping to the base food on consumers' calorie estimation and consumption of the augmented food (base food plus topping). We show that consumers underestimate the calorie content of augmented food with an unhealthy base, especially when the topping is healthy. However, consumers are less likely to underestimate the calorie content of augmented food with a healthy base, regardless of whether the topping is healthy or unhealthy. Further, we show that adding a healthy topping to an unhealthy base makes consumers not only underestimate the calorie content but also eat more of the augmented food.     

Examining associations between self-conscious emotions and implicit and reflective processes among adolescent girls and boys

ObjectiveExamine the association between self-conscious emotions and adolescents’ implicit attitudes (i.e., automatic evaluations or “gut reactions”) towards sport.DesignIn this cross-sectional design, 162 adolescents completed self-report questionnaires and a single category implicit association task. Study protocols transitioned from in-person to online at the start of the COVID-19 pandemic.MeasuresSelf-conscious emotions (i.e., fitness-related shame and pride) were assessed using the BSE-FIT (Castonguay et al., 2016), and sport enjoyment (i.e., reflective attitudes) was assessed using four items from Scanlan and colleagues measure of adolescents’ sport experiences (Scanlan et al., 1993). The Single-Category Implicit Association Task using images was used to measure their implicit attitudes towards sport (Karpinski & Steinman, 2006).ResultsBased on a structural equation model accounting for age and gender, higher levels of fitness pride related to higher sport enjoyment and higher levels of fitness shame related to less favourable implicit attitudes towards sport.ConclusionOverall, results might reflect differences in when self-conscious emotions are most relevant, whereby pride is more relevant when reflecting on experiences in sport, however fitness shame is related to implicit attitudes elicited from the sport context. Interventions that target affective judgments of the self may be useful to incorporate into strategies to reduce sport drop out in adolescents.     

潜变量交互效应标准化估计：方法比较与选用策略

标准化估计对模型的解释和效应大小的比较有重要作用。虽然潜变量交互效应的恰当标准化估计公式已经面世超过10年, 国内外都在使用和引用, 但至今未见到关于不同估计方法得到的恰当标准化估计的系统比较。通过模拟实验, 比较了乘积指标法、潜调节结构方程(LMS)、无先验信息和有先验信息的贝叶斯法的潜变量交互效应标准化估计在不同条件下的表现。结果发现, 在正态条件下, LMS和有信息贝叶斯法表现较好; 而在非正态条件下, 乘积指标法比较稳健, 但需要较大的样本(不小于500), 小样本且外生潜变量之间相关很低时可使用无信息贝叶斯法。