Multilevel Reliability Measures of Latent Scores Within an Item Response Theory Framework

Abstract

This paper evaluated multilevel reliability measures in two-level nested designs (e.g., students nested within teachers) within an item response theory framework. A simulation study was implemented to investigate the behavior of the multilevel reliability measures and the uncertainty associated with the measures in various multilevel designs regarding the number of clusters, cluster sizes, and intraclass correlations (ICCs), and in different test lengths, for two parameterizations of multilevel item response models with separate item discriminations or the same item discrimination over levels. Marginal maximum likelihood estimation (MMLE)-multiple imputation and Bayesian analysis were employed to evaluate the accuracy of the multilevel reliability measures and the empirical coverage rates of Monte Carlo (MC) confidence or credible intervals. Considering the accuracy of the multilevel reliability measures and the empirical coverage rate of the intervals, the results lead us to generally recommend MMLE-multiple imputation. In the model with separate item discriminations over levels, marginally acceptable accuracy of the multilevel reliability measures and empirical coverage rate of the MC confidence intervals were found in a limited condition, 200 clusters, 30 cluster size, .2 ICC, and 40 items, in MMLE-multiple imputation. In the model with the same item discrimination over levels, the accuracy of the multilevel reliability measures and the empirical coverage rate of the MC confidence intervals were acceptable in all multilevel designs we considered with 40 items under MMLE-multiple imputation. We discuss these findings and provide guidelines for reporting multilevel reliability measures.     

Links between self-reported and laboratory behavioral impulsivity

Havik, M., Jakobson, A., Tamm, M., Paaver, M., Konstabel, K., Uusberg, A., Allik, J., Ööpik, V. & Kreegipuu, K. (2012). Links between self‐reported and laboratory behavioral impulsivity. Scandinavian Journal of Psychology 53, 216–223. A major problem in the research considering impulsivity is the lack of mutual understanding on how to measure and define impulsivity. Our study examined the relationship between self‐reported impulsivity, behavioral excitatory and inhibitory processes and time perception. Impulsivity – fast, premature, thoughtless or disinhibited behavior – was assessed in 58 normal, healthy participants (30 men, mean age 21.9 years). Self‐reported impulsivity as measured by Adaptive and Maladaptive Impulsivity Scale (AMIS) and behavioral excitatory and inhibitory processes as measured by Stop Signal Task were not directly related. Time perception, measured by the retrospective Time Estimation Task, was related to both. The length of the perceived time interval was positively correlated to AMIS Disinhibition subscale and negatively to several Stop Signal Task parameters. The longer subjects perceived the duration to last, the higher was their score on Disinhibition scale and the faster were their reactive responses in the Stop Signal Task. In summary our findings support the idea of cognitive tempo as a possible mechanism underlying impulsive behavior.     

An IRT analysis of motive questionnaires: The Unified Motive Scales

Multiple inventories claiming to assess the same explicit motive (achievement, power, or affiliation) show only mediocre convergent validity. In three studies (N = 1685) the structure, nomological net, and content coverage of multiple existing motive scales was investigated with exploratory factor analyses. The analyses revealed four approach factors (achievement, power, affiliation, and intimacy) and a general avoidance factor with a facet structure. New scales (the Unified Motive Scales; UMS) were developed using IRT, reflecting these underlying dimensions. In comparison to existing questionnaires, the UMS have the highest measurement precision and provide short (six-item) and ultra-short (three-item) scales. In a fourth study (N = 96), the UMS demonstrated incremental validity over existing motive scales with respect to several outcome criteria.     

Happy to See Me,Aren’t You,Sally? Signal Detection Analysis of Emotion Detection in Briefly Presented Male and Female Faces

Using signal detection methods, possible effects of emotion type (happy, angry), gender of the stimulus face, and gender of the participant on the detection and response bias of emotion in briefly presented faces were investigated. Fifty-seven participants (28 men, 29 women) viewed 90 briefly presented faces (30 happy, 30 angry, and 30 neutral, each with 15 male and 15 female faces) answering yes if the face was perceived as emotional and no if it was not perceived as emotional. Sensitivity [d', z(hit rate) minus z(false alarm rate)] and response bias (β, likelihood ratio of "signal plus noise" vs. "noise") were measured for each face combination for each presentation time (6.25, 12.50, 18.75, 25.00, 31.25 ms). The d' values were higher for happy than for angry faces and higher for angry-male than for angry-female faces, and there were no effects of gender-of-participant. Results also suggest a greater tendency for participants to judge happy-female faces as emotional, as shown by lower β values for these faces as compared to the other emotion-gender combinations. This happy-female response bias suggests, at least, a partial explanation to happy-superiority effects in studies where performance is only measured as percent correct responses, and, in general, that women are expected to be happy.     

On the development and mechanics of delayed matching-to-sample performance

Despite its frequent use to assess effects of environmental and pharmacological variables on short-term memory, little is known about the development of delayed matching-to-sample (DMTS) performance. This study was designed to examine the dimensions and dynamics of DMTS performance development over a long period of exposure to provide a more secure foundation for assessing stability in future research. Six pigeons were exposed to a DMTS task with variable delays for 300 sessions (i.e., 18,000 total trials; 3,600 trials per retention interval). Percent-correct and log-d measures used to quantify the development of conditional stimulus control under the procedure generally and at each of five retention intervals (0, 2, 4, 8 and 16-s) individually revealed that high levels of accuracy developed relatively quickly under the shorter retention intervals, but increases in accuracy under the longer retention intervals sometimes were not observed until 100-150 sessions had passed, with some still increasing at Session 300. Analyses of errors suggested that retention intervals induced biases by shifting control from the sample stimulus to control by position, something that was predicted by observed response biases during initial training. These results suggest that although it may require a great deal of exposure to DMTS prior to obtaining asymptotic steady state, quantification of model parameters may help predict trends when extended exposure is not feasible.     

The effects of romantic partners’ goal congruence on affective well-being

Affective well-being in romantic couples was examined from the perspective of interdependence theory. The independent variables were (a) presence of partner, (b) whether an activity met the actor’s goals, and (c) goals of the actor’s partner. Dependent variables were feelings of closeness and affective well-being (happiness, sadness, anger, anxiety). We predicted a three-way interaction with the highest affective well-being when partners are together and activities meet both partners’ goals. In Study 1, data from 194 married individuals who participated in an experience sampling study supported our predictions. Feelings of closeness partially mediated the effect on affective well-being. Study 2 replicated the findings with 112 participants in dating relationships who recalled specific events and made ratings about goals and affective well-being.     

Musicians and tone-language speakers share enhanced brainstem encoding but not perceptual benefits for musical pitch

Behavioral and neurophysiological transfer effects from music experience to language processing are well-established but it is currently unclear whether or not linguistic expertise (e.g., speaking a tone language) benefits music-related processing and its perception. Here, we compare brainstem responses of English-speaking musicians/non-musicians and native speakers of Mandarin Chinese elicited by tuned and detuned musical chords, to determine if enhancements in subcortical processing translate to improvements in the perceptual discrimination of musical pitch. Relative to non-musicians, both musicians and Chinese had stronger brainstem representation of the defining pitches of musical sequences. In contrast, two behavioral pitch discrimination tasks revealed that neither Chinese nor non-musicians were able to discriminate subtle changes in musical pitch with the same accuracy as musicians. Pooled across all listeners, brainstem magnitudes predicted behavioral pitch discrimination performance but considering each group individually, only musicians showed connections between neural and behavioral measures. No brain-behavior correlations were found for tone language speakers or non-musicians. These findings point to a dissociation between subcortical neurophysiological processing and behavioral measures of pitch perception in Chinese listeners. We infer that sensory-level enhancement of musical pitch information yields cognitive-level perceptual benefits only when that information is behaviorally relevant to the listener.     

Impact of motivation on cognitive control in the context of vigilance lowering: an ERP study

We assessed the effects of time-on-task on cognitive control expressed by the CRN/Nc and the extent to which motivation modulates this relationship. We utilized two groups of participants, who were told that their performance would (evaluation condition) or would not (control condition) be evaluated online. Both groups performed a version of the Eriksen Flanker Task for 60 min. We observed classical vigilance lowering, manifested by a progressive performance decline with time-on-task, in the control, but not in the evaluation, condition. In the latter, performance remained stable throughout the task. ERP analysis indicated the same interaction in our main component of interest, the CRN/Nc, whose amplitude decreased from the first to the last period in the control condition but remained stable over time in the evaluation condition. To our knowledge, this study is the first to demonstrate the impact of motivation on monitoring processes as indexed by the correct response negativity, in the context of a prolonged task. Vigilance lowering caused by a repetitive and prolonged flanker task, results in compromised response control and compromised control of correct responses. Our results suggest that alterations in ACC functioning may underlie vigilance decline and can be viewed as evidence that the action monitoring functions of the ACC can be positively affected by motivation.     

The size coding of responses: The size of switches and the force feedback

We aimed to understand which factors have a functional role in the size coding of responses, either the size of the switches or the force required to trigger each switch. This question is of relevance because it allows a better understanding of processes underlying action coding. In each trial, participants saw a small or large object. Depending on its colour, the participants had to press one of two switches. In the “size” condition, the response device consisted of two switches of different visual size, but both required the same amount of force. In the “force-feedback” condition, the response device consisted in two switches of identical visual size, but one switch required more force than the other. We found a compatibility effect in the “size,” not in the “force-feedback” condition, supporting that the size-coding of responses would be due to the size of the switches.     

情绪Stroop效应：威胁驱动还是反应偏差？

为了考查情绪Stroop效应的产生机制是威胁驱动还是反应偏差,本研究进行了两个实验.实验一用一般负性情绪词语和中性词语作为实验材料,实验二用威胁性的情绪图片和中性图片作为实验材料.每个实验均要求一组被试做按键反应,另一组被试做摇杆反应.结果发现,当实验材料为词语时,未出现情绪Stroop效应;当实验材料为威胁性图片时,无论是趋向性反应方式还是回避性反应方式,都出现了情绪Stroop效应.本研究结果支持威胁驱动机制.