The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments

Automated essay scoring engines (AESEs) are becoming increasingly popular as an efficient method for performance assessments in writing, including many language assessments that are used worldwide. Before they can be used operationally, AESEs must be “trained” using machine-learning techniques that incorporate human ratings. However, the quality of the human ratings used to train the AESEs is rarely examined. As a result, the impact of various rater effects (e.g., severity and centrality) on the quality of AESE-assigned scores is not known. In this study, we use data from a large-scale rater-mediated writing assessment to examine the impact of rater effects on the quality of AESE-assigned scores. Overall, the results suggest that if rater effects are present in the ratings used to train an AESE, the AESE scores may replicate these effects. Implications are discussed in terms of research and practice related to automated scoring.     

Investigating How Test-Takers Change Their Strategies to Handle Difficulty in Taking a Reading Comprehension Test: Implications for Score Validation

This article investigates how test-takers change their strategies to handle increased test difficulty. An adult sample reported their test-taking strategies immediately after completing the tasks in a reading test. Data were analyzed using structural equation modeling specifying a measurement-invariant, ability-moderated, latent transition analysis in Mplus (Muthén & Asparouhov, 2011). It was found that almost half of the test-takers (47%) changed their strategies when encountering increased task-difficulty. The changes were characterized by augmenting comprehending-meaning strategies with score-maximizing and test-wiseness strategies. Moreover, test-takers' ability was the driving influence that facilitated and/or buffered the changes. The test outcomes, when reviewed in light of adjusted test-taking strategies, demonstrated a form of process-based validity evidence.     

Intersectional experiences of discrimination in a low‐resource urban community: An exploratory latent class analysis

There have been increasing calls for the application of an intersectionality framework to understand and address discrimination and health inequities among diverse communities. Yet there have been theoretical debates regarding to whom intersectionality applies and how intersectional experiences of discrimination are associated with health outcomes. The current study aimed to contribute to these theoretical debates and inform practical applications to reduce health inequities. Data were drawn from a community health survey in New Haven, CT (N = 1,293 adults) and analysed using latent class analysis. Results yielded 4 classes. Members of the 4 classes were similar sociodemographically. Three classes of participants reported experiencing discrimination, and members of these classes had greater stress, higher rates of smoking and sleep disruption, and worse overall health than members of the class reporting no discrimination. Members of 2 classes made multiple, or intersectional, attributions for discrimination, and members of these classes reported the most frequent discrimination. Findings suggest that community members who are sociodemographically similar may have diverse discrimination experiences. Multilevel interventions that address multiple forms of discrimination (e.g., racism and sexism) may hold promise for reducing discrimination and, ultimately, health inequities within low‐resource urban community settings.     

Exploring the relationship between executive functions and self-reported media-multitasking in young adults

Media-multitasking involves simultaneous engagement with information streams from multiple media sources, and is most prevalent in young adults. Heavy media-multitasking has been associated with differential performance on tasks involving attentional control and working memory relative to light media-multitasking. The aim of the present study was to systematically investigate relationships between executive functions and self-reported media-multitasking. Healthy participants (N?=?112, aged 18–25, male N?=?36) completed a battery of 10 traditional executive function tasks, that included assessments of attentional inhibition, response inhibition, working memory, and cognitive flexibility. Scores on the individual executive function tasks were correlated against frequency of self-reported media-multitasking, but no significant relationships were found. Trait anxiety, however, was found to be significantly associated with greater frequency of self-reported media-multitasking. The present study found no evidence of a relationship between the frequency of self-reported media-multitasking and executive functioning. The possible reasons for this are discussed.     

Two-Stage maximum likelihood estimation in the misspecified restricted latent class model

The maximum likelihood classification rule is a standard method to classify examinee attribute profiles in cognitive diagnosis models (CDMs). Its asymptotic behaviour is well understood when the model is assumed to be correct, but has not been explored in the case of misspecified latent class models. This paper investigates the asymptotic behaviour of a two-stage maximum likelihood classifier under a misspecified CDM. The analysis is conducted in a general restricted latent class model framework addressing all types of CDMs. Sufficient conditions are proposed under which a consistent classification can be obtained by using a misspecified model. Discussions are also provided on the inconsistency of classification under certain model misspecification scenarios. Simulation studies and a real data application are conducted to illustrate these results. Our findings can provide some guidelines as to when a misspecified simple model or a general model can be used to provide a good classification result.     

Subjective,physiological, and behavioural responses towards evaluatively conditioned stimuli

Evaluative Conditioning (EC) is commonly defined as the change in liking of a stimulus (conditioned stimulus, CS) due to its pairings with an affective unconditioned stimulus (US). In Experiment 1, we investigated effects of repeated stimulus pairings on affective responses, i.e. valence and arousal ratings, pupil size, and duration estimation. After repeatedly pairing the CSs with affective USs, a consistent pattern of affective responses emerged: The CS_negative was rated as being more negative and more arousing, resulted in larger pupils, and was temporally overestimated compared to the CS_neutral. In Experiment 2, the influence of a mere instruction about the contingency between a CS and US on affective responses was examined. After mere instruction about upcoming pairings between the CS and US, subjective ratings also changed, but there was neither evidence for differential pupillary responses nor for differential temporal processing. The results indicate that EC via pairings or instructions can change the affective responses towards formerly neutral stimuli and introduce pupil size as a physiological measure in EC research. However, Experiment 2 suggests that there might be moderating factors based on the type of EC procedure involved.     

农村留守儿童焦虑情绪的异质性:基于潜在剖面分析

采用中学生心理健康诊断量表（MHT）对湖南省1292名留守儿童的焦虑情绪进行施测,并用潜在剖面分析方法探究留守儿童的焦虑情绪问题是否存在异质性,以及留守儿童相关的人口学因素对焦虑情绪异质性的作用机制。结果发现,留守儿童的焦虑情绪存在明显的异质性,可以分为三种潜在类别：“严重焦虑情绪型”、“中等程度焦虑情绪型”和“低焦虑情绪型”;留守男生、小学五年级、高中二年级、留守时间越长、留守初始年龄越小的留守儿童更容易焦虑情绪严重;无论是父亲外出、母亲外出还是双亲外出对留守儿童的焦虑情绪严重性的影响是一致的。由此,应根据留守儿童的性别、年级、留守时长和留守的初始年龄等情况特点,开展针对性的预防和干预措施,缓解留守儿童焦虑。     

Latent class analysis of indicators of intolerance of uncertainty

Intolerance of Uncertainty (IU ) is a transdiagnostic vulnerability factor involved in depression and anxiety symptoms and disorders. IU encompasses Prospective IU (“Unforeseen events upset me greatly”) and Inhibitory IU (“The smallest doubt can stop me from acting”). Research has yet to explore whether subgroups or classes of people exist characterized by different profiles of IU . This study used latent class analysis to identify such subgroups and examined if different classes of IU were distinct in terms of several cognitive vulnerabilities and psychological symptoms. Data were obtained from 519 students completing a 12‐item measure of IU . Four subgroups were identified, characterized by low IU , predominantly Prospective IU , predominantly Inhibitory IU , and high IU , respectively. People in the high IU class reported cognitive vulnerabilities and depression and anxiety more than people in the low IU class. Inhibitory IU was more strongly associated with poor outcomes than was Prospective IU .