The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments

Automated essay scoring engines (AESEs) are becoming increasingly popular as an efficient method for performance assessments in writing, including many language assessments that are used worldwide. Before they can be used operationally, AESEs must be “trained” using machine-learning techniques that incorporate human ratings. However, the quality of the human ratings used to train the AESEs is rarely examined. As a result, the impact of various rater effects (e.g., severity and centrality) on the quality of AESE-assigned scores is not known. In this study, we use data from a large-scale rater-mediated writing assessment to examine the impact of rater effects on the quality of AESE-assigned scores. Overall, the results suggest that if rater effects are present in the ratings used to train an AESE, the AESE scores may replicate these effects. Implications are discussed in terms of research and practice related to automated scoring.     

建构反应题中能力估计准确性的影响因素:评分者人数和项目个数的交互作用

采用康春花、孙小坚和曾平飞(2016)提出的等级反应多水平侧面模型探讨了评分者人数和项目个数对被试能力估计准确性的影响。模拟研究的结果表明:(1)随着项目个数的增加,估计值与真值之间的相关也不断增加;(2)评分者人数和项目个数在平均绝对偏差(MAB)和误差均方根(RMSE)上的主效应均显著,两者间的交互效应也显著;(3)简单效应分析发现,当项目较少时,3个评分者条件下的能力估计准确性最好; 随着项目个数的增加,4个评分者的估计误差迅速下降,且表现变为最好。     

Serotonin transporter 5-HTTLPR genotype is associated with intrusion and avoidance symptoms of DSM-5 posttraumatic stress disorder (PTSD) in Chinese earthquake survivors

Background and Objectives: Prior studies have found that the serotonin transporter gene-linked polymorphic region (5-HTTLPR) interacts with trauma exposure to increase general risk for Posttraumatic Stress Disorder (PTSD). However, there is little knowledge about the effects of the interaction on distinct symptom clusters of PTSD. This study aimed to investigate the relation between the interaction of 5-HTTLPR and earthquake-related exposures and a contemporary phenotypic model of DSM-5 PTSD symptoms in a traumatised adult sample from China.

Design: A cross-sectional design with gene-environment interaction (G?×?E) approach was adopted. Methods: Participants were 1131 survivors who experienced 2008 Wenchuan earthquake. PTSD symptoms were assessed with the PTSD Checklist for DSM-5 (PCL-5). The 5-HTTLPR polymorphism was genotyped with capillary electrophoresis (CE) in ABI 3730xl genetic Analyzer.

Results: Although there was no significant interaction between 5-HTTLPR and traumatic exposure on total PTSD symptoms, respondents with the LL genotype of 5-HTTLPR who were highly exposed to the earthquake experienced lower intrusion and avoidance symptoms than those with the S-allele carriers.

Conclusions: The findings suggest that the 5-HTTLPR may have an important impact on the development of PTSD and add to the extant knowledge on understanding and treating of posttraumatic psychopathology.     

Discounting: A practical guide to multilevel analysis of choice data

Multilevel modeling provides the ability to simultaneously evaluate the discounting of individuals and groups by examining choices between smaller sooner and larger later rewards. A multilevel logistic regression approach is advocated in which sensitivity to relative reward magnitude and relative delay are considered as separate contributors to choice. Examples of how to fit choice data using multilevel logistic models are provided to help researchers in the adoption of these methods.     

The longitudinal association between internalizing symptoms and academic achievement among immigrant and non‐immigrant children in Norway

The aim of this study was to explore the longitudinal relation between internalizing symptoms and academic achievement, as two processes of children and youth development, among children in Norway, and whether having an immigrant background moderated this association. Data collected from 4,458 students in Norway in four waves over three years were analyzed with multi‐group latent growth curve modeling (LGM ). Results showed that internalizing symptoms level remained unchanged over time both for immigrant and non‐immigrant children, while levels of academic achievement increased only for children of immigrants with both parents born outside of Norway. Further analyses supported a reciprocal relation between internalizing symptoms and academic achievement and revealed that the initial level of academic achievement predicted the rate of change in internalizing symptoms over time, but not vice versa. Moreover, immigrant background did not moderate the associations in the model, however, children of immigrants with both parents born abroad initially had lower levels of academic achievement, but showed an increase in academic achievement, compared to their non‐immigrant peers as well as to peers with one native‐born parent after controlling for gender and their grade at the first observation. The implications for policy and practice were discussed.     

Latent Growth Curve and Repeated Measures ANOVA Contrasts: What the Models are Telling You

The relationship between the latent growth curve and repeated measures ANOVA models is often misunderstood. Although a number of investigators have looked into the similarities and differences among these models, a cursory reading of the literature can give the impression that they are very different models. Here we show that each model represents a set of contrasts on the occasion means. We demonstrate that the fixed effects parameters of the estimated basis vector latent growth curve model are merely a transformation of the repeated measures ANOVA fixed effects parameters. We further show that differences in fit in models that estimate the same means structure can be due to the different error covariance structures implied by the model. We show these relationships both algebraically and through using data from a simulation.     

死亡心理:外部防御还是内在成长?

对西方主要的死亡心理研究进行了回顾和比较分析,不同的死亡心理研究在研究传统、研究对象和研究方法方面存在差异。但是根据研究的侧重点,可以总结出死亡心理的核心内涵:外部防御和内在成长。对死亡的抽象认知产生死亡焦虑,死亡焦虑产生死亡恐惧、自我保护、外在价值导向等外部防御反应;对死亡的具体认知产生死亡反省,死亡反省产生死亡接受、亲社会动机和行为、内在价值导向等内在成长反应。作者以死亡心理的两类反应为框架,综述了近5年来新的研究证据,并对该领域当前研究的局限和未来研究方向进行了分析。     

Investigating Individual Differences in Toddler Search with Mixture Models

Children's performance on cognitive tasks is often described in categorical terms in that a child is described as either passing or failing a test, or knowing or not knowing some concept. We used binomial mixture models to determine whether individual children could be classified as passing or failing two search tasks, the DeLoache model room task and the Berthier et al. door task. The data support categorical classification of the children and suggest that the increase in average proportion correct with age is the result of an increasing proportion of children who can solve the tasks. Performance on the two tasks was concordant, and improving performance could be due to advances in a single psychological ability, such as cognitive control. Copyright © 2014 John Wiley & Sons, Ltd.