The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments

Automated essay scoring engines (AESEs) are becoming increasingly popular as an efficient method for performance assessments in writing, including many language assessments that are used worldwide. Before they can be used operationally, AESEs must be “trained” using machine-learning techniques that incorporate human ratings. However, the quality of the human ratings used to train the AESEs is rarely examined. As a result, the impact of various rater effects (e.g., severity and centrality) on the quality of AESE-assigned scores is not known. In this study, we use data from a large-scale rater-mediated writing assessment to examine the impact of rater effects on the quality of AESE-assigned scores. Overall, the results suggest that if rater effects are present in the ratings used to train an AESE, the AESE scores may replicate these effects. Implications are discussed in terms of research and practice related to automated scoring.     

Association splitting of the sexual orientation-OCD-relevant semantic network

There is little research on treating symptoms of sexual orientation-obsessive-compulsive disorder (SO-OCD). Semantic networks represent a new cognitive approach for understanding cognitive mechanisms of SO-OCD. Specifically, we tested whether the self-help cognitive technique of association splitting (AS) developed from this approach would be efficacious in reducing SO-OCD symptoms and thought suppression. One hundred and twenty heterosexual undergraduates (82 females, 38 males) were randomly assigned to either the AS or waitlist control group. At baseline and four weeks later, participants completed items assessing SO-OCD symptoms, measures of sexual obsessions and thought suppression, and an association task in which they generated associations to different cue words. Generated associations were coded based on SO-OCD relevance and emotional valence. Results indicated reductions in SO-OCD-relevant associations across levels of emotional valence and SO-OCD-irrelevant negative associations, and increases in SO-OCD-irrelevant positive and neutral associations, only in the AS group. Furthermore, there were reductions in SO-OCD symptoms, sexual obsessions, and thought suppression only in the AS group. Importantly, these findings were obtained with overall large effect sizes. AS appears to be an efficacious self-help technique in reducing SO-OCD symptoms, sexual obsessions, and thought suppression. Clinical implications, limitations, and suggestions for future research are discussed.     

Serotonin transporter 5-HTTLPR genotype is associated with intrusion and avoidance symptoms of DSM-5 posttraumatic stress disorder (PTSD) in Chinese earthquake survivors

Background and Objectives: Prior studies have found that the serotonin transporter gene-linked polymorphic region (5-HTTLPR) interacts with trauma exposure to increase general risk for Posttraumatic Stress Disorder (PTSD). However, there is little knowledge about the effects of the interaction on distinct symptom clusters of PTSD. This study aimed to investigate the relation between the interaction of 5-HTTLPR and earthquake-related exposures and a contemporary phenotypic model of DSM-5 PTSD symptoms in a traumatised adult sample from China.

Design: A cross-sectional design with gene-environment interaction (G?×?E) approach was adopted. Methods: Participants were 1131 survivors who experienced 2008 Wenchuan earthquake. PTSD symptoms were assessed with the PTSD Checklist for DSM-5 (PCL-5). The 5-HTTLPR polymorphism was genotyped with capillary electrophoresis (CE) in ABI 3730xl genetic Analyzer.

Results: Although there was no significant interaction between 5-HTTLPR and traumatic exposure on total PTSD symptoms, respondents with the LL genotype of 5-HTTLPR who were highly exposed to the earthquake experienced lower intrusion and avoidance symptoms than those with the S-allele carriers.

Conclusions: The findings suggest that the 5-HTTLPR may have an important impact on the development of PTSD and add to the extant knowledge on understanding and treating of posttraumatic psychopathology.     

Investigating Individual Differences in Toddler Search with Mixture Models

Children's performance on cognitive tasks is often described in categorical terms in that a child is described as either passing or failing a test, or knowing or not knowing some concept. We used binomial mixture models to determine whether individual children could be classified as passing or failing two search tasks, the DeLoache model room task and the Berthier et al. door task. The data support categorical classification of the children and suggest that the increase in average proportion correct with age is the result of an increasing proportion of children who can solve the tasks. Performance on the two tasks was concordant, and improving performance could be due to advances in a single psychological ability, such as cognitive control. Copyright © 2014 John Wiley & Sons, Ltd.     

Thinking bigger: The Cronbachian paradigm & personality theory integration

Personality theories largely mirror disparities between fundamental paradigms that guide most psychology: one based upon Galton’s emphasis on inter-individual differences and the lexical hypothesis; the other inspired by Wundt is focused upon intra-individual processes, such as temperament and Reinforcement Sensitivity Theory. Previous integrations of personality models using these paradigms failed to account for observed variance, suggesting model incommensurability. We propose that a broader conception—developed from Cronbach and colleagues’ generalisability theory—provides an effective integrating framework, and show how Galtonian and Wundtian models may be assimilated within Cronbachian approaches. Additionally, Cronbachian models have novel practical implications. Application of the Cronbachian paradigm to personality will be challenging, but provides an opportunity to achieve genuine coherence in personality research.     

Identifying expectations about the strength of causal relationships

When we try to identify causal relationships, how strong do we expect that relationship to be? Bayesian models of causal induction rely on assumptions regarding people’s a priori beliefs about causal systems, with recent research focusing on people’s expectations about the strength of causes. These expectations are expressed in terms of prior probability distributions. While proposals about the form of such prior distributions have been made previously, many different distributions are possible, making it difficult to test such proposals exhaustively. In Experiment 1 we used iterated learning—a method in which participants make inferences about data generated based on their own responses in previous trials—to estimate participants’ prior beliefs about the strengths of causes. This method produced estimated prior distributions that were quite different from those previously proposed in the literature. Experiment 2 collected a large set of human judgments on the strength of causal relationships to be used as a benchmark for evaluating different models, using stimuli that cover a wider and more systematic set of contingencies than previous research. Using these judgments, we evaluated the predictions of various Bayesian models. The Bayesian model with priors estimated via iterated learning compared favorably against the others. Experiment 3 estimated participants’ prior beliefs concerning different causal systems, revealing key similarities in their expectations across diverse scenarios.     

Fitting direct covariance structures by the MSTRUCT modeling language of the CALIS procedure

This paper demonstrates the usefulness and flexibility of the general structural equation modelling (SEM) approach to fitting direct covariance patterns or structures (as opposed to fitting implied covariance structures from functional relationships among variables). In particular, the MSTRUCT modelling language (or syntax) of the CALIS procedure (SAS/STAT version 9.22 or later: SAS Institute, 2010) is used to illustrate the SEM approach. The MSTRUCT modelling language supports a direct covariance pattern specification of each covariance element. It also supports the input of additional independent and dependent parameters. Model tests, fit statistics, estimates, and their standard errors are then produced under the general SEM framework. By using numerical and computational examples, the following tests of basic covariance patterns are illustrated: sphericity, compound symmetry, and multiple‐group covariance patterns. Specification and testing of two complex correlation structures, the circumplex pattern and the composite direct product models with or without composite errors and scales, are also illustrated by the MSTRUCT syntax. It is concluded that the SEM approach offers a general and flexible modelling of direct covariance and correlation patterns. In conjunction with the use of SAS macros, the MSTRUCT syntax provides an easy‐to‐use interface for specifying and fitting complex covariance and correlation structures, even when the number of variables or parameters becomes large.