The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments

Automated essay scoring engines (AESEs) are becoming increasingly popular as an efficient method for performance assessments in writing, including many language assessments that are used worldwide. Before they can be used operationally, AESEs must be “trained” using machine-learning techniques that incorporate human ratings. However, the quality of the human ratings used to train the AESEs is rarely examined. As a result, the impact of various rater effects (e.g., severity and centrality) on the quality of AESE-assigned scores is not known. In this study, we use data from a large-scale rater-mediated writing assessment to examine the impact of rater effects on the quality of AESE-assigned scores. Overall, the results suggest that if rater effects are present in the ratings used to train an AESE, the AESE scores may replicate these effects. Implications are discussed in terms of research and practice related to automated scoring.     

Navigating Comics II: Constraints on the Reading Order of Comic Page Layouts

Although readers typically believe that comic page layouts should be read following the left to right and down ‘Z‐path’ inherited from written language, several spatial arrangements can push readers to deviate from this order. These manipulations include separating panels from each other, overlapping one panel onto another, and using a long vertical panel to the right of a vertical column to ‘block’ a horizontal row. We asked participants to order empty panels in comic page layouts that manipulated these factors. All manipulations caused participants to deviate from the conventional Z‐path, and this departure was modulated by incremental changes to spatial arrangements: The more layouts deviated from a grid, the less likely participants were to use the Z‐path. Overall, these results reinforce that various constraints push comic readers to engage with panels in predictable ways, even when deviating from the traditional Z‐path of written language. Copyright © 2014 John Wiley & Sons, Ltd.     

Task switching and action sequencing

We investigated if task switching affects late response processes that occur after the selection of a response. Subjects performed a sequence of two responses. The first and second response were selected, and then executed in close succession. The interresponse interval (IRI) was taken as a measure of late response processes. The two responses could either belong to different tasks (task-switch condition), or to the same task that was performed twice (task-repetition condition). In all three experiments, the IRI was found to be longer in the task-switch condition than in the task-repetition condition, consistent with the idea that task switching affected late response processes. However, the effects of the manipulation of the stimulus-onset asynchrony revealed that the tendency to perform the two responses as a sequence was reduced in the task-switch condition relative to the task-repetition condition. Thus, the data do not provide unequivocal evidence for task switching affecting late response processes. The data show, however, that task switching affected action sequencing. Two actions that do not belong to the same task context are less likely to be performed as an action sequence. We suggest that task switching interacts with higher-order control processes that cannot be studied within the traditional task-switching paradigm.     

Standardizing the psychological autopsy: addressing the Daubert standard

The psychological autopsy, as a research, clinical, and forensic tool, has gained widespread usage in suicidology over the last half century. In forensic settings, the lack of standardization and problems determining the procedure's validity and reliability pose significant issues for the procedure's admissibility under the Daubert standard of evidence. The Daubert standard requires that evidence must be founded on scientific knowledge and established five guidelines for judicial decisions regarding the admissibility of expert testimony. In this paper we examine expert opinion regarding the psychological autopsy and recommend a standardized protocol or template regarding areas and lines of inquiry for a psychological autopsy used in legal cases.     

Attenuating the link between threatened egotism and aggression

Research has found that narcissists behave aggressively when they receive a blow to their ego. The current studies examined whether narcissistic aggression could be reduced by inducing a unit relation between the target of aggression and the aggressor. Experimental participants were told that they shared either a birthday (Study 1) or a fingerprint type (Study 2) with a partner. Control participants were not given any information indicating similarity to their partner. Before aggression was measured, the partners criticized essays written by the participants. Aggression was measured by allowing participants to give their partner loud blasts of noise through a pair of headphones. In the control groups, narcissists were especially aggressive toward their partner. However, narcissistic aggression was completely attenuated, even under ego threat, when participants believed they shared a key similarity with their partner.     

Why walkers slip: shine is not a reliable cue for slippery ground

In a series of four studies, we investigated the visual cues that walkers use to predict slippery ground surfaces and tested whether visual information is reliable for specifying low-friction conditions. In Study 1, 91% of participants surveyed responded that they would use shine to identify upcoming slippery ground. Studies 2-4 confirmed participants' reliance on shine to predict slip. Participants viewed ground surfaces varying in gloss, paint color, and viewing distance under indoor and outdoor lighting conditions. Shine and slip ratings and functional walking judgments were related to surface gloss level and to surface coefficient of friction (COF). However, judgments were strongly affected by surface color, viewing distance, and lighting conditions--extraneous factors that do not change the surface COF. Results suggest that, although walkers rely on shine to predict slippery ground, shine is not a reliable visual cue for friction. Poor visual information for friction may underlie the high prevalence of friction-related slips and falls.     

Remembering all that and then some: Recollection of autobiographical memories after a 1-year delay

We have previously shown that repeated retrievals of remote autobiographical memories over the course of one month led to an overall increase in reported detail (Nadel, Campbell, & Ryan, 2007). The current study examined the retrieval of those same memories 1 year later in order to determine whether the level of detail remained stable or whether the memories returned to their original state. Participants reported even more details than they had recalled at least 1 year earlier, including new details that were reported for the first time. This finding was consistent across both multiple and single retrieval conditions, suggesting that the critical factor leading to the increase in recall was the passage of time. These findings provide evidence for long-term effects of repeated retrieval on memory content.     

Importance of having agreeable friends in adolescence (especially when you are not)

Using the actor–partner interdependence model, we examined whether adolescent's agreeableness, best friend's agreeableness, and the interaction between adolescent‐friend agreeableness are important for interpersonal functioning. Adolescents (N = 158) in fifth to eighth grades who were part of best friend pairs completed personality and friendship measures. Adolescents' adjustment and victimization experiences were assessed by peer nominations. For boys, best friend's agreeableness moderated the relationship between the target boy's agreeableness and overt victimization, relational victimization, and externalizing problems. For girls, best friend's agreeableness moderated the relationship between the target girl's agreeableness and internalizing problems and prosocial skills. This study provides an initial glimpse into how traits of friends can influence outcomes beyond what would be expected from adolescents' personality alone.