The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments

Automated essay scoring engines (AESEs) are becoming increasingly popular as an efficient method for performance assessments in writing, including many language assessments that are used worldwide. Before they can be used operationally, AESEs must be “trained” using machine-learning techniques that incorporate human ratings. However, the quality of the human ratings used to train the AESEs is rarely examined. As a result, the impact of various rater effects (e.g., severity and centrality) on the quality of AESE-assigned scores is not known. In this study, we use data from a large-scale rater-mediated writing assessment to examine the impact of rater effects on the quality of AESE-assigned scores. Overall, the results suggest that if rater effects are present in the ratings used to train an AESE, the AESE scores may replicate these effects. Implications are discussed in terms of research and practice related to automated scoring.     

Crossing Boundaries Online in Romantic Relationships: An Exploratory Study of The Perceptions of Impact on Partners by Cyberstalking Offenders

As technology has become an ever-present facet in the lives of young people, they have become reliant on it to form and maintain relationships. The present study explored the perceptions of cyberstalking offenders on their own romantic relationships of their offending behaviors utilizing General Theory of Crime and Social Learning Theory. Results indicated that low self-control is a significant predictor of perceiving a negative impact by a victim, such as breaking up the relationship or taking revenge on the offender, as offenders with low self-control do not believe their behaviors would have a negative impact.     

Eight-Year Latent Class Trajectories of Academic and Social Functioning in Children with Attention-Deficit/Hyperactivity Disorder

We examined trajectories of academic and social functioning in children with attention-deficit/hyperactivity disorder (ADHD) to identify those who might be at risk for especially severe levels of academic and social impairment over time. We estimated a series of growth mixture models using data from two subsamples of children participating in the NIMH Collaborative Multisite Multimodal Treatment Study of Children with ADHD (MTA) including those with at least baseline and 96-month data for reading and mathematics achievement (n = 392; 77.3% male; M age = 7.7; SD = 0.8) or social skills ratings from teachers (n = 259; 74.9% male; M age = 7.6; SD = 0.8). We compared latent trajectories for children with ADHD to mean observed trajectories obtained from a local normative (i.e., non-ADHD) comparison group (n = 289; 80.6% male; M age = 9.9; SD = 1.1). Results indicated six latent trajectory classes for reading and mathematics and four classes for teacher social skills ratings. There was not only a relationship between trajectories of inattention symptoms and academic impairment, but also a similarly strong association between trajectory classes of hyperactive-impulsive symptoms and achievement. Trajectory class membership correlated with socio-demographic and diagnostic characteristics, inattention and hyperactive-impulsive symptom trajectories, externalizing behavior in school, and treatment receipt and dosage. Although children with ADHD display substantial heterogeneity in their reading, math, and social skills growth trajectories, those with behavioral and socio-demographic disadvantages are especially likely to display severe levels of academic and social impairment over time. Evidence-based early screening and intervention that directly address academic and social impairments in elementary school-aged children with ADHD are warranted. The ClinicalTrials.gov identifier is NCT00000388.     

Handedness and Reach-to-Place Kinematics in Adults: Left-Handers Are Not Reversed Right-Handers

The primary goal of this study was to examine the relations between limb control and handedness in adults. Participants were categorized as left or right handed for analyses using the Edinburgh Handedness Inventory. Three-dimensional recordings were made of each arm on two reach-to-place tasks: adults reached to a ball and placed it into the opening of a toy (fitting task), or reached to a Cheerio inside a cup, which they placed on a designated mark after each trial (cup task). We hypothesized that limb control and handedness were related, and we predicted that we would observe side differences favoring the dominant limb based on the dynamic dominance hypothesis of motor lateralization. Specifically, we predicted that the dominant limb would be straighter and smoother on both tasks compared with the nondominant limb (i.e., right arm in right-handers and left arm in left-handers). Our results only partially supported these predictions for right-handers, but not for left-handers. When differences between hands were observed, the right hand was favored regardless of handedness group. Our findings suggest that left-handers are not reversed right-handers when compared on interlimb kinematics for reach-to-place tasks, and reaffirm that task selection is critical when evaluating manual asymmetries.     

A Review of Best Practice Recommendations for Text Analysis in R (and a User-Friendly App)

In recent decades, the amount of text available for organizational science research has grown tremendously. Despite the availability of text and advances in text analysis methods, many of these techniques remain largely segmented by discipline. Moreover, there is an increasing number of open-source tools (R, Python) for text analysis, yet these tools are not easily taken advantage of by social science researchers who likely have limited programming knowledge and exposure to computational methods. In this article, we compare quantitative and qualitative text analysis methods used across social sciences. We describe basic terminology and the overlooked, but critically important, steps in pre-processing raw text (e.g., selection of stop words; stemming). Next, we provide an exploratory analysis of open-ended responses from a prototypical survey dataset using topic modeling with R. We provide a list of best practice recommendations for text analysis focused on (1) hypothesis and question formation, (2) design and data collection, (3) data pre-processing, and (4) topic modeling. We also discuss the creation of scale scores for more traditional correlation and regression analyses. All the data are available in an online repository for the interested reader to practice with, along with a reference list for additional reading, an R markdown file, and an open source interactive topic model tool (topicApp; see https://github.com/wesslen/topicApp, https://github.com/wesslen/text-analysis-org-science, https://dataverse.unc.edu/dataset.xhtml?persistentId=doi:10.15139/S3/R4W7ZS).