The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments

Automated essay scoring engines (AESEs) are becoming increasingly popular as an efficient method for performance assessments in writing, including many language assessments that are used worldwide. Before they can be used operationally, AESEs must be “trained” using machine-learning techniques that incorporate human ratings. However, the quality of the human ratings used to train the AESEs is rarely examined. As a result, the impact of various rater effects (e.g., severity and centrality) on the quality of AESE-assigned scores is not known. In this study, we use data from a large-scale rater-mediated writing assessment to examine the impact of rater effects on the quality of AESE-assigned scores. Overall, the results suggest that if rater effects are present in the ratings used to train an AESE, the AESE scores may replicate these effects. Implications are discussed in terms of research and practice related to automated scoring.     

When Intertemporal Rewards Are Hedonic,Larger Units of Wait Time Boost Patience

In intertemporal choices between smaller‐sooner (SS) and larger‐later (LL) rewards, five studies reveal that patience for the LL option is influenced by an interactive effect of the measurement units used to express wait time (large vs. small) and the type of the reward (hedonic vs. utilitarian). Specifically, larger time units boost patience, but more so when rewards are hedonic rather than utilitarian. In line with the numerosity heuristic, the effect of time units on patience is mediated by larger time units shrinking wait time perception. This effect arises because hedonic (vs. utilitarian) rewards promote a reliance on heuristics rather than systematic calculations. Therefore, a more calculative mindset diminishes the effect of units even for hedonic rewards and eliminates the hedonic‐utilitarian asymmetry. These results contribute to research on numerosity, intertemporal choice, and hedonic‐utilitarian differences, and offer a simple tool for practitioners to influence patience.     

Sexual‐perception processes in acquaintance‐targeted sexual aggression

This study analyzes data from seven published studies to examine whether three performance‐based indices of men's misperception of women's sexual interest (MSI), derived from a self‐report questionnaire, are associated with sexual‐aggression history, rape‐supportive attitudes, sociosexuality, problem drinking, and self‐reported MSI. Almost 2000 undergraduate men judged the justifiability of a man's increasingly unwanted advances toward a woman on the Heterosocial Perception Survey‐Revised. Participants self‐reported any sexual‐aggression history, and some completed questionnaires assessing rape‐supportive attitudes, sociosexuality, problem drinking, and self‐reported MSI. A three‐parameter logistic function was fitted to participants’ justifiability ratings within a non‐linear mixed‐effects framework, which provided precise participant‐specific estimates of three sexual‐perception processes (baseline justifiability, bias, and sensitivity). Sexual‐aggression history and rape‐supportive attitudes predicted: (a) reduced sensitivity to women's affect; (b) more liberal biases, such that the woman's affect had to be more negative before justifiability ratings dropped substantially; and (c) greater baseline justifiability of continued advances after a positive response. Sexual‐aggression history and attitudes correlated more strongly with sensitivity than baseline justifiability; remaining variables showed the opposite pattern. This work underscores the role of sexual‐perception processes in sexual aggression and illustrates the derivation of performance‐based estimates of sexual‐perception processes from questionnaire responses.

     

An external review of the conclusions regarding the peak direct training module

Promoting the Emergence of Advanced Knowledge—Direct Training Module (PEAK‐DTM) is a commercially available assessment and curriculum for individuals whose language skills are not developmentally adequate. In their review of PEAK‐DTM, Reed and Luiselli (2016) analyzed the extant literature on PEAK‐DTM and concluded that it has a sound and growing body of empirical support on its efficacy, usability, and psychometric properties. Similar conclusions are mirrored in the PEAK‐DTM literature and promotional material. I review these conclusions and contend that many overrate the research that backs them. Suggestions for a more rigorous research agenda on PEAK‐DTM and its related modules are provided.     

MLK Day and Racial Attitudes: Liking the Group More but Its Members Less

Intuition suggests that the Martin Luther King holiday (MLK Day) should improve racial attitudes toward African Americans. However, its influence may depend on whether African Americans are evaluated as a group or individually. In two studies, we assessed racial attitudes either on MLK Day or on a control day. As might be expected, participants had more sympathetic attitudes towards African Americans as a group on MLK Day compared to control days; however, they evaluated individual African American exemplars more negatively on MLK Day compared to control days, who presumably seemed worse by comparison to the eminent political figure.     

Are Automatic Conceptual Cores the Gold Standard of Semantic Processing? The Context‐Dependence of Spatial Meaning in Grounded Congruency Effects

According to grounded cognition, words whose semantics contain sensory‐motor features activate sensory‐motor simulations, which, in turn, interact with spatial responses to produce grounded congruency effects (e.g., processing the spatial feature of up for sky should be faster for up vs. down responses). Growing evidence shows these congruency effects do not always occur, suggesting instead that the grounded features in a word's meaning do not become active automatically across contexts. Researchers sometimes use this as evidence that concepts are not grounded, further concluding that grounded information is peripheral to the amodal cores of concepts. We first review broad evidence that words do not have conceptual cores, and that even the most salient features in a word's meaning are not activated automatically. Then, in three experiments, we provide further evidence that grounded congruency effects rely dynamically on context, with the central grounded features in a concept becoming active only when the current context makes them salient. Even when grounded features are central to a word's meaning, their activation depends on task conditions.     

Self‐congruity effects: A critical review and an integrative model

This research conducts an extensive review study on the extant research on self‐congruity, and presents some recommendations and propositions for future research. It describes the differences between two constructs (i.e., self‐congruity and brand personality) regarding the symbolic benefits of a brand. It also suggests an integrative model to encompass important cognitive and motivational factors to moderate the relationship between self‐congruity and brand evaluation. Although there should be meaningful moderators of self‐congruity effects that are not addressed in the model, this research can be a starting point for future research in order to develop a better research model for analyzing self‐congruity effects.     

The Academic Advantage of Devotion: Measuring Variation in the Value of Weekly Worship in Late Adolescence on Educational Attainment Using Propensity Score Matching

This study measures the effect of regular worship attendance at age 17 on total years of schooling by age 25, using data from the National Longitudinal Survey of Youth 1997. Expanding on previous work, this study estimates differences in the impact of worship attendance by race and family income status using propensity score matching. Individuals who frequently attend religious services complete .69 more years of schooling than similar individuals who do not frequently attend services. There are significantly greater returns to attendance for low‐income youth and no significant difference in returns by religious affiliation. These findings suggest that religious observance provides greater benefits for low‐income individuals or perhaps provides resources high‐income individuals have access to elsewhere. Moreover, this study extends previous work by examining a more recent and nationally representative sample of youth and by using methods that allow for greater causal inference.