Neutral Responses to Structured Questionnaires within Organizational Settings: Roles of Rater Affective Feelings and Demographic Characteristics

This study is a follow-up to a prior publication by the authors dealing with influences behind neutral responses to structured questionnaires. The earlier study found rater affective feelings, nationality, and age to be significant influences, while sex was not. The questionnaire used in the first study dealt primarily with supervision and relied on a three-point scale. The present instrument covers a broader range of topics and uses a five-point Likert-type scale. The results on the influence of rater affective feelings are consistent with previous studies, but there is some variation in the demographic influences. Implications of the findings are discussed and suggestions are offered for additional research.     

THE INFLUENCE OF RATER AND RATEE AGE ON TWO PERFORMANCE JUDGMENTS

The influences of rater age and rater age on performance evaluations of 513 exempt managers from a large manufacturing organization were examined. Two general performance criteria and six specific performance criteria were used. Results showed small but significant age influences on two of the specific performance measures, Self-Development and Interpersonal Skills. Older workers received lower ratings than younger workers on both of these dimensions. In addition, younger raters gave significantly higher ratings than older raters on Interpersonal Skills. Small but significant age interactions were found in both Interpersonal Skills ratings and Self-Development ratings. The findings were replicated in a second sample ( N =178). The influence of age on performance ratings is consistent with previous research and is compared. The extent of the influence is smaller in the present study than in previous research.     

Distinguishing between perceiver and wearer effects in clothing color-associated attributions

Recent studies have noted positive effects of red clothing on success in competitive sports, perhaps arising from an evolutionary predisposition to associate the color red with dominance status. Red may also enhance judgments of women's attractiveness by men, perhaps through a similar association with fertility. Here we extend these studies by investigating attractiveness judgments of both sexes and by contrasting attributions based on six different colors. Furthermore, by photographing targets repeatedly in different colors, we could investigate whether color effects are due to influences on raters or clothing wearers, by either withholding from raters information about clothing color or holding it constant via digital manipulation, while retaining color-associated variation in wearer's expression and posture. When color cues were available, we found color-attractiveness associations when males were judged by either sex, or when males judged females, but not when females judged female images. Both red and black were associated with higher attractiveness judgments and had approximately equivalent effects. Importantly, we also detected significant clothing color-attractiveness associations even when clothing color was obscured from raters and when color was held constant by digital manipulation. These results suggest that clothing color has a psychological influence on wearers at least as much as on raters, and that this ultimately influences attractiveness judgments by others. Our results lend support for the idea that evolutionarily-derived color associations can bias interpersonal judgments, although these are limited neither to effects on raters nor to the color red.     

Putting Raters in Ratees’ Shoes: Perspective Taking and Assessment of Creative Products

This study reported 2 experiments that studied the effect of perspective taking on assessment of creative products by using human raters. Forty responses of 2 alternative uses tasks (AUTs) and 15 alien stories generated by 6th-grade students were used as assessment materials. Undergraduate students as the novice raters assessed the products under 3 experimental conditions: assessing without any information of the ratees, assessing only with age information of the ratees, and assessing with age information and taking the perspective of the ratees. Results of Experiment 1 showed significant differences in creativity ratings between group 1 and 2. But no significant difference was found between group 2 and 3. In Experiment 2, raters in group 1 used objective perception and raters in group 3 were asked to take the perspective for more time. Raters in group 3 assigned higher ratings than the other 2 groups but no difference was found between group 1 and 2. Overall, the results showed the effect of perspective taking on assessment of creative products. Possible mechanisms of the effect and implications for creativity assessment and creativity teaching practice were discussed.     

Predictors of Differential Responding on a Sentence Completion Task in Traumatized Children

Information-processing theories posit that persons who have experienced trauma may have disturbed emotional processing and therefore exhibit an excess of negative responses to otherwise neutral cues. The role of this mechanism in traumatized children has yet to be fully investigated. This study examined the relationship of varied clinical characteristics and theoretically relevant moderating variables to increased trauma related responding on a non-idiographic sentence completion task. One hundred and eighty-nine children (ages 6–17) residing at a residential home between 1996 and 2011 who had experienced physical, emotional and/or sexual abuse were administered a sentence completion task and the Trauma Symptom Checklist for Children (TSCC) shortly after their admission. Three independent raters determined whether sentence completion responses were trauma related (TRR), non-trauma related, or ambiguous. Trauma related responses were then reviewed for either avoidant/denial style responding or loss related responding. The TSCC posttraumatic stress subscale (PTS) was the only subscale that was uniquely related to TRR. A significant interaction between abuse type and PTS was found with sexual abuse moderating the effect of PTS on TRR. Additionally, age at assessment was positively correlated with both TRR and loss related responding. Time since removal from home was negatively correlated with TRR. Gender differences were found for TRR and loss related responding. These findings highlight the empirical relevance of the sentence completion task in clinical research. Results are discussed in terms of their implications for research, assessment, and intervention.     

Pain in the eye of the beholder: Variations in pain visual representations as a function of face ethnicity and culture

Pain experienced by Black individuals is systematically underestimated, and recent studies have shown that part of this bias is rooted in perceptual factors. We used Reverse Correlation to estimate visual representations of the pain expression in Black and White faces, in participants originating from both Western and African countries. Groups of raters were then asked to evaluate the presence of pain and other emotions in these representations. A second group of White raters then evaluated those same representations placed over a neutral background face (50% White; 50% Black). Image-based analyses show significant effects of culture and face ethnicity, but no interaction between the two factors. Western representations were more likely to be judged as expressing pain than African representations. For both cultural groups, raters also perceived more pain in White face representations than in Black face representations. However, when changing the background stimulus to the neutral background face, this effect of face ethnic profile disappeared. Overall, these results suggest that individuals have different expectations of how pain is expressed by Black and White individuals, and that cultural factors may explain a part of this phenomenon     

托幼机构教育质量评价中评委偏差的多侧面Rasch分析

采用多侧面Rasch模型对28位评委在托幼机构教育质量评价中的评委偏差进行了分析。分析结果显示：28名评委评分宽严度差异显著;3名评委内部一致性较差,其余25名评委内部一致性较稳定;评委与评价班级的交互作用不显著,与评价项目的交互作用显著。研究结果表明MFRM可以对托幼机构教育质量评价的评委偏差进行个体层面的具体分析,从项目反应理论的视角为托幼机构教育质量评价的评委针对性培训、评估评委的合格性从而建立合格评委库等提供现代教育、心理测量学依据。     

Validation of affective and neutral sentence content for prosodic testing

Conducting a study of emotional prosody often requires that one have a valid set of stimuli for assessing perceived emotion in vocal intonation. In this study, we created a list of sentences with both affective and neutral content, and then validated them against rater opinion. Participants read sentences with content that implied happiness, sadness, anger, fear, or neutrality and rated how well they could imagine each sentence being expressed in each emotion. Coefficients of variation and intraclass correlations were calculated to narrow the list to affective sentences that had high agreement and neutral sentences that had low agreement. We found that raters could easily identify most emotional content and did not ascribe any unique emotion to most neutral content. We also found differences between the intensity of male and female ratings. The final list of sentences is available on the Internet (www.med.upenn.edu/bbl/) and can be recorded for use as stimuli for prosodic studies.     

Unconscious transfer of meaning to brands

We examine semantic conditioning in a consumer context. We subliminally paired neutral ideographs with attributes. In experiment 1, the ideographs served as primes during a lexical decision task and slowed down response times to target words with the opposite semantic meaning. In experiment 2, the ideographs served as brand names of beverages, and attitudinal responses to them were less favorable when the associated attributes were incongruent with existing schemas. These results showed that semantic conditioning (1) can occur unconsciously, (2) can have significant and meaningful consequences for brand evaluation, and (3) influences subsequent attitudinal responses via conceptual disfluency processes.     

The reliability and concurrent validity of alternative methods for assessing ego development

This study demonstrates that an unstructured interview (INT) and the Thematic Apperception Test (TAT) are suitable alternatives to Loevinger's Sentence Completion Test of Ego Development. Seventy subjects were solicited from six groups varying widely with respect to age and education level. Each subject was asked to complete the SCI, an INT and the TAT. Two raters using Loevinger and Wessler's self-training exercises and Loevinger, Wessler, and Redmore's scoring manual rated subjects' responses to each instrument. Reliability was found between raters and concurrent validity between instruments. Subjects scoring high on the SCT were found to score higher on the INT and TAT.     

The Machines Take Over: A Comparison of Various Supervised Learning Approaches for Automated Scoring of Divergent Thinking Tasks

Traditionally, researchers employ human raters for scoring responses to creative thinking tasks. Apart from the associated costs this approach entails two potential risks. First, human raters can be subjective in their scoring behavior (inter-rater-variance). Second, individual raters are prone to inconsistent scoring patterns (intra-rater-variance). In light of these issues, we present an approach for automated scoring of Divergent Thinking (DT) Tasks. We implemented a pipeline aiming to generate accurate rating predictions for DT responses using text mining and machine learning methods. Based on two existing data sets from two different laboratories, we constructed several prediction models incorporating features representing meta information of the response or features engineered from the response’s word embeddings that were obtained using pre-trained GloVe and Word2Vec word vector spaces. Out of these features, word embeddings and features derived from them proved to be particularly effective. Overall, longer responses tended to achieve higher ratings as well as responses that were semantically distant from the stimulus object. In our comparison of three state-of-the-art machine learning algorithms, Random Forest and XGBoost tended to slightly outperform the Support Vector Regression.     

Evaluations of the quality of coping reported by prisoners who have self-harmed and those who have not.

Yufit and Bongar (1992) argued that a deficiency in coping skills is an important risk factor for suicidal behavior. Dear, Thomson, Hall, and Howells (1998) found that prisoners who had self-harmed in the past 3 days were less likely than a comparison group to have used problem-solving or active cognitive coping strategies to handle the most significant stressor of the past week, but it was not clear whether this represented a difference in the quality of coping responses used. In this study, three groups of blind raters (prisoners, prison officers, and forensic psychologists) rated the coping responses of the participants in Dear et al.'s study. The coping responses of self-harmers were judged less beneficial and more risky. Problem-solving strategies were most often cited as contributing to beneficial outcomes and the catharsis strategies employed by self-harmers were most often judged to be counterproductive. It remains unclear whether prisoners who self-harmed routinely employ poor quality coping strategies or if they simply used poor quality coping on this occasion.     

Being better than others but otherwise perfectly normal: Perceptions of uniqueness and similarity in close relationships

The current research addresses individuals' self-generated thoughts regarding their own and others' relationships, examining the ways in which perceptions of uniqueness and similarity are manifested in judgments regarding own and others' responses to dissatisfying incidents. Consistent with the uniqueness bias, participants characterized their own relationships by a greater number of constructive responses and a smaller number of destructive responses relative to characterizations of others' relationships. Moreover, external raters judged own constructive responses to be more constructive than others' constructive responses. Consistent with the similarity bias, external raters judged items describing others' responses to be less frequently occurring and more extreme than their own responses. Also, this research revealed support for the claim that the similarity bias is more pronounced for destructive responses than for constructive responses. A recall task corroborated these findings, revealing very good recall for destructive responses enacted by others and poor recall for destructive responses enacted by oneself.     

Effect of Quality Characteristics of Peer Raters on Rating Errors in Peer Assessment

Little research has been done on the effects of peer raters’ quality characteristics on peer rating qualities. This study aims to address this gap and investigate the effects of key variables related to peer raters’ qualities, including content knowledge, previous rating experience, training on rating tasks, and rating motivation. In an experiment where training and motivation interventions were manipulated, 24 classes with 838 high school students were randomly assigned to study conditions. Inter-rater error, intra-rater error and criterion error indices for peer ratings on four selected essays were analyzed using hierarchical linear models. Results indicated that peer raters’ content knowledge, previous rating experience, and rating motivation were associated with rating errors. This study also found some significant interactions between peer raters’ quality characteristics. Implications for in-person and online peer assessments as well as future directions are discussed.     

Physical stature influences character perception in women

It is well established that height influences the perceived character of men but very little evidence exists that the same influences of height exist for women. To investigate the effects of female height on perception of their character, we used a novel technique of digitally altering the perceived height of two women depicted in photographs such that the same targets appeared to be of tall or short stature to different participants. We asked participants to rate each of the two women on personal qualities and characteristics that reflected the instrumental/expressive distinction in masculine/feminine stereotypes. Overall, tall women were rated as more intelligent, affluent, assertive and ambitious than shorter women. A perceiver-sex difference was found such that men rated short females more highly than tall females on expressive characteristics while women raters did not. Thus, we provide the first direct evidence that female height influences perception of their character.     

The impact of changes in spatial distance on emotional responses

Drawing on motivational approaches to emotion, the authors propose that the perceived change in spatial distance to pictures that arouse negative emotions exerts an influence on the significance of these pictures. Two experiments induced the illusion that affective pictures approach toward the observer, recede from the observer, or remain static. To determine the motivational significance of the pictures, emotional valence and arousal ratings as well as startle responses were assessed. Approaching unpleasant pictures were found to exert an influence on both the valence and the arousal elicited by the pictures. Furthermore, movement of pleasant or neutral pictures did not influence startle responses, while the second experiment showed that approaching unpleasant pictures elicited enhanced startle responses compared to receding unpleasant pictures. These findings support the view that a change of spatial distance influences motivational significance and thereby shapes emotional responses.     

Beyond "happy, angry, or sad?": age-of-poser and age-of-rater effects on multi-dimensional emotion perception

Young, middle-aged, and older raters (N=154) evaluated 1,026 prototypical facial poses of neutrality, happiness, anger, disgust, fear, and sadness stemming from 171 young, middle-aged, and older posers. The majority of poses were rated as multi-faceted, that is, to comprise several expressions of varying intensities. Consistent with the notion of age-related increases in negativity-avoidance/positivity effects, crossed-random effects analyses showed an age-related decrease in the attributions of negative, but not positive and neutral, target expressions (that the poser intended to show), and an age-related increase in the attributions of positive and neutral, but not negative, non-target expressions (that the posers did not intend to show). Expressions were more difficult to read the older the posers, particularly for male posers. These age-of-poser effects were independent of the valence of the expression, but partly differed across age groups of raters. The study supports the idea of multi-dimensionality and age-dependency of emotion perception.     

Personality and lexical decision times for evaluative words

We studied personality influences on accessibility of pleasant and unpleasant stimuli in a sample of 129 students. Self‐reports and reports by knowledgeable informants on extraversion, neuroticism, approach temperament and avoidance temperament were combined with a go/no‐go lexical decision task that included pleasant, unpleasant and neutral words, and two response modes, manual and vocal. The data were analysed using multilevel modelling. Extraversion and approach temperament predicted faster identification of pleasant words than of neutral and of unpleasant words. Vocal responses took longer than manual responses, but mode of response did not interact with the valence of the words. Copyright © 2009 John Wiley & Sons, Ltd.     

等级反应多水平侧面模型及其在主观题评分中的应用

探讨了康春花,孙小坚和曾平飞(2016)提出的等级反应多水平侧面模型(GR-MLFM)在包含被试及评分者层面预测变量(完整模型)下的返真性和适用性。结果表明:(1)GR-MLFM完整模型具有逻辑上和数理上的合理性,可用于主观题的评分情境,能较好地检测出评分者效应、影响因素及其影响程度;(2)在数学问题解决的评分实践中,评分员存在两种类型的评分倾向(宽松和严格效应),但绝大多数评分员的宽严度不明显;评分者的责任心可正向预测其严格程度,自信心可正向预测其宽松程度,而情绪稳定性和评分经验的预测作用不显著。