A Comparison of Assessment Methods and Raters in Product Creativity

Although previous studies have attempted to use different experiences of raters to rate product creativity by adopting the Consensus Assessment Method (CAT) approach, the validity of replacing CAT with another measurement tool has not been adequately tested. This study aimed to compare raters with different levels of experience (expert ves. nonexpert raters) using both CAT and the product creativity measurement instrument (PCMI) to assess the product creativity of 56 design works based on a design competition. The results showed that nonexpert raters who used either CAT or PCMI had higher interreliability than expert raters. Using PCMI was found to result in higher correlation than using CAT for the expert and nonexpert raters, although the correlation between the CAT and PCMI methods was statistically insignificantly different. After regression analysis, the results showed that all PCMI items had higher explanatory power for the creativity scores using CAT and, moreover, the nonexpert raters were found to have higher explanatory power than the expert raters. Based on these results, it is recommended that the use of both nonexpert raters and PCMI is an alternative way of enhancing the flexibility of product creativity assessment.     

新手考官的工作记忆和人格对其评分有效性的影响——以无领导小组讨论为例

相对于其它评价中心技术而言,在无领导小组讨论中考官因素对评分结果的影响尤为重要.本研究主要探讨无领导小组讨论中新手考官的工作记忆与人格对其评分有效性的影响.结果发现,首先,新手考官的评分者一致性较低,评分准确度较差.其次,工作记忆和人格的部分因素分别从不同方面影响新手考官的评分有效性,具体表现在：（1）利他性越强,新手考官评分总均值的准确性越高,且评分结果越宽松;（2）新手考官的决断性越强,对所有应聘者做出有效区分的准确性越高;（3）新手考官的沉稳性越高,对各维度的区分越有效;（4）注意转换和抑制能力对新手考官的晕轮效应及其在各个维度上进行区分的准确度有抑制作用.     

Rating ability in performance judgments: The joint influence of implicit theories and intelligence

The purpose of this study was to approach the issue of rating ability by examining the influence of rater implicit theories and rater intelligence on rating outcomes. Using the inferential accuracy model (Jackson, 1972), raters were identified as either possessing a normative or idiosyncratic implicit theory of the occupation of college instructor. In a laboratory setting, 50 normative and 50 idiosyncratic raters judged the videotaped performance of either a good or poor lecturer. Results showed that (a) intelligence was positively related to rating accuracy and to possessing a normative implicit theory, (b) rater type moderated the relationship between intelligence and rating accuracy, and (c) controlling for intelligence, normative raters committed stronger halo effects than idiosyncratic raters. These results were discussed in relation to furthering the understanding of rating ability.     

Racial bias in behavioral assessment of children's social skills

The social skills of 20 second- and sixth-grade students were assessed by 20 trained raters using the Social Skills Test for Children (SST-C). Rater and child characteristics were examined to determine whether differences in social skills ratings were due to the race of the rater or the race of the children being rated or due to the interactive effects of these characteristics, which would suggest racial bias in the ratings procedure. The results showed that the race of the rater did affect some behavioral observations. Black raters gave higher scores than white raters on four behavioral categories: response latency, appropriate assertion, effective assertion, and smiling. White raters gave higher scores for head position and gestures. The results of this study replicated earlier findings of significant differences in social skills ratings due to the race and age of the child being rated. The results also showed modest racial bias effects in that black and white raters scored black and white children differentially on two behavioral categories: overall skill ratings and smiling. These results suggested that most behavioral categories of the SST-C were not systematically affected by racial bias. However, the most subjective rating, overall skill, did evidence racial bias effects. This finding is consistent with previous data showing that subjective ratings may be most affected by racial bias.     

An examination of raters' and ratees' preferences in process and feedback in performance appraisal

This paper examined the raters' and the ratees' preferences in Malaysia regarding the performance appraisal process and feedback. A total of 52 managers (raters) and 122 subordinates (ratees) participated. Analysis by t test and correlation showed significant differences between raters and the ratees on most factors. Ratees preferred to have more frequent appraisals than once a year and preferred to have more frequent feedback from the managers about their performance. Ratees' preference for the purpose of information on performance was towards salary increase, promotion, training, and career development whereas the raters gave more importance to training and career development.     

Putting Raters in Ratees’ Shoes: Perspective Taking and Assessment of Creative Products

This study reported 2 experiments that studied the effect of perspective taking on assessment of creative products by using human raters. Forty responses of 2 alternative uses tasks (AUTs) and 15 alien stories generated by 6th-grade students were used as assessment materials. Undergraduate students as the novice raters assessed the products under 3 experimental conditions: assessing without any information of the ratees, assessing only with age information of the ratees, and assessing with age information and taking the perspective of the ratees. Results of Experiment 1 showed significant differences in creativity ratings between group 1 and 2. But no significant difference was found between group 2 and 3. In Experiment 2, raters in group 1 used objective perception and raters in group 3 were asked to take the perspective for more time. Raters in group 3 assigned higher ratings than the other 2 groups but no difference was found between group 1 and 2. Overall, the results showed the effect of perspective taking on assessment of creative products. Possible mechanisms of the effect and implications for creativity assessment and creativity teaching practice were discussed.     

托幼机构教育质量评价中评委偏差的多侧面Rasch分析

采用多侧面Rasch模型对28位评委在托幼机构教育质量评价中的评委偏差进行了分析。分析结果显示：28名评委评分宽严度差异显著;3名评委内部一致性较差,其余25名评委内部一致性较稳定;评委与评价班级的交互作用不显著,与评价项目的交互作用显著。研究结果表明MFRM可以对托幼机构教育质量评价的评委偏差进行个体层面的具体分析,从项目反应理论的视角为托幼机构教育质量评价的评委针对性培训、评估评委的合格性从而建立合格评委库等提供现代教育、心理测量学依据。     

Reliability of Ratings of Children’s Expressive Reading

This study examined ratings of fourth graders’ oral reading expression. Randomly assigned participants (n = 36) practiced repeated readings using narrative or informational passages for 7 weeks. After this period raters used the Multidimensional Fluency Scale (MFS) on two separate occasions to rate students’ expressive reading of four equivalent passages. Results of this generalizability study showed that a minimum of two and preferably three equivalent passages, two raters, and one rating occasion are recommended to obtain reliable ratings. This research substantiates the reliability of the MFS and demonstrates the importance of raters collaborating and finding texts at students’ independent reading levels.     

The design of the MBT‐G adherence and quality scale

Few group psychotherapy studies focus on therapists' interventions, and instruments that can measure group psychotherapy treatment fidelity are scarce. The aim of the present study was to evaluate the reliability of the Mentalization‐based Group Therapy Adherence and Quality Scale (MBT‐G‐AQS), which is a 19‐item scale developed to measure adherence and quality in mentalization‐based group therapy (MBT‐G). Eight MBT groups and eight psychodynamic groups (a total of 16 videotaped therapy sessions) were rated independently by five raters. All groups were long‐term, outpatient psychotherapy groups with 1.5 hours weekly sessions. Data were analysed by a Generalizability Study (G‐study and D‐study). The generalizability models included analyses of reliability for different numbers of raters. The global (overall) ratings for adherence and quality showed high to excellent reliability for all numbers of raters (the reliability by use of five raters was 0.97 for adherence and 0.96 for quality). The mean reliability for all 19 items for a single rater was 0.57 (item range 0.26–0.86) for adherence, and 0.62 (item range 0.26–0.83) for quality. The reliability for two raters obtained mean absolute G‐coefficients on 0.71 (item range 0.41–0.92 for the different items) for adherence and 0.76 (item range 0.42–0.91) for quality. With all five raters the mean absolute G‐coefficient for adherence was 0.86 (item range 0.63–0.97) and 0.88 for quality (item range 0.64–0.96). The study demonstrates high reliability of ratings of MBT‐G‐AQS. In models differentiating between different numbers of raters, reliability was particularly high when including several raters, but was also acceptable for two raters. For practical purposes, the MBT‐G‐AQS can be used for training, supervision and psychotherapy research.     

Rater Personality,Rating Format,and Social Context: Implications for Performance Appraisal Ratings

This study investigates the effects of rater personality (Conscientiousness and Agreeableness), rating format (graphic rating scale vs. behavioral checklist), and the rating social context (face‐to‐face feedback vs. no face‐to‐face feedback) on rating elevation of performance ratings. As predicted, raters high on Agreeableness showed more elevated ratings than those low on Agreeableness when they expected to have the face‐to‐face feedback meeting. Furthermore, rating format moderated the relationship between Agreeableness and rating elevation, such that raters high on Agreeableness provided less elevated ratings when using the behavioral checklist than the graphic rating scale, whereas raters low on Agreeableness showed little difference in elevation across different rating formats. Results also suggest that the interactive effects of rater personality, rating format, and social context may depend on the performance level of the ratee. The implications of these findings will be discussed.     

Rater agreement,accuracy, and experienced cognitive load: Comparison of distributional and traditional assessment approaches to rating performance

Error in performance ratings is typically believed to be due to the cognitive complexity of the rating task. Distributional assessment (DA) is proposed to improve rater accuracy by reducing cognitive load. In two laboratory studies, raters reported perceptions of cognitive effort and difficulty while assessing rating targets using DA or the traditional assessment approach. Across both studies, DA raters showed greater interrater agreement, and Study 2 findings provide some support for DA being associated with greater true score rating accuracy. However, DA raters also reported experiencing greater cognitive load during the rating task, and cognitive load did not mediate the relationship between rating format and rater accuracy. These findings have important implications regarding our understanding of cognitive load in the rating process.     

Gender-based influences: The promotion recommendation

This study used a between-subjects analysis of variance design (2 × 2 × 2 × 2) to estimate the effects of rater gender, ratee gender, the gender stereotype of the job, and the gender stereotype of the ratee's personal characteristics on a promotion decision. The results indicated that female employees with masculine characteristics were evaluated as most promotable regardless of the gender stereotype of the job or the gender of the rater. We also investigated the personal characteristics raters reported using in making their decisions. We asked raters to select ratee personal characteristics that most influenced their promotion decisions. Discriminant analyses showed that male and female raters approached the rating process differently and that it is predominantly the gender stereotype of the ratee's personal characteristics rather than the ratee's gender that influences the promotion process.     

The relative contribution of profile body shape and weight to judgements of women's physical attractiveness in Britain and Malaysia

Ninety-six Malaysian and British men rated for physical attractiveness a set of photographs of real women in profile, with known body mass index (BMI) and waist-to-hip ratio (WHR). Results showed that BMI accounted for the greater amount of variance in all settings. There were also significant differences in preferences for body weight, with low resource, low socioeconomic status (SES) raters preferring a significantly heavier partner than high resource, high SES raters. The disparity with previous findings using line drawings of women in profile was discussed in terms of the weaknesses of line-drawn stimuli.     

Do all raters value task,citizenship, and counterproductive behaviors equally: An investigation of cultural values and performance evaluations

We investigated whether individualistic and collectivistic cultural values influence the extent to which raters consider task, citizenship (OCB), and counterproductive work behaviors (CWB) when evaluating overall employee performance. Participants completed a managerial role-play exercise in which they read employee performance vignettes and rated the overall performance of each employee. A relative weights approach was used to determine to what extent raters considered task, OCB, and CWB information when evaluating employee performance. Results indicated that as compared to individualistic raters, collectivistic raters placed higher weights on OCBs and less weight on task performance when assigning an overall performance rating. However, contrary to expectations, collectivistic raters did not place significantly higher weights on CWBs as compared to individualistic raters. Future research directions and practical implications are discussed.     

The exact variance of weighted kappa with multiple raters

Weighted kappa described by Cohen in 1968 is widely used in psychological research to measure agreement between two independent raters. Everitt then provided the exact variance for weighted kappa for two raters. In this paper, Everitt's exact variance is extended to three or more raters.     

Resampling probability values for weighted kappa with multiple raters

A new procedure to compute weighted kappa with multiple raters is described. A resampling procedure to compute approximate probability values for weighted kappa with multiple raters is presented. Applications of weighted kappa are illustrated with an example analysis of classifications by three independent raters.     

国家公务员结构化面试中评委偏差的IRT分析

使用项目反应理论(IRT)中的多面Rasch模型,对两组共12名评委在国家公务员结构化面试中的评委偏差进行了分析。提出并验证了两种评委偏差：评委之间在宽严程度上的差异和评委自身的一致性问题。结果发现：不同评委之间在宽严程度上差异显著,且不同评委评定行为的跨考生、跨维度、跨性别、跨时间的自身一致性也存在差异。研究表明,这种进入到评委个体层次的分析突破了经典测量理论(CTT)定位于评委群体进行分析的局限,针对每位评委的偏差行为提供了详细具体的诊断信息,从而为评委的针对性培训和评委库的建立提供了现代测量学的新方法     

Accuracy of National Stereotypes in Central Europe: Outgroups Are Not Better than Ingroup in Considering Personality Traits of Real People

In a study on national stereotypes in central Europe—composed of Austria, the Czech Republic, Germany, Poland and Slovakia—2241 participants rated their autostereotype (a typical representative of their own country) and heterostereotypes (typical representatives of the other countries) by using National Character Survey (NCS). Existing data from 17377 participants including self‐reports or observer ratings on Revised NEO Personality Inventory and NCS were compared with the national autostereotypes and heterostereotypes. Although national autostereotypes converged with personality traits of real people in Poland and an adult subsample in the Czech Republic, national heterostereotypes did not correspond to personality traits of real people in any of the studied countries. National stereotypes were shared within as well as across countries. In heterostereotypes, raters from similar cultural backgrounds speaking similar languages agreed better as compared with raters from more distant cultures. Target country played a role in agreement of raters from different countries, showed in the highest convergence between autostereotypes and heterostereotypes of a typical German. Sharing of national stereotypes is influenced by political and economic significance of the target country. Although national autostereotypes clearly differentiated between typical representatives of central European countries, the comparison of personality profiles of their inhabitants showed remarkable resemblance. Copyright © 2013 European Association of Personality Psychology.     

Explaining the surprisingly weak relationship between organizational constraints and job performance

Organizational constraints are contextual factors that interfere with task performance. Study 1 was a meta-analysis of constraints-performance studies that found a smaller than expected relationship. Two additional studies were designed to understand why constraints fail to relate as expected to performance. Study 2 found support for the rater-bias hypothesis that raters take constraints into consideration when rating performance. Study 3 showed that constraints were both a challenge and a hindrance, and the two components tend to cancel one another. Two important conclusions that arise are that the constraints–performance relationship observed in field studies using raters is an underestimate of the true relationship, and constraints are not simply a hindrance, but can have a challenging effect that enhances motivation, as well.