Understanding the latent structure of job performance ratings

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n = 2,350 and n = 2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%.     

SINCE WHEN IS NO NEWS GOOD NEWS? THE RELATIONSHIP BETWEEN PERFORMANCE AND RESPONSE RATES IN MULTIRATER FEEDBACK

Despite the popularity of multirater feedback in practice and research, few studies have examined the issue of response rates in these efforts. This study explored the relationship between performance, vis-á-vis a measure of service quality, and feedback response rates in a large-scale developmental multirater feedback initiative using data from 538 senior service providers, 4,446 coworkers and supervisors, and 1,617 clients. The number of rater responses that the focal individual received was largely unrelated to his or her performance level as rated by his or her clients. More specifically, less than 2% of the variance in response rates was explained by the focal individual's performance. Data representativeness and feedback acceptance implications are discussed.     

Sources of Variance in Personality Facets: A Multiple‐Rater Twin Study of Self‐Peer,Peer‐Peer,and Self‐Self (Dis)Agreement

ABSTRACT This study considered the validity of the personality structure based on the Five‐Factor Model using both self‐ and peer reports on twins' NEO‐PI‐R facets. Separating common from specific genetic variance in self‐ and peer reports, this study examined genetic substance of different trait levels and rater‐specific perspectives relating to personality judgments. Data of 919 twin pairs were analyzed using a multiple‐rater twin model to disentangle genetic and environmental effects on domain‐level trait, facet‐specific trait, and rater‐specific variance. About two thirds of both the domain‐level trait variance and the facet‐specific trait variance was attributable to genetic factors. This suggests that the more personality is measured accurately, the better these measures reflect the genetic structure. Specific variance in self‐ and peer reports also showed modest to substantial genetic influence. This may indicate not only genetically influenced self‐rater biases but also substance components specific for self‐ and peer raters' perspectives on traits actually measured.     

Reconsidering the Equivalence of Multisource Performance Ratings: Evidence for the Importance and Meaning of Rater Factors

Purpose

The study specified an alternate model to examine the measurement invariance of multisource performance ratings (MSPRs) to systematically investigate the theoretical meaning of common method variance in the form of rater effects. As opposed to testing invariance based on a multigroup design with raters aggregated within sources, this study specified both performance dimension and idiosyncratic rater factors.

Design/Methodology/Approach

Data was obtained from 5,278 managers from a wide range of organizations and hierarchical levels, who were rated on the BENCHMARKS^® MSPR instrument.

Findings

Our results diverged from prior research such that MSPRs were found to lack invariance for raters from different levels. However, same level raters provided equivalent ratings in terms of both the performance dimension loadings and rater factor loadings.

Implications

The results illustrate the importance of modeling rater factors when investigating invariance and suggest that rater factors reflect substantively meaningful variance, not bias.

Originality/Value

The current study applies an alternative model to examine invariance of MSPRs that allowed us to answer three questions that would not be possible with more traditional multigroup designs. First, the model allowed us to examine the impact of paramaterizing idiosyncratic rater factors on inferences of cross-rater invariance. Next, including multiple raters from each organizational level in the MSPR model allowed us to tease apart the degree of invariance in raters from the same source, relative to raters from different sources. Finally, our study allowed for inferences with respect to the invariance of idiosyncratic rater factors.     

AN INVESTIGATION OF FACTORS EXPECTED TO AFFECT FEEDBACK SEEKING: A LONGITUDINAL FIELD STUDY

This study of 137 helicopter pilot trainees investigated individual strategies used to obtain performance feedback during two consecutive phases of their training. Individual and situational factors cited in previous research were investigated as predictors of two feedback seeking behaviors: eliciting (asking directly for feedback) and monitoring (using indirect techniques, such as observing, to gain additional feedback). Both individual and situational factors were significant predictors of feedback seeking behaviors. Feedback seeking costs and the student pilots'external propensity (an individual difference measure assessing their desire for external feedback) were found to be the most consistent predictors of feedback eliciting and monitoring, both within and across the two training phases. In addition, the results point to higher feedback eliciting when performance was rated as low. The implications of this research are discussed, especially with respect to training.     

Evidence for genetic influences on personality from self-reports and informant ratings.

Self-report data on Extraversion (E) and Neuroticism (N), together with ratings by the co-twin, were obtained from a sample of 826 adult female twin pairs ascertained through a population-based twin register. Data were analyzed using a model that allowed for the contributions to personality ratings of the rater's personality (rater bias) as well as of the personality of the person being rated. For E, but not for N, significant rater bias was found, with extraverted respondents tending to underestimate, and introverted respondents tending to overestimate, the Extraversion of their co-twins. Good agreement between self-reports and ratings by the respondent's co-twin was found for both E and N. Substantial genetic influences were found for both personality traits, confirming findings from genetic studies of personality that have relief only on self-reports of respondents.     

多面Rasch模型在结构化面试中的应用

使用项目反应理论中的多面Rasch模型,对66名考生在结构化面试中的成绩进行分析,剔除了由于评委等具体测量情境因素引入的误差对原始分数的影响,得到考生的能力估计值以及个体水平的评分者一致性信息。对基于考生能力估计值和考生面试分得到的决策结果进行比较,发现测量误差的确对决策造成影响,对个别考生的影响甚至相当巨大。进一步使用Facets偏差分析以及评委宽严程度的Facets分析追踪误差源。结果表明,将来自不同面试组的被试进行面试原始成绩的直接比较,评委的自身一致性和评委彼此之间在宽严程度上的差异均将导致误差。研究表明,采用Facets的考生能力估计值作为决策的依据将提高选拔的有效性。同时,Facets分析得到的考生个体层次的评分者一致性指标,以及评委与考生的偏差分析等研究结果还可以为面试误差来源的定位提供详细的诊断信息     

Responses to verifying and enhancing appraisals from romantic partners: the role of trait importance and trait visibility

An experiment assessed when people respond more positively to verifying and enhancing appraisals from romantic partners. Two‐hundred and fifty‐eight individuals comprising 129 dating couples participated in this research. Couples privately rated their self‐concept on traits that were either high or low on trait visibility, rated how important each trait was to them, and rated their partners. A computer program ostensibly compared their self‐ratings with appraisals from their partners on traits they selected as being high or low in personal importance, and participants received either verifying or enhancing feedback. Confirming predictions, people believed their partners understood them more when they received verifying feedback, but felt their partners saw the best in them when they received enhancing feedback. Additionally, people responded more positively to verifying appraisals on important, less visible traits, and enhancing appraisals on important, highly visible traits. Results are discussed in terms of preferences for enhancing and verifying feedback in romantic relationships. Copyright © 2005 John Wiley & Sons, Ltd.     

Differences in Inter‐Rater Reliability and Accuracy for a Treatment Adherence Scale

Inter‐rater reliability and accuracy are measures of rater performance. Inter‐rater reliability is frequently used as a substitute for accuracy despite conceptual differences and literature suggesting important differences between them. The aims of this study were to compare inter‐rater reliability and accuracy among a group of raters, using a treatment adherence scale, and to assess for factors affecting the reliability of these ratings. Paired undergraduate raters assessed therapist behavior by viewing videotapes of 4 therapists' cognitive behavioral therapy sessions. Ratings were compared with expert‐generated criterion ratings and between raters using intraclass correlation (2,1). Inter‐rater reliability was marginally higher than accuracy (p = 0.09). The specific therapist significantly affected inter‐rater reliability and accuracy. The frequency and intensity of the therapists' ratable behaviors of criterion ratings correlated only with rater accuracy. Consensus ratings were more accurate than individual ratings, but composite ratings were not more accurate than consensus ratings. In conclusion, accuracy cannot be assumed to exceed inter‐rater reliability or vice versa, and both are influenced by multiple factors. In this study, the subject of the ratings (i.e. the therapist and the intensity and frequency of rated behaviors) was shown to influence inter‐rater reliability and accuracy. The additional resources needed for a composite rating, a rating based on the average score of paired raters, may be justified by improved accuracy over individual ratings. The additional time required to arrive at a consensus rating, a rating generated following discussion between 2 raters, may not be warranted. Further research is needed to determine whether these findings hold true with other raters and treatment adherence scales.     

Rater bias in the EASI temperament scales: a twin study

Under trait theory, ratings may be modeled as a function of the temperament of the child and the bias of the rater. Two linear structural equation models are described, one for mutual self- and partner ratings, and one for multiple ratings of related individuals. Application of the first model to EASI temperament data collected from spouses rating each other shows moderate agreement between raters and little rating bias. Spouse pairs agree moderately when rating their twin children, but there is significantly rater bias, with greater bias for monozygotic than for dizygotic twins. MLE's of heritability are approximately .5 for all temperament scales with no common environmental variance. Results are discussed with reference to trait validity, the person-situation debate, halo effects, and stereotyping. Questionnaire development using ratings on family members permits increased rater agreement and reduced rater bias.     

Evidence that projection of a feared trait can serve a defensive function

Two experiments tested the notion that allowing people to project a feared trait onto another individual would facilitate denial of the trait. In Study 1, participants were given feedback that they were high or low in repressed anger and were allowed to rate an ambiguous target on anger or not. Participants who received high (vs. low) anger feedback rated the target especially high on anger. In addition, participants who received high anger feedback and who were allowed to project their anger had the lowest anger accessibility on a word completion exercise. Study 2 replicated these basic findings using a different trait dimension (dishonesty) and a direct measure of denial (self-attributions of dishonesty). Specifically, in Study 2, participants who received high dishonesty feedback and who were allowed to project dishonesty reported having an especially low level of dishonesty. Discussion focused on the relationship between classic projection and other forms of psychological defense.     

Differential item functioning in an international 360-degree assessment: Evidence of gender stereotype,environmental complexity,and organizational contingency

This research used logistic regression to model item responses from a popular 360-degree-for-development survey used in a leadership development programme given to middle and upper level European managers in Brussels. The survey contained 106 items on 16 scales. The model used gender of ratee and rater group to identify items that exhibited differential item functioning (DIF). The rater groups were self, boss, peer, and direct report. The sample consisted of 356 survey families where a survey family consisted of a matched set of four surveys: one self, one boss, one peer, and one direct report. The sample contained 88% male and 12% female raters. The sample contained 1424 total surveys. The procedure for flagging items exhibiting differential functioning used effect size computed from Wald chi-square statistics rather than statistical significance, resulting in fewer flagged items. One item exhibited rating anomalies due to the gender of the ratee; 55 items exhibited DIF attributable to rater group. The apparent effect of the DIF was small with each item. An examination of the maximum likelihood parameter estimates suggested the rater group DIF was the result of either hierarchical complexity or organizational contingency. The DIF due to gender conformed to prior expectations of gender-related stereotypical interpretations. This research further suggested that DIF due to environmental complexity or organizational contingency could be a naturally occurring phenomenon in some 360-degree assessment, and that the interpretation of some 360-degree feedback could need to include the potential for such DIF to exist.     

创造力测评中的评分者效应

创造力测评中的评分者效应(rater effects)是指在创造性测评过程中, 由于评分者参与而对测评结果造成的影响.评分者效应本质上源于评分者内在认知加工的不同, 具体体现在其评分结果的差异.本文首先概述了评分者认知的相关研究, 以及评分者,创作者,社会文化因素对测评的影响.其次在评分结果层面梳理了评分者一致性信度的指标及其局限, 以及测验概化理论和多面Rasch模型在量化,控制该效应中的应用.最后基于当前研究仍存在的问题, 指出了未来可能的研究方向, 包括深化评分者认知研究,整合不同层面评分者效应的研究, 以及拓展创造力测评方法和技术等.     

Construction by Interpersonal Context and Relationship to Psychological Outcomes

The repertory grid was used to elicit personal constructs with 10 elements, including three interpersonal self roles, in 33 participants (age M = 20.79, SD = 2.70). Each participant also rated a selection of supplied personality trait constructs and completed several psychological outcome measures. The distance between the self roles was associated with higher levels of anxiety for both personal and the supplied trait constructs, and was also related to greater cognitive complexity for personal constructs. The lack of statistical association between anxiety and cognitive complexity, however, suggested the distance relationships to each outcome are due to some other factor. Based on previous research findings, the overall pattern of results suggests that the grid distances between each interpersonal self is a due to the individual's behavioral flexibility or situational changeability. The findings demonstrate the importance of distinguishing between personal and supplied trait constructs.     

All smiles perceived equally: Facial expressions trump target characteristics in impression formation

Race, gender, and emotionally expressive facial behavior have been associated with trait inferences in past research. However, it is unclear how interactions among these factors influence trait perceptions. In the current research, we test the roles of targets’ race, gender, and facial expression along with participants’ culture in predicting personality ratings. Caucasian and Asian-American participants rated the big-5 personality traits of either smiling or inexpressive photographs of Caucasian and Asian male and female faces. Ratings of extraversion, agreeableness, and conscientiousness differed significantly across inexpressive targets as a function of race and gender categorization and individual characteristics. Smiling was associated with reduced variation in perceptions of targets’ extraversion and agreeableness relative to ratings made of inexpressive targets. In addition, participant culture generally did not significantly impact trait ratings. Results suggest that emotionally expressive facial behavior reduces the use of information based on race or gender in forming impressions of interpersonally relevant traits.     

Decomposing a Sense of Superiority: The Differential Social Impact of Self-Regard and Regard for Others

The tendency to self-enhance has been related to a host of beneficial psychological outcomes (Taylor & Brown, 1988), although some negative social consequences have also been identified (Colvin et al., 1995, Paulhus, 1998). One operationalization of self-enhancement is derived by subtracting the rater's evaluations of others from his or her self-ratings to yield a measure of the rater's sense of superiority/inferiority, i.e., rater-derived self-enhancement. The present research assessed the psychological and social correlates of a person's sense of superiority in groups whose members worked on tasks together for 3 months. A sense of superiority was scored as a composite but also separated into its two components, self-regard and regard for others, to determine if these components of a sense of superiority have separate relationships to psychological and social processes. A sense of superiority evidenced the same self-rated psychological benefits as had been found in Western research, though it showed both positive and negative social outcomes, as assessed on an eight-factor measure of the target's personality rated by his or her other group members. Positive psychological characteristics and a stereotypically masculine reputation were associated with higher levels of self-regard; lower levels of self-rated Agreeableness, a stereotypically nonfeminine reputation, and lower liking were associated with lower levels of regard for others. Given their different functions, it is proposed that self-regard and regard for others be separated in future research and attention directed toward characterizing the behavioral profiles of those high and low in these two measures of basic personality orientation.     

Rating scales for hyperactivity: Concurrent validity,reliability, and decisions to label for the Conners and Davids abbreviated scales

Rating scales have become the instrument of choice in labeling and assessing change in behavior of hyperactive children. However, several criticisms have recently have levied against their use. The present investigation examined the concurrent validity, and inter- and intrarater reliability for the Abbreviated Teacer Questionnaire (ATQ, Conners, 1973) and the Rating Scales for Hyperkinesis (Davids, 1971). Sixteen teachers from two special and two regular schools (grades 1-4) rated 211 normal and 49 special children using both scales. High correlations were found suggesting excellent predictability between scales and considerable stability across time and rater. Lower scores on a subsequent rating relative to an initial rating were demonstrated, dependent on time between ratings but independent of (a) teacher expectation of treatment gains, (b) bias produced by rating selected children, and (c) whether children were hyperactive or normal. Use of initial and infrequent rating scores versus subsequent, closely spaced ratings was related to the rater's objective (e.g., diagnosis, treatment, or assessment).     

Hindsight bias in insight and mathematical problem solving: Evidence of different reconstruction mechanisms for metacognitive versus situational judgments

This article presents two experiments that used insight and mathematical problems to investigate whether different factors would affect hindsight bias on metacognitive and situational judgments. In both studies, participants initially rated their likelihood of solving each problem within a certain amount of time (metacognitive judgments) and rated the importance of each component of the problem for finding the solution (situational judgments). Next, participants attempted to solve each problem. In Experiment 1, all participants were given solution feedback information, but in Experiment 2, participants were not given any solution feedback. After 1 week, participants were asked to recall their original judgments. Hindsight bias was assessed by comparing the initial with the final ratings. Insight problems and math problems showed different patterns of hindsight bias effects on the metacognitive and situational judgments. The results suggest that two competing models of hindsight effects are actually complementary explanations for judgment reconstruction on different types of judgment tasks.     

The Effects of Source Credibility and Performance Rating Discrepancy on Reactions to Multiple Raters1

Using multiple feedback sources, the present study investigated the effects of source credibility and performance rating discrepancy on recipients' reactions. Individuals performed an ambiguous group task, rated their own performance on the task, and were later provided bogus feedback ostensibly from their peers and an expert rater. Individuals reacted toward the feedback and the source of the feedback as a function of the rating discrepancy and credibility of the feedback source. Generally, more credible sources and their feedback were evaluated more favorably. However, as predicted, this effect was overcome by performance rating discrepancy in the predicted conditions. The results show the importance of studying the interactive effects of message and source characteristics on individuals' reactions.