Examining contrast effects in performance appraisals: using appropriate controls and assessing accuracy

In research on contrast effects in performance appraisals, control conditions or measures of accuracy have rarely been used. In the present study, the authors included appropriate controls and used expert ratings to develop "true scores" for assessing accuracy. The study is an examination of the influence of 3 variables on performance ratings: (a) the sequence of viewing and rating performance, (b) the delay between viewing and rating performance, and (c) whether the target of the performance rating was the same person as the anchor. Experimental conditions did not have the expected differential effects on target ratings, but target ratings in all the experimental conditions showed contrast effects when compared with ratings in relevant control conditions. The target ratings in experimental conditions were accurate, however, as assessed by comparisons with true scores, thus raising questions about the relationship between contrast effects and accuracy.     

A META-ANALYSIS OF SELF-SUPERVISOR, SELF-PEER, AND PEER-SUPERVISOR RATINGS

Reviews of self–supervisor, self–peer, and peer–supervisor ratings have generally concluded that there is at best a modest correlation between different rating sources. Nevertheless, there has been much inconsistency across studies. Accordingly, a meta-analysis was conducted. The results indicated a relatively high correlation between peer and supervisor ratings (ρ= .62) but only a moderate correlation between self-supervisor (ρ= .35) and self-peer ratings (ρ= .36). While rating format (dimensional versus global) and rating scale (trait versus behavioral) had little impact as moderators, job type (managerial/professional versus blue-collar/service) did seem to moderate self-peer and self-supervisor ratings.     

The Influence of Candidate Social Effectiveness on Assessment Center Performance Ratings: A field study

The present research examined the influence of constructs representing social effectiveness on assessment center (AC) ratings in two samples. We expected different effects of self‐monitoring (SM) on different dimension ratings, a positive effect of the ability to identify criteria (ATIC) on the overall AC rating and a moderating effect of the ATIC on the relationship between SM and the dimension rating. Forty‐six (Study 1) and 115 (Study 2) applicants participated in ACs in field settings. Across both studies, SM had a negative effect on the integrity rating. No relationship was identified between SM and social sensitivity or problem solving ratings. In Study 1, the ATIC had a positive effect on the overall AC rating. No support was identified for a moderating effect of the ATIC on the relationship between SM and the social sensitivity rating.     

Friend or Foe? Differential Use of the Self‐Based Heuristic as a Function of Relationship Satisfaction

ABSTRACT We report two studies investigating whether relationship satisfaction differentially influences the use of the "self-based heuristic" (SBH) or the degree to which an individual's own characteristics contribute to ratings of another's personality. Individuals rated themselves, a friend, and a person with whom they have experienced significant conflict (a "foe"); ratings were made on measures of the Big Five and trait affectivity. Replicating previous research, judges made greater use of the SBH when rating trait affectivity than when rating the Big Five. In addition, individuals were more likely to utilize the SBH when rating friends than when rating foes. Further, relationship satisfaction made significant independent contributions in accounting for the variance in trait ratings of others. These findings extend our understanding of the mechanisms involved with person perception beyond observable trait-related information.     

Three method factors explaining the low correlations between assessment center dimension ratings and scores on personality inventories

In general, correlations between assessment centre (AC) ratings and personality inventories are low. In this paper, we examine three method factors that may be responsible for these low correlations: differences in (i) rating source (other versus self), (ii) rating domain (general versus specific), and (iii) rating format (multi‐ versus single item). This study tests whether these three factors diminish correlations between AC exercise ratings and external indicators of similar dimensions. Ratings of personality and performance were combined in an analytical framework following a 2 × 2 × 2 (source, domain, format) completely crossed, within subjects design. Results showed partial support for the influence of each of the three method factors. Implications for future research are discussed. Copyright © 2004 John Wiley & Sons, Ltd.     

Assessing social desirability in family self-report

This study focused on social desirability in family members' self-reports. 32 clinical families (93 family members) were given self-report measures from the McMaster and Circumplex family-assessment models and a measure of social desirability. Clinicians assessed these families on clinical rating scales from the same models. Regression analyses were used to examine the relationship between self-reports, social desirability scores, and clinicians' ratings. It was expected that social desirability would be a suppressor variable (i.e., when accounted for, the similarity between clinicians' and family members' ratings would be enhanced). This did not occur; instead, social desirability was significantly but negatively correlated with ratings of pathology. Results provide evidence that correcting for social desirability on clinical pencil-and-paper tests is not supported.     

Words high and low in pleasantness as rated by male and female college students

No catalog of words currently available contains normative data for large numbers of words rated low or high in affect. A preliminary sample of 1,545 words was rated for pleasantness by 26–33 college students. Of these words, 274 were selected on the basis of their high or low ratings. These words, along with 125 others (Rubin, 1981), were then rated by additional groups of 62–76 college students on 5-point rating scales for the dimensions of pleasantness, imagery, and familiarity. The resulting mean ratings were highly correlated with the ratings obtained by other investigators using some of the same words. However, systematic differences in the ratings were found for male versus female raters. Females tended to use more extreme ratings than did males when rating words on the pleasantness scale. Also, females tended to rate words higher on the imagery and familiarity scales. Whether these sex differences in ratings represent cognitive differences between the sexes or merely differences in response style is a question that can be determined only by further research.     

A FIELD STUDY OF PERFORMANCE APPRAISAL PURPOSE: RESEARCH- VERSUS ADMINISTRATIVE- BASED RATINGS

Many researchers have discussed the theoretical and practical importance of rating purpose. Nevertheless, the body of empirical studies, the majority of which were conducted in a laboratory setting, focus on leniency. There has been little research on other effects of rating purpose. The present study examines 223 ratees in a field setting for whom there were both administrative-based performance appraisal ratings (which were actually used for personnel decisions) and research-based performance appraisal ratings (obtained for a validation study). Two of the hypotheses were supported; administrative ratings were more lenient than research-based ratings. The administrative-based ratings demonstrated a statistically significant relationship with ratee seniority, while the research-based ratings did not. There was mixed support for a third hypothesis: Research ratings were significantly correlated with a predictor, while the administrative ratings were not. The difference between the validity coefficients, however, was not significant. Contrary to the hypothesis, the rank order between administrative-based and research-based ratings was relatively high ( r = 33).     

The Marital Communication Rating Schedule: An instrument for clinical assessment

The Marital Communication Rating Schedule (MCRaS) is presented as an observationally based clinical rating system for assessing verbal behavior in marital communication. Data from 35 response display discussions lasting from 20 to 30 min each, which took place between 11 married couples, were used to examine aspects of the reliability and validity of the instrument. Three raters made independent ratings of 37 MCRaS categories for each husband and wife for each discussion period. Reliability among the raters was shown to be high when calculated within one scale point. Concurrent validity was assessed by comparing MCRaS ratings for four categories with observationally based validation criteria independently coded and measured. Results indicated that for three categories — negative statements, overgeneralizations, and amount of talk — ratings produced results that were similar to those yielded by laborious coding of audiotapes. For one category, opinions requested, a relationship between the ratings and coded data was not found. The validation results were discussed in terms of possible differences in the basis of ratings for the categories subjected to validation. Although further research is needed, it was concluded that MCRaS has many of the desirable qualities needed in a clinically useful, observationally based rating system.This investigation was conducted in connection with the Sociobehavioral Research Project at The University of Michigan when Joyce Borkin and Claude L. Walter were affiliated with the project.     

Attitudes, self-monitoring, and appraisal behaviors

This study examined the influence of attitudes and self-monitoring on leniency (elevation accuracy) of performance ratings and personnel decisions. In addition, moderating effects of self-monitoring on the relationship between attitudes and accuracy of ratings and decisions were investigated. Attitudes and self-monitoring tendency of 210 managers-professionals were measured, and ratings provided and decisions made by them were used to test 3 sets of hypotheses. Moderated regression and follow-up split-group analyses indicated that self-monitoring moderated the relationship between attitudes toward accurate appraisal and rating accuracy. Self-monitoring significantly influenced rating and decision accuracy such that accuracy declined with increasing level of self-monitoring. Results highlight the influence of rater's personality on appraisal behaviors. Implications of results and directions for future research are discussed.     

结构化面试中评分量表对评分准确性的影响

采用2(组内变量:量尺大小(25分和9分))×2(组间变量:评分方法(相对和绝对))的混合实验设计探讨评分量表对115名大学生新手评委评分准确性的影响。对于评分准确性,采用Cronbach1955年提出的四个指标,Elevation(EL)、Differential elevation(DE)、Stereotype accuracy(SA)、Differential Accuracy(DA)。结果发现,评分方法只在SA上主效应显著,量尺大小在只在DA上主效应边缘显著,评分方法和量尺大小在DE、SA和DA三个指标上均有交互作用。总体上看,在结构化面试评分中,对于评分准确性,相对评分量表优于绝对评分量表,小量尺量表优于大量尺量表。     

Rater Personality,Rating Format,and Social Context: Implications for Performance Appraisal Ratings

This study investigates the effects of rater personality (Conscientiousness and Agreeableness), rating format (graphic rating scale vs. behavioral checklist), and the rating social context (face‐to‐face feedback vs. no face‐to‐face feedback) on rating elevation of performance ratings. As predicted, raters high on Agreeableness showed more elevated ratings than those low on Agreeableness when they expected to have the face‐to‐face feedback meeting. Furthermore, rating format moderated the relationship between Agreeableness and rating elevation, such that raters high on Agreeableness provided less elevated ratings when using the behavioral checklist than the graphic rating scale, whereas raters low on Agreeableness showed little difference in elevation across different rating formats. Results also suggest that the interactive effects of rater personality, rating format, and social context may depend on the performance level of the ratee. The implications of these findings will be discussed.     

Rating scales for hyperactivity: Concurrent validity,reliability, and decisions to label for the Conners and Davids abbreviated scales

Rating scales have become the instrument of choice in labeling and assessing change in behavior of hyperactive children. However, several criticisms have recently have levied against their use. The present investigation examined the concurrent validity, and inter- and intrarater reliability for the Abbreviated Teacer Questionnaire (ATQ, Conners, 1973) and the Rating Scales for Hyperkinesis (Davids, 1971). Sixteen teachers from two special and two regular schools (grades 1-4) rated 211 normal and 49 special children using both scales. High correlations were found suggesting excellent predictability between scales and considerable stability across time and rater. Lower scores on a subsequent rating relative to an initial rating were demonstrated, dependent on time between ratings but independent of (a) teacher expectation of treatment gains, (b) bias produced by rating selected children, and (c) whether children were hyperactive or normal. Use of initial and infrequent rating scores versus subsequent, closely spaced ratings was related to the rater's objective (e.g., diagnosis, treatment, or assessment).     

Self‐Supervisor Agreement: The Influence of Feedback Seeking on the Relationship Between Self and Supervisor Ratings of Performance1

Self‐assessment research has continued to search for those factors that increase self‐other rating agreement. The current field study investigated the feedback‐seeking strategies (i. e., monitoring and inquiry) used by 125 employees to obtain performance information, as well as the relationship between feedback‐seeking strategy use and self‐supervisor performance‐rating agreement. Results indicate that the frequency of monitoring reported by employees significantly moderated the relationship between self and supervisor ratings of performance. Individuals who reported higher levels of feedback seeking through monitoring were more likely to have self‐assessments that were congruent with their supervisors' ratings of performance.     

The Influence of Situational and Interpersonal Variables in Projective Testing

Evidence from 85 adult medical outpatients supported psychometric comparability of the 2 halves of the Washington University Sentence Completion Test (SCT) Form 81 and of the female and male forms of the SCT. There was slightly stronger internal consistency for the first versus the second half of the SCT. Each half correlated highly with the ogive total protocol rating and 36-item-sum rating. Intercorrelations of the 2 halves with external measures also suggested essentially equivalent relations. For the 30 identical items across gender, the median correlation between individual item ratings with the item-sum ratings was nearly equal for women and men. When the 6 nonidentical items were considered with the identical items, the median item-total correlation was slightly higher for men (45) than women (41). This difference was accounted for by the slightly larger variability in the mate subsample. Practically speaking, the 2 halves and the female and male forms may be used with minimal concern regarding psychometric comparability in similar medical outpatient settings.     

The Washington University Sentence Complete Test: are the two halves alternate forms? Are the female and male forms comparable?

Evidence from 85 adult medical outpatients supported psychometric comparability of the 2 halves of the Washington University Sentence Completion Test (SCT) Form 81 and of the female and male forms of the SCT. There was slightly stronger internal consistency for the first versus the second half of the SCT. Each half correlated highly with the ogive total protocol rating and 36-item-sum rating. Intercorrelations of the 2 halves with external measures also suggested essentially equivalent relations. For the 30 identical items across gender, the median correlation between individual item ratings with the item-sum ratings was nearly equal for women and men. When the 6 nonidentical items were considered with the identical items, the median item-total correlation was slightly higher for men (45) than women (41). This difference was accounted for by the slightly larger variability in the mate subsample. Practically speaking, the 2 halves and the female and male forms may be used with minimal concern regarding psychometric comparability in similar medical outpatient settings.     

Pseudotaxonicity in MAMBAC and MAXCOV analyses of rating-scale data: turning continua into classes by manipulating observer's expectations

Taxometric procedures such as mean above minus below a cut and maximum covariance can determine whether a trait is distributed as a discrete latent class. These methods have been used to infer taxonic structure in several personality and psychopathology constructs, often from analyses of rating scale data. This is problematic given (a) well established biases in ratings, (b) the human tendency to think categorically, and (c) implicit typological models of personality and psychopathology among expert raters. Using an experimental method in which the cognitive sets of raters were manipulated as dimensional versus categorical, it is demonstrated that pseudotaxonicity can be created readily with rating scale measures. This suggests that researchers avoid an exclusive reliance on rating scales when conducting taxometrics investigations.     

Confidence and accuracy of lineup selections and rejections: Postdicting rejection accuracy with confidence

Correlation and calibration approaches show meaningful, positive confidence-accuracy relations for witnesses making selections from lineups, but rarely for rejections (Brewer and Wells, 2006, Sauerland and Sporer, 2009). This disparity may reflect the difference between selecting a single photo versus rejecting a set of photos. Participants (N = 101) in two experiments made selections from and rejections of lineups in situations requiring either a single confidence rating about a single face (typical of “choosers”) or a single confidence rating about multiple faces (typical of “nonchoosers”). Mean confidence ratings were significantly higher for accurate versus inaccurate decisions for both selections and rejections when decisions were based on single faces. Single decisions about multiple faces produced no significant difference in confidence between correct and incorrect rejections but a significant difference for selections.