Multisource Feedback and Leaders' Goal Performance: Moderating effects of rating purpose,rater perspective,and performance dimension

The main purposes of this study are to examine whether multisource feedback ratings predict leaders' organizational goal performance, and whether the relationships are consistent across the two rating purposes (developmental, administrative), two leadership dimensions behaviors (Consideration, Initiating Structure), and three rating perspectives (supervisor, self, and ‘other’ raters, i.e., peers and subordinates). Leaders (n=396) in a large organization in the transportation industry participated in two multisource feedback programs, the first for developmental purposes and the second 8 months later for administrative purposes. Approximately 1 month later, they were rated by their supervisor on their effectiveness in attaining five organizational performance goals (financial, safety, customer satisfaction, employee satisfaction, diversity). Results revealed that both developmental ratings and administrative ratings uniquely predicted leaders' goal performance. However, both leadership dimension and rater perspective moderated these relationships. Leadership behaviors associated with Consideration were stronger predictors of goal performance for supervisor ratings, whereas behaviors associated with Initiating Structure were stronger predictors of goal performance for self and other ratings.     

RATER-RATEE RACE EFFECTS IN DEVELOPMENTAL PERFORMANCE RATINGS OF MANAGERS

The effects of rater and ratee race on performance ratings of managers were examined. Ratings were obtained from peers, subordinates and bosses as part of a multirater, developmental feedback program for managers. Two data sets were created for purposes of this study. The between-subjects data set consisted of ratings from over 20,000 bosses, over 50,000 peers, and over 40,000 subordinates. The repeated measures data set was substantially smaller because it included only those Black and White managers who were rated by both a Black and White rater from each of the three perspectives. Results for rater race indicated that Black raters from all perspectives (peers, subordinates, and bosses) assigned more favorable ratings to ratees of their own race. Results for White raters differed according to the particular rating source. White bosses assigned more favorable ratings to ratees of their own race, but White subordinates did not. White peers assigned more favorable ratings to Whites in the repeated measures analysis, but not in the between-subjects analysis. Results for ratee race indicated that both White and Black managers received higher ratings from Black raters than from White raters, and the effect was more pronounced for ratings assigned to Black managers.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.     

Understanding the latent structure of job performance ratings

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n = 2,350 and n = 2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%.     

Who Will Evaluate Me? Rater Selection in Multi‐Source Assessment Contexts

This study investigates the factors that motivate the preference of individuals, or ratees, participating in multi‐source assessment (MSA) processes for some raters over others. Two rater characteristics were assessed to attempt to identify these preferences: rater familiarity and the affect toward the rater. Two separate studies were conducted to assess the extent to which these characteristics are used. The extent to which the purpose of the appraisal (developmental vs. administrative) and the rating source (peer vs. subordinate) influenced the use of these characteristics was also investigated. Evidence from these two studies suggests that ratees selected their raters based on rater familiarity but not on affect. In addition, while the purpose of the appraisal did not influence selection patterns, the preference for peers was motivated by different factors than the preference for subordinates. The implications of these results for research and practice are discussed.     

TRAIT, RATER AND LEVEL EFFECTS IN 360-DEGREE PERFORMANCE RATINGS

Method and trait effects in multitrait-multirater (MTMR) data were examined in a sample of 2,350 managers who participated in a developmental feedback program. Managers rated their own performance and were also rated by two subordinates, two peers, and two bosses. The primary purpose of the study was to determine whether method effects are associated with the level of the rater (boss, peer, subordinate, self) or with each individual rater, or both. Previous research which has tacitly assumed that method effects are associated with the level of the rater has included only one rater from each level; consequently, method effects due to the rater's level may have been confounded with those due to the individual rater. Based on confirmatory factor analysis, the present results revealed that of the five models tested, the best fit was the 10-factor model which hypothesized 7 method factors (one for each individual rater) and 3 trait factors. These results suggest that method variance in MTMR data is more strongly associated with individual raters than with the rater's level. Implications for research and practice pertaining to multirater feedback programs are discussed.     

A Meta-Analysis of Incremental Validity and Nomological Networks for Subordinate and Peer Rating

Use of subordinates and peers for job performance ratings has increased markedly in recent years, on the assumption that these sources cover different parts of the criterion space and provide incremental validity over and above supervisors. This study directly tested this assumption using a meta-analytically derived correlation matrix including supervisor-, subordinate-, and peer-ratings as well as objective performance measures. Hierarchical regression results showed that both subordinates and peers accounted for significant variance in objective measures, over and above other sources. Further meta-analyses investigated nomological networks of subordinate and peer ratings, specifically relations with personality and ability measures. Ability correlations for peers and especially subordinates were lower than those previously found for supervisors. Correlations for personality dimensions affiliation and agreeableness, although low, were somewhat higher than previous supervisor values. Results provide justification for the added expense of including subordinates and peers in a rating system.     

Leniency effect as a function of rating format,purpose for appraisal,and rater individual differences

The effect of rating format and non-performance variables on rating leniency were studied in two law enforcement organizations. One of these variables, trust in the appraisal process, was defined as the extent to which a rater believes that fair and accurate appraisal will be made in the organization. A measure of trust in appraisal accounted for a significant proportion of variance in performance ratings. The purpose of appraisal (i.e., feedback or promotion) also accounted for rating variance. A mixed-standard rating format showed less susceptibility to the non-performance variables on the extent of leniency. Discussion centers on the usefulness of rater and organizational variables in performance appraisal research.     

Rating leniency and halo in multisource feedback ratings: testing cultural assumptions of power distance and individualism-collectivism

This study extends multisource feedback research by assessing the effects of rater source and raters' cultural value orientations on rating bias (leniency and halo). Using a motivational perspective of performance appraisal, the authors posit that subordinate raters followed by peers will exhibit more rating bias than superiors. More important, given that multisource feedback systems were premised on low power distance and individualistic cultural assumptions, the authors expect raters' power distance and individualism-collectivism orientations to moderate the effects of rater source on rating bias. Hierarchical linear modeling on data collected from 1,447 superiors, peers, and subordinates who provided developmental feedback to 172 military officers show that (a) subordinates exhibit the most rating leniency, followed by peers and superiors; (b) subordinates demonstrate more halo than superiors and peers, whereas superiors and peers do not differ; (c) the effects of power distance on leniency and halo are strongest for subordinates than for peers and superiors; (d) the effects of collectivism on leniency were stronger for subordinates and peers than for superiors; effects on halo were stronger for subordinates than superiors, but these effects did not differ for subordinates and peers. The present findings highlight the role of raters' cultural values in multisource feedback ratings.     

EXPLORING RELATIONS BETWEEN TYPICAL AND MAXIMUM PERFORMANCE RATINGS AND THE FIVE FACTOR MODEL OF PERSONALITY

The study tests the distinction between typical and maximum criteria with ratings of transformational leadership performance, and examines whether the criterion-related validities of the five factor model differ for the two types of criteria. Using an East Asian military sample ( n = 1,259) where multiple ratings of typical and maximum performance were obtained from different sources, we used structural equation modeling to test the typical/maximum performance distinction. Results found that typical and maximum performance are different latent constructs and that this distinction is present even after considering rating method factors (i.e., rater source, time). The importance of this distinction is shown by the fact that validities for the personality constructs were not equally predictive of both criteria: Openness was most predictive of maximum performance, Neuroticism was most predictive of typical performance, and Extroversion was predictive of both. By distinguishing typical from maximum performance constructs, relationships between personality and transformational leadership were found to be stronger than previous research suggested.     

MODESTY BIAS IN SELF-RATINGS OF PERFORMANCE: A TEST OF THE CULTURAL RELATIVITY HYPOTHESIS

Although self-ratings of performance are usually higher than ratings obtained from supervisors, Farh, Dobbins, and Cheng (1991) found that Taiwanese workers exhibited modesty bias (i.e., self-ratings that were lower than supervisory ratings). They explained their findings in terms of broad cultural differences between Taiwanese and Western workers. To test this cultural relativity hypothesis, we replicated their study using data from several organizations in mainland China. As is typically found in Western research, Chinese workers showed leniency in self-ratings (i.e., self-ratings that were higher than supervisor or peer ratings), which suggests that broad cultural factors do not explain the modesty bias reported by Farh et al.     

WHERE ALL THE CHILDREN ARE ABOVE AVERAGE: THE PERFORMANCE APPRAISAL PURPOSE EFFECT

More than 40 years ago, Taylor and Wherry (1951) hypothesized that performance appraisal ratings obtained for administrative purposes, such as pay raises or promotions, would be more lenient than ratings obtained for research, feedback, or employee development purposes. However, research on appraisal purpose has yielded inconsistent results, with roughly half of such studies supporting this hypothesis and the other half refuting it. To account for those differences, a meta-analysis of performance appraisal purpose research was conducted with 22 studies and a total sample size of 57,775. Our results support Taylor and Wherry's hypothesis as performance evaluations obtained for administrative purposes were, on average, one-third of a standard deviation larger than those obtained for research or employee development purposes. In addition, moderator analyses indicated larger differences between ratings obtained for administrative and research purposes when performance evaluations were made in field settings, by practicing managers, and for real world subordinates. Implications for researchers and practitioners are discussed.     

A New Old Question: Do contextual factors relate to rating behavior: An investigation with peer evaluations

The personality (dispositional) characteristic of self‐monitoring, beliefs and orientations toward the performance appraisal of 62 peer raters were collected in an attempt to predict peer rating behavior (i.e., mean ratings, rater discrimination, and performance appraisal dimensions discrimination). The beliefs and orientation toward appraisal systems emerged moderately related to rating behavior measures. Specific variables included in the category termed rating context revealed further illuminating effects. Furthermore, the moderating effect of self‐monitoring on the relationship rating the context‐rating behavior of peers was solidly supported. Theoretical interpretations of the results were discussed.     

FORMAL STRUCTURAL CHARACTERISTICS OF ORGANIZATIONS AND SUPERIOR-SUBORDINATE COMMUNICATION

This study attempted to determine the effects of several structural characteristics of organizations (i.e., organizational size, organizational level, and span of control) on subordinates' perceptions of openness in superior-subordinate communication. Data were collected from 15 different organizations and over 800 subordinates. While not accounting for a large amount of variance, results indicate that subordinates in the lowest levels of their organizational hierarchies perceive significantly less openness in superior-subordinate communication than subordinates at the highest levels of their hierarchies. In addition, although not statistically significant, the findings of the investigation suggest that subordinates in very large organizations perceive less openness than subordinates in small organizations. Implications of findings are discussed and recommendations for future research provided.     

PSYCHOMETRIC PROPERTIES OF SELF-APPRAISALS OF JOB PERFORMANCE

This paper reviews literature of psychometric properties of self-appraisals of work performance. It summarizes evidence of leniency, variability, halo, bias, and construct validity. Comparisons with appraisals by supervisors, peers, and subordinates suggest that self-appraisals tend to show more leniency, less variability, and less discriminant validity. Different factor structures have been found among self, supervisor and peer-ratings. On the other hand, self-appraisals showed less halo. Self-appraisals were significantly correlated with other sources in some studies and failed to correlate in many others. Existing data do not allow any conclusion whether the quality of self-appraisals is a function of scale format, amount or rater training, type of judgment, or purpose of appraisal. The effects of the observed psychometric qualities of self-appraisals on various applications are discussed. Problems may exist when they are used for administrative decision making, diagnosis of training needs, applied criterion measurement, measurement of constructs in basic research, or for selection purposes.     

Developmental relationships and managerial promotability in organizations: A multisource study

Managers are now focusing on developmental relationships by providing career-related mentoring to their direct reports, but research is lacking in showing whether such mentoring is in fact related to outcomes that benefit the manager. This study investigates whether self- and direct report ratings of the extent to which focal-managers provide career-related mentoring are associated with perceptions of their promotability as perceived by their bosses (n = 1623) and peers (n = 1597). Results of hierarchical regression indicated that both self- and direct report ratings of focal-managers' career-related mentoring were significant and positively related to boss and peer ratings of focal-managers' promotability. Within a self-other rating agreement framework, results of polynomial regression indicated that higher ratings of career-related mentoring by focal-managers and their direct reports were positively related to both boss and peer ratings of focal-managers' promotability. Furthermore, underrating (i.e., when self-ratings are lower than direct report ratings) was more positively related to promotability than overrating (i.e., when self-ratings are higher than direct report ratings).     

Gay male attraction toward muscular men: Does mating context matter?

The purpose of this study was to examine gay men's perceived attractiveness of male figures based on short-term and long-term partner contexts. A sample of 190 gay adult men rated the attractiveness of line-drawings depicting male figures varying systematically in muscularity and body fat percentage in both short-term and long-term dating contexts. Mixed effects modeling was used to estimate the effects of figure (muscularity and body fat), dating context (short-term vs. long-term), and individual rater characteristics on attractiveness ratings. Results indicated that figure muscularity and body-fat had significant non-linear (i.e., quadratic) relationships with attractiveness ratings, and short-term dating context was associated with more discriminating ratings of attractiveness. Interactions between individual characteristics and figure characteristics indicated that the more available the individual and lower body fat, the more discriminating they were in ratings of attractiveness. The implications for future investigations considering both object and observer characteristics of attractiveness preferences are discussed.     

In the Eye of the Beholder? Parental Ratings of Externalizing and Internalizing Symptoms

Variations in ratings of externalizing and internalizing symptoms may contain a trait (i.e., shared view) component when behavioral symptoms that generalize across context are perceived and an individual view component when they are misperceived or when each informant has access to different symptoms. Using a LISREL model, we estimated the trait and the informant-specific, individual view components in parental ratings of externalizing and internalizing symptoms of adolescent siblings. The model demonstrated that mothers' and fathers' ratings contained a substantial individual view component (from 21% to 50% of total rating variance, depending on rater and trait). Except for fathers' ratings of internalizing symptoms (13%), parental ratings also contained a substantial trait component (42% to 58%). Mother's, father's, and child's ratings may be averaged to estimate a trait of externalizing. To estimate an internalizing trait, it may be best to combine just the mother's rating with the child's self-rating.     

Sources of Variance in Personality Facets: A Multiple‐Rater Twin Study of Self‐Peer,Peer‐Peer,and Self‐Self (Dis)Agreement

ABSTRACT This study considered the validity of the personality structure based on the Five‐Factor Model using both self‐ and peer reports on twins' NEO‐PI‐R facets. Separating common from specific genetic variance in self‐ and peer reports, this study examined genetic substance of different trait levels and rater‐specific perspectives relating to personality judgments. Data of 919 twin pairs were analyzed using a multiple‐rater twin model to disentangle genetic and environmental effects on domain‐level trait, facet‐specific trait, and rater‐specific variance. About two thirds of both the domain‐level trait variance and the facet‐specific trait variance was attributable to genetic factors. This suggests that the more personality is measured accurately, the better these measures reflect the genetic structure. Specific variance in self‐ and peer reports also showed modest to substantial genetic influence. This may indicate not only genetically influenced self‐rater biases but also substance components specific for self‐ and peer raters' perspectives on traits actually measured.