Competencies,Personality Traits,and Organizational Rewards of Middle Managers: A Motive-Based Approach

Previous literature suggests that performance ratings are saturated with rater-related idiosyncratic variance. Given that modern psychometric theories relegate this source of variance to measurement error, it has not been the subject of much previous research. Of importance, identifying and estimating the variance components underlying idiosyncratic rater variance will inform our understanding of the nature of this variance. In a sample of managerial performance ratings we report on components of variance and find that the idiosyncratic rater variance component is about one third rater main effects variance, one third Rater × Ratee interaction effects variance, and one third upper-bound Rater × Ratee × Dimension interaction effects variance. Further, results indicate that variance components are moderated by the acquaintanceship time between the rater and the ratee.     

Are performance appraisal ratings from different rating sources comparable?

The purpose of this study was to test whether a multisource performance appraisal instrument exhibited measurement invariance across different groups of raters. Multiple-groups confirmatory factor analysis as well as item response theory (IRT) techniques were used to test for invariance of the rating instrument across self, peer, supervisor, and subordinate raters. The results of the confirmatory factor analysis indicated that the rating instrument was invariant across these rater groups. The IRT analysis yielded some evidence of differential item and test functioning, but it was limited to the effects of just 3 items and was trivial in magnitude. Taken together, the results suggest that the rating instrument could be regarded as invariant across the rater groups, thus supporting the practice of directly comparing their ratings. Implications for research and practice are discussed, as well as for understanding the meaning of between-source rating discrepancies.     

Reconsidering the Equivalence of Multisource Performance Ratings: Evidence for the Importance and Meaning of Rater Factors

Purpose

The study specified an alternate model to examine the measurement invariance of multisource performance ratings (MSPRs) to systematically investigate the theoretical meaning of common method variance in the form of rater effects. As opposed to testing invariance based on a multigroup design with raters aggregated within sources, this study specified both performance dimension and idiosyncratic rater factors.

Design/Methodology/Approach

Data was obtained from 5,278 managers from a wide range of organizations and hierarchical levels, who were rated on the BENCHMARKS^® MSPR instrument.

Findings

Our results diverged from prior research such that MSPRs were found to lack invariance for raters from different levels. However, same level raters provided equivalent ratings in terms of both the performance dimension loadings and rater factor loadings.

Implications

The results illustrate the importance of modeling rater factors when investigating invariance and suggest that rater factors reflect substantively meaningful variance, not bias.

Originality/Value

The current study applies an alternative model to examine invariance of MSPRs that allowed us to answer three questions that would not be possible with more traditional multigroup designs. First, the model allowed us to examine the impact of paramaterizing idiosyncratic rater factors on inferences of cross-rater invariance. Next, including multiple raters from each organizational level in the MSPR model allowed us to tease apart the degree of invariance in raters from the same source, relative to raters from different sources. Finally, our study allowed for inferences with respect to the invariance of idiosyncratic rater factors.     

Assessing measurement equivalence across rating sources: a multitrait-multirater approach

Research examining the structure of multisource performance ratings has demonstrated that ratings are a direct function of both who is doing the rating (rating source) as well as what is being rated (performance dimension). A separate line of research has focused on the extent to which performance ratings are equivalent across sources. To date no research has examined the measurement equivalence of multisource ratings within the context of both dimension and rating source direct effects on ratings. We examine the impact of both performance dimension and rating source as well as the degree of measurement equivalence across sources. Results indicate that (a) the impact of the underlying performance dimension is the same across rating sources, (b) the impact of rating source is substantial and only slightly smaller than the impact of the underlying performance dimension, and (c) the impact of rating source differs substantially depending on the source.     

TRAIT, RATER AND LEVEL EFFECTS IN 360-DEGREE PERFORMANCE RATINGS

Method and trait effects in multitrait-multirater (MTMR) data were examined in a sample of 2,350 managers who participated in a developmental feedback program. Managers rated their own performance and were also rated by two subordinates, two peers, and two bosses. The primary purpose of the study was to determine whether method effects are associated with the level of the rater (boss, peer, subordinate, self) or with each individual rater, or both. Previous research which has tacitly assumed that method effects are associated with the level of the rater has included only one rater from each level; consequently, method effects due to the rater's level may have been confounded with those due to the individual rater. Based on confirmatory factor analysis, the present results revealed that of the five models tested, the best fit was the 10-factor model which hypothesized 7 method factors (one for each individual rater) and 3 trait factors. These results suggest that method variance in MTMR data is more strongly associated with individual raters than with the rater's level. Implications for research and practice pertaining to multirater feedback programs are discussed.     

Rating leniency and halo in multisource feedback ratings: testing cultural assumptions of power distance and individualism-collectivism

This study extends multisource feedback research by assessing the effects of rater source and raters' cultural value orientations on rating bias (leniency and halo). Using a motivational perspective of performance appraisal, the authors posit that subordinate raters followed by peers will exhibit more rating bias than superiors. More important, given that multisource feedback systems were premised on low power distance and individualistic cultural assumptions, the authors expect raters' power distance and individualism-collectivism orientations to moderate the effects of rater source on rating bias. Hierarchical linear modeling on data collected from 1,447 superiors, peers, and subordinates who provided developmental feedback to 172 military officers show that (a) subordinates exhibit the most rating leniency, followed by peers and superiors; (b) subordinates demonstrate more halo than superiors and peers, whereas superiors and peers do not differ; (c) the effects of power distance on leniency and halo are strongest for subordinates than for peers and superiors; (d) the effects of collectivism on leniency were stronger for subordinates and peers than for superiors; effects on halo were stronger for subordinates than superiors, but these effects did not differ for subordinates and peers. The present findings highlight the role of raters' cultural values in multisource feedback ratings.     

Understanding the latent structure of job performance ratings

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n = 2,350 and n = 2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%.     

Measurement Equivalence and Multisource Ratings for Non-managerial Positions: Recommendations for Research and Practice

ABSTRACT: The present investigation applies a comprehensive sequence of confirmatory factor analysis tests (Vandenberg and Lance, Organisational Research Methods, 3, 4–69, 2000) to the examination of the measurement equivalence of self, peer, and supervisor ratings of non-managerial targets across several performance dimensions. Results indicate a high degree of measurement equivalence across rater sources and performance dimensions. The paper illustrates how this procedure can identify very specific areas of non-equivalence and how the complexity of a multisource feedback system may be represented using such procedures. Implications of these results and recommendations for both research and practice are offered.     

EXERCISES AND DIMENSIONS ARE THE CURRENCY OF ASSESSMENT CENTERS

A novel assessment center (AC) structure that models broad dimension factors, exercise factors, and a general performance factor is proposed and supported in 4 independent samples of AC ratings. Consistent with prior research, the variance attributable to dimension and exercise factors varied widely across ACs. To investigate the construct validity of these empirically supported components of AC ratings, the nomological network of broad dimensions, exercises, and general performance was examined. Results supported the criterion‐related validity of broad dimensions and exercises as predictors of effectiveness and success criteria as well as the incremental validity of broad dimensions beyond exercises and general performance. Finally, the relationships between individual differences and AC factors supported the construct validity of broad dimension factors and provide initial insight as to the meaning of exercise specific variance and general AC performance.     

Evidence for the Effectiveness of an Alternative Multisource Performance Rating Methodology

Despite persistent concerns as to the quality of performance information obtained from multisource performance ratings (MSPRs), little research has sought ways to improve the psychometric properties of MSPRs. Borrowing from past methodologies designed to improve performance ratings, we present a new method of presenting items in MSPRs, frame‐of‐reference scales (FORS), and test the efficacy of this method in a field and lab study. The field study used confirmatory factor analysis to compare the FORS to traditional rating scales and revealed that FORS are associated with increased variance due to dimensions, decreased overlap among dimensions, and decreased error. The laboratory study compared rating accuracy associated with FORS relative to frame‐of‐reference training (FORT) and a control group and demonstrated that FORS are associated with higher levels of accuracy than the control group and similar levels of accuracy as FORT. Implications for the design and implementation of FORS are discussed.     

DISENTANGLING THE MEANING OF MULTISOURCE PERFORMANCE RATING SOURCE AND DIMENSION FACTORS

We extend multisource performance rating (MSPR) construct validity research by examining the pattern of relationships between factor analytically derived MSPR rating source and performance dimension factors and externally measured constructs (e.g., assessment center dimensions, personality constructs, and intelligence). The pattern of relationships among MSPR dimensions and external constructs provides modest construct validity evidence for the MSPR dimensions. In addition, MSPR source factors were differentially correlated with externally measured constructs, suggesting that MSPR source effects represent substantively meaningful source specific variance, as opposed to bias. These findings are discussed in the context of managerial skill diagnosis and the efficacy of collecting performance data from multiple sources.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.     

Managerial Experience and the Measurement Equivalence of Performance Ratings

ABSTRACT: Establishing the measurement equivalence of instruments is a prerequisite to making meaningful comparisons between individuals or within individuals over time. Whereas previous research has investigated the effects of rater characteristics on the measurement equivalence of performance ratings, the current study investigated a ratee characteristic—ratee job experience. Using confirmatory factor analysis and item response theory methods with replication, the measurement equivalence of supervisor ratings of 7,200 managers with differing levels of managerial experience was assessed. Overall, results indicated a high degree of measurement equivalence suggesting that meaningful comparison may be made across ratees with different levels of job experience.     

PLACING PEER RATINGS IN CONTEXT: SYSTEMATIC INFLUENCES BEYOND RATEE PERFORMANCE

Performance evaluation research indicates that variance in ratings may be attributable to systematic sources beyond the actual performance of the ratee. However, the majority of prior work compares ratings across sources and uses ratings from a single rating event. Using confirmatory factor analysis and multivariate latent growth modeling (MLGM), we specifically examine peer ratings from 740 participants on 5 performance dimensions across 3 distinct performance situations for systematic sources of variance beyond ratee performance. Results demonstrate that both ratee performance and the performance context have systematic effects, with contextual effects varying by how "strong" or "weak" the situation is for a given performance dimension. Furthermore, MLGM results suggest that the influence of performance dynamism is only meaningfully interpreted when contextual effects are modeled.     

Multisource Assessments of Behavioral Competencies and Selection Interview Performance

This study sought to determine whether and how ratings of leadership in a developmental multisource feedback program compare against those obtained by leadership selection decision makers. Seventy‐seven senior managers received feedback on eight competencies that also formed the basis of a subsequent promotional interview for senior executive positions. Each rating source displayed a unique perspective on the participants, but only the supervisor and peer ratings predicted performance on the interview. As well, peer, subordinate, and self‐ratings failed to improve upon the supervisors' predictions. The results also suggested a general performance factor underlying both the multisource assessments and the interview ratings.     

The Performance Appraisal Milieu: A Multilevel Analysis of Context Effects in Performance Ratings

Purpose

The purpose of this study was to take an inductive approach in examining the extent to which organizational contexts represent significant sources of variance in supervisor performance ratings, and to explore various factors that may explain contextual rating variability.

Design/Methodology/Approach

Using archival field performance rating data from a large state law enforcement organization, we used a multilevel modeling approach to partition the variance in ratings due to ratees, raters, as well as rating contexts.

Findings

Results suggest that much of what may often be interpreted as idiosyncratic rater variance, may actually reflect systematic rating variability across contexts. In addition, performance-related and non-performance factors including contextual rating tendencies accounted for significant rating variability.

Implications

Supervisor ratings represent the most common approach for measuring job performance, and understanding the nature and sources of rating variability is important for research and practice. Given the many uses of performance rating data, our findings suggest that continuing to identify contextual sources of variability is particularly important for addressing criterion problems, and improving ratings as a form of performance measurement.

Originality/Value

Numerous performance appraisal models suggest the importance of context; however, previous research had not partitioned the variance in supervisor ratings due to omnibus context effects in organizational settings. The use of a multilevel modeling approach allowed the examination of contextual influences, while controlling for ratee and rater characteristics.

     

Designs for evaluating the validity and accuracy of performance ratings

For many jobs, objective measures of performance are not available or are impractical to obtain, such that ratings are the sole source of information. Unfortunately, performance ratings are a distorted source of information. The multitrait-multimethod and person perception designs have been used to investigate the distortions in ratings. The purpose of using these research designs is to isolate the factors that distort the ratings and to use this knowledge to improve the quality of performance ratings. The goal of the present research was to develop a design that combined both the multitrait-multimethod and person perception designs. Each design was discussed, and examples were presented to illustrate that design. The combination design was used to isolate the influence of rater, ratee, and context factors on the quality of performance ratings. Future research was recommended to understand and utilize these factors to improve performance ratings.     

Quantifying with words: An investigation of the validity of narrative‐derived performance scores

Performance appraisal research has focused almost entirely on traditional numerical ratings despite narrative text comments regularly being collected within appraisals. This study investigated the theory and utility of leveraging narrative comments to better understand employee performance. Narrative sentiment scores were derived using text mining on a large sample of narrative comments. These scores were then applied to an independent set of 2 years of performance data. It was assumed that narrative comments would reflect true performance variance that overlaps with traditional ratings, but also that they would capture incremental variance due to increases in total information and a reduction in rater‐motivated biases in contexts in which narrative data were not explicitly linked to administrative outcomes. The derived narrative scores were reliable across years, converged with traditional numerical ratings and explained incremental variance in future performance outcomes (performance ratings, involuntary turnover, promotions, and pay increases). Collectively, this study highlights how narratives can enhance performance measurement and demonstrates how these data can be economically scored in applied settings.     

Sex Effects in Performance--Related Judgments

This study examined the relationships among objective performance, performance ratings, and pay allocations made by male and female managers. Because rater sex has been found to affect performance ratings and pay allocations, but little research has explored the causes of these sex differences, this study explored the effects of contextual factors that may moderate differences between the performance ratings and pay allocations of male and female raters. Two hundred twenty-nine managers assigned performance ratings and merit pay allocations to performance profiles that varied in levels of objective performance. It was found that: (a) sex effects on performance ratings were moderated by contextual variables, and b) sex and objective performance did not influence merit pay allocations independent of their separate effects on performance ratings.