Too Old to Train or Reprimand: The Role of Intergroup Attribution Bias in Evaluating Older Workers

Purpose

The purpose of this study was to explore the role rater and target age play in the evaluations of poorly performing workers. Intergroup attribution theory suggests that rater age predicts the attributions made for the poor performance of older workers.

Design/Methodology/Approach

In this study, 203 supervisors in various industries completed measures of causal attribution and evaluations for a poorly performing hypothetical subordinate.

Findings

Compared to the poor performance of younger targets, the poor performance of older targets was more likely to be attributed to external and controllable causes by older raters and more likely to be attributed to stable causes by younger raters. These attributions predicted willingness to punish and likelihood to provide training.

Implications

Our findings were partially supported by intergroup attribution theory and suggest that this theory may be useful in understanding how older workers’ performance is evaluated.

Originality/Value

This is one of the first studies to utilize intergroup attribution theory among supervisors in exploring how older workers are evaluated in the workplace and to demonstrate that the theory predicts how older workers’ poor performance will be attributed. Our study provides evidence that when evaluating a poorly performing older target, older raters will be more inclined to attribute this performance to controllable causes and thus be more punitive than younger raters. Further, we provide some evidence that raters will be more punitive and less willing to provide training when evaluating poorly performing targets to whom they are similar.     

Rater training revisited: An updated meta‐analytic review of frame‐of‐reference training

The present study updates Woehr and Huffcutt's (1994) rater training meta‐analysis and demonstrates that frame‐of‐reference (FOR) training is an effective method of improving rating accuracy. The current meta‐analysis includes over four times as many studies as included in the Woehr and Huffcutt meta‐analysis and also provides a snapshot of current rater training studies. The present meta‐analysis also extends the previous meta‐analysis by showing that not all operationalizations of accuracy are equally improved by FOR training; Borman's differential accuracy appears to be the most improved by FOR training, along with behavioural accuracy, which provides a snapshot into the cognitive processes of the raters. We also investigate the extent to which FOR training protocols differ, the implications of protocol differences, and if the criteria of interest to FOR researchers have changed over time.     

Effects of Organizational Perspective on Implicit Trait Policies About Correctional Officers' Job Performance

This study tests the general idea that people in different organizational positions can have different perspectives on what performance means in a particular job and that these different perspectives are reflected in their implicit trait policies (ITPs) about performance in that job. We test the hypothesis that prison inmates harbor ITPs about the importance of agreeableness and conscientiousness in correctional officers' performance that are different from the ITPs harbored by officers and supervisors. Our results show that critical incidents of officers' performance that are generated by inmates emphasize agreeableness more and conscientiousness less than critical incidents generated by officers and supervisors. Our results also show that inmates' ratings of the effectiveness of officers' behavior described in critical incidents are more strongly associated with the agreeableness of the behavior and less strongly associated with the conscientiousness of the behavior than are officers' and supervisors' ratings.     

The Effect of Feedback Packaging on Ratee Reactions1

In performance‐feedback situations, reactions to the rater have been examined rarely. A clearer understanding of what causes negative reactions toward raters might be used by them to better control feedback outcomes without having to distort the feedback message. In Study I, ratee reactions to the packaging of feedback messages were examined in a laboratory experiment. A legitimizing statement included in the feedback message resulted in more positive reactions to the rater than when no such statement was presented. In addition, the use of less personal feedback language resulted in more positive reactions to the rater than when more personal language was used. Neither legitimization statement nor type of language significantly impacted reactions to the feedback message or perceptions of performance, indicating that they did not distort the feedback message. In Study 2, a laboratory observation, the use of more personal language by the rater was related negatively to ratee confidence in rater judgment and rater likability. More research on feedback packaging, with the goal of training raters in how best to convey the feedback message, is needed.     

Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice.     

Reconsidering the Equivalence of Multisource Performance Ratings: Evidence for the Importance and Meaning of Rater Factors

Purpose

The study specified an alternate model to examine the measurement invariance of multisource performance ratings (MSPRs) to systematically investigate the theoretical meaning of common method variance in the form of rater effects. As opposed to testing invariance based on a multigroup design with raters aggregated within sources, this study specified both performance dimension and idiosyncratic rater factors.

Design/Methodology/Approach

Data was obtained from 5,278 managers from a wide range of organizations and hierarchical levels, who were rated on the BENCHMARKS^® MSPR instrument.

Findings

Our results diverged from prior research such that MSPRs were found to lack invariance for raters from different levels. However, same level raters provided equivalent ratings in terms of both the performance dimension loadings and rater factor loadings.

Implications

The results illustrate the importance of modeling rater factors when investigating invariance and suggest that rater factors reflect substantively meaningful variance, not bias.

Originality/Value

The current study applies an alternative model to examine invariance of MSPRs that allowed us to answer three questions that would not be possible with more traditional multigroup designs. First, the model allowed us to examine the impact of paramaterizing idiosyncratic rater factors on inferences of cross-rater invariance. Next, including multiple raters from each organizational level in the MSPR model allowed us to tease apart the degree of invariance in raters from the same source, relative to raters from different sources. Finally, our study allowed for inferences with respect to the invariance of idiosyncratic rater factors.     

托幼机构教育质量评价中评委偏差的多侧面Rasch分析

采用多侧面Rasch模型对28位评委在托幼机构教育质量评价中的评委偏差进行了分析。分析结果显示：28名评委评分宽严度差异显著;3名评委内部一致性较差,其余25名评委内部一致性较稳定;评委与评价班级的交互作用不显著,与评价项目的交互作用显著。研究结果表明MFRM可以对托幼机构教育质量评价的评委偏差进行个体层面的具体分析,从项目反应理论的视角为托幼机构教育质量评价的评委针对性培训、评估评委的合格性从而建立合格评委库等提供现代教育、心理测量学依据。     

督导重要事件对新手咨询师专业成长的影响

目的：了解个别督导过程中重要事件的类型、内涵及对新手咨询师专业成长的影响。方法：采用重要事件法，访谈4对受督者和督导师，并用共识性质性研究法分析。结果：有益重要事件包括督导关系、概念化、历程化、个人化4大类；督导促进了新手咨询师干预技巧能力、治疗计划目标等8个专业领域的成长。结论：督导对于新手咨询师专业成长非常重要。各类督导事件对于新手咨询师的促进作用并不均衡。     

督导重要事件对新手咨询师专业成长的影响

目的：了解个别督导过程中重要事件的类型、内涵及对新手咨询师专业成长的影响。方法：采用重要事件法,访谈4对受督者和督导师,并用共识性质性研究法分析。结果：有益重要事件包括督导关系、概念化、历程化、个人化4大类;督导促进了新手咨询师干预技巧能力、治疗计划目标等8个专业领域的成长。结论：督导对于新手咨询师专业成长非常重要。各类督导事件对于新手咨询师的促进作用并不均衡。     

Frame-of-reference training and cognitive categorization: an empirical investigation of rater memory issues.

We considered the effects of frame-of-reference (FOR) training on raters' ability to correctly classify ratee performance as well as their ability to recognize previously observed behaviors. The purpose was to examine the cognitive changes associated with FOR training to better understand why such training generally improves rating accuracy. We trained college students (N = 94) using either FOR or control procedures, had them observe three managers on videotape, and rate the managers on three performance dimensions. Results supported our hypotheses that, compared with control training, FOR training led to better rating accuracy and better classification accuracy. Also consistent with predictions, FOR training resulted in lower decision criteria (i.e., higher bias) and lower behavioral accuracy on a recognition memory task involving impression-consistent behaviors. The implications of these results are discussed, particularly in terms of the ability of FOR-trained raters to provide accurate performance feedback to ratees.     

Rater Model Using Signal Detection Theory for Latent Differential Rater Functioning

Differential rater functioning (DRF) occurs when raters show evidence of exercising differential severity or leniency when scoring examinees within different subgroups. Previous studies of DRF have examined rater bias using manifest variables (e.g., use of covariates) to determine the subgroups. These manifest variables include gender and the ethnicity of the examinee. For example, a rater may score males more severely. Ideally, each rater’s severity should be invariant across subgroups. This study examines DRF in the context of latent subgroups that classify possible sources of DRF based on raters’ scoring behavior rather than manifest factors. An extension of the latent class signal detection theory (LC-SDT) model for identifying DRF is proposed and examined using real-world data and simulations. Results from real-world data show that the signal detection approach leads to an effective method to identify latent DRF. Simulations with varying sample sizes and conditions of rater precision were shown to recover parameters at an adequate level, supporting its use to identify latent DRF in large-scale data. These findings suggest that the DRF extension of the LC-SDT can be a useful model to examine characteristics of raters and add information that can aid rater training.     

INFLUENCES BEHIND NEUTRAL RESPONSES IN SUBORDINATE RATINGS OF SUPERVISORS: A METHODOLOGICAL NOTE

Isolation of the influences behind neutral responses is the chief concern of this paper. A sample of 416 assembly-line operators were asked to rate their supervisors according to an instrument which contained a list of positive supervisory traits and behaviors. Three rating options were provided: agree, don't know and disagree. It was hypothesized that over-all dissatisfaction with the subjects would be a significant influence behind neutral responses. Several methods of classifying raters into satisfied/dissatisfied groups were experimented with. The results generally supported the hypothesis, with some qualifications. Dissatisfied raters were found to give significantly more neutral responses than satisfied raters. However, rater nationality and age were also found to be significant influences.     

USING WEB-BASED FRAME-OF-REFERENCE TRAINING TO DECREASE BIASES IN PERSONALITY-BASED JOB ANALYSIS: AN EXPERIMENTAL FIELD STUDY

We identify sources of biases in personality-based job analysis (PBJA) ratings and offer a Web-based frame-of-reference (FOR) training program to mitigate these biases. Given the use of job analysis data for the development of staffing, performance management, and many other human resource management systems, using biased PBJA ratings is likely to lead to a workforce that is increasingly homogenous in terms of personality but not necessarily a workforce with improved levels of performance. We conducted a field experiment (i.e., full random assignment) using 2 independent samples of employees in a city government and found evidence in support of the presence of biases as well as the effectiveness of the proposed solution. Specifically, FOR training was successful at decreasing the average correlation between job incumbents' self-reported personality and PBJA ratings from .27 to .07 (administrative support assistants) and from .30 to .09 (supervisors). Also, FOR training was successful at decreasing mean PBJA ratings by d = .44 (administrative support assistants) and by d = .68 (supervisors). We offer the entire set of Web-based FOR training materials for use in future research and applications.     

Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment

A quasi-experiment was conducted to investigate the effects of frame-of-reference training on the quality of competency modeling ratings made by consultants. Human resources consultants from a large consulting firm were randomly assigned to either a training or a control condition. The discriminant validity, interrater reliability, and accuracy of the competency ratings were significantly higher in the training group than in the control group. Further, the discriminant validity and interrater reliability of competency inferences were highest among an additional group of trained consultants who also had competency modeling experience. Together, these results suggest that procedural interventions such as rater training can significantly enhance the quality of competency modeling.     

A Meta-Analysis of the Relationship Between Rater Liking and Performance Ratings

This meta-analysis reviewed the magnitude and moderators of the relationship between rater liking and performance ratings. The results revealed substantial overlap between rater liking and performance ratings (ρ = .77). Although this relationship is often interpreted as indicative of bias, we review studies that indicate that to some extent the relationship between liking and performance ratings potentially reflects “true” differences in ratee performance. Moderator analyses indicated that the relationship between liking and performance ratings was weaker for ratings of organizational citizenship behaviors, ratings made by peer raters, ratings in nonsales jobs, and ratings made for development; however, the relationship was strong across moderator levels, underscoring the robustness of this relationship. Implications for the interpretation of performance ratings are discussed.     

The inconsistency with which raters weight and combine information across targets

To assign an overall performance rating to a target, a rater must weight and combine various pieces of specific performance information about that target. Policy‐capturing research has demonstrated that individual differences in raters can influence the way raters combine specific performance information. The current study examined information processing from a different perspective by exploring the possibility that target differences may also influence the way raters weight and combine performance information. Raters (N = 146) rated each of six targets on six specific performance dimensions and on overall performance. Sequential moderation analyses indicated that targets influenced the way raters, as a group, combined information across targets. These results lend support to the inference that overall performance ratings may not be comparable across targets, that is, they may not reflect the same underlying performance across targets.     

Frames of Reference in Quantity Estimations by Groups and Individuals

The superiority of group performance over performance of the average individual is relatively greater on world knowledge tasks than on quantity estimation tasks. Previous research on quantity estimations has involved judgments without an explicit frame of reference. We propose that a frame of reference converts a quantity estimation into a world knowledge inference by embedding the estimation in a larger cognitive structure. Individuals first estimated 30 pairs of quantities, such as the length of the Ohio River and the length of the Arkansas River, given either 2 statements as a frame of reference (the Mississippi River is 2340 miles long; the Colorado River is 1450 miles long), 1 of these statements as a frame of reference, or no frame of reference. Then they made the same 30 pairs of estimations again as 3-person groups or as individuals under the same frame-of-reference conditions. As predicted, group estimations were more accurate than individual estimations, both group and individual estimations were more accurate with either a 2-statement or a 1-statement frame of reference than without a frame of reference, and the frame of reference improved group estimations relatively more than individual estimations.     

Rating ability in performance judgments: The joint influence of implicit theories and intelligence

The purpose of this study was to approach the issue of rating ability by examining the influence of rater implicit theories and rater intelligence on rating outcomes. Using the inferential accuracy model (Jackson, 1972), raters were identified as either possessing a normative or idiosyncratic implicit theory of the occupation of college instructor. In a laboratory setting, 50 normative and 50 idiosyncratic raters judged the videotaped performance of either a good or poor lecturer. Results showed that (a) intelligence was positively related to rating accuracy and to possessing a normative implicit theory, (b) rater type moderated the relationship between intelligence and rating accuracy, and (c) controlling for intelligence, normative raters committed stronger halo effects than idiosyncratic raters. These results were discussed in relation to furthering the understanding of rating ability.     

国家公务员结构化面试中评委偏差的IRT分析

使用项目反应理论(IRT)中的多面Rasch模型,对两组共12名评委在国家公务员结构化面试中的评委偏差进行了分析。提出并验证了两种评委偏差：评委之间在宽严程度上的差异和评委自身的一致性问题。结果发现：不同评委之间在宽严程度上差异显著,且不同评委评定行为的跨考生、跨维度、跨性别、跨时间的自身一致性也存在差异。研究表明,这种进入到评委个体层次的分析突破了经典测量理论(CTT)定位于评委群体进行分析的局限,针对每位评委的偏差行为提供了详细具体的诊断信息,从而为评委的针对性培训和评委库的建立提供了现代测量学的新方法