Frame-of-Reference Training: Overcoming the Effects of Organizational Citizenship Behavior on Performance Rating Accuracy

Frame-of-reference (FOR) training has been shown to increase rating accuracy. Studies suggest, however, that organizational citizenship behavior (OCB) can influence performance ratings insomuch as they are considered in the rating process. We examined the effects of OCB on the effectiveness of FOR training for improving performance rating accuracy. Results from 77 managers showed that OCB attenuates FOR training effectiveness because of rating inflation, unless the managers are trained to observe and evaluate the OCB.     

A Cognitive Evaluation of Frame-of-Reference Rater Training: Content and Process Issues

A multivariate, cognitive evaluation model was used to examine the previously noted effects of frame-of-reference (FOR) training on rating accuracy. The tested model emphasized two content-related issues associated with FOR training: (a) the content of the training itself and the extent that raters agreed with it, which provided an index of the amount of overlap between the theory of performance taught in training and a rater's implicit theory of performance following training; and (b) the content of raters' performance impressions of ratees based on a comprehensive and integrative model of the cognitive representations of persons— Associated Systems Theory (Carlston, 1992, 1994). Undergraduates (N= 172) were trained with FOR or control procedures using two different performance theories, observed and rated videotaped manager performance on three performance dimensions, and engaged in written free-recall of target performance vignettes. A structural model was tested incorporating a latent differential rating accuracy construct. FOR training was associated with higher demonstrated levels of agreement and with changes in ratee representation that were more abstract and more target-referent (i.e., less idiosyncratic) than control training. Agreement was found to be associated with enhanced rating accuracy and more self-referent impressions; however, target-referent impressions were shown to be related to better rating accuracy. Discussion focuses on the interrelated nature of content and process issues in performance appraisal. Using the AST framework, it is also proposed that longstanding manager/subordinate dyads could be expected to benefit the most from FOR training.     

Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice.     

Rater training revisited: An updated meta‐analytic review of frame‐of‐reference training

The present study updates Woehr and Huffcutt's (1994) rater training meta‐analysis and demonstrates that frame‐of‐reference (FOR) training is an effective method of improving rating accuracy. The current meta‐analysis includes over four times as many studies as included in the Woehr and Huffcutt meta‐analysis and also provides a snapshot of current rater training studies. The present meta‐analysis also extends the previous meta‐analysis by showing that not all operationalizations of accuracy are equally improved by FOR training; Borman's differential accuracy appears to be the most improved by FOR training, along with behavioural accuracy, which provides a snapshot into the cognitive processes of the raters. We also investigate the extent to which FOR training protocols differ, the implications of protocol differences, and if the criteria of interest to FOR researchers have changed over time.     

Effects of training intensity on observers' ratings of anxiety, social skills, and alcohol-specific coping skills

Few studies have addressed the amount of training needed to obtain reliable ratings in behavioral observation data. The current study examined the effects of differing intensities of frame-of-reference (FOR) rater training on observers' ratings of anxiety, social skills, and alcohol-specific coping skills in community volunteers with and without social anxiety and alcohol use disorders. Interrater reliability was assessed by comparing three training conditions (no-training, moderate FOR, and intensive FOR) on discrepancies between raters'scores and the strength of association between raters'scoring patterns. The discrepancies between raters' scores were significantly larger in the control condition than in the intensive and moderate training conditions. Generally, small and nonsignificant differences were found between intensive and moderate training's discrepancy scores. Strength of association results showed significantly lower correlations in the control group compared to the intensive group. However, these correlational results showed less consistent differences between the moderate and other training conditions; differences when found were in the expected directions. Study findings suggest that differing training intensities can affect rating scores and that interrater reliability may be meaningfully assessed through multiple methods.     

Effects of Accountability on Rating Behavior and Rater Accuracy

We tested whether improvements in rater accuracy on a performance-appraisal task attributed to rater accountability could be explained by variance in rater behavior. Data from Mero and Motowidlo (1995) were used initially to test whether observed improvements in rater accuracy could be explained by rating process behaviors of attending to relevant subordinate performance and taking better notes to record that performance. Results showed that both behaviors were correlated with accuracy and that these behaviors partially mediated the relationship between accountability and improved decision quality for participants who were held accountable for their rating decision and did not have any preconceived information about that rating from their supervisor. We then tested this process model on a new sample of participants and found that attentiveness and notetaking mediated the relationship between accountability and accuracy.     

Two for the Price of One: Assessment Center Training to Focus on Behaviors Can Transfer to Performance Appraisals

The purpose of this study was to evaluate whether the training and experience of assessors to record behaviors as part of an actual organizational assessment center (AC) would transfer to another context: performance appraisals. Specifically, we examined the effects of transfer of training on managers' behavioral specificity in comments on performance evaluations. Performance appraisal comments from managers who had received training and participated as an assessor and from managers who had not were compared. Analyses revealed significant differences between managers who were trained as assessors compared with managers who were not, suggesting transfer of training effects. Results provide the first empirical evidence for an added benefit of AC assessor participation and training.     

A LONGITUDINAL STUDY OF UPWARD FEEDBACK

This study followed 92 managers during four administrations of an upward feedback program over 2.5 years. Managers' whose initial level of performance (defined as the average rating from subordinates) was low improved between Administrations 1 and 2, and sustained this improvement 2 years later. Most of the performance improvement occurred between the first and second administrations of the program. The observed performance improvement could not be attributed solely to regression to the mean and was unrelated to the number of times, or when, the managers received feedback. Our results suggest that the continued administration of an upward feedback program can result in sustained change over a fairly long period of time and that actually receiving feedback may be less important than the exposure to the valued behaviors. We use control theory and goal setting theory as frameworks for explaining the results and suggesting directions for future research.     

Rater training: Understanding effects of training content,practice ratings,and feedback

Despite the popularity of frame‐of‐reference training (FORT), it is not clear how different structural elements of FORT work in concert to improve rating accuracy. Furthermore, past rater training studies have lacked rigorous control groups leading to low thresholds for showing improvements in rating accuracy due to FORT. The current study allowed for the isolation of components of rater training that increase rating accuracy when compared to a rigorously designed control group. Results indicated that repeated rendering of practice ratings improve rating accuracy and this practice effect was amplified by practice rating feedback. Although accuracy‐based training content improved interrater agreement, it did not contribute to improvements rating accuracy over and above the control group. We discuss the implications of the findings in relation to best practices for designing rater training programs.     

ACTION TRAINING FOR CHARISMATIC LEADERSHIP: TWO EVALUATIONS OF STUDIES OF A COMMERCIAL TRAINING MODULE ON INSPIRATIONAL COMMUNICATION OF A VISION

An action training program that teaches inspirational communication of a vision as part of a training of charismatic leadership for managers is presented (1½ days) and evaluated in 2 studies ( N = 25 and N = 22). We used the research design "nonequivalent dependent variable design" (Cook & Campbell, 1979, p, 118) or "internal referencing strategy" (Haccoun & Hamtieux, 1994), which compared the trained behaviors (charismatic inspirational communication) with behaviors that were not trained (public speech) to control for testing and Hawthorne effects. The training had specific positive effects on those behaviors that were trained but not on those variables that were not trained. Good to excellent effect sizes appeared as a result of the training. We suggest that this research design is useful for evaluation of training effects within the constraints of commercial settings and, moreover, we argue that this design is in many ways superior to a nonequivalent nontraining control group design because it controls for testing effects and for effects that otherwise would need a pseudo-training control group.     

Training elementary aged peer behavior managers to control small group programmed mathematics

The effects of a training procedure and two maintenance contingencies on consequence-dispensing behavior were investigated. Four peer behavior managers were trained to supervise small groups of subjects (four to six per group) working in programmed math materials and were compared with a teacher skilled in the use of social and point reinforcement and response cost. Manager training was differentially effective in accelerating manager's rates of appropriate social and point dispensing. Having manager reinforcement contingent upon manager consequence-dispensing resulted in moderately higher rates of appropriate social and point dispensing for three of four subjects than did having manager reinforcement contingent upon group study behavior. Two managers exposed to the group performance contingency before the manager performance contingency increased inappropriate social and point-dispensing behaviors to pretraining baseline levels. Subsequent change to the manager performance contingency was effective in reducing the inappropriate dispensing behavior of only one of the two managers.     

The accuracy of roommate ratings of behaviors versus beliefs

We evaluated the accuracy of peer ratings of roommates’ personality characteristics, against roommate self-ratings, as a function of rating domain observability. Instead of the usual ratings of broad personality traits, however, our domains represented peer ratings of narrow exemplars of personality traits. Specifically, we compared roommate ratings on (a) observable trait-related behaviors with (b) unobservable trait-related attitudes or beliefs. We observed greater self-peer agreement in rating behaviors, in general, than in rating beliefs. We also observed greater tendency of raters to adopt an assumed similarity heuristic when judging their roommates’ attitudes and beliefs than their behaviors. We discuss the contribution of these findings to understanding the determinants of accuracy in personality judgments and developing best practices for personality assessment.     

USING WEB-BASED FRAME-OF-REFERENCE TRAINING TO DECREASE BIASES IN PERSONALITY-BASED JOB ANALYSIS: AN EXPERIMENTAL FIELD STUDY

We identify sources of biases in personality-based job analysis (PBJA) ratings and offer a Web-based frame-of-reference (FOR) training program to mitigate these biases. Given the use of job analysis data for the development of staffing, performance management, and many other human resource management systems, using biased PBJA ratings is likely to lead to a workforce that is increasingly homogenous in terms of personality but not necessarily a workforce with improved levels of performance. We conducted a field experiment (i.e., full random assignment) using 2 independent samples of employees in a city government and found evidence in support of the presence of biases as well as the effectiveness of the proposed solution. Specifically, FOR training was successful at decreasing the average correlation between job incumbents' self-reported personality and PBJA ratings from .27 to .07 (administrative support assistants) and from .30 to .09 (supervisors). Also, FOR training was successful at decreasing mean PBJA ratings by d = .44 (administrative support assistants) and by d = .68 (supervisors). We offer the entire set of Web-based FOR training materials for use in future research and applications.     

The social validation and training of conversational skills

Three reliably measured components of conversation-questioning, providing positive feedback, and proportion of time spent talking-were identified and validated as to their social importance. The social validity of the three conversational behaviors was established with five female university students and five female junior-high students. Each was videotaped in conversations with previously unknown adults. The conversational ability of each girl was evaluated by a group of 13 adult judges who viewed each tape and rated each conversant "poor" to "excellent" on a seven-point rating scale. The average ratings of the girls correlated at r = 0.85 with the specified behavioral measures. These procedures were replicated with additional subjects and judges and yielded a correlation of r = 0.84. The high correlations between ratings and the objective measures suggested that the specified conversational behaviors were socially important aspects of conversational ability. Employing a multiple-baseline design across the behaviors of asking questions and providing positive feedback, an attempt was made to train four girls who used these behaviors minimally to engage in the behaviors in conversations with adults. Adult judges were again employed to rate randomly selected samples of the girls' skills in pre- and posttraining conversations. The average ratings of the girls before training were lower than both the university girls and the junior high-school girls. After training, the girls' conversational abilities were rated substantially higher than those of their junior high-school peers. These rating data validated the benefits of the training and the social importance of the behavioral components of questions and feedback in conversation. The authors suggest that it may be necessary for traditional behavior analysis measurement systems to be supplemented by social-validation procedures in order to establish the relationship between "objectively" measured behaviors and complex classes of behavior of interest to society.     

Evidence for the Effectiveness of an Alternative Multisource Performance Rating Methodology

Despite persistent concerns as to the quality of performance information obtained from multisource performance ratings (MSPRs), little research has sought ways to improve the psychometric properties of MSPRs. Borrowing from past methodologies designed to improve performance ratings, we present a new method of presenting items in MSPRs, frame‐of‐reference scales (FORS), and test the efficacy of this method in a field and lab study. The field study used confirmatory factor analysis to compare the FORS to traditional rating scales and revealed that FORS are associated with increased variance due to dimensions, decreased overlap among dimensions, and decreased error. The laboratory study compared rating accuracy associated with FORS relative to frame‐of‐reference training (FORT) and a control group and demonstrated that FORS are associated with higher levels of accuracy than the control group and similar levels of accuracy as FORT. Implications for the design and implementation of FORS are discussed.     

冲动特质对青少年认知控制的影响——基于双重认知控制理论

根据双重认知控制理论,抑制控制分为主动性控制和反应性控制。为了探索冲动特质青少年究竟在哪个抑制控制成分存在不足,本研究用冲动性人格量表(BIS-11)选取了高冲动特质和低冲动特质青少年,并记录了他们在完成AX-CPT任务时的反应时和正确率。结果显示:与低冲动个体相比,高冲动个体在BX试次(主动性控制)上的反应时和正确率以及线索辨别力指数上的表现均差于低冲动个体,而在AY试次(反应性控制)上的反应时和正确率以及探测敏感性指标上的表现与低冲动个体没有显著差异。这些结果说明,与低冲动特质青少年相比,高冲动特质青少年的反应性控制保存较好,但主动性控制能力表现出不足。     

Examining contrast effects in performance appraisals: using appropriate controls and assessing accuracy

In research on contrast effects in performance appraisals, control conditions or measures of accuracy have rarely been used. In the present study, the authors included appropriate controls and used expert ratings to develop "true scores" for assessing accuracy. The study is an examination of the influence of 3 variables on performance ratings: (a) the sequence of viewing and rating performance, (b) the delay between viewing and rating performance, and (c) whether the target of the performance rating was the same person as the anchor. Experimental conditions did not have the expected differential effects on target ratings, but target ratings in all the experimental conditions showed contrast effects when compared with ratings in relevant control conditions. The target ratings in experimental conditions were accurate, however, as assessed by comparisons with true scores, thus raising questions about the relationship between contrast effects and accuracy.     

Cognitive categories of raters and rating accuracy

This research investigated cognitive processes affecting the accuracy of performance evaluation. One hundred and twenty-five nurses from three hospitals completed questionnaires measuring their cognitive categorization processes and then rated a videotaped nurse's performance. Results focusing on the match between raters' cognitive categories and rating scales indicated that: a) rating accuracy was related to the match between raters' cognitive category dimensions and rating scale dimensions, but b) not related to the match between rating scales and raters' general tendency to describe ratees in terms of behavioral or trait characteristics. Raters' ability to differentiate clearly between rating dimensions was related to accuracy and to halo in ratings. Finally, work experience was related to raters' cognitive systems while prior rating experience was important for accuracy. Limitations of the study and implications for future research are discussed.     

Rating ability in performance judgments: The joint influence of implicit theories and intelligence

The purpose of this study was to approach the issue of rating ability by examining the influence of rater implicit theories and rater intelligence on rating outcomes. Using the inferential accuracy model (Jackson, 1972), raters were identified as either possessing a normative or idiosyncratic implicit theory of the occupation of college instructor. In a laboratory setting, 50 normative and 50 idiosyncratic raters judged the videotaped performance of either a good or poor lecturer. Results showed that (a) intelligence was positively related to rating accuracy and to possessing a normative implicit theory, (b) rater type moderated the relationship between intelligence and rating accuracy, and (c) controlling for intelligence, normative raters committed stronger halo effects than idiosyncratic raters. These results were discussed in relation to furthering the understanding of rating ability.