Reactions to Appraisal Discrepancies: Performance Ratings and Attributions

The traditional top-down approach to performance appraisal has been criticized as not keeping pace with the move toward more participative organizations of the last decade. This has led many organizations to adopt newer performance appraisal systems that use multiple feedback sources (e.g., 360-degree feedback systems). However, research indicates that multiple feedback sources often disagree on performance ratings and attributions. The current paper reports the results of one laboratory and one field study that show the importance of performance and attributional rating discrepancies on the appraisal process.     

Performance Appraisal Process Research in the 1980s: What Has It Contributed to Appraisals in Use?

In the early 1980s, Landy and Farr (1980) and Feldman (1981) redirected performance appraisal research from issues related to the development of psychometrically sound rating scales to those involving the cognitive processes of raters. Since that time, several reviews have attempted to translate principles from social cognition and cognitive psychology to the specific conditions of formal appraisal systems in work-oriented organizations. In addition, a number of empirical studies have been conducted on this topic. This article reviews empirical research during the 1980s that focused on performance appraisal processes, particularly the research that has focused upon rating accuracy. The review is structured around a three-stage process model of gathering, storing, and retrieving information about social stimuli for the purposes of rating performance. Factors affecting this process are clustered into four categories: appraisal settings, ratees, raters, and the nature of the scales used for the appraisal. Once reviewed, the research is evaluated in terms of its contributions to improving the quality of appraisal systems as they are used in organizations.     

Predictors of Participation in Performance Appraisal: A Voluntary System in a Blue-Collar Work Environment

Organisations have long acknowledged the value of performance appraisal in both administration decisions and in motivating employees. However, it would appear that most empirical research has focused on compulsory systems designed for use with executives and managers (Bannister and Balkin 1990; Ilgen, Fisher and Taylor 1979; Klein 1989). The reluctance to implement appraisal systems with lower level and base-grade employees may be due to several factors, including employee and union resistance to compulsory systems of appraisal (Nankervis 1990). Faced with this problem, appraisal can be viewed by appraisees and appraisers as an `innovation'. In a climate with growing emphasis on participative management styles, self-regulated or voluntary systems of performance appraisal may become increasingly prevalent and important. Thus, 476 base-grade (shop floor) employees of an Australian government business enterprise completed a survey regarding issues such as attitude to performance appraisal, source credibility, system satisfaction, anticipated feedback, perceived consequences of participation and intention to enter the system. Roger's (1983) innovation-decision model was utilised in the structure and analysis of the data and provided useful insights regarding the factors which are related to participation in a voluntary system of performance appraisal. Findings indicated that attitude towards, and support of, performance appraisal in general, perception of consequences of performance appraisal, satisfaction with the supervisor, anticipated feedback, and awareness and knowledge of the system in place in the organisation, were important in predicting the decision to adopt or reject the performance appraisal scheme.     

360 degree feedback: accuracy, reactions, and perceptions of usefulness

This study examined how 360 degree feedback ratings and self-other rating discrepancies related to reactions to feedback, perceptions of feedback accuracy, perceived usefulness of the feedback, and recipients' receptivity to development. The results indicated that less favorable ratings were related to beliefs that feedback was less accurate and to negative reactions. Negative reactions and perceptions that feedback was less accurate were related to beliefs that the feedback was less useful. Those who found feedback less useful were perceived by a facilitator as less development-focused. Goal orientation did not moderate the relationship between ratings and perceptions of accuracy or reactions to feedback. Goal orientation was related to perceptions of usefulness of the process several weeks after receipt of feedback. The results question widely held assumptions about 360 degree feedback that negative and discrepant feedback motivates positive change.     

The role of cognitive appraisal in media-induced presence and emotions

In the present study, we applied Valins’ (1966) bogus bodily feedback paradigm to investigate the roles of cognitive appraisal (tele)presence, and emotion in the context of media usage. Participants (N=30) viewed short sequences of a TV show, whereby false heart rate feedback was provided. In line with Valins’ paradigm, the manipulation we used in this study included two levels: high vs. low false heart rate feedback. We assessed presence, enjoyment, and the real heart rate. Participants reported stronger experiences of presence and enjoyed the clips more when they wrongly believed that they had been aroused. Since bogus feedback increases the sensation of presence, the results indicate that presence—like emotions—can be influenced by cognitive appraisals. It is likely that the false heart rate feedback was attributed to the stimulus, which in turn enhanced the sensation of presence.     

Practical Considerations and Alternative Research Methods for Evaluating HR Programs

Program evaluation methods capitalizing on quasi-experimental designs are introduced as useful alternatives for evaluating human resources (HR) programs such as selection systems, training courses, performance measurement systems, and 360 degree feedback. A case study is presented to illustrate the benefits of program evaluation methods for evaluating HR programs. In addition, several post-hoc considerations that often moderate results from HR evaluations are described. It is suggested that when these considerations are addressed, quasi-experimental program evaluation methods provide a promising tool for demonstrating bottom line impact of HR initiatives.     

SELF-APPRAISAL BASED UPON SUPERVISORY FEEDBACK

Self-appraisals of job performance have historically proven to be weakly related to appraisals conducted by supervisory personnel. Two studies sought to facilitate rater agreement by invoking supervisory performance feedback as a frame of reference for subordinate raters. In the first study, a sample of 401 branch managers from a large lending institution performed a conventional self-appraisal and an appraisal containing instructions referencing supervisory feedback (Feedback Based Self-Appraisal). Feedback Based Self-Appraisals exhibited significantly smaller leniency error, greater total rating variance, and more agreement with superior ratings than conventional self-appraisals. FBSAs produced small improvements over conventional self-appraisals in predicting objective criteria of managerial job performance. Study 2 examined FBSA-supervisory appraisal agreement for two samples of military organization personnel. The relative degree of feedback available in a work environment was found to moderate FBSA-superior agreement.     

Cognitive appraisals and emotional experience: Further evidence

Abstract

The present study uses methods derived from Smith and Ellsworth (1985) to investigate the degree to which emotions are associated with distinctive patterns of cognitive appraisal. Subjects described past situations in which they had unambiguously experienced each of eight emotions: guilt. embarrassment, shame. anxiety. anger, hope. joy, and pride. These situations were then rated on each of 10 appraisal scales: unpleasantness. unexpectedness of events from own and from others' perspective, inconsistency of own actions with own and others' behavioural standards, benefit to self and to others, own and others' responsibility for events. and degree to which events were beyond anyone's control. Analysis of these ratings revealed distinctive patterns of appraisal associated with the eight emotions. including discriminable patterns for guilt and shame—emotions that were not distinguished in terms of the appraisal dimensions recovered by Smith and Ellsworth. Thus the present study demonstrates that appraisal dimensions (such as unexpectedness, benefit, and inconsistency with behavioural standards) in addition than those used by Smith and Ellsworth have utility in distinguishing among emotions.     

The predictive validity of individual psychological assessments in selecting UK public sector senior managers

Individual Psychological Assessment (IPA) is a very widely offered service in Organizational Psychology. It generally consists of a psychologist or HR practitioner using a combination of interview and psychometrics to arrive at a detailed assessment of an individual's capabilities in relation to a job they are being considered for. Although much used, this practice has limited supporting evidence of its validity—not least because of the methodological difficulties in conducting research on this subject—and has been criticized accordingly. The current study examines the use of IPAs with 115 middle and senior management level candidates in a civil service context. All candidates completed a set of psychometric measures and had an in‐depth interview with a psychologist as part of a standardized process. The ratings made by the assessors were correlated with a criterion measure of potential for promotion derived from multisource feedback ratings obtained on these candidates some months later. Analysis of the results indicated that three of the four attributes rated by assessors correlated significantly with the criterion measure. Further, assessors’ ratings were found to show incremental validity over that provided by psychometric test scores alone. These findings are discussed in terms of the use of IPAs in senior level assessment.     

Some Factors Explaining the Reliability of a Structured Interview System at a Work Site

Recent meta-analytic research has demonstrated structured interviewing to hold acceptable validity and reliability. While the emphasis has been on refining psychometric properties, there is sufficient evidence to suggest a discrepancy between the manner in which interviewing systems should be used and how they are actually used. The present study examined the use of a commercially available structured interviewing system based on past behaviour. 112 candidates were interviewed on two separate occasions by 28 interviewers. Inter-rater reliability was 0.55. The system required the derivation of a consensus score which was found not to differ significantly from the arithmetical mean of the original scores suggesting the process was not undertaken as required. Follow up discussions with interviewers reported three main areas of misunderstanding; lack of role clarity, different interpretations of job specification and inconsistent use of the rating system. Data also suggested interviewers were inexperienced. A critical finding involved female interviewers making assumptions about female applicant's motivations and suitability to the position. This finding was explained by employing ‘person-in-job’ prototypes. The data support the conclusion that although structured interviews may contain appropriate psychometric properties, the application of the system is critical.     

Leader personality and 360‐degree assessments of leader behavior

To investigate the relationship between personality and multi‐source feedback, we assessed 190 health care managers by applying the Understanding Personal Potential personality test, which provides comprehensive measurement of the Big Five dimensions and eight narrower personality traits. Managers' leadership behaviors were assessed by colleagues, supervisors, a random sample of each manager's subordinates as well as the managers themselves using a 360‐degree change, production, employee (CPE) instrument. Hierarchical multivariate regression analysis showed that the Big Five variables were significantly related to the Managers' leadership behavior in all CPE dimensions. Also, addition of narrow personality variables to the Big Five increased explained variance in leadership behavior. This study is the first of its kind to include the full range of viewpoints in a 360‐degree instrument, along with a large number of subordinate assessments. We found that both the strength of the relationship between personality and behavior and the configuration of different predictors varied depending on who did the rating and what leadership orientation was investigated, and this observation merits further investigation.     

Using 360 Degree Feedback Instruments up,down and around the world: Implications for global implementation and use of Multi-Rater Feedback

This report highlights some of the challenges faced when using multi-rater feedback around the world. It is based on the observations of coaches who have worked with a variety of managers from different cultures using a 360 degree feedback instrument — the PROFILOR®. Reference is made to some of the research on cultural differences in order to clarify, or otherwise, some of the issues raised.     

REACTION OF EMPLOYEES TO PERFORMANCE APPRAISAL INTERVIEWS AS A FUNCTION OF THEIR PARTICIPATION IN RATING SCALE DEVELOPMENT

A substantial amount of research has been conducted on the performance appraisal feedback interview. The purpose of the present field study was to see whether employee involvement in the development of rating scales used in the feedback interview affected ratees' perceptions of the interview. Behaviorally anchored rating scales (BARS) were developed for five job families in a midwestem hospital. A participation group consisted of employees from each of the job families who were involved in all phases of BARS construction. A nonparticipation group involved an equal number of employees from each of the job families who were not involved in the BARS development. The results showed that participation in BARS construction led to favorable perceptions regarding the performance appraisal interview process as well as positive outcomes.     

Rating leniency and halo in multisource feedback ratings: testing cultural assumptions of power distance and individualism-collectivism

This study extends multisource feedback research by assessing the effects of rater source and raters' cultural value orientations on rating bias (leniency and halo). Using a motivational perspective of performance appraisal, the authors posit that subordinate raters followed by peers will exhibit more rating bias than superiors. More important, given that multisource feedback systems were premised on low power distance and individualistic cultural assumptions, the authors expect raters' power distance and individualism-collectivism orientations to moderate the effects of rater source on rating bias. Hierarchical linear modeling on data collected from 1,447 superiors, peers, and subordinates who provided developmental feedback to 172 military officers show that (a) subordinates exhibit the most rating leniency, followed by peers and superiors; (b) subordinates demonstrate more halo than superiors and peers, whereas superiors and peers do not differ; (c) the effects of power distance on leniency and halo are strongest for subordinates than for peers and superiors; (d) the effects of collectivism on leniency were stronger for subordinates and peers than for superiors; effects on halo were stronger for subordinates than superiors, but these effects did not differ for subordinates and peers. The present findings highlight the role of raters' cultural values in multisource feedback ratings.     

Feedback type as a moderator of the relationship between achievement goals and feedback reactions

The aim of the current study is to shed new light on the inconsistent relationship between performance‐approach (PAp) goals and feedback reactions by examining feedback type as a moderator. Results of a field experiment (N = 939) using a web‐based work simulation task showed that the effect of achievement‐approach goals was moderated by feedback type. Relative to individuals pursuing mastery‐approach goals, individuals pursuing PAp goals responded more negatively to comparative feedback but not to task‐referenced feedback. In line with the hypothesized mediated moderation model, the interaction between achievement goals and feedback type also indirectly affected task performance through feedback reactions. Providing employees with feedback is a key psychological principle used in a wide range of human resource and performance management instruments (e.g., developmental assessment centres, multi‐source/360° feedback, training, selection, performance appraisal, management education, computer‐adaptive testing, and coaching). The current study suggests that organizations need to strike a balance between encouraging learning and encouraging performance, as too much emphasis on comparative performance (both in goal inducement and in feedback style) may be detrimental to employees' reactions and rate of performance improvement.     

A Systematic Review of the Psychometric Properties of Trait Cognitive Self-Report Measures in Social Anxiety

Cognitive factors, including beliefs, thoughts and assumptions have been found to play an important role in the development and maintenance of Social Anxiety Disorder. Trait cognitive self-report measures of social anxiety are widely used in research and clinical settings. It is imperative that only measures with good psychometric properties are used in order to interpret assessment scores accurately, and to make valid and reliable conclusions. The present systematic review evaluated the psychometric properties of trait cognitive self-report measures of social anxiety. Relevant studies were identified via a comprehensive and systematic search of academic databases. The reported psychometric properties of included studies were analysed by applying an appraisal of adequacy tool developed by Terwee et al. (2007). Of the 3091 studies identified, 50 studies met the inclusion criteria, and they included 21 measures. Included studies demonstrated that a number of measures had some adequate psychometric properties, however, no measure fulfilled criteria for all psychometric properties according to the appraisal tool. Findings highlight the need to further establish the psychometric properties of cognitive self-report measures of social anxiety in clinical and research settings through additional empirical studies.     

The Development of an Integrated Performance Category System for Supervisory Jobs in the U.S. Navy

When a single job is the target, established job analysis approaches provide relatively straightforward methods for identifying performance requirements. However, when multiple jobs are involved, the identification of a single set of relevant performance dimensions can be more daunting. In the application here, there was also a compelling requirement to develop a dimension set that conformed with U.S. Navy fleet personnel perceptions of the performance domain. Accordingly, the behavioral performance constructs that experienced Navy officers believed differentiate effective from ineffective supervisory performance were gathered using a personal construct theory protocol. We then used a methodology suggested by Borman and Brush (1993) to integrate and summarize these personal work constructs, to provide—through a series of qualitative and quantitative strategies—a dimension set targeted toward supervisors in all Navy communities. The resulting dimensions revealed certain themes that might not have emerged if traditional job analysis strategies had been used. The dimension set and the resulting behaviorally anchored rating scales appear widely relevant to Navy supervisor jobs, and the performance appraisal system that employs the scales is nearing implementation. In addition, performance feedback and development tools were developed to complement the performance appraisal system.     

Do Student Evaluations of University Reflect Inaccurate Beliefs or Actual Experience? A Relative Rank Model

It was shown that student satisfaction ratings are influenced by context in ways that have important theoretical and practical implications. Using questions from the UK's National Student Survey, the study examined whether and how students' expressed satisfaction with issues such as feedback promptness and instructor enthusiasm depends on the context of comparison (such as possibly inaccurate beliefs about the feedback promptness or enthusiasm experienced at other universities) that is evoked. Experiment 1 found strong effects of experimentally provided comparison context—for example, satisfaction with a given feedback time depended on the time's relative position within a context. Experiment 2 used a novel distribution‐elicitation methodology to determine the prior beliefs of individual students about what happens in universities other than their own. It found that these beliefs vary widely and that students' satisfaction was predicted by how they believed their experience ranked within the distribution of others' experiences. A third study found that relative judgement principles also predicted students' intention to complain. An extended model was developed to show that purely rank‐based principles of judgement can account for findings previously attributed to range effects. It was concluded that satisfaction ratings and quality of provision are different quantities, particularly when the implicit context of comparison includes beliefs about provision at other universities. Quality and satisfaction should be assessed separately, with objective measures (such as actual times to feedback), rather than subjective ratings (such as satisfaction with feedback promptness), being used to measure quality wherever practicable. © 2014 The Authors. Journal of Behavioral Decision Making published by John Wiley & Sons Ltd.     

Automatic Generation of Rasch-Calibrated Items: Figural Matrices Test GEOM and Endless-Loops Test EC

The future of test construction for certain psychological ability domains that can be analyzed well in a structured manner may lie—at the very least for reasons of test security—in the field of automatic item generation. In this context, a question that has not been explicitly addressed is whether it is possible to embed an item response theory (IRT) based psychometric quality control procedure directly into the process of automatic item generation. Research in this area was conducted using 2 item generators (for the 2 domains of reasoning and spatial ability) that were developed and based on relevant models of cognitive psychology. During the course of the 4 studies reported here, those parts of the generators that check for possible violations of psychometric quality ("constraints") were improved. The main findings indicate that quality control procedures can be embedded in automatic item generators depending on (a) the degree to which the domain to be measured can be structured; (b) item-specific, content-based analyses; and (c) the degree to which the constraints can be implemented in software. Furthermore, beyond the global check of given model fit via IRT, the content-based analysis of items may be valuable in terms of finding such item properties that may lead to violations of psychometric quality.     

THE USE OF SELF-ASSESSMENTS IN EXAMINING

Self-assessments combined with other evidences such as achievements and test scores lead to increased accuracy and fairness in examining. The use of self-ratings to evaluate an applicant requires adjustment for degree of understanding of the element to be rated and for applicant's subjective base for rating. However, raw ratings may be used for psychometric analysis. Since minorities are usually represented in significant numbers among applicants but not in employed groups, this kind of analysis appears promising for studies designed to reduce adverse impact of tests and other measuring devices. An initial exploratory example of such analysis is described in detail.