The Influences of Performance Cues and Observational Focus on Performance Rating Accuracy1

The present study examined the effects of observational focus and performance cues on rating accuracy. It was hypothesized that these two factors would affect ratings independently: a) subjects given a good performance cue would rate the target more positively than subjects given a poor performance cue and, b) subjects using an event focus would rate the target person more accurately than subjects using a person focus. One hundred twenty undergraduates viewed the same videotape and subsequently stated which of a set of 48 behaviors were exhibited by the target person. The results supported the hypothesized performance cue effect. Observational focus did not have the hypothesized main effect on rating accuracy but was involved in a four-way interaction. The interaction indicated that, as the focus of the subject's attention broadened to the entire event, the errors associated with the performance cue effect were lessened. The subjects using an event focus were likely to generate ratings characterized by a positivity bias, whereas those using a person focus were more likely to generate ratings biased by performance cues.     

The Systematic Distortion Hypothesis: A Confirmatory Test of the Implicit Covariance and General Impression Models

Supervisory ratings of subordinate performance were used to compare two variations of the systematic distortion hypothesis: an implicit covariance model and a general impression (GI) model. Our purpose was to assess whether a model specifying a set of correlated performance dimensions or a model specifying a single, GI factor provided the best representation of actual supervisory performance ratings (N = 491). Confirmatory factor analysis was used to test a first-order correlated factor (i.e., implicit covariance) model against a second-order hierarchical (i.e., GI) model. The results generally supported the latter perspective whereby a GI is thought to serve as the basis for subsequent dimensional judgments, which in turn serve as the basis for individual item ratings. The implications of these results with respect to rater training and rating accuracy are discussed.     

The Deliberation-Without-Attention Effect: Evidence for an Artifactual Interpretation

ABSTRACT— Proponents of unconscious-thought theory assert that letting the unconscious "mull it over" can enhance decisions. In a series of recent studies, researchers demonstrated that participants whose attention was focused on solving a complex problem (i.e., those using conscious thought) made poorer choices, decisions, and judgments than participants whose attention was distracted from the problem (i.e., those purportedly using unconscious thought). We argue that this finding, rather than establishing the existence of a deliberation-without-attention effect, is explained more compellingly in terms of the well-established distinction between on-line and memory-based judgments. In Experiment 1 , we reversed the recent finding by simply changing participants' on-line processing goal from impression formation to memorization. Experiment 2 provided a replication and further established that some cognitive effort appears necessary to produce both the original pattern of results and its reversal, suggesting that such judgments are ultimately a product of conscious, rather than unconscious, thinking.     

Examining contrast effects in performance appraisals: using appropriate controls and assessing accuracy

In research on contrast effects in performance appraisals, control conditions or measures of accuracy have rarely been used. In the present study, the authors included appropriate controls and used expert ratings to develop "true scores" for assessing accuracy. The study is an examination of the influence of 3 variables on performance ratings: (a) the sequence of viewing and rating performance, (b) the delay between viewing and rating performance, and (c) whether the target of the performance rating was the same person as the anchor. Experimental conditions did not have the expected differential effects on target ratings, but target ratings in all the experimental conditions showed contrast effects when compared with ratings in relevant control conditions. The target ratings in experimental conditions were accurate, however, as assessed by comparisons with true scores, thus raising questions about the relationship between contrast effects and accuracy.     

The effect of dimension content on observation and ratings of job performance

Recent theory and research in the performance appraisal area (e.g., Denisi, Cafferty, & Meglino, 1984; Feldman, 1986; Ilgen & Feldman, 1983; Williams, Denisi, & Blencoe, 1985) have suggested that providing information regarding the performance dimension to be rated will cause raters to select appropriate observational schemata and, as a result, produce higher quality ratings. A study was conducted to determine if giving raters dimension-relevant information prior to performance observation would affect their attention processes and rating quality. Prior to watching a videotape of an instructor giving a lecture, 156 subjects were given either: 1) correct information, 2) incorrect information, or 3) no information regarding dimensions of performance they would subsequently be asked to rate. The results indicated that giving prior information regarding dimension content affected subjects' attention processes. Further, raters receiving no information and those receiving misinformation prior to performance observation produced less accurate ratings compared to expert raters. Ratings produced by subjects receiving correct information did not differ significantly from experts' ratings. These results are discussed in terms of both their practical and theoretical implications.     

Context and Prior Impression Effects on Attention, Judgment Standards, and Ratings: Contrast Effects Revisited

The present study extended research on contrast effects by (a) examining the effect of context performances on ratings of a target performance when a prior impression of the target performer already exists, and (b) clarifying the issue of whether contrast is caused by attention to context-discrepant behavior or shifts in judgment standards. The results demonstrated that the existence of a prior impression mitigates the influence of context performances on ratings. Judgment standards were found to be unstable and dependent on information provided to raters by the experimental manipulations. Regression analyses showed that both attention and standards of judgment mediate the relationship between context and ratings. Implications of these findings for contrast effects, performance ratings, and the importance of reliable judgment standards for real-world performance appraisals are discussed.     

Factors predictive of a fetal alcohol spectrum disorder diagnosis: Parent and teacher ratings

Diagnostic assessment in Fetal Alcohol Spectrum Disorder (FASD) is informed by multidisciplinary assessment incorporating objective (i.e., test measures) and subjective means, such as parent and teacher behavior ratings. The purpose of this study was to extend our previous neuropsychological test findings by identifying parent and teacher ratings of academic achievement, attention, executive functioning, and adaptive functioning as predictors of an FASD diagnosis. The charts of 315 children and adolescents with prenatal alcohol exposure (PAE) who underwent assessment for FASD were retrospectively reviewed. Direct logistic regressions analyzed the contribution of different ratings on the likelihood of an FASD diagnosis. The results suggest that a number of rating measures do contribute toward accurately differentiating those with FASD from within a PAE population, including teacher ratings of learning problems, inattention, and adaptive skills. The classification accuracy for each regression was clinically significant (59.1–70.8%). Children with worse ratings on these variables are approximately 1.5 to 2 times more likely to receive an FASD diagnosis. Only teacher ratings (not parent) significantly contributed to whether a diagnosis was made, suggesting that teacher observational rating scales are a critical component of an FASD assessment. Together with our previous research examining neuropsychological evaluation and FASD diagnostic assessment, this study helps to further guide decisions to streamline care in multidisciplinary assessment and intervention planning.     

Role of Behavior Level,Behavioral Variability,and Ratee Order in the Formation of Appraisal Ratings

Two laboratory studies were conducted to assess the role of behavioral data in the formation of appraisal ratings. In the first study, which included 200 participants, significant main effects using a split-plot factorial analysis of variance design were found for performance variability, performance level, and ratee order. A central drift phenomenon also was discovered with ratees whose performance profiles were both similar and extreme (extremely high or extremely low), receiving ratings that drifted toward average as a function of rating queue position. A second study, which included 126 participants, replicated this drift phenomenon and isolated the cause as profile similarity rather than profile extremity. Implications of these findings for rating accuracy and future "truescore" research designs are discussed.     

Eye movements,word familiarity,and vocabulary acquisition

Two experiments examined the effects of word familiarity on word recognition and text comprehension during silent reading. Readers' eye movements were monitored as they read sentences containing words that varied in familiarity as assessed by printed estimates of word frequency, subjective ratings of familiarity, and a multiple‐choice test of meaning knowledge. Effects of word frequency were unaffected by differences in subjective familiarity rating for high frequency words. Differential effects of familiarity rating were observed in low frequency conditions. In addition, processing time on high and low frequency words did not differ when familiarity was held constant for moderately familiar words. Readers spent more initial processing time on novel words than familiar words. Performance on a vocabulary test administered after the reading session demonstrated that readers successfully acquired and retained new word meanings. Finally, reanalysis of word processing time as a function of vocabulary test performance demonstrated a systematic relationship between online processing patterns and memory for novel word meaning.     

A FIELD STUDY OF PERFORMANCE APPRAISAL PURPOSE: RESEARCH- VERSUS ADMINISTRATIVE- BASED RATINGS

Many researchers have discussed the theoretical and practical importance of rating purpose. Nevertheless, the body of empirical studies, the majority of which were conducted in a laboratory setting, focus on leniency. There has been little research on other effects of rating purpose. The present study examines 223 ratees in a field setting for whom there were both administrative-based performance appraisal ratings (which were actually used for personnel decisions) and research-based performance appraisal ratings (obtained for a validation study). Two of the hypotheses were supported; administrative ratings were more lenient than research-based ratings. The administrative-based ratings demonstrated a statistically significant relationship with ratee seniority, while the research-based ratings did not. There was mixed support for a third hypothesis: Research ratings were significantly correlated with a predictor, while the administrative ratings were not. The difference between the validity coefficients, however, was not significant. Contrary to the hypothesis, the rank order between administrative-based and research-based ratings was relatively high ( r = 33).     

Differences in inter-rater reliability and accuracy for a treatment adherence scale

Inter-rater reliability and accuracy are measures of rater performance. Inter-rater reliability is frequently used as a substitute for accuracy despite conceptual differences and literature suggesting important differences between them. The aims of this study were to compare inter-rater reliability and accuracy among a group of raters, using a treatment adherence scale, and to assess for factors affecting the reliability of these ratings. Paired undergraduate raters assessed therapist behavior by viewing videotapes of 4 therapists' cognitive behavioral therapy sessions. Ratings were compared with expert-generated criterion ratings and between raters using intraclass correlation (2,1). Inter-rater reliability was marginally higher than accuracy (p = 0.09). The specific therapist significantly affected inter-rater reliability and accuracy. The frequency and intensity of the therapists' ratable behaviors of criterion ratings correlated only with rater accuracy. Consensus ratings were more accurate than individual ratings, but composite ratings were not more accurate than consensus ratings. In conclusion, accuracy cannot be assumed to exceed inter-rater reliability or vice versa, and both are influenced by multiple factors. In this study, the subject of the ratings (i.e. the therapist and the intensity and frequency of rated behaviors) was shown to influence inter-rater reliability and accuracy. The additional resources needed for a composite rating, a rating based on the average score of paired raters, may be justified by improved accuracy over individual ratings. The additional time required to arrive at a consensus rating, a rating generated following discussion between 2 raters, may not be warranted. Further research is needed to determine whether these findings hold true with other raters and treatment adherence scales.     

Reducing the Effects of Performance Expectations on Behavioral Ratings

In this study, we develop and test two strategies for reducing the effects of performance expectations on behavioral ratings. A 3 × 3 experimental design (N = 169), manipulating preobservation performance cues (positive, negative, or none) and the type of intervention (halo error training, structured recall memory, or none), was conducted. The results of this study indicate that both interventions reduced the effects of performance expectations on behavioral ratings. However, analyses of rating accuracy and measures from both recognition memory and recall memory tests suggest that the structured recall memory intervention has distinct advantages. These analyses indicate that the structured recall intervention can reduce raters' reliance on heuristics and increase the correspondence between raters' memory and their subsequent ratings.     

再认条件下元监测判断的影响因素

探讨了再认条件下元监测判断的影响因素。以中文双字词对作为实验材料 ,安排在不同预先呈现时间、有无进行有意识记 ,以及不同识记遍数等条件下进行元监测判断。用计算机随机呈现项目 ,再认之前先让被试做元监测判断 ,判断是否是记过的项目的把握程度 ,再对有意识记过的项目做迫选再认 ,作为标准测验 ,以验证元监测判断的准确性。结果是预先呈现时间长、有意识记过的和识记遍数多的项目 ,对其再认的反应时较短 ,相应的元监测判断等级也较高 ;对只看而未有意记过的项目的判断是呈现时间长的判断等级反而低于呈现时间短的。此结果表明 ,再认条件下 ,元监测判断的等级的高低与识记项目的激活程度有关     

Frame-of-Reference Training: Overcoming the Effects of Organizational Citizenship Behavior on Performance Rating Accuracy

Frame-of-reference (FOR) training has been shown to increase rating accuracy. Studies suggest, however, that organizational citizenship behavior (OCB) can influence performance ratings insomuch as they are considered in the rating process. We examined the effects of OCB on the effectiveness of FOR training for improving performance rating accuracy. Results from 77 managers showed that OCB attenuates FOR training effectiveness because of rating inflation, unless the managers are trained to observe and evaluate the OCB.     

Gradedness of visual awareness depends on attentional scope: Global perception is more graded than local perception

The gradedness or discreteness of our visual awareness has been debated. Here, we investigate the influence of spatial scope of attention on the gradedness of visual awareness. We manipulated scope of attention using hierarchical letter-based tasks (global: broad scope; local: narrow scope). Participants reported the identity of a masked hierarchical letter either at the global level or at the local level. We measured subjective awareness using the perceptual awareness scale ratings and objective performance. The results indicate more graded visual awareness (lesser slope for the awareness rating curve) at the global level compared to the local level. Graded perception was also observed in visibility ratings usage with global level task showing higher usage of the middle PAS ratings. Our results are in line with the prediction of level of processing hypothesis and show that global/local attentional scope and contextual endogenous factors influence the graded nature of our visual awareness.     

Differences in Inter‐Rater Reliability and Accuracy for a Treatment Adherence Scale

Inter‐rater reliability and accuracy are measures of rater performance. Inter‐rater reliability is frequently used as a substitute for accuracy despite conceptual differences and literature suggesting important differences between them. The aims of this study were to compare inter‐rater reliability and accuracy among a group of raters, using a treatment adherence scale, and to assess for factors affecting the reliability of these ratings. Paired undergraduate raters assessed therapist behavior by viewing videotapes of 4 therapists' cognitive behavioral therapy sessions. Ratings were compared with expert‐generated criterion ratings and between raters using intraclass correlation (2,1). Inter‐rater reliability was marginally higher than accuracy (p = 0.09). The specific therapist significantly affected inter‐rater reliability and accuracy. The frequency and intensity of the therapists' ratable behaviors of criterion ratings correlated only with rater accuracy. Consensus ratings were more accurate than individual ratings, but composite ratings were not more accurate than consensus ratings. In conclusion, accuracy cannot be assumed to exceed inter‐rater reliability or vice versa, and both are influenced by multiple factors. In this study, the subject of the ratings (i.e. the therapist and the intensity and frequency of rated behaviors) was shown to influence inter‐rater reliability and accuracy. The additional resources needed for a composite rating, a rating based on the average score of paired raters, may be justified by improved accuracy over individual ratings. The additional time required to arrive at a consensus rating, a rating generated following discussion between 2 raters, may not be warranted. Further research is needed to determine whether these findings hold true with other raters and treatment adherence scales.     

Effect of conducting behavioral observations and ratings on staff implementation of a paired-stimulus preference assessment

We assessed the effects of trainees observing and rating the accuracy of others implementing a paired-stimulus preference assessment from video on the trainees' subsequent implementation accuracy. These observational experiences increased implementation accuracy with both a confederate and an actual consumer. These results suggest conducting observations and accuracy ratings may prepare staff to implement behavioral procedures without need of a dedicated trainer.     

Performance-based and parent ratings of attention in children treated for a brain tumor: The significance of radiation therapy and tumor location on outcome

Children are at increased risk for cognitive difficulties following the diagnosis and treatment of a brain tumor. Radiation therapy (RT) and tumor location are commonly-cited predictors of neurocognitive functioning. Disruptions to foundational neurocognitive processes such as attention, working memory, and processing speed underlie declines on measures of general intellectual functioning. While several studies have examined visual sustained attention in pediatric brain tumor patients, auditory sustained attention has yet to be examined. This study employs a longitudinal design to examine performance-based and parent ratings of attention in children undergoing surgical resection of a brain tumor (n = 29) and treated with (n = 11) or without (n = 18) RT at 6.79 months post-surgery (baseline) and then again at 30.56 months post-surgery (follow-up). The measures include an auditory continuous performance test (CPT) and parent ratings of attention and hyperactivity on a behavior rating scale. Ultimately, children treated with and without RT performed similarly on performance-based and parent ratings of attention. However, the performance on the auditory CPT differed according to tumor location, with children with infratentorial tumors committing more inattention and inhibitory control errors compared to children with supratentorial tumors. Parent ratings did not differ according to tumor location, and parent ratings and auditory sustained attention performance are not significantly correlated. The findings are interpreted in the context of neurocognitive and brain development.     

Do alcoholic Korsakoff's syndrome patients acquire affective reactions?

In this study we report two experiments that investigate the acquisition of affective reactions. In Experiment 1, unfamiliar melodies were played to Korsakoff's syndrome patients and alcoholic and nonalcoholic control subjects who were matched with them according to age and education. Following a retention interval of 5 min, subjects received a preference test on old and new melodies. Korsakoff's syndrome patients showed the same increase in preference for old melodies as a consequence of prior exposures as control subjects did, but their recognition of melodies was significantly impaired in comparison with controls. In Experiment 2, the same subjects saw photographs of two men. Fictional biographical information depicted one as a "good guy" and the other as a "bad guy." After a retention interval of approximately 20 days, Korsakoffs recalled virtually none of the biographical information; however, 78% preferred the good guy, and impression ratings were less favorable for the bad guy. Korsakoff patients developed preferences and impressions even though they did not have voluntary access to the information on which the preferences were based. However, their impression ratings were less extreme than those of controls. The pattern of results of the two studies is discussed in terms of Johnson's (1983) MEM model of memory.     

Rater agreement,accuracy, and experienced cognitive load: Comparison of distributional and traditional assessment approaches to rating performance

Error in performance ratings is typically believed to be due to the cognitive complexity of the rating task. Distributional assessment (DA) is proposed to improve rater accuracy by reducing cognitive load. In two laboratory studies, raters reported perceptions of cognitive effort and difficulty while assessing rating targets using DA or the traditional assessment approach. Across both studies, DA raters showed greater interrater agreement, and Study 2 findings provide some support for DA being associated with greater true score rating accuracy. However, DA raters also reported experiencing greater cognitive load during the rating task, and cognitive load did not mediate the relationship between rating format and rater accuracy. These findings have important implications regarding our understanding of cognitive load in the rating process.