Evidence of the construct validity of developmental ratings of managerial performance

The construct validity of developmental ratings of managerial performance was assessed by using 2 data sets, each based on a different 360 degrees rating instrument. Specifically, the authors investigated the nature of the constructs measured by developmental ratings, the structural relationships among those constructs, and the generalizability of results across 4 rater perspectives (boss, peer, subordinate, and self). A structure with 4 lower order factors (Technical Skills, Administrative Skills, Human Skills, and Citizenship Behaviors) and 2 higher order factors (Task Performance and Contextual Performance) was tested against competing models. Results consistently supported the lower order constructs, but the higher order structure was problematic, indicating that the structure of ratings is not yet well understood. Multisample analyses indicated few practically significant differences in factor structures across perspectives.     

Improving the validity of global ratings

To examine the effects of behavior sampling on global ratings, four groups of 10 teachers each received varying amounts of observation training, practice, and feedback. Teachers viewed a series of seven videotapes depicting two boys whose percentage of distractible behavior was systematically manipulated. Ratings of distractibility were obtained for each taped vignette. Results showed that teachers who received observation training and who routinely collected data in their own classroom submitted ratings which corresponded to actual levels of distractible behavior. Teachers who received no training, or who were trained but did not practice, submitted ratings that were significantly less accurate.This research was supported by the Molson Foundation, Montreal, Quebec, Canada.     

Accountability and need for cognition effects on contrast, halo, and accuracy in performance ratings

In the present study, the authors investigated the effects of accountability and need for cognition on contrast errors, halo, and accuracy of performance ratings examined in good and poor performance context conditions, as well as in a context-free control condition. The accountability manipulation reduced the contrast effect and also modified rater recall of good ratee behavior. Accountability reduced halo in ratings and increased rating accuracy in a poor performance context. Accountability also interacted with need for cognition in predicting individual rater halo.     

A camera-based scoring system for evaluating performance accuracy during a golf putting task

Golf putting accuracy is often evaluated by measuring the distance that the ball finishes from the hole. However, accuracy is a function of line and length, and distance-from-hole measures confound these two factors. A scoring system for evaluating putting accuracy is described that enables the efficient measurement of errors in line and length. A camera placed above the hole takes digital photographs of the final position of the ball. A custom-developed program written in the National Instruments LabVIEW graphical programming language derives a variety of accuracy measures from these photographs, including distance from the hole, angle of error, distance short or long from the hole, and distance left or right from the hole. Evaluation of the system indicated that the measures were as accurate as manual measurements and were reliable when rescored on separate occasions. The camera-based scoring system presents a number of advantages in the evaluation of putting accuracy and may be extended to examine performance in other sports. The ScorePutting program may be downloaded from the Psychonomic Society's Archive of Norms, Stimuli, and Data, www.psychonomic.org/archive.     

On the accuracy of ratings of personality by strangers

Past studies in person perception have shown unexpectedly high correlations between targets' self-ratings of personality traits and ratings of those targets made by stranger judges. Possible reasons for the apparent accuracy of strangers' ratings are examined, including personality judgments that are based on (a) the perceived social desirability of targets, (b) known sex-linked base rates of behavior, and (c) the inevitable presence of valid personality cues in the typical stranger rating study. A computer simulation study is then described that illustrates that a relatively small number of nonrandom accurate judgments among many random judgments can produce substantial overall correlations between self-ratings and stranger ratings. The data of the study provide some clarification of past empirical results of both stranger and acquaintance ratings of personality.     

Further evidence for the validity of assessment center dimensions: a meta-analysis of the incremental criterion-related validity of dimension ratings

This study investigates the incremental variance in job performance explained by assessment center (AC) dimensions over and above personality and cognitive ability. The authors extend previous research by using meta-analysis to examine the relationships between AC dimensions, personality, cognitive ability, and job performance. The results indicate that the 7 summary AC dimensions postulated by W. Arthur, Jr., E. A. Day, T. L. McNelly, & P. S. Edens (2003) are distinguishable from popular individual difference constructs and explain a sizeable proportion of variance in job performance beyond cognitive ability and personality. (PsycINFO Database Record (c) 2008 APA, all rights reserved).     

Construct validity of parent ratings of inhibitory control.

Recent literature has emphasized the need to examine executive functions (EF) in children using multiple sources, including both parent rating and performance-based measures. Computerized Go/No-Go tests, including commercially available continuous performance tests (CPTs), represent one of the most commonly used methods of assessing inhibitory control - a variable central to the executive function construct. We examined the relationship between parent ratings of inhibitory control and CPT performance in two mixed clinical samples. Experiment 1 examined 109 children ages 6-18 using the Behavior Rating Inventory of Executive Function (BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2000) and the Conners' CPT-II (Conners, 2000). In this sample, ratings on the BRIEF Inhibit scale (mean T-score = 62.3) were significantly higher than the CPT-II commissions score (mean T-score = 50.7; p < .0001); and the BRIEF and CPT-II scores were not highly correlated (r = - .12). Experiment 2 examined a sample of 131 children ages 7-18 using the BRIEF and the Tests of Variables of Attention (TOVA; Greenberg, 1996). In this sample, parent ratings on the BRIEF Inhibit scale (mean T-score = 56.8) were similar to TOVA commissions scores (mean T-score = 58.6; p = .33), although still poorly correlated (r = -.02). Factor analyses exploring covariance between BRIEF scales CPT-II variables (Experiment 1) and between BRIEF and TOVA (Experiment 2) yielded similar findings. In both experiments, all eight BRIEF scales loaded on a single factor, with no overlap with either the CPT-II or the TOVA. In mixed outpatient clinical samples, the BRIEF appears to measure different elements of inhibitory control than those assessed by computerized continuous performance tests.     

The accuracy of roommate ratings of behaviors versus beliefs

We evaluated the accuracy of peer ratings of roommates’ personality characteristics, against roommate self-ratings, as a function of rating domain observability. Instead of the usual ratings of broad personality traits, however, our domains represented peer ratings of narrow exemplars of personality traits. Specifically, we compared roommate ratings on (a) observable trait-related behaviors with (b) unobservable trait-related attitudes or beliefs. We observed greater self-peer agreement in rating behaviors, in general, than in rating beliefs. We also observed greater tendency of raters to adopt an assumed similarity heuristic when judging their roommates’ attitudes and beliefs than their behaviors. We discuss the contribution of these findings to understanding the determinants of accuracy in personality judgments and developing best practices for personality assessment.     

Understanding the latent structure of job performance ratings

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n = 2,350 and n = 2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%.     

The differential validity of teacher ratings of Inattention/Overactivity and Aggression

The ability of school-based measures of child behavior to predict differentially the IOWA Conners teacher rating scale Inattention/Overactivity (IO) and Aggression (A) factors was evaluated in a sample of 71 school-aged boys. Multiple measures and multiple methods of assessment were utilized to provide a comprehensive assessment of social and academic behavior. These included direct observations of children in the classroom and on the playground, examination of the organization of children's desks, measures of academic performance, peer nominations of popularity and rejection, and sociometric ratings using the Pupil Evaluation Inventory. Despite moderately high correlations between the teacher rating factors (r = .60), considerable evidence was provided for differential validity on measures of academic performance, peer rating measures, and measures of disruptive or inappropriate classroom and playground behavior. These differences between IO and A factors indicated that the factors were logically independent in important ways, supporting prior work validating separate hyperactivity and aggression dimensions.     

Teachers' ratings of gross motor skills suffer from low concurrent validity

In this study an attempt was made to construct a reliable and valid unifactorial teachers' rating scale for gross motor ability. Study 1 (132 children from 3 to 7 years) revealed that reliability of the scale was acceptable and that the scale represented an unifactorial dimension. Two studies on concurrent validity of the scale with an experimental gross motor task (stepping-stone crossing), the unifactorial subtest Locomotion of the Test of Gross Motor Development and the subtest Balance of the Movement Assessment Battery for Children as criterion measures, did not produce acceptable validity coefficients. In both validation studies an age effect was found. It was concluded that factor specificity does not seem the answer to the usual low validity coefficients of multifactorial teachers' rating scales. An alternative approach is suggested in which the assessment of functional activities in daily situations is stressed. Finally, the inclusion of atypical groups in random samples, which is common practice in research on concurrent validity of screening instruments for children's motor problems, is discussed.     

True intercorrelation among job components, time delay in rating, and rater intelligence as determinants of accuracy in performance ratings

Ninety subjects viewed one of three sets of videotapes which presented simulated work performances by five ratees. Subjects viewed videotapes where the true intercorrelation among the three job components was either high, moderate, or low. Half of the subjects completed ratings immediately after viewing all five ratees and again, 24 h later. The remaining subjects completed ratings only once, 24 h after viewing the videotapes. Rater intelligence was assessed via the Wesman Personnel Classification Test. Performance ratings for each ratee on each of the three job components were completed using a magnitude estimation scale. Subjects' ratings were compared to ratee true scores (based on objective worker output) to obtain four components of rater accuracy: elevation, differential elevation, stereotype accuracy, and differential accuracy as well as a measure of overall accuracy. Results indicated that subjects' ratings were more accurate with respect to overall accuracy and differential accuracy when the true intercorrelation among job components was high rather than low. Rater intelligence was significantly correlated with stereotype accuracy. In addition, rater intelligence was significantly related to overall accuracy, differential elevation, and elevation, but only when the true intercorrelation among job components was moderate or high. Also, there was a curvilinear component to the relationship between intelligence and both differential accuracy and stereotype accuracy such that the most intelligent raters tended to be less accurate than more moderately intelligent raters who were, in turn, more accurate than the least intelligent raters. Finally, subjects' immediate ratings were not more accurate than ratings provided by subjects who completed only delayed ratings. However, repeated measures analyses found that for subjects who completed both immediate and delayed ratings, delayed ratings were less accurate than immediate ratings with respect to overall accuracy, differential accuracy, and differential elevation.     

An investigation of factors influencing accountability and performance ratings

The authors explored whether accountability has implications for performance ratings and investigated factors that may influence both accountability and performance ratings. Specifically, they investigated (a) whether feelings of accountability are directly related to performance ratings, (b) whether experimental manipulations that have been proposed to manipulate accountability in fact do so, and (c) the role of motivation-related constructs. They developed and tested a model of factors proposed to influence accountability and performance ratings. Results from 334 raters in an upward feedback situation in a classroom context provide empirical support that feelings of accountability influence rating level, as do perceptions of anonymity and the importance raters place on their jobs. However, neither anonymity nor importance perceptions were related to feelings of accountability.     

Evaluating dynamic performance: the influence of salient Gestalt characteristics on performance ratings

It is well recognized that performance changes over time. However, the effect of these changes on overall assessments of performance is largely unknown. In a laboratory experiment, we examined the influence of salient Gestalt characteristics of a dynamic performance profile on supervisory ratings. We manipulated performance trend (flat, linear-improving, linear-deteriorating, U-shaped, and inverted U-shaped), performance variation (small, large), and performance mean (negative, zero, positive) within subjects and display format (graphic, tabular) between subjects. Participants received and evaluated information about the weekly performance of different employees over a simulated 26-week period. Results showed strong main effects on performance ratings of both performance mean and performance trend, as well as interactions with display format. Theoretical and practical implications of the results are discussed.