Designs for evaluating the validity and accuracy of performance ratings

For many jobs, objective measures of performance are not available or are impractical to obtain, such that ratings are the sole source of information. Unfortunately, performance ratings are a distorted source of information. The multitrait-multimethod and person perception designs have been used to investigate the distortions in ratings. The purpose of using these research designs is to isolate the factors that distort the ratings and to use this knowledge to improve the quality of performance ratings. The goal of the present research was to develop a design that combined both the multitrait-multimethod and person perception designs. Each design was discussed, and examples were presented to illustrate that design. The combination design was used to isolate the influence of rater, ratee, and context factors on the quality of performance ratings. Future research was recommended to understand and utilize these factors to improve performance ratings.     

Evidence of the construct validity of developmental ratings of managerial performance

The construct validity of developmental ratings of managerial performance was assessed by using 2 data sets, each based on a different 360 degrees rating instrument. Specifically, the authors investigated the nature of the constructs measured by developmental ratings, the structural relationships among those constructs, and the generalizability of results across 4 rater perspectives (boss, peer, subordinate, and self). A structure with 4 lower order factors (Technical Skills, Administrative Skills, Human Skills, and Citizenship Behaviors) and 2 higher order factors (Task Performance and Contextual Performance) was tested against competing models. Results consistently supported the lower order constructs, but the higher order structure was problematic, indicating that the structure of ratings is not yet well understood. Multisample analyses indicated few practically significant differences in factor structures across perspectives.     

Construct validity of parent ratings of inhibitory control.

Recent literature has emphasized the need to examine executive functions (EF) in children using multiple sources, including both parent rating and performance-based measures. Computerized Go/No-Go tests, including commercially available continuous performance tests (CPTs), represent one of the most commonly used methods of assessing inhibitory control - a variable central to the executive function construct. We examined the relationship between parent ratings of inhibitory control and CPT performance in two mixed clinical samples. Experiment 1 examined 109 children ages 6-18 using the Behavior Rating Inventory of Executive Function (BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2000) and the Conners' CPT-II (Conners, 2000). In this sample, ratings on the BRIEF Inhibit scale (mean T-score = 62.3) were significantly higher than the CPT-II commissions score (mean T-score = 50.7; p < .0001); and the BRIEF and CPT-II scores were not highly correlated (r = - .12). Experiment 2 examined a sample of 131 children ages 7-18 using the BRIEF and the Tests of Variables of Attention (TOVA; Greenberg, 1996). In this sample, parent ratings on the BRIEF Inhibit scale (mean T-score = 56.8) were similar to TOVA commissions scores (mean T-score = 58.6; p = .33), although still poorly correlated (r = -.02). Factor analyses exploring covariance between BRIEF scales CPT-II variables (Experiment 1) and between BRIEF and TOVA (Experiment 2) yielded similar findings. In both experiments, all eight BRIEF scales loaded on a single factor, with no overlap with either the CPT-II or the TOVA. In mixed outpatient clinical samples, the BRIEF appears to measure different elements of inhibitory control than those assessed by computerized continuous performance tests.     

Improving the concurrent validity of the Bender-Gestalt test

30 individuals diagnosed as showing Korsakoff's psychosis, 30 as paranoid schizophrenics and 30 as normal controls, each had the Bender-Gestalt test administered by four different methods. All four showed good concurrent validity. The so-called perfect method which requires the individual to compare his drawings on the standard administration with the Bender-Gestalt test cards and to redraw the design eliminating errors, showed some potential for improving the predictive validity of the Bender-Gestalt test.     

Further evidence for the validity of assessment center dimensions: a meta-analysis of the incremental criterion-related validity of dimension ratings

This study investigates the incremental variance in job performance explained by assessment center (AC) dimensions over and above personality and cognitive ability. The authors extend previous research by using meta-analysis to examine the relationships between AC dimensions, personality, cognitive ability, and job performance. The results indicate that the 7 summary AC dimensions postulated by W. Arthur, Jr., E. A. Day, T. L. McNelly, & P. S. Edens (2003) are distinguishable from popular individual difference constructs and explain a sizeable proportion of variance in job performance beyond cognitive ability and personality. (PsycINFO Database Record (c) 2008 APA, all rights reserved).     

Teachers' ratings of gross motor skills suffer from low concurrent validity

In this study an attempt was made to construct a reliable and valid unifactorial teachers' rating scale for gross motor ability. Study 1 (132 children from 3 to 7 years) revealed that reliability of the scale was acceptable and that the scale represented an unifactorial dimension. Two studies on concurrent validity of the scale with an experimental gross motor task (stepping-stone crossing), the unifactorial subtest Locomotion of the Test of Gross Motor Development and the subtest Balance of the Movement Assessment Battery for Children as criterion measures, did not produce acceptable validity coefficients. In both validation studies an age effect was found. It was concluded that factor specificity does not seem the answer to the usual low validity coefficients of multifactorial teachers' rating scales. An alternative approach is suggested in which the assessment of functional activities in daily situations is stressed. Finally, the inclusion of atypical groups in random samples, which is common practice in research on concurrent validity of screening instruments for children's motor problems, is discussed.     

The differential validity of teacher ratings of Inattention/Overactivity and Aggression

The ability of school-based measures of child behavior to predict differentially the IOWA Conners teacher rating scale Inattention/Overactivity (IO) and Aggression (A) factors was evaluated in a sample of 71 school-aged boys. Multiple measures and multiple methods of assessment were utilized to provide a comprehensive assessment of social and academic behavior. These included direct observations of children in the classroom and on the playground, examination of the organization of children's desks, measures of academic performance, peer nominations of popularity and rejection, and sociometric ratings using the Pupil Evaluation Inventory. Despite moderately high correlations between the teacher rating factors (r = .60), considerable evidence was provided for differential validity on measures of academic performance, peer rating measures, and measures of disruptive or inappropriate classroom and playground behavior. These differences between IO and A factors indicated that the factors were logically independent in important ways, supporting prior work validating separate hyperactivity and aggression dimensions.     

Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

Improving the validity and reliability of medical-legal certifications of suicide

On a very basic level, those who study suicide are either explicitly or implicitly concerned with the certification of suicide deaths by coroners and medical examiners. Although many authors have questioned the reliability and validity of officially reported rates, these statistics continue to be quoted as fact--a potentially problematic practice for researchers who seek accurate and true causative factors for suicidal behavior. In relation to this topic, this article provides an overview of the existing vital statistics registry system and describes and proposes innovations that could ultimately improve the quality of officially reported suicide statistics.     

The validity of using prior ratings of perceive exertion to regulate resistance exercise intensity

This investigation examined the validity of using ratings of perceived exertion (RPE) to regulate intensity during resistance exercise. 19 male (M age = 22.2 yr., SD = 2.7) and 19 female (M age = 21.4 yr., SD = 2.3) participants estimated exertion when performing the knee extension at 40% to 90% of one-repetition maximum. One week later, participants were asked to produce weights for the knee extension that elicited the feelings of exertion associated with 9, 13, and 17 on the Borg RPE scale. The weight produced at 9, 13, and 17 during the production session was compared to the weight lifted at each RPE during the estimation session. The effect sizes associated with the differences in weight lifted across sessions at each RPE were generally small in magnitude, especially at the higher intensities (ES = .40, -.09, .15, respectively). The results of this investigation support the use of RPE as a method of prescribing the intensity of resistance exercise.     

On the invalidity of validity scales: evidence from self-reports and observer ratings in volunteer samples

Because of the potential for bias and error in questionnaire responding, many personality inventories include validity scales intended to correct biased scores or identify invalid protocols. The authors evaluated the utility of several types of validity scales in a volunteer sample of 72 men and 106 women who completed the Revised NEO Personality Inventory (NEO-PI-R; P. T. Costa & R. R. McCrae, 1992) and the Multidimensional Personality Questionnaire (MPQ; A. Tellegen, 1978/1982) and were rated by 2 acquaintances on the observer form of the NEO-PI-R. Analyses indicated that the validity indexes lacked utility in this sample. A partial replication (N = 1,728) also failed to find consistent support for the use of validity scales. The authors illustrate the use of informant ratings in assessing protocol validity and argue that psychological assessors should limit their use of validity scales and seek instead to improve the quality of personality assessments.     

CPQ validity: The relationship between Children's Personality Questionnaire scores and teacher ratings

Scores on the Children's Personality Questionnaire (CPQ) were correlated with scores from rating scale items selected by a panel of raters measuring the same behavior as a CPQ scale. Data were obtained on a normal sample of school children in grades three to six and on two referred groups, underachievers and chilren with behavior or social problems. The teachers of the children were asked to complete two rating scales on each child. Correlations between rating scale scores and CPQ scores suggest that, in general, the CPQ does not measure the traits it purports to measure.     

Improving communicative understanding: The benefits of global processing

The current research investigated whether a global processing style, affording a focus on the broader context, promotes communicative understanding and, in turn, communicative success. Results confirmed that compared to priming local processing, priming global processing enhanced the deciphering of potentially sarcastic statements of which participants were the receivers (Experiment 1). Experiment 2 extended these findings to oral communication: Global processing increased the correct deciphering of meanings that speakers’ attempted to convey by ambiguous sentences to listeners. Overall ‘seeing the forest instead of the trees’ allowed receivers to attend more perceptively to the larger communicative context, thereby facilitating successful communication.     

Response style and the revised NEO personality inventory: validity scales and spousal ratings in a Chinese psychiatric sample

The effects of response style bias on profile scores from the family of NEO scales and the resultant influence of response style on the predictive capacity of these scales continues to be debated. In this study, a large sample of Chinese psychiatric patients were categorized into four response style groups based on their scores from recently developed "validity" scales for the revised NEO Personality Inventory (NEO PI-R). Mean differences and correlations between self-report and spousal ratings of these patients were examined for the NEO PI-R domain and facet scales. Excessive positive self-presentation bias resulted in mean differences between the self-report and spousal ratings for N and E. Correlations between self-report and spousal ratings were reduced in patients engaging in positive self-presentational bias compared to those who were not so categorized on three of the five NEO PI-R scales. However, these results were manifest only in a sub-sample of psychotic patients. Negative self-presentational bias did not affect mean differences or diminish the correlations between the self-report and spousal ratings.