AN EVALUATION OF ALTERNATE SCORING METHODS FOR THE MIXED STANDARD SCALE

This study investigated the psychometric properties of three methods of scoring a Mixed Standard Scale (MSS) performance evaluation: the patterned procedure as corrected by Saal (1979); a simple nonpatterned scoring procedure suggested by Prien, Jones, and Miller (1977), which gives equal weights to the performance statements; and a procedure that assigned differential weights to each statement on the basis of scale values provided by a panel of subject matter experts. Interrater reliabilities, scale variances for averaged ratings, and a convergent/discriminant validity analysis, which included an alternate method of job skill ratings, indicated no difference in the score distribution variance, interrater reliability, or validity of different method scores.     

THE EFFECT OF QUALITY CONTROL REVISIONS ON MIXED STANDARD SCALE RATING ERRORS

Evaluations of performance evaluation ratings conventionally use various psychometric criteria, with reliability being the measure most commonly reported. Rating errors have been attributed to a variety of sources, including the statistical properties of the format or content, ratee performance variability, rater biases, and characteristics of the work setting. This article reports research on error statistics that can be generated using the Mixed Standard Scale (MSS) evaluation format. The reduction of system error rates following revision guided by statistical analysis is a clear indication that the quality control features of this format are useful in the management of performance evaluation systems.     

Prediction of job satisfactoriness for workers with severe handicaps from aptitudes,personality, and training ratings

The predictive validity of General Aptitude Test Battery and Sixteen Personality Factor Questionnaire scores were compared to standard training ratings made by vocational instructors against the criterion of work performance measured by the Minnesota Satisfactoriness Scales for a sample of 106 employees with severe handicaps. The psychometric test variables were not correlated with the criterion; however, the training ratings were consistently predictive of the job satisfactoriness scores. These results suggest that the employment potential of job applicants with disabilities can be assessed more accurately using situational training ratings, as opposed to standardized psychometric test scores.     

An experimental investigation into judgment and behavioral implications of disability‐based stereotypes in simulated work decisions: Evidence of shifting standards

We investigated the explanatory relevance of the shifting standards model to the role of employee disability in two simulated organizational processes: (a) job performance appraisal and (b) resource allocation. Two experimental studies found evidence of the shifting standards effect—employees with and without a disability received similar ratings when subjective judgment standards were employed, but those without a disability received significantly higher performance ratings than their counterparts with a disability when rated with objective standards. Furthermore, the analysis of the ranking process on the dimension of deservingness of limited resources yielded evidence that disability‐related stereotypes can differentially affect perceptions of deservingness through the allocation of organizational resources.     

INTERPERSONAL AFFECT AND PERFORMANCE APPRAISAL: A FIELD STUDY

Interpersonal affect has been found, in prior laboratory research, to be related to ratings of job performance. Such findings have been taken to mean that affect creates bias in ratings. The present study was conducted to determine if this relationship would hold up in a field setting. The present study was also designed to examine how structured diary-keeping, and the nature of the appraisal instrument, might be related to affect-appraisal relationships. The results for 85 raters, and 404 ratees, suggested that affect was significantly related to all ratings, but more strongly related to trait-like ratings than task/outcome-like ratings, and that having raters keep performance diaries actually increased the strength of the relationship between affect and ratings. We concluded that affect may not be a biasing influence on ratings, but may be a result of better subordinate performance. Results from an analysis of the diary content supported this conclusion. Implications for the role of affect on ratings and the nature of the relationship between past performance and interpersonal affect in field settings are discussed.     

Context and Prior Impression Effects on Attention, Judgment Standards, and Ratings: Contrast Effects Revisited

The present study extended research on contrast effects by (a) examining the effect of context performances on ratings of a target performance when a prior impression of the target performer already exists, and (b) clarifying the issue of whether contrast is caused by attention to context-discrepant behavior or shifts in judgment standards. The results demonstrated that the existence of a prior impression mitigates the influence of context performances on ratings. Judgment standards were found to be unstable and dependent on information provided to raters by the experimental manipulations. Regression analyses showed that both attention and standards of judgment mediate the relationship between context and ratings. Implications of these findings for contrast effects, performance ratings, and the importance of reliable judgment standards for real-world performance appraisals are discussed.     

Validation of a Swedish short self-report measure of core affect

This research aimed at validating two self-report composite rating scales of core affect that are useful when quick assessments are required, for instance, of current mood or recalled, anticipated, and experienced emotional reactions. The ratings were derived from the Swedish Core Affect Scales (SCAS) comprising six self-report rating scales of the two orthogonal dimensions of core affect, valence (unpleasantness-pleasantness) and activation (quietness-excitement). In three samples of university students who were requested to rate current moods, affect-inducing sounds, or affect-inducing pictures, the composite ratings were compared to SCAS, two widely used graphical rating scales of valence and activation, and skin conductance responses and the acceleratory peak of heart rate. Three different rating formats were also compared. The results showed that the composite ratings were reliable and yielded the expected correlations with the other ratings and with the physiological affect indicators. No effects of rating format were detected. It is concluded that the composite ratings of valence and activation may be used if quick assessments are called for. Choices can be made of any of three rating formats depending on purpose with the assessments.     

On Being the Same in Different Places: Evaluating Frame‐of‐Reference Effects Across Two Social Contexts

Frame‐of‐reference (FOR) effects in personality assessment are demonstrated when self‐rated items oriented to specific contexts (e.g., workplace) show better predictive validity than noncontextualized items. Empirical support of FOR effects typically relies on job performance ratings or academic grades for criteria. The current study evaluates FOR effects using ratings of personality provided by informants from the home or school context. Items from the NEO Five‐Factor Inventory (NEO‐FFI; Costa & McCrae, 1992) were contextualized to the home and school contexts to create NEO‐Home and NEO‐School versions. One hundred fifty‐eight college students completed the NEO‐Home and NEO‐School questionnaires, and 161 college students completed the standard, noncontextualized NEO‐FFI. All participants recruited one peer from college and at least one parent to complete standard rater versions of the NEO‐FFI. Contextualized self‐ratings did not show FOR effects. NEO‐Home self‐ratings did not correlate higher with parent ratings than with peer ratings, and NEO‐School self‐ratings did not correlate higher with peer ratings than with parent ratings. Standard NEO‐FFI self‐ratings generally showed higher self‐informant agreement with both types of informants than contextualized self‐ratings. The pattern of correlations suggests that validity is enhanced more by specific trait‐informant combinations than by the contextualization of items to social contexts.     

Comparison of the Vineland Adaptive Behavior Scales, Second Edition, and the Bayley Scales of Infant and Toddler Development, Third Edition

The Vineland Adaptive Behavior Scales, Second Edition (Vineland-II), and Bayley Scales of Infant and Toddler Development, Third Edition (Bayley-III) were administered to 65 children between the ages of 12 and 42 months referred for developmental delays. Standard scores and age equivalents were compared across instruments. Analyses showed no statistical difference between Vineland-II ABC standard scores and cognitive levels obtained from the Bayley-III. However, Vineland-II Communication and Motor domain standard scores were significantly higher than corresponding scores on the Bayley-III. In addition, age equivalent scores were significantly higher on the Vineland-II for the fine motor subdomain. Implications for early intervention are discussed.     

Sex Effects in Performance--Related Judgments

This study examined the relationships among objective performance, performance ratings, and pay allocations made by male and female managers. Because rater sex has been found to affect performance ratings and pay allocations, but little research has explored the causes of these sex differences, this study explored the effects of contextual factors that may moderate differences between the performance ratings and pay allocations of male and female raters. Two hundred twenty-nine managers assigned performance ratings and merit pay allocations to performance profiles that varied in levels of objective performance. It was found that: (a) sex effects on performance ratings were moderated by contextual variables, and b) sex and objective performance did not influence merit pay allocations independent of their separate effects on performance ratings.     

Practicum and On-the-job Ratings of School Counselors

Practicum ratings of 50 school counselors were examined in relation to administrators' subsequent on-the-job ratings of their performance. The relationship proved negligible. The practicum ratings did relate, however, to retention of cognitive material in guidance and counseling (scores on a comprehensive examination). Such a relationship did not exist between scores on the same comprehensive examination and the administrators' ratings. It was suggested that cognitive standards for counselors are probably best established and maintained by training institutions. Administrators seeking assurance of at least minimum cognitive competency in counselors ought to seek endorsement of candidates from those who observe their practicum performance.     

Reading-disabled hyperactive children: A distinct subgroup of Attention Deficit Disorder with Hyperactivity?

Mixed hyperactive/reading-disabled children, selected from a large sample of hyperactive children, were compared to pure hyperactive children on demographic, behavioral, and neuropsychological measures. Children in the Mixed group were reading-disabled relative to age and IQ; those in the Pure group had achievement scores that were average relative to age and IQ. The Mixed group was significantly older than the Pure group; there were no differences in gender, race, or socioeconomic status. Behaviorally, the groups did not differ significantly on teacher ratings or on psychiatric ratings of aggression. The Mixed group had a significantly higher Performance IQ, whereas the Pure group had a significantly higher Verbal IQ and performed better on measures of cognitive impulsivity. Several other measures failed to distinguish the groups. The data give minimal support to the notion that pure hyperactive and mixed hyperative/reading-disabled children constitute distinct subgroups of Attention Deficit Disorder with Hyperactivity.This research was supported in part by Mental Health Clinical Research Grant MH-30906 and by U. S. Public Health Service Grant MH-18579.     

The Relationship Between Measures of Cognitive Attention and Behavioral Ratings of Attention in Typically Developing Children

In the present study, we explored the relation between performance on cognitive measures of attention (selection, sustained, and control) and behavioral ratings of inattention and hyperactivity in a sample of typically developing children aged 3 to 7 years. We also examined the influence of chronological age and IQ on both task performance and behavior ratings. Four well-documented attention paradigms were employed, the Visearch (single-target search) task as a measure of selective attention, the Continuous Performance Test (CPT) as a measure of sustained attention, the Day-Night task as a measure of response inhibition, and the Visearch (dual-target search) task as a measure of inhibitory control. The Conners' Rating Scales (Cognitive/Inattention and Hyperactivity subscales) were used to allow for a finer tuned comparison of cognitive performance as related to inattentive behaviors versus hyperactive behaviors. Findings indicate that accuracy and speed in the Visearch dual search task were the most sensitive measures relating respectively to inattentive and hyperactive rated behaviors.     

Role of Behavior Level,Behavioral Variability,and Ratee Order in the Formation of Appraisal Ratings

Two laboratory studies were conducted to assess the role of behavioral data in the formation of appraisal ratings. In the first study, which included 200 participants, significant main effects using a split-plot factorial analysis of variance design were found for performance variability, performance level, and ratee order. A central drift phenomenon also was discovered with ratees whose performance profiles were both similar and extreme (extremely high or extremely low), receiving ratings that drifted toward average as a function of rating queue position. A second study, which included 126 participants, replicated this drift phenomenon and isolated the cause as profile similarity rather than profile extremity. Implications of these findings for rating accuracy and future "truescore" research designs are discussed.     

Self-Ratings of Personality: A Naturalistic Comparison of Normative, Ipsative, and Idiothetic Standards

ABSTRACT The purpose of this research is to examine some of the implicit standards that individuals may use when they rate themselves on personality dimensions Specifically, we compared ratings based on instructions that specified normative, ipsative, or idiothetic creteria with ratings obtained when subjects used their own (implicit) standards In a series of three studies, we found that each of the explicit ratings was highly similar to the implicit-standard ratings and that there were substantial individual differences in the relative similarity of each of the explicit ratings to the implicit ratings However, because the normative instructions elicited less extreme responses than any of the other instructions, there was a small but consistent tendency for them to be less similar to the ratings based on implicit standards than the ipsative or idiothetic ratings The implications of this research for personality assessment and self-evaluation are discussed     

External correlates of the MMPI-2 content component scales in mental health inpatients

External correlates of the Minnesota Multiphasic Personality Inventory-2 (MMPI-2) Content Component Scales were identified using an inpatient sample of 544 adults. The Brief Psychiatric Rating Scale (BPRS) and Symptom Checklist 90-Revised (SCL-90-R) produced correlates of the Content Component Scales, demonstrating external validity with clinician-rated and self-report scales. Relationships between MMPI-2 Content Component Scales and patient hospital chart variables were also examined. Results demonstrated cross-criterion validity in that most MMPI-2 Content Component Scales correlated with appropriate BPRS dimensions, SCL-90-R items, and other patient variables. Discriminant validity of the Content Component Scales was also demonstrated. In addition, the finding that similar patterns of correlates are produced when the component scales are correlated with a self-report measure, as well as clinician ratings and medical chart variables, provides converging lines of evidence supporting the construct validity of the Content Component Scales.     

Effects of Organizational Citizenship Behaviours: Evidence from Production Supervisors

Based on a sample of production workers in a textile firm, supervisory ratings of employee performance were found to be influenced by supervisors' perceptions of employees' organizational citizenship behaviours as well as by employees' objectively measured performance. Regression analysis indicated that supervisory ratings were influenced by two dimensions of organizational citizenship behaviour — conscientiousness and altruism — as well as by objective performance. These three variables explained 53% of the variance in ratings. A third dimension of OCB, sportsmanship, did not affect supervisory ratings.     

Observing home-visiting quality across time: A longitudinal reliability study of the Home Visit Rating Scales

Empirical examination of home-visiting quality over time is lacking in the research literature. To this end, this study examined the most recent edition of a widely used observational measure of home-visiting quality, the Home Visit Rating Scales (HOVRS), over four repeated assessments spanning an average of 6 months. Psychometric quality for the HOVRS Total and Home Visitor Practice Scales, including item ratings and calculated scores, across time was demonstrated. Stability indicators for the Family Engagement Scales varied, with superior reliability often indicated for individual ratings over the calculated score. The exploration of interrelationships among family demographics and change in HOVRS scores over time were largely insignificant. Interestingly, a decline in the Home Visitor Practice Scale significantly related to increased symptoms of maternal depression, possibly suggesting that home-visitors’ focus shifted from children's needs to those of parent s when mental health concerns were observed. Although this study is an examination of one program model and poses limited generalizability, it sets the stage for advancing the HOVRS and longitudinal measurement of home-visiting quality.     

When fit is fundamental: performance evaluations and promotions of upper-level female and male managers

Using archival organizational data, the authors examined relationships of gender and type of position (i.e., line or staff) to performance evaluations of 448 upper-level managers, and relationships of performance evaluations to promotions during the subsequent 2 years. Consistent with the idea that there is a greater perceived lack of fit between stereotypical attributes of women and requirements of line jobs than staff jobs, women in line jobs received lower performance ratings than women in staff jobs or men in either line or staff jobs. Moreover, promoted women had received higher performance ratings than promoted men and performance ratings were more strongly related to promotions for women than men, suggesting that women were held to stricter standards for promotion.