EFFECTS OF TRAINING AND RATING SCALES ON RATING ERRORS

Ninety business students were randomly assigned to one of three conditions where they used behavioral observation scales (BOS), behavioral expectation scales (BES), or trait scales in observing people on videotape. Half the individuals received four hours of training to minimize rating errors. Rating errors were reduced significantly regardless of the rating scale that was used. However, behavioral criteria were more resistant to rating errors than trait scales. There was no significant difference between BOS and BES on this dimension. With regard to practicality, BOS were evaluated as significantly better than BES and trait scales. BES and trait scales did not differ significantly on this measure.     

A SECOND LOOK AT BEHAVIORAL OBSERVATION SCALES

Latham, Fay, and Saari (1979) discussed the development of behavioral observation scales (BOS) for the appraisal of foremen and managers. They present arguments and data in support of BOS and made several conclusions regarding the relative superiority of BOS versus other appraisal formats, particularly behavioral expectation scales. This critique of their article suggests that most of their conclusions are beset by conceptual problems, questionable statistical analyses, and/or a disregard of previous research.     

THE PRACTICALITY OF BEHAVIORAL OBSERVATION SCALES, BEHAVIORAL EXPECTATION SCALES, AND TRAIT SCALES

The practicality of three appraisal instruments were measured in terms of user preference, namely, behavioral observation scales (BOS), behavioral expectation scales (BES), and trait scales. A questionnaire containing items pertaining to differentiating good from poor performers, objectivity, providing feedback, suggesting training needs, and ease of use was administered to managers and their subordinates. In all instances, BOS were preferred to BES, and in all but two instances, BOS were viewed as superior to trait scales. Trait scales were felt to be as good if not better than BES. A second questionnaire administered to attorneys indicated that BOS would be easier to defend in the courtroom than either BES or trait scales.     

How many fidgets in a pretty much: A critique of behavior rating scales for identifying students with ADHD

The school psychologist's role and responsibilities for meeting the needs of students with Attention-Deficit Hyperactivity Disorder (ADHD) have been expanding in recent years. Students with this disorder present refractory problems that may involve the school psychologist in the assessment and diagnosis of ADHD. Rating scales commonly are used, sometimes in conjunction with other techniques, for assessing and diagnosing ADHD. They often are presented as an objective way to quantify the severity of a child's behavior in comparison with a normative standard. Because rating scales have become such an integral component in the identification of children with ADHD, school psychologists should understand the limitations associated with this methodology. In this article we first describe behavior rating scales and difficulties in the use of cutoff scores to identify students as ADHD. Second, we describe how problems with interobserver agreement hamper the validity of rating scales and the subsequent conclusions that can be drawn about students' behavior. Finally, we present recommendations for obtaining more reliable and valid information from rating scales.     

Effects of instructions and rating scales on item selection for the BSRI scales

Walkup and Abbott (1978) stated that Edwards and Ashworth's (1977) failure to replicate Bem's (1974) selection of items for the Masculinity and Femininity Scales of the Bern Sex Role Inventory (BSRI) may be attributed to differences in the instructions and anchored rating scales used in the two studies. The present study tested the hypothesis that presence of various interaction effects involving instructions and rating scales would influence the acceptability of items for the BSRI Masculinity and Femininity Scales. Results based on the evaluation of individual items by Bem's item selection criteria in each of the four experimental conditions obtained by systematically manipulating two instructions (Bem's and Edwards' instructions) and two rating scales (Bem's and Edwards' rating scales) and also those based on the analysis of variance of item mean desirability ratings from the four experimental conditions supported the hypothesis.     

THE DEVELOPMENT OF BEHAVIORAL OBSERVATION SCALES FOR APPRAISING THE PERFORMANCE OF FOREMEN

Behavioral observation scales (BOS) were developed for first line foremen. BOS are similar to behavioral expectation scales (BES) in that both are based on a job analysis procedure known as the critical incident technique. However, the BOS differ from BES in that, in developing BOS, (a) a group of individuals is observed and rated on a five-point scale as to the frequency with which they engage in the behavior described by each incident/statement, (b) a total score for each individual is determined by summing the observer's responses for each behavioral item, and (c) an item analysis (or factor analysis, depending upon the sample size) is conducted to select the most discriminating items. Those items with the highest correlations with the total score on a scale are retained to form one behavioral criterion or scale (BOS).     

BEHAVIORAL OBSERVATION SCALES FOR PERFORMANCE APPRAISAL PURPOSES

Behavioral items ( N = 78) critical to the job success of logging supervisors were developed from 1204 critical incidents, the frequency with which a supervisor ( N = 300) engaged in each behavior was rated on a 5-point Likert type scale by two sets of observers. A factor analysis reduced the items to 38 and 33, respectively, for the two sets of observers which in turn constituted 10 and 11 factors or criteria for performance evaluation purposes. Multiple regression equations based on composite scores were used to predict cost-related measures of logging crew effectiveness. The shrinkage in Rs after double cross-validation was moderately small. Moreover, the behavioral observation scales (BOS) that were developed by factor analyzing the observation ratings had moderately high reliability and accounted for more variance in the cost-related measures than did the BOS developed by traditional judgmental clustering techniques. The similarities and differences between BOS and BES procedures are discussed.     

Effects of Instructions and Rating Scales on Item Selection for the BSRI Scales

Walkup and Abbott (1978) stated that Edwards and Ashworth's (1977) failure to replicate Bem's (1974) selection of items for the Masculinity and Femininity Scales of the Bern Sex Role Inventory (BSRI) may be attributed to differences in the instructions and anchored rating scales used in the two studies. The present study tested the hypothesis that presence of various interaction effects involving instructions and rating scales would influence the acceptability of items for the BSRI Masculinity and Femininity Scales. Results based on the evaluation of individual items by Bem's item selection criteria in each of the four experimental conditions obtained by systematically manipulating two instructions (Bem's and Edwards' instructions) and two rating scales (Bem's and Edwards' rating scales) and also those based on the analysis of variance of item mean desirability ratings from the four experimental conditions supported the hypothesis.     

A Systematic Review and Psychometric Evaluation of Adaptive Behavior Scales and Recommendations for Practice

Adaptive behavior scales are vital in assessing children and adolescents who experience a range of disabling conditions in school settings. This article presents the results of an evaluation of the design characteristics, norming, scale characteristics, reliability and validity evidence, and bias identification studies supporting 14 norm-referenced, informant-based interviews and rating scales designed to measure adaptive behaviors. To derive these results, the manuals for each of these scales were reviewed using a standardized coding procedure, and information about each scale was double-coded by reviewers. Findings reveal that several evidence-based adaptive behavior scales are available to school psychologists. Concluding recommendations address selection and use of adaptive behavior scales as part of a comprehensive assessment, using the optimal methods of administration of adaptive behavior scales, and interpreting resultant scores that have demonstrated the highest levels of reliability and the largest body of validity evidence.     

Evidence for the Effectiveness of an Alternative Multisource Performance Rating Methodology

Despite persistent concerns as to the quality of performance information obtained from multisource performance ratings (MSPRs), little research has sought ways to improve the psychometric properties of MSPRs. Borrowing from past methodologies designed to improve performance ratings, we present a new method of presenting items in MSPRs, frame‐of‐reference scales (FORS), and test the efficacy of this method in a field and lab study. The field study used confirmatory factor analysis to compare the FORS to traditional rating scales and revealed that FORS are associated with increased variance due to dimensions, decreased overlap among dimensions, and decreased error. The laboratory study compared rating accuracy associated with FORS relative to frame‐of‐reference training (FORT) and a control group and demonstrated that FORS are associated with higher levels of accuracy than the control group and similar levels of accuracy as FORT. Implications for the design and implementation of FORS are discussed.     

Computer-administered interviews and rating scales

To evaluate the value of computer-administered interviews and rating scales, the following topics are reviewed in the present article: (a) strengths and weaknesses of structured and unstructured assessment instruments, (b) advantages and disadvantages of computer administration, and (c) the validity and utility of computer-administered interviews and rating scales. Computer-administered evaluations are more comprehensive and reliable and less biased than evaluations routinely conducted in clinical practice. Also, the use of continuous monitoring systems, which increasingly entail the use of computer administration, has been related to improved treatment outcome. However, the use of computer-administered interviews and rating scales will sometimes lead to false positive diagnoses, and for this reason, it is recommended that computer assessment be combined with clinical judgment.     

Affective meanings of 1,469 Bengali concepts

This article provides semantic differential ratings of 1,469 concepts in Bengali, a language spoken by about 250 million individuals in eastern India and Bangladesh. These data were collected from 20 male and 20 female Calcutta respondents who rated stimuli on three culturally universal affective dimensions: evaluation–potency–activity (EPA). This study employs pan-respondent component analyses as a means of examining the respondents’ usage of the standard EPA scales. The pan-respondent component analyses indicate that some respondents used the rating scales in unexpected ways, recording their feelings about one component of concepts’ EPA with ratings on a scale intended to measure a different dimension. When scores were based only on respondents who used the scales appropriately, several interesting patterns were found. For respondents of both genders, potency scores have a curvilinear relation with evaluation, such that very good and very bad concepts are mostly seen as very potent, whereas evaluatively neutral concepts are seen as somewhat impotent or just slightly potent. A moderate linear correlation exists between activity and evaluation, and a modest positive relation exists between potency and activity. Gender correlations are high on evaluation, .93, but much lower for potency scores, with a correlation of .55, and even lower for activity, .30. In this article we examine several explanations for why scales denoting potency and activity were reinterpreted as indicating goodness by certain respondents, and consider the matter of including data collected from respondents who used scales in this way.     

The Relationships Between Life Satisfaction, Happiness, and Current Mood

The present research proposes and tests a conceptualization of happiness as a situation-dependent evaluation of hedonic experiences, specifically the valence and activation of current mood, as well as being related to cognitive judgments of life satisfaction. In Experiment 1 86 undergraduates rated their life satisfaction (five scales forming the Satisfaction With Life Scale), happiness on five rating scales, and valence and activation of current mood, each on three rating scales, before and after attending a lecture. The results showed that happiness was related to life satisfaction mediated by valence of current mood. The change in happiness from before to after attending the lecture was only influenced by the change in valence of the current mood. In Experiment 2 which used scenarios to induce valence and activation of current mood in four randomized groups with a total of 135 undergraduates, happiness was related to preference for current mood rated on three scales which in turn was directly affected by both the valence and activation induction. Valence was in the mood-induction scenarios related to the expected rate of goal pursuit and activation to incidental events. Experiment 3 demonstrated for another 59 undergraduates that happiness stayed related to preference for a positive mood despite that its relation to a positive mood was reversed in an incongruent situation.     

Is there a Preferred Performance Rating Format? A Non-psychometric Perspective

Les échelles d'appréciation constituent une forme d'évaluation des performances qui a rencontré un grand succès ces dernières décennies. En particulier, de gros efforts ont été consacrés au développement d'échelles relativement insensibles aux erreurs et biais cognitifs. Mais on s'est beaucoup moins intéressé au fait de savoir si et comment le type d'évaluation affecte les attitudes professionnelles et les réactions des personnes évaluées. Des données en provenance de quatre études différentes avec des échantillons tirés dans deux pays (lsraël et le Canada) apportent des éléments en faveur de l'idée selon laquelle une évaluation de la performance basée sur le BOS peut être supérieure aux autres méthodes en ce sens qu'elle entraîne des conséquences plus positives au niveau des attitudes.
One aspect of performance appraisal that has received considerable attention over past decades is the rating format. In particular, much effort has been devoted to developing rating scales that are relatively impervious to cognitive rating errors and biases. However, much less attention has been accorded the issues of whether and how an appraisal's format affects work attitudes and reactions of ratees. Data collected in four separate studies and with samples in two nations (Israel, Canada) lend credence to the proposition that a BOS-based performance appraisal and review may be superior to other appraisal methods in terms of yielding more favorable attitudinal effects.     

结构化面试中评分量表对评分准确性的影响

采用2(组内变量:量尺大小(25分和9分))×2(组间变量:评分方法(相对和绝对))的混合实验设计探讨评分量表对115名大学生新手评委评分准确性的影响。对于评分准确性,采用Cronbach1955年提出的四个指标,Elevation(EL)、Differential elevation(DE)、Stereotype accuracy(SA)、Differential Accuracy(DA)。结果发现,评分方法只在SA上主效应显著,量尺大小在只在DA上主效应边缘显著,评分方法和量尺大小在DE、SA和DA三个指标上均有交互作用。总体上看,在结构化面试评分中,对于评分准确性,相对评分量表优于绝对评分量表,小量尺量表优于大量尺量表。