Computer-administered interviews and rating scales

To evaluate the value of computer-administered interviews and rating scales, the following topics are reviewed in the present article: (a) strengths and weaknesses of structured and unstructured assessment instruments, (b) advantages and disadvantages of computer administration, and (c) the validity and utility of computer-administered interviews and rating scales. Computer-administered evaluations are more comprehensive and reliable and less biased than evaluations routinely conducted in clinical practice. Also, the use of continuous monitoring systems, which increasingly entail the use of computer administration, has been related to improved treatment outcome. However, the use of computer-administered interviews and rating scales will sometimes lead to false positive diagnoses, and for this reason, it is recommended that computer assessment be combined with clinical judgment.     

Effects of instructions and rating scales on item selection for the BSRI scales

Walkup and Abbott (1978) stated that Edwards and Ashworth's (1977) failure to replicate Bem's (1974) selection of items for the Masculinity and Femininity Scales of the Bern Sex Role Inventory (BSRI) may be attributed to differences in the instructions and anchored rating scales used in the two studies. The present study tested the hypothesis that presence of various interaction effects involving instructions and rating scales would influence the acceptability of items for the BSRI Masculinity and Femininity Scales. Results based on the evaluation of individual items by Bem's item selection criteria in each of the four experimental conditions obtained by systematically manipulating two instructions (Bem's and Edwards' instructions) and two rating scales (Bem's and Edwards' rating scales) and also those based on the analysis of variance of item mean desirability ratings from the four experimental conditions supported the hypothesis.     

Adaptive rating scales in sequential decisions

The benefits gained in decision research through the use of computer-controlled procedures, as opposed to the use of traditional pencil-and-paper techniques, can exceed the expected gains of efficiency and standardization. Computers extend the scope of a sequential decision making paradigm by enhancing (1) the strength of experimental design, (2) reliability of measurement, (3) procedural flexibility, and (4) single-subject data analysis. Data are reported that indicate the necessity of using computers to obtain reliable measurements and valid results for this type of decision research.     

On the history of rating scales

Rating scales constitute one of the most widely employed techniques in research on personality and individual differences. The historical background of rating scales is therefore a matter of considerable interest. Though Galton has generally been given credit for originating rating scale methodology, several applications of rating scales prior to Galton can be identified, and the seminal idea of rating scales can be traced back to Galen.     

Multiple anchoring of category rating scales

A psychophysical experiment compared the effects of two different kinds of anchoring upon category ratings of the sizes of squares: (1) single anchoring in which the same square was presented on every anchoring trial, and (2) multiple anchoring in which squares of different sizes were presented on anchoring trials. Subjects did not rate the anchors, only those squares presented on alternate trials as the series stimuli. The major finding was that the two kinds of anchoring have similar effects. As with the single anchor, the multiple anchor establishes a new endpoint for the scale of judgment. The previously demonstrated relationship of increasing and then decreasing contrast as a function of the remoteness of the single anchor (Sarris, 1967, 1976) was found also for multiple anchoring.     

Behavior rating scales: need for refining normative data

The purpose of this study was to generate normative data by grade and sex to accompany behavior rating scales. Teachers rate 483 boys and girls in Grades 1 through 4. The findings suggest rating scales be re-examined since norms by grade level and sex may be desirable attributes.     

Interrater reliability of teacher rating scales for children with attention-deficit hyperactivity disorder

Interrater reliability of eight teacher rating scales designed to assess characteristics of attention-deficit hyperactivity disorder was investigated. Coteachers of 46 students completed the rating scales. The students, ages 8–17, were designated as having a Serious Emotional Disturbance. The resulting interrater reliability correlation coefficients ranged from .62 to .87. The percentage of variance shared between raters ranged from a low of 38.4% (the ACTeRS Oppositional factor and the CBCL-TRF Attention Problems factor) to 75.7% (ADHD Rating Scale). The percent of shared variance was higher for younger children. Kappa scores evaluating rater agreement were highest at the two standard deviations above the mean cutoff. The reliability coefficients were consistent with those reported in prior research.     

Factor analysis of the Iowa family interaction rating scales

Observational coding systems are uniquely suited for investigating interactional processes in couples and families, but their validity in diverse populations is unknown. We addressed this issue by applying factor analysis to interactional data collected from couples in low-income neighborhoods and coded with the widely used Iowa Family Interaction Rating Scales (IFIRS). Our sample of 414 low-income, ethnically diverse newlywed couples each provided 24-min samples of problem-solving and social support behavior. Interrater reliabilities were strong, and the resultant factors--reflecting positive, negative, and effective communication--were very similar to those obtained with White middle-class samples. Additionally, couples were more negative, less positive, and less effective in problem-solving conversations than in socially supportive conversations, further supporting the validity of the IFIRS in this population. We conclude by discussing the strengths and shortcomings of the IFIRS when used in a low-income, ethnically diverse population.     

Format effects in two teacher rating scales of hyperactivity

The object of this study was to investigate the effect of differences in format on the precision of teacher ratings and thus on the reliability and validity of two teacher rating scales of children's hyperactive behavior. Teachers (N=242) rated a sample of children in their classrooms using rating scales assessing similar attributes with different formats. For a sub-sample the rating scales were readministered after 2 weeks. The results indicated that improvement can be made in the precision of teacher ratings that may be reflected in improved reliability and validity.This research was supported by grants from the Spencer Foundation and the National Institute of Mental Health (MH 29495). A somewhat different version of this paper was presented at the annual convention of the American Educational Research Association, Toronto, Canada, March 1978. The author thanks Nadine Lambert and Leonard Marascuilo for their assistance in the preparation of this paper.     

Individually tailored behavioral rating scales: a possible alternative

Individually tailored behavioral ratings were introduced as a possible alternative to time sampling or counting each occurrence of the behavior. Interrater reliability was found to be good for the more specific behavior categories and reliability between ratings obtained from direct observation and those based on unit records was acceptable. The usefulness of individually tailored behavioral rating scales in demonstrating behavior change is demonstrated through the presentation of a representative case.     

Review of clinical validation of ADHD behavior rating scales

The purpose of this review is to assess the range of overall accuracies for Attention Deficit/Hyperactivity Disorder (ADHD) behavior rating scales evaluated in clinical validation studies. Studies were characterized according to the evidence standards of the American Academy of Neurology (AAN). Studies were excluded due to major design problems such as overfitting by discriminant analysis. The 13 included evaluations of rating scales revealed overall accuracy in the range of 59%-79% with a pooled mean of 69% (+/-7%, standard deviation) and a pooled sample size of 2228 subjects from nine studies. While some of the excluded studies demonstrated higher overall accuracies (>79%), these studies were observed to have factors in experimen tal design and statistics that are known to unduly inflate accuracy. We recommend further research following the full AAN standards, namely well-designed, blinded, pro spective studies of rating scales applied to clinically representative samples evaluated with a clinical standard.     

Natural anchoring at the neutral point of category rating scales

Contextual theories of judgment assume that the rating of any stimulus depends on its relationship to a context of similar stimuli. For example, any specific act of behavior would be rated less favorably when considered in the context of good behaviors then when considered in the context of bad ones. However, two experiments suggest an absolute anchoring of the natural neutral point, an exception to this generalization. In Experiment 1, morally indifferent acts remained “neutral,” regardless of context; and in Experiment 2 the break-even or zero-point in a simulated game of chance was always rated “neutral.” In both experiments, contextual manipulations had powerful effects upon ratings of other stimuli, though never shifting them across the neutral point. Furthermore, both experiments suggest that neutral-point anchoring also affects the use of the most extreme categories in a manner that is unique to this phenomena. In Experiment 2 for example, the introduction of extremely positive “wins” not only made other wins seem less favorable while leaving ratings of the zero-point unchanged, but also made the most negative losses seem more favorable. It is as if the introduction of an extremely positive “win” also introduces the possibility of an extremely negative “loss.” Taken together these findings contradict the basic adaptation-level premise that the entire scale is determined by the neutral point; they are also inconsistent with the assumption of range theories that the scale is anchored by the endpoints of the range of stimuli actually experienced.