Computing interrater reliability on the Apple Macintosh computer

A program is described for computing interrater reliability by averaging, for each rater, the correlations between one rater’s ratings and every other rater’s ratings. For situations in which raters rate more than one ratee, raters’ reliabilities can be computed for either each item or each ratee. The program reads data from a text file and puts the reliability coefficients in a text file. The standard Macintosh interface is implemented. The Quick-BASIC program is distributed both as a listing and in compiled form; it can be run with advantage with math coprocessors.     

Interrater reliability of teacher rating scales for children with attention-deficit hyperactivity disorder

Interrater reliability of eight teacher rating scales designed to assess characteristics of attention-deficit hyperactivity disorder was investigated. Coteachers of 46 students completed the rating scales. The students, ages 8–17, were designated as having a Serious Emotional Disturbance. The resulting interrater reliability correlation coefficients ranged from .62 to .87. The percentage of variance shared between raters ranged from a low of 38.4% (the ACTeRS Oppositional factor and the CBCL-TRF Attention Problems factor) to 75.7% (ADHD Rating Scale). The percent of shared variance was higher for younger children. Kappa scores evaluating rater agreement were highest at the two standard deviations above the mean cutoff. The reliability coefficients were consistent with those reported in prior research.     

Iterative techniques for unbiased rating scales

Three iterative techniques for neutralizing the effects of stimulus bias in category rating experiments were examined with a wide variety of stimulus variables. Under all conditions examined, the iterative techniques quickly led to a stable category estimation. This result was obtained for stimulus variables with strong measurement properties, e.g. length and weight; for stimulus variables with only ordinal properties, e.g. emery papers; and for stimulus variables with only nominal properties, where an ordered set is obtained only in the course of the category scaling, e.g. female profiles.     

Effects of instructions and rating scales on item selection for the BSRI scales

Walkup and Abbott (1978) stated that Edwards and Ashworth's (1977) failure to replicate Bem's (1974) selection of items for the Masculinity and Femininity Scales of the Bern Sex Role Inventory (BSRI) may be attributed to differences in the instructions and anchored rating scales used in the two studies. The present study tested the hypothesis that presence of various interaction effects involving instructions and rating scales would influence the acceptability of items for the BSRI Masculinity and Femininity Scales. Results based on the evaluation of individual items by Bem's item selection criteria in each of the four experimental conditions obtained by systematically manipulating two instructions (Bem's and Edwards' instructions) and two rating scales (Bem's and Edwards' rating scales) and also those based on the analysis of variance of item mean desirability ratings from the four experimental conditions supported the hypothesis.     

A computer program to synthesize American Sign Language

A program is described that allows experimenters to generate American Sign Language forms by computer. The computer synthesis of such signs could allow major advances in the experimental investigation of the perception of sign language, much as computer-generated speech has done for the study of speech perception.     

Behavior rating scales: need for refining normative data

The purpose of this study was to generate normative data by grade and sex to accompany behavior rating scales. Teachers rate 483 boys and girls in Grades 1 through 4. The findings suggest rating scales be re-examined since norms by grade level and sex may be desirable attributes.     

A multitrait-multisource confirmatory factor analytic approach to the construct validity of ADHD rating scales

Confirmatory factor analysis was used to model a multitrait-multisource design to evaluate the construct validity of attention-deficit/hyperactivity disorder (ADHD) rating scales. The 2 trait factors were the ADHD inattention and hyperactivity/impulsivity dimensions. The 2 source factors were parents and teachers. In Study 1, parents and teachers rated 1,475 Australian elementary school children on the ADHD symptoms. In Study 2, parents and teachers rated 285 Brazilian elementary school children on the ADHD symptoms. Similar results occurred in both studies with most of the ADHD symptoms containing more source than trait variance, thus providing weak evidence for the convergent and discriminant validity of the symptoms as measured by rating scales. The study outlines the implications of such strong source effects for understanding ADHD.     

Ill-structured measurement designs in organizational research: implications for estimating interrater reliability

Organizational research and practice involving ratings are rife with what the authors term ill-structured measurement designs (ISMDs)--designs in which raters and ratees are neither fully crossed nor nested. This article explores the implications of ISMDs for estimating interrater reliability. The authors first provide a mock example that illustrates potential problems that ISMDs create for common reliability estimators (e.g., Pearson correlations, intraclass correlations). Next, the authors propose an alternative reliability estimator--G(q,k)--that resolves problems with traditional estimators and is equally appropriate for crossed, nested, and ill-structured designs. By using Monte Carlo simulation, the authors evaluate the accuracy of traditional reliability estimators compared with that of G(q,k) for ratings arising from ISMDs. Regardless of condition, G(q,k) yielded estimates as precise or more precise than those of traditional estimators. The advantage of G(q,k) over the traditional estimators became more pronounced with increases in the (a) overlap between the sets of raters that rated each ratee and (b) ratio of rater main effect variance to true score variance. Discussion focuses on implications of this work for organizational research and practice.