Validity and reliability of visual ratings of the vertical jump

The validity and reliability of visual estimates of the kinematics of the vertical jump as would be common in qualitative analysis of human movement was studied. Sagittal plane videotapes of 12 females performing vertical jumps were rated on two occasions by three samples of subjects: 6 basketball coaches, 10 kinesiology students, and 5 kinesiology professors. Visual ratings were compared to values quantified by biomechanical analysis using the Peak Performance Technologies system. Assistant collegiate basketball coaches were unable to rate discrete body angles in the vertical jump accurately or consistently. Six of 10 college student raters could accurately and consistently rate over-all range of motion. Since only one of the kinesiology professors could accurately and consistently rate range of motion compared to the majority of the students, professional experience did not affect the ability to rate range of motion in the vertical jump in these subjects.     

Training raters to assess adult ADHD: reliability of ratings

The standardization of ADHD ratings in adults is important given their differing symptom presentation. The authors investigated the agreement and reliability of rater standardization in a large-scale trial of atomoxetine in adults with ADHD. Training of 91 raters for the investigator-administered ADHD Rating Scale (ADHDRS-IV-Inv) occurred prior to initiation of a large, 31-site atomoxetine trial. Agreement between raters on total scores was established in two ways: (a) by Kappa coefficient (rater agreement for each item with the percentage of raters that had identical item-by-item scores) and (b) intraclass correlation coefficients (reliability). For the ADHDRS-IV-Inv, rater agreement was moderate, and reliability, as measured by Cronbach's alpha, was substantial. The data indicate that clinicians can be trained to reliably evaluate ADHD in adults using the ADHDRS-IV-Inv.     

Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

The reliability of ratings by linguistically untrained subjects in response to stress in speech

To determine the reliability with which untrained raters could identify stress in the speech of a single person, two forms of the same material, (1) speech broken into short utterances and (2) speech in its conversational context, were presented to 40 linguistically naive psychology students who were asked to underline those syllables that they perceived as stressed. High reliabilities were obtained from both interrater measures (r=0.96 for each treatment) and a test-retest estimate (r=0.88). However, significantly larger total stress scores were recorded under the short utterance presentation than under the context condition. It was suggested that this result occurred because each of the few syllables in short utterances received greater attention than did the stream of syllables in context. Subsequent regression analysis led to the prediction that, for a short passage to attain a mean score equal to that which it would receive if rated in context, it should contain approximately 40 syllables.     

Estimation of the reliability of a test divided into two parts of unequal length

In some situations where reliability must be estimated it is impossible to divide the measuring instrument into more than two separately scoreable parts. When this is the case, the parts may be homogeneous in content but clearly unequal in length. The resultant scores will not be essentially τ-equivalent, and hence total test reliability cannot be satisfactorily estimated via Cronbach's coefficient alpha. Limitation on the number of parts rules out Kristof's three-part approach. A technique is developed for estimating reliability in such situations. The approach is shown to function very well when applied to five achievement tests.     

Improving the validity of global ratings

To examine the effects of behavior sampling on global ratings, four groups of 10 teachers each received varying amounts of observation training, practice, and feedback. Teachers viewed a series of seven videotapes depicting two boys whose percentage of distractible behavior was systematically manipulated. Ratings of distractibility were obtained for each taped vignette. Results showed that teachers who received observation training and who routinely collected data in their own classroom submitted ratings which corresponded to actual levels of distractible behavior. Teachers who received no training, or who were trained but did not practice, submitted ratings that were significantly less accurate.This research was supported by the Molson Foundation, Montreal, Quebec, Canada.     

Artifact in pain ratings,its implications for test-retest reliability,and correction by a new scaling procedure

The conventionally employed procedure for rating ischemic pain was found to produce a degree of response bias associated with the ceiling points of the scale used. A new approach permitting open-ended ratings followed by transformation of these ratings into a common decile scale provided far greater test-retest reliability. This was explained largely in terms of the attenuation of rating artifact. The new procedure also gave rise to consistently linear functions for ischemic pain. Implications are raised for the measurement of pain as well as other psychological continua.