Interdisciplinary agreement in the diagnosis of headache types

The degree of agreement in independent diagnosis of headache by a neurologist and a clinical psychologist was determined to be 86.4% in a series of 66 consecutive patients. Fifty-five percent of the disagreements were due to the patients' giving different information to the two assessors. This source of disagreement was associated with the patients' having had previous psychiatric treatment.This research was supported in part by Grant NS-15235 from the National Institute of Neurological and Communicative Disorders and Stroke.     

Measures of interobserver agreement: Calculation formulas and distribution effects

Seventeen measures of association for observer reliability (interobserver agreement) are reviewed and computational formulas are given in a common notational system. An empirical comparison of 10 of these measures is made over a range of potential reliability check results. The effects on percentage and correlational measures of occurrence frequency, error frequency, and error distribution are examined. The question of which is the best measure of interobserver agreement is discussed in terms of critical issues to be considered     

Evaluating interobserver reliability of interval data

Previous recommendations to employ occurrence, nonoccurrence, and overall estimates of interobserver reliability for interval data are reviewed. A rationale for comparing obtained reliability to reliability that would result from a random-chance model is explained. Formulae and graphic functions are presented to allow for the determination of chance agreement for each of the three indices, given any obtained per cent of intervals in which a response is recorded to occur. All indices are interpretable throughout the range of possible obtained values for the per cent of intervals in which a response is recorded. The level of chance agreement simply changes with changing values. Statistical procedures that could be used to determine whether obtained reliability is significantly superior to chance reliability are reviewed. These procedures are rejected because they yield significance levels that are partly a function of sample sizes and because there are no general rules to govern acceptable significance levels depending on the sizes of samples employed.     

Accuracy of observational data with whole interval,partial interval,and momentary time-sampling recording techniques

The purpose of the study was to compare the accuracies obtained from whole interval, partial interval, and momentary time-sample recording procedures. Two types of accuracies were defined. Between-methods accuracy is the degree to which an observer using a particular recording method agrees with a standard using a continuous recording method. Within-methods accuracy is the degree to which an observer and a standard agree when both use the same recording method. Fifty-four undergraduate students viewed a videotape of a woman twisting her hair and recorded the occurrence or nonoccurrence of the behavior using one of the three recording methods. Tapes were divided into segments with low, intermediate, and high rates of behavior. Partial and whole interval recording obtained poorer between-methods accuracies than momentary time sampling. Results showed that whole interval and momentary time-sampling procedures yielded higher within-method accuracies than partial interval recording. Thus, momentary time-sampling recording provided the advantages of both greater representativeness and as few or fewer observer errors than whole or partial interval recording procedures. Use of the interval recording methods by researchers must be justifiable within this framework.     

Social skills training for juvenile delinquents

Twenty-seven incarcerated juvenile delinquents matched on the number of previous offenses, age, WISC-R IQ, and locus of control were assigned to a Social Skills. Discussion or Control group. Social skills training consisted of instruction, feedback, modeling, behavior rehearsal, social reinforcement and graduated homework assignments. All subjects were assessed before and after treatment on a variety of self-report, role-play and behavioral measures. Analyses of variance for difference scores indicated that the Social Skills group improved significantly more than the Discussion and Control groups, which did not differ. Appropriate interpersonal skills were learned, state anxiety was reduced, internal locus of control was increased, and significant shifts in adjustment to the institutional program were evidenced for the Social Skills group.     

Children's use of imagery in accessing different types of semantic information

Semantic retrieval was examined in second graders under an induced imagery and a control condition. Two types of animal properties, perceptual and functional, were presented to children for verification. Properties were high and low in rated association strength; the size of the property (i.e., large vs small) was also manipulated in the case of perceptual meaning. Results suggest that young children rely on imagery in semantic retrieval; however, second graders are also adept at processing functional as well as perceptual meaning, and at using an abstract memory accessing system. Thus, considerable flexibility is evidenced by subjects in semantic retrieval.     

Rorschach interrater agreement estimates: An empirical evaluation

A standardized estimation of Rorschach interrater agreement is needed. Percentage agreement, although widely used, is found to be unsuitable. Forty-one protocols from adults in both a normal and a psychiatric sample were scored by two or three scorers, making 85 scoring pairs. Percentage agreement, correlations (phi and Pearson's r ), and kappa were computed on single response, total score, and category level. Percentage agreement shows minimal variation. Even when exceeding 0.80, it can obscure major disagreements. Kappa and correlations both vary in a similar way with level of disagreement. Total score level does not give additional information compared to single score and category levels. Kappa proved to be conservative and reliable and is therefore suggested as a standard estimate.     

Principles of good clinical practice (GCP) in clinical research

Good Clinical Practice is an international quality standard for conducting trials that involve participation of human subjects. Currently, the most widely accepted international document forming the base for GCP is the ICH Harmonised Tripartite Guideline for GCP, which defines in detail the responsibilities and obligations of parties engaged in clinical research. The purpose of this paper is to analyse how compliance with GCP provides protection of the trial subjects and assures quality and credibility of the data obtained. An earlier version of this paper was presented at a symposium, Scientific Misconduct: An International Perspective, organised by The Medical University of Warsaw, 16 November, 1998.     

Reliability and expected loss: A unifying principle

We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the proportional reduction in loss (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.Bruce Cooil is an Associate Professor of Statistics, and Roland T. Rust is a Professor and area head for Marketing. The authors thank three anonymous reviewers and an Associate Editor for their helpful comments and suggestions. This work was supported in part by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.     

Reversibility of single-incentive selective associations.

Rats were trained to press a lever in the presence of a tone-light compound stimulus and not to press in its absence. In each of two experiments, schedules were designed to make the compound a conditioned punisher for one group and a conditioned reinforcer for the other. In Experiment 1, one group's responding produced food in the presence of the compound but not in its absence. The other group's responding terminated the compound stimulus, and food was presented only in its absence. When tone and light were later presented separately, light controlled more responding than did tone in the former group, but tone gained substantial control in the latter. The same effects were also observed within subjects when the training schedules were switched over groups. In Experiment 2, two groups avoided shock in the presence of the compound stimulus. In the absence of the compound, one group was not shocked, and the other received both response-independent and response-produced shock. When tone and light were presented separately, the former group's responding was mainly controlled by tone, but the latter group's responding was almost exclusively controlled by light. These effects were also observed within subjects when the training schedules were switched over groups. Thus, these single-incentive selective association effects (appetitive in Experiment 1 and aversive in Experiment 2) were completely reversible. The schedules in which the compound should have been a conditioned reinforcer consistently produced visual control, and auditory control increased when the compound should have become a conditioned punisher. Currently accepted accounts of selective associations based on affinities between shock and auditory stimuli and between food and visual stimuli (i.e., stimulus-reinforcer interactions) do not adequately address these results. The contingencies of reinforcement most recently associated with the compound and with its absence, rather than the nature of the reinforcer, determined whether auditory or visual stimulus control developed.