The effect of witnessing consequences on the behavioral recordings of experimental observers

The cueing effects of interviewer praise contingent on a target behavior and expectation of behavior change were examined with six observers. Experiment I investigated the effect of cues in conjunction with expectation. Experiment II assessed the relative contributions of cues and expectation, and Experiment III examined the effect of cues in the absence of expectation. The frequencies of two behaviors, client eye contact and face touching, were held constant throughout a series of videotaped interviews between an "interviewer" and a "client". A within-subjects design was used in each experiment. During baseline conditions, praise did not follow eye contact by the client on the videotape. In all experimental conditions, praise statements from the interviewer followed each occurrence of eye contact with an equal number of praises delivered at random times when there was no eye contact. Three of the six observers dramatically increased their recordings of eye contact during the first experimental phase, but these increases were not replicated in a second praise condition. There were no systematic changes in recorded face touching. Witnessing the delivery of consequences, rather than expectation seemed to be responsible for the effect. This potential threat to the internal validity of studies using observational data may go undetected by interobserver agreement checks.     

Interdisciplinary agreement in the diagnosis of headache types

The degree of agreement in independent diagnosis of headache by a neurologist and a clinical psychologist was determined to be 86.4% in a series of 66 consecutive patients. Fifty-five percent of the disagreements were due to the patients' giving different information to the two assessors. This source of disagreement was associated with the patients' having had previous psychiatric treatment.This research was supported in part by Grant NS-15235 from the National Institute of Neurological and Communicative Disorders and Stroke.     

Measures of interobserver agreement: Calculation formulas and distribution effects

Seventeen measures of association for observer reliability (interobserver agreement) are reviewed and computational formulas are given in a common notational system. An empirical comparison of 10 of these measures is made over a range of potential reliability check results. The effects on percentage and correlational measures of occurrence frequency, error frequency, and error distribution are examined. The question of which is the best measure of interobserver agreement is discussed in terms of critical issues to be considered     

Convergent and concurrent validation of four measures of assertion

This study examined the relationship among the College Self-Expression Scale (CSES), the Dominance Scale of the California Psychological Inventory (Do), role-play assessment, and peer ratings of assertion. Peer ratings served as the criterion measure. Fifty-three members of a college sorority completed the CSES and the Do Scale, role played eight scenes requiring an assertive response, and rated each other on a five-point scale of assertion. Peer ratings were significantly correlated with the Do Scale and the CSES. A significant correlation was also obtained between the CSES and the Do. Role-play ratings were not significantly related to any measure. Results are discussed in terms of convergent and concurrent validity for the Do and CSES as measures of assertion. A precautionary note is presented regarding the use of role-play assessment for measuring assertion and as a behavioral validation procedure.     

Evaluating interobserver reliability of interval data

Previous recommendations to employ occurrence, nonoccurrence, and overall estimates of interobserver reliability for interval data are reviewed. A rationale for comparing obtained reliability to reliability that would result from a random-chance model is explained. Formulae and graphic functions are presented to allow for the determination of chance agreement for each of the three indices, given any obtained per cent of intervals in which a response is recorded to occur. All indices are interpretable throughout the range of possible obtained values for the per cent of intervals in which a response is recorded. The level of chance agreement simply changes with changing values. Statistical procedures that could be used to determine whether obtained reliability is significantly superior to chance reliability are reviewed. These procedures are rejected because they yield significance levels that are partly a function of sample sizes and because there are no general rules to govern acceptable significance levels depending on the sizes of samples employed.     

Accuracy of observational data with whole interval,partial interval,and momentary time-sampling recording techniques

The purpose of the study was to compare the accuracies obtained from whole interval, partial interval, and momentary time-sample recording procedures. Two types of accuracies were defined. Between-methods accuracy is the degree to which an observer using a particular recording method agrees with a standard using a continuous recording method. Within-methods accuracy is the degree to which an observer and a standard agree when both use the same recording method. Fifty-four undergraduate students viewed a videotape of a woman twisting her hair and recorded the occurrence or nonoccurrence of the behavior using one of the three recording methods. Tapes were divided into segments with low, intermediate, and high rates of behavior. Partial and whole interval recording obtained poorer between-methods accuracies than momentary time sampling. Results showed that whole interval and momentary time-sampling procedures yielded higher within-method accuracies than partial interval recording. Thus, momentary time-sampling recording provided the advantages of both greater representativeness and as few or fewer observer errors than whole or partial interval recording procedures. Use of the interval recording methods by researchers must be justifiable within this framework.     

Book reviews

Zinkin, Hindle, Gordon, Rosemary & Haynes, Jane (eds.). Dialogue in the Analytic Setting: Selected Papers of Louis Zinkin on Jungian and Group Analysis
Lyard, Denyse. Les Analyses d'Enfants: Une Clinique Jungienne.
Bion, W. R. Taming Wild Thoughts.
Eigen, Michael. The Psychoanalytic Mystic.
Langs, Robert. Death Anxiety and Clinical Practice.
Etkind, Alexander. The Eros of the Impossible: The History of Psychoanalysis in Russia.     

Rorschach interrater agreement estimates: An empirical evaluation

A standardized estimation of Rorschach interrater agreement is needed. Percentage agreement, although widely used, is found to be unsuitable. Forty-one protocols from adults in both a normal and a psychiatric sample were scored by two or three scorers, making 85 scoring pairs. Percentage agreement, correlations (phi and Pearson's r ), and kappa were computed on single response, total score, and category level. Percentage agreement shows minimal variation. Even when exceeding 0.80, it can obscure major disagreements. Kappa and correlations both vary in a similar way with level of disagreement. Total score level does not give additional information compared to single score and category levels. Kappa proved to be conservative and reliable and is therefore suggested as a standard estimate.     

Reliability and expected loss: A unifying principle

We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the proportional reduction in loss (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.Bruce Cooil is an Associate Professor of Statistics, and Roland T. Rust is a Professor and area head for Marketing. The authors thank three anonymous reviewers and an Associate Editor for their helpful comments and suggestions. This work was supported in part by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.     

Graphic comparisons of interobserver agreement indices

How changes in the interobserver agreement and disagreement cells in the reliability matrix are reflected differently in eight commonly used reliability indices is shown graphically. Indices which take into account expected chance difference are compared to those indices which do not. Differences between indices which do and do not treat the agreement and the disagreement cells equally are also illustrated.