A stochastic model for test-retest correlations

A simple stochastic model is formulated in order to determine the optimal time between the first test and the second test when the test-retest method of assessing reliability is used. A forgetting process and a change in true score process are postulated. The optimal time between tests is derived by maximizing the probability that the respondent has not remembered the response on the first test and has not had a change in true score. The resulting test-retest correlation is then found to be a linear function of the true reliability of the test, where the slope of this function is the key probability of not remembering and having no change in true score. Some numerical examples and suggestions for using the results in empirical studies are given. Specific recommendations are presented for improved design and analysis of intentions data.This research was made possible by a grant from the Center for Food Policy Research, Graduate School of Business, Columbia University, New York, New York, 10027.     

ANALYSIS OF THE CONSISTENCY OF OBJECTIVE MEASURES OF SEXUAL AROUSAL IN WOMEN

Eight adult women volunteers viewed the same erotic film in two different sessions. Their resulting genital responses were recorded simultaneously by three different measures: vaginal pressure pulse, vaginal blood volume, and labial temperature change. During the erotic stimulation, the vaginal pressure pulse and labial responses always increased, and the vaginal blood volume response increased on all but two occasions. Although there was considerable intersubject variability in each genital measure, all three measures were found to have some intrasubject consistency over sessions with respect to either their response amplitudes or patterns, with labial temperature being the most consistent on both parameters. The relationship between the response patterns of the three measures during the film was also relatively consistent across sessions, as was the correspondence between subjective ratings of arousal and both vaginal pressure pulse and labial responses but not vaginal blood volume response. To overcome the problem of considerable intrasubject variability of response amplitudes, it was suggested that the inclusion in the data analysis of several parameters of response patterns, which were relatively stable over sessions, might facilitate the evaluation of a treatment.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Comparison of observational recordings in vivo, via mirror, and via television

Simultaneous observational recordings were made in vivo, via an observation mirror, and via closed circuit television. Three of nine observers had extensive experience recording behavior in vivo; three had extensive experience recording behavior via mirror; and three had extensive experience recording via television. Observers recorded nine categories of disruptive behavior for children in a special class setting. Frequencies of behavior recorded in vivo, via mirror, and via television differed significantly for only one category, vocalization. There were no significant main effects or interactions involving the observers' previous experience. Occurrence reliability coefficients computed within and between media demonstrated the similarity of observer agreement in all three media. Data collection procedures using an observational mirror or closed circuit television appear to be reasonable alternatives to in vivo observation in circumstances similar to those in the present study.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Considerations in the choice of interobserver reliability estimates

Two types of interobserver reliability values may be needed in treatment studies in which observers constitute the primary data-acquisition system: trial reliability and the reliability of the composite unit or score which is subsequently analyzed, e.g., daily or weekly session totals. Two approaches to determining interobserver reliability are described: percentage agreement and "correlational" measures of reliability. The interpretation of these estimates, factors affecting their magnitude, and the advantages and limitations of each approach are presented.     

Children's strategies in producing three-dimensional relationships on a two-dimensional surface

It is a common task to give children a picture containing implicit depth cues and to require them to extract depth information from it. The cues are always selected from the adult repertoire; little is known about children's production of their own cues. In this experiment, 5- to 10-year-old children were required to draw one object behind another in a situation in which adults invariably produce the further object partially occluded by the nearer. The results were an age-related decline in the tendency to segregate the objects and an increase in the tendency to group the objects using partial occlusion, with a cross-over at 8 years. At all ages some children drew one object inside the boundary of the other. It is argued that the results are composed of two tendencies, a gradual mastery of discrete scaling phenomena (e.g., “up” on the page means “further”) within a given style, and a set of decisions to be made between incompatible styles.     

The processing of fundamental frequency in a dichotic matching task

The temporal characteristics of auditory memory were investigated using a reaction time paradigm. The study attempted to determine whether or not there are functionally distinct left and right hemisphere components of auditory memory with different capacities for the retention of nonlinguistic information, such as fundamental frequency, over the course of several seconds. The results indicated that fundamental frequency information in auditory memory remains substantially unchanged over the course of 2 sec. No evidence was found for the existence of separate left and right hemisphere components of auditory memory with different temporal characteristics. The implications of this finding for the organization of auditory memory in the brain are discussed.     

EMPLOYING TASK ARRANGEMENTS AND VERBAL CONTINGENCIES TO PROMOTE VERBALIZATIONS BETWEEN RETARDED CHILDREN1

This study investigated the effects of arranging task events for interdependence, to increase the probability of social responding. During task interdependence, the subjects, participating in dyads and a four-person group, obtained task materials (a puzzle piece) from their partner before completing their task (appropriately placing the puzzle piece). The verbal contingency required a verbal request to precede a subject's receiving a task material from his partner. The verbal contingency yoked with task interdependence made task completion contingent on the appropriate verbalization. The findings from two experiments suggested that task interdependence was sufficient to increase partner-directed verbalizations for three of the four subjects. When the verbal contingency was added, all subjects increased their requests and other verbalizations to partner. Applied to a four-person group, the verbal contingency yoked with varying levels of task interdependence correspondingly affected the pattern and level of group communications. The greater the task interdependence, i.e., the more members each subject depended on to complete his task, the more complex the social network of verbal contacts, and the higher the level of both requests and other verbalizations for the group.