A critical review of theS/L reliability index

The meaning and properties of a commonly used index of reliability, S/L,were examined critically. It was found that the index does not reflect any conventional concept of reliability. When used for an identical behavioral observation session, it is not statistically correlated with other reliability indices. Within an observation session, the standardizing measure of Lis beyond the control of the investigator. Furthermore, the reason for the choice of Las the standard is unclear. The role of chance agreement in S/Lis not known. The exact interpretation of the index depends on which observer reports L.Overall the conceptual and mathematical meaning of S/Lis dubious. It is suggested that the S/Lindex should not be used until its nature is shown to be a measure of reliability. Other approaches such as the intraclass correlations and generalizability coefficients should be used instead.The authors are indebted to Johnny Matson for his critique of an earlier version of this paper.     

A stochastic model for test-retest correlations

A simple stochastic model is formulated in order to determine the optimal time between the first test and the second test when the test-retest method of assessing reliability is used. A forgetting process and a change in true score process are postulated. The optimal time between tests is derived by maximizing the probability that the respondent has not remembered the response on the first test and has not had a change in true score. The resulting test-retest correlation is then found to be a linear function of the true reliability of the test, where the slope of this function is the key probability of not remembering and having no change in true score. Some numerical examples and suggestions for using the results in empirical studies are given. Specific recommendations are presented for improved design and analysis of intentions data.This research was made possible by a grant from the Center for Food Policy Research, Graduate School of Business, Columbia University, New York, New York, 10027.     

ANALYSIS OF THE CONSISTENCY OF OBJECTIVE MEASURES OF SEXUAL AROUSAL IN WOMEN

Eight adult women volunteers viewed the same erotic film in two different sessions. Their resulting genital responses were recorded simultaneously by three different measures: vaginal pressure pulse, vaginal blood volume, and labial temperature change. During the erotic stimulation, the vaginal pressure pulse and labial responses always increased, and the vaginal blood volume response increased on all but two occasions. Although there was considerable intersubject variability in each genital measure, all three measures were found to have some intrasubject consistency over sessions with respect to either their response amplitudes or patterns, with labial temperature being the most consistent on both parameters. The relationship between the response patterns of the three measures during the film was also relatively consistent across sessions, as was the correspondence between subjective ratings of arousal and both vaginal pressure pulse and labial responses but not vaginal blood volume response. To overcome the problem of considerable intrasubject variability of response amplitudes, it was suggested that the inclusion in the data analysis of several parameters of response patterns, which were relatively stable over sessions, might facilitate the evaluation of a treatment.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Comparison of observational recordings in vivo, via mirror, and via television

Simultaneous observational recordings were made in vivo, via an observation mirror, and via closed circuit television. Three of nine observers had extensive experience recording behavior in vivo; three had extensive experience recording behavior via mirror; and three had extensive experience recording via television. Observers recorded nine categories of disruptive behavior for children in a special class setting. Frequencies of behavior recorded in vivo, via mirror, and via television differed significantly for only one category, vocalization. There were no significant main effects or interactions involving the observers' previous experience. Occurrence reliability coefficients computed within and between media demonstrated the similarity of observer agreement in all three media. Data collection procedures using an observational mirror or closed circuit television appear to be reasonable alternatives to in vivo observation in circumstances similar to those in the present study.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Considerations in the choice of interobserver reliability estimates

Two types of interobserver reliability values may be needed in treatment studies in which observers constitute the primary data-acquisition system: trial reliability and the reliability of the composite unit or score which is subsequently analyzed, e.g., daily or weekly session totals. Two approaches to determining interobserver reliability are described: percentage agreement and "correlational" measures of reliability. The interpretation of these estimates, factors affecting their magnitude, and the advantages and limitations of each approach are presented.     

EMPLOYING TASK ARRANGEMENTS AND VERBAL CONTINGENCIES TO PROMOTE VERBALIZATIONS BETWEEN RETARDED CHILDREN1

This study investigated the effects of arranging task events for interdependence, to increase the probability of social responding. During task interdependence, the subjects, participating in dyads and a four-person group, obtained task materials (a puzzle piece) from their partner before completing their task (appropriately placing the puzzle piece). The verbal contingency required a verbal request to precede a subject's receiving a task material from his partner. The verbal contingency yoked with task interdependence made task completion contingent on the appropriate verbalization. The findings from two experiments suggested that task interdependence was sufficient to increase partner-directed verbalizations for three of the four subjects. When the verbal contingency was added, all subjects increased their requests and other verbalizations to partner. Applied to a four-person group, the verbal contingency yoked with varying levels of task interdependence correspondingly affected the pattern and level of group communications. The greater the task interdependence, i.e., the more members each subject depended on to complete his task, the more complex the social network of verbal contacts, and the higher the level of both requests and other verbalizations for the group.     

Correlation and prediction in ordinal test theory

Based on the test theory model for ordinal measurements proposed by Schulman and Haden, the present paper considers correlations between tests, attenuation, regressions involving true and observed scores, and prediction of test reliability.The population correlation between tests is shown to be related to the expected sample correlation for samples of sizen ₁ andn ₂. Errors of estimation, measurement and prediction are found to be similar to their counterparts in interval test theory, while attenuation is identical to its counterpart. The bias in estimating population reliability from sample data is compared for Kendall's tau and Spearman's rho.The author wishes to thank the referees for their helpful comments on an earlier draft of this paper, and in particular, for the suggested alternative methods of establishing some of the presented results.     

Reliability of variable message signs information: A field experiment study in Chongqing

A variable message sign (VMS), which is a key component of intelligent transportation systems, has been frequently used in the management of urban roads and motorways to provide drivers with real-time traffic condition information about a road section or area. Nevertheless, there is a lack of unified regulations for VMS design, which makes it difficult to accurately determine the understandability and legibility of VMS information. In practice, inappropriate designs of VMSs are common, such as overload of VMS information and excessive number of phases, particularly on the urban roads of China. Building on our earlier findings obtained by surveys, in this study, field tests were conducted to assess the reliability of VMS information in terms of the visual perception characteristics of drivers. For a full visual perception, the information displayed on a VMS must be reliable, i.e., it should be easy for drivers to detect, obtain, and understand the messages they require. Two basic theories—information theory and visual perception theory—were introduced to build a quantification model of VMS information, select evaluation indices, and design the experimental process. A total of 24 drivers participated in the field experiments and questionnaires. Five indices—information obtainment rate, periodic validity, driver subjective scoring and the corresponding UI (represents information intelligibility), and CE (represents information legibility)—were analyzed to evaluate their relationships to VMS information reliability. The results confirmed that the amount of information, and the scrolling period considering redundant distance, significantly affect the reliability of VMSs information. For static VMSs, the information obtainment rate and the subjective scoring decrease with increasing amount of information. The recollection accuracy of the drivers significantly declines when the amount of information shown increases to 90 bits, corresponding to an information obtainment rate of less than 0.8 and UI equal to 0. For dynamic VMSs, the information reliability deteriorates as the scrolling period shortens and the number of phases increases. Unreasonable reliability is found when the periodic validity is less than 0.9, i.e., the actual scrolling period is more than 10% less than the calculated one, corresponding to CE equal to 0. The reliability of information was evaluated by combining the subjective scoring of the drivers and a data-based statistical analysis and considering driving safety. Accordingly, it was recommended that 90 bit is the maximum amount of information to be shown on a VMS and that the preferred scrolling period of each phase of a dynamic VMS is 5 s. The results of this study support the objective of providing reliable information to drivers by addressing the problems related to the amount of information and presentation time for VMSs. These findings provide a basis for determining the thresholds for VMS information to promote practical and user-friendly designs of VMSs on urban roads.