Reliability of self-reported service use: Test-retest consistency of children's responses to the Child and Adolescent Services Assessment (CASA)

This paper reports on the reliability of children's responses on the Child and Adolescent Services Assessment (CASA) — a self-report instrument for use with 8– to 18-year-olds that gathers information about services used to address mental health problems. Findings were based on interviews completed by 77 children at a one week test-retest interval. Results showed that reports of lifetime service use were as reliable as were reports of service use in the preceding three months. Children reported restrictive and intrusive services more reliably than services that were provided in their natural environment. Reliability appeared to be associated more strongly with characteristics of the type of service than with characteristics of the child. Children also could report reliably on some details about their encounters with service providers (e.g., length of stay, number of visits, and onset of service use).     

三种心理测量理论的信度观

目前,心理测量领域中主要存在三大理论派别。本文分别对这三种理论即经典测验理论、可概括性理论和项目反应理论作了简要介绍,着重分析这三种理论的信度观。文章讨论了这三种信度观的理论基础和研究方法,比较了它们的异同,指出经典测验理论存在的一些不足及概化理论和项目反应理论所作的改进。概化理论是对经典测验理论的扩展,它用多维的信度指标（概化系数）替代了经典测验理论的信度系数,项目反应理论则从信息量的角度出发,用项目信息函数、测验信息函数等指标更具体深入地反映项目、测验的测量可靠程度。     

Influence of subject and interviewer characteristics on the reliability of young adults' self-reports of drinking

The influence of subject and interviewer characteristics on the reliability of self-reports of alcohol consumption among young adults was investigated. Data were gathered from black and white college students of both sexes (N's=24) using a time-line procedure. The results snowed that these young adults provided highly reliable self-reports regarding their use of alcohol. Test-retest correlations for a criterion interval of 90 days were .96, .93, and .97 for the numbers of abstinent, moderate-drinking, and heavy-drinking days, respectively. Analyses also showed that white female subjects generally provided more reliable reports of abstinent and heavy-drinking days and that white female interviewers gathered more reliable reports across the three drinking disposition categories. These findings suggest that nonalcoholic young adults' retrospective reports of their drinking behavior can be reliably assessed using the time-line methodology. Future research is required to determine the validity of these self-reports and to understand the differential influence of subject and interviewer characteristics on the levels of reliability found.     

A critical review of theS/L reliability index

The meaning and properties of a commonly used index of reliability, S/L,were examined critically. It was found that the index does not reflect any conventional concept of reliability. When used for an identical behavioral observation session, it is not statistically correlated with other reliability indices. Within an observation session, the standardizing measure of Lis beyond the control of the investigator. Furthermore, the reason for the choice of Las the standard is unclear. The role of chance agreement in S/Lis not known. The exact interpretation of the index depends on which observer reports L.Overall the conceptual and mathematical meaning of S/Lis dubious. It is suggested that the S/Lindex should not be used until its nature is shown to be a measure of reliability. Other approaches such as the intraclass correlations and generalizability coefficients should be used instead.The authors are indebted to Johnny Matson for his critique of an earlier version of this paper.     

A test of the hypothesis that Cronbach's alpha reliability coefficient is the same for two tests administered to the same sample

In measurement studies the researcher may wish to test the hypothesis that Cronbach's alpha reliability coefficient is the same for two measurement procedures. A statistical test exists for independent samples of subjects. In this paper three procedures are developed for the situation in which the coefficients are determined from the same sample. All three procedures are computationally simple and give tight control of Type I error when the sample size is 50 or greater.The author is indebted to Jerry S. Gilmer for development of the computer programs used in this study.     

A stochastic model for test-retest correlations

A simple stochastic model is formulated in order to determine the optimal time between the first test and the second test when the test-retest method of assessing reliability is used. A forgetting process and a change in true score process are postulated. The optimal time between tests is derived by maximizing the probability that the respondent has not remembered the response on the first test and has not had a change in true score. The resulting test-retest correlation is then found to be a linear function of the true reliability of the test, where the slope of this function is the key probability of not remembering and having no change in true score. Some numerical examples and suggestions for using the results in empirical studies are given. Specific recommendations are presented for improved design and analysis of intentions data.This research was made possible by a grant from the Center for Food Policy Research, Graduate School of Business, Columbia University, New York, New York, 10027.     

ANALYSIS OF THE CONSISTENCY OF OBJECTIVE MEASURES OF SEXUAL AROUSAL IN WOMEN

Eight adult women volunteers viewed the same erotic film in two different sessions. Their resulting genital responses were recorded simultaneously by three different measures: vaginal pressure pulse, vaginal blood volume, and labial temperature change. During the erotic stimulation, the vaginal pressure pulse and labial responses always increased, and the vaginal blood volume response increased on all but two occasions. Although there was considerable intersubject variability in each genital measure, all three measures were found to have some intrasubject consistency over sessions with respect to either their response amplitudes or patterns, with labial temperature being the most consistent on both parameters. The relationship between the response patterns of the three measures during the film was also relatively consistent across sessions, as was the correspondence between subjective ratings of arousal and both vaginal pressure pulse and labial responses but not vaginal blood volume response. To overcome the problem of considerable intrasubject variability of response amplitudes, it was suggested that the inclusion in the data analysis of several parameters of response patterns, which were relatively stable over sessions, might facilitate the evaluation of a treatment.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Comparison of observational recordings in vivo, via mirror, and via television

Simultaneous observational recordings were made in vivo, via an observation mirror, and via closed circuit television. Three of nine observers had extensive experience recording behavior in vivo; three had extensive experience recording behavior via mirror; and three had extensive experience recording via television. Observers recorded nine categories of disruptive behavior for children in a special class setting. Frequencies of behavior recorded in vivo, via mirror, and via television differed significantly for only one category, vocalization. There were no significant main effects or interactions involving the observers' previous experience. Occurrence reliability coefficients computed within and between media demonstrated the similarity of observer agreement in all three media. Data collection procedures using an observational mirror or closed circuit television appear to be reasonable alternatives to in vivo observation in circumstances similar to those in the present study.