How well do ratings of academic performance by mothers and their sons correspond to grades,achievement test scores,and teachers' ratings?

Accurate perceptions of their child's academic performance are pivotal for parents' encouragement of their child's academic efforts. In research studies, parents' reports of their child's academic achievement, as well as the child's own reports, are often used as a proxy for teachers' ratings or grades. However, are parents' reports or their child's reports accurate and unbiased? We investigated whether ratings of academic performance made by mothers and their sons corresponded more closely to final grades and achievement test scores than did similar ratings made by teachers for three large samples of boys. Mothers' ratings correlated with final grades as well or better than teachers' ratings but more poorly with achievement test scores than did teachers' ratings. The seventh-grade boys' ratings correlated more poorly with both final grades and achievement test scores than did teachers' or mothers' ratings. Across all three samples combined, mothers were more likely to over-estimate rather than under-estimate the final grade. The same was true of the seventh grade sample of boys. Once the effect of final grade had been controlled, little evidence of a consistent pattern of associations between maternal demographic, parenting, and child behavior variables and either over- or underestimation was found.     

TOWARD THE DEVELOPMENT OF STRUCTURED CRITERIA FOR INTERPRETATION OF FUNCTIONAL ANALYSIS DATA

Using functional analysis results to prescribe treatments is the preferred method for developing behavioral interventions. Little is known, however, about the reliability and validity of visual inspection for the interpretation of functional analysis data. The purpose of this investigation was to develop a set of structured criteria for visual inspection of multielement functional analyses that, when applied correctly, would increase interrater agreement and agreement with interpretations reached by expert consensus. In Study 1, 3 predoctoral interns interpreted functional analysis graphs, and interrater agreement was low (M = .46). In Study 2, 64 functional analysis graphs were interpreted by a panel of experts, and then a set of structured criteria were developed that yielded interpretive results similar to those of the panel (exact agreement = .94). In Study 3, the 3 predoctoral interns from Study 1 were trained to use the structured criteria, and the mean interrater agreement coefficient increased to .81. The results suggest that (a) the interpretation of functional analysis data may be less reliable than is generally assumed, (b) decision-making rules used by experts in the interpretation of functional analysis data can be operationalized, and (c) individuals can be trained to apply these rules accurately to increase interrater agreement. Potential uses of the criteria are discussed.     

A decision tree approach to selecting an appropriate observation reliability index

Based on the conceptual framework outlined by Cone (1986) and Suen (1988), a practical decision tree is developed as an aid for the selection of observational reliability indices.     

Some empirical justification for a modest proposal regarding data acquisition via intermittent direct observation

Computer simulations of behavior and event records of a child's visual fixation on an adult person were generated and sampled using a modified time sampling procedure. The modified approach required making two independent scoring judgments per observation interval: if the behavior was initiated within the interval, the interval was scored; if the behavior was ongoing precisely at the end of the interval, the interval was again scored. The results showed that the within-interval and end-of-interval scores yielded, respectively, estimates of behavioral frequency and duration. The accuracy of the estimates was a function of the number of observations. Several field studies demonstrated high agreement among observers who employed the measurement system.     

Shaping data collection congruent with experimental hypotheses

The influence of instructions and feedback from an experimenter on observational recordings of disruptive behavior was evaluated. Four subject-observers recorded four categories of disruptive behavior from videotapes of children in a classroom setting. Two sets of videotapes, labelled "baseline" and "treatment", were matched for rates of disruptive behavior in each category. The observers were told that two target behaviors were expected to decrease and the other two control behaviors were not expected to change during the treatment condition. During observational recording of treatment videotapes, the observers were given positive feedback when they reported decreases in the two target behaviors, and negative feedback when they reported either to change or increases in the two control behaviors. The target behaviors were recorded less frequently by observers, while recordings of control behaviors were unaffected during the treatment phase. These results suggest that contingent feedback to observers should be avoided as a possible source of bias in behavioral recordings.     

Multidimensional rotation and scaling of configurations to optimal agreement

An integrated method for rotating and rescaling a set of configurations to optimal agreement in subspaces of varying dimensionalities is developed. The approach relates existing orthogonal rotation techniques as special cases within a general framework based on a partition of variation which provides convenient measures of agreement. In addition to the well-known Procrustes and inner product optimality criteria, a criterion which maximizes the consensus among subspaces of the configurations is suggested. Since agreement of subspaces of the configurations can be examined and compared, rotation and rescaling is extended from a data transformation technique to an analytical method.     

A cautionary note on the use of probability values to evaluate interobserver agreement

Proposed methods of assessing the statistical significance of interobserver agreements provide erroneous probability values when conducted on serially correlated data. Investigators who wish to evaluate interobserver agreements by means of statistical significance can do so by limiting the analysis to every k(th) interval of data, or by using Markovian techniques which accommodate serial correlations.     

Measures of observer agreement when binomial data are collected in free operant situations

This paper examines a model and defines reasonable assumptions underlying different measures of observer agreement for categorical data collected in free operant situations. It is assumed that two or more observers classify operant behaviors of subjects into occurrences and nonoccurrences by recognition by validated response classes (categories) such that the rates of false positives and observer biases are acceptably low. Thus errors are mostly omissions, i.e., failing to observe events that occur. Four alternative cases are derived, together with formulas for calculating significance tests, variances, and standard errors, three of which do not depend on knowledge of the proportion of time points at which the event does not occur.We wish to acknowledge NICHD Grant HD-10570, The Neuropharmacology of Developmental Disorders, George Breese, Ph.D., and C. T. Gualtieri, M.D., Principal Investigators; NIEHS Grant ES-01104; USPHS Grant HD-03110; and MCH Project 916 to the Division for Disorders of Development and Learning.     

Evaluation of self-contained training procedures for the Time-Sample Behavioral Checklist

The comparative effectiveness of two time-limited modes of training observers to code the behavior of clients in residential treatment programs on the Time-Sample Behavioral Checklist (TSBC) was evaluated. The susceptibility of training procedures to consensual observer drift and the predictability of TSBC mastery from trainee characteristics were also examined. Two equated groups of undergraduate students (N=15 each) participated in full-time training on the TSBC and another instrument for 27 days, followed by criterion testingin vivo and on videotapes. One group was trained by experienced personnel using procedures known to be effective but potentially subject to consensual observer drift. The other group was trained using a previously untested set of written and videotape procedures that do not rely on experienced personnel. Comparative effectiveness and observer drift were evaluated by multivariate and univariate ANOVAs on mastery scores reflecting both pattern agreement and level differences between each trainee and criterion codings. The new, more efficient training procedures were found to be as effective as the original training procedures in the degree of mastery achieved by trainees. Original training procedures were found to be resistant to consensual observer drift, with such a phenomenon appearing in only 1 instance of 156 opportunities. The high degree of mastery achieved by trainees during the time-limited training period was comparable to that previously achieved with the original training procedures. No meaningful predictions of coding mastery were found, with only one trainee characteristic being significantly related to one of six mastery criteria. The results document procedures that are both efficient and resistant to invalidity for training observers in the use of multidimensional observational systems, as well as providing guidelines for the development of standardized procedures.This article is based on a thesis submitted to the Graduate College of the University of Illinois at Urbana-Champaign in partial fulfillment of the requirements for the Ph.D. degree in psychology by the first author and under direction of the second author. The third and fourth authors also participated as supervisors. Appreciation is extended to other members of the thesis committee, Fred Kanfer, Ross Parke, and Julian Rappaport, and to John Gottman and W. Robert Nay for their comments and recommendations. This study was partially supported by Public Health Service Grants MH-25464 and MH-14257 from the National Institute of Mental Health and by grants from the Joyce Foundation, the MacArthur Foundation, and the Illinois Department of Mental Health and Developmental Disabilities.     

The comparative effectiveness of two modes of observer training on the Staff-Resident Interaction Chronograph

The comparative effectiveness of two time-limited modes of training observers to code activity on the Staff-Resident Interaction Chronograph (SRIC) in residential treatment programs for mentally disabled adults was evaluated. The susceptibility of training procedures for consensual observer drift was also examined, as was the predictability of SRIC mastery from trainee characteristics. Two equated groups of undergraduate student trainees (N=15 each) participated in full-time training for 27 days, followed by two weeks of criterion testing in vivo and on videotapes. One group received training by experience personnel using procedures known to be effective (original method). The other group received training via a previously untested set of written and videotaped procedures that do not rely on experienced personnel (package method). Multivariate and univariate analyses of variance found both methods to be equally effective in the degree of mastery achieved by trainees, without evidence of observer drift. No meaningful predictions of coding mastery were found, but conceptual mastery was predictable from individual characteristics. Differences were obtained for both groups between in vivo versus videotaped criterion tests. The results document procedures that are both efficient and resistant to invalidity for complex observational methodology as well as feasible for standardizing assessment of staff functioning across residential settings.This article is based on a thesis submitted to the Graduate College of the University of Illinois at Urbana-Champaign in partial fulfillment of the requirements for the Ph.D. degree in psychology by the first author under direction of the second author. The third and fourth authors also participated as supervisors. Appreciation is extended to other members of the thesis committee, Fred Kanfer, W. Robert Nay, Julian Rappaport, and James Wardrop, for their comments and recommendations. This study was partially supported by Public Health Service Grants MH-25464 and MH-14257 from the National Institute of Mental Health, and by grants from The Joyce Foundation and the Illinois Department of Mental Health and Developmental Disabilities.