Interrater reliability of teacher rating scales for children with attention-deficit hyperactivity disorder

Interrater reliability of eight teacher rating scales designed to assess characteristics of attention-deficit hyperactivity disorder was investigated. Coteachers of 46 students completed the rating scales. The students, ages 8–17, were designated as having a Serious Emotional Disturbance. The resulting interrater reliability correlation coefficients ranged from .62 to .87. The percentage of variance shared between raters ranged from a low of 38.4% (the ACTeRS Oppositional factor and the CBCL-TRF Attention Problems factor) to 75.7% (ADHD Rating Scale). The percent of shared variance was higher for younger children. Kappa scores evaluating rater agreement were highest at the two standard deviations above the mean cutoff. The reliability coefficients were consistent with those reported in prior research.     

上海地区青少年道德判断能力测验的编制及常模制定

上海地区青少年道德判断能力测验量表共有四个道德行为故事,分别反映自由与自律、公与私、义与利、平等与公正四对道德冲突中的一对矛盾。常模样本系随机选取上海市区、郊区四所完全中学,每校从初一至高三年级各一个班共1324名学生。常模形式包括总分常模、P分常模和道德判断能力水平等级划分三种。其信度和效度均令人满意。本量表适用于12－17岁青少年学生。     

Models for measurement,precision, and the nondichotomization of graded responses

It is common in educational, psychological, and social measurement in general, to collect data in the form of graded responses and then to combine adjacent categories. It has been argued that because the division of the continuum into categories is arbitrary, any model used for analyzing graded responses should accommodate such action. Specifically, Jansen and Roskam (1986) enunciate ajoining assumption which specifies that if two categoriesj andk are combined to form categoryh, then the probability of a response inh should equal the sum of the probabilities of responses inj andk. As a result, they question the use of the Rasch model for graded responses which explicitly prohibits the combining of categories after the data are collected except in more or less degenerate cases. However, the Rasch model is derived from requirements of invariance of comparisons of entities with respect to different instruments, which might include different partitions of the continuum, and is consistent with fundamental measurement. Therefore, there is a strong case that the mathematical implication of the Rasch model should be studied further in order to understand how and why it conflicts with the joining assumption. This paper pursues the mathematics of the Rasch model and establishes, through a special case when the sizes of the categories are equal and when the model is expressed in the multiplicative metric, that its probability distribution reflects the precision with which the data are collected, and that if a pair of categories is collapsed after the data are collected, it no longer reflects the original precision. As a consequence, and not because of a qualitative change in the variable, the joining assumption is destroyed when categories are combined. Implications of the choice between a model which satisfies the joining assumption or one which reflects on the precision of the data collection considered are discussed.     

TOWARD THE DEVELOPMENT OF STRUCTURED CRITERIA FOR INTERPRETATION OF FUNCTIONAL ANALYSIS DATA

Using functional analysis results to prescribe treatments is the preferred method for developing behavioral interventions. Little is known, however, about the reliability and validity of visual inspection for the interpretation of functional analysis data. The purpose of this investigation was to develop a set of structured criteria for visual inspection of multielement functional analyses that, when applied correctly, would increase interrater agreement and agreement with interpretations reached by expert consensus. In Study 1, 3 predoctoral interns interpreted functional analysis graphs, and interrater agreement was low (M = .46). In Study 2, 64 functional analysis graphs were interpreted by a panel of experts, and then a set of structured criteria were developed that yielded interpretive results similar to those of the panel (exact agreement = .94). In Study 3, the 3 predoctoral interns from Study 1 were trained to use the structured criteria, and the mean interrater agreement coefficient increased to .81. The results suggest that (a) the interpretation of functional analysis data may be less reliable than is generally assumed, (b) decision-making rules used by experts in the interpretation of functional analysis data can be operationalized, and (c) individuals can be trained to apply these rules accurately to increase interrater agreement. Potential uses of the criteria are discussed.     

Norms for Parental Ratings on Conners' Abbreviated Parent-Teacher Questionnaire: Implications for the Design of Behavioral Rating Inventories and Analyses of Data Derived from Them

Despite the wide usage of Conners' 10-item Abbreviated Parent-Teacher Questionnaire (APTQ), normative data for parental ratings on the APTQ are rare. Moreover, its psychometric properties as a stand alone behavioral rating instrument have yet to be examined in detail. Using data from parental ratings for 6,841 children aged 5 to 14 years, this paper reports the psychometric properties of the APTQ and provides normative and prevalence estimates for four age cohorts (5 to 6, 7 to 8, 9 to 11, and 12 to 14 years) by gender. In presenting the findings, the paper highlights key methodological issues endemic to the design of behavioral rating inventories like the APTQ and the analysis of data derived from them. These issues are illustrated and discussed in terms of their implications for the measurement and determination of prevalence estimates of problem behaviors in child psychology and child psychiatry. The need to revise the design, content, and response formats of child behavior rating inventories is stressed.     

A decision tree approach to selecting an appropriate observation reliability index

Based on the conceptual framework outlined by Cone (1986) and Suen (1988), a practical decision tree is developed as an aid for the selection of observational reliability indices.     

Some empirical justification for a modest proposal regarding data acquisition via intermittent direct observation

Computer simulations of behavior and event records of a child's visual fixation on an adult person were generated and sampled using a modified time sampling procedure. The modified approach required making two independent scoring judgments per observation interval: if the behavior was initiated within the interval, the interval was scored; if the behavior was ongoing precisely at the end of the interval, the interval was again scored. The results showed that the within-interval and end-of-interval scores yielded, respectively, estimates of behavioral frequency and duration. The accuracy of the estimates was a function of the number of observations. Several field studies demonstrated high agreement among observers who employed the measurement system.     

Multidimensional rotation and scaling of configurations to optimal agreement

An integrated method for rotating and rescaling a set of configurations to optimal agreement in subspaces of varying dimensionalities is developed. The approach relates existing orthogonal rotation techniques as special cases within a general framework based on a partition of variation which provides convenient measures of agreement. In addition to the well-known Procrustes and inner product optimality criteria, a criterion which maximizes the consensus among subspaces of the configurations is suggested. Since agreement of subspaces of the configurations can be examined and compared, rotation and rescaling is extended from a data transformation technique to an analytical method.     

Parent-perceived child behavior problems: Comparison of normative and clinical factors of the Adelaide-Conners Parent Rating Scale

The Adelaide-Conners Parent Rating Scale (APRS), an instrument developed by studying a large, representative group of schoolchildren, was used with a group of psychiatry attenders. Multimethod factor analysis found satisfactory agreement between the factor structures of the clinical and the normative groups. The patterns of scores on the 12 APRS scales were also compared. Two higher-order factors (Conflict with the Environment and Conflict within the Self) were identified in the clinical sample as previously found in the normative group. Comparison of the factor solutions with previous empirical efforts to identify parent-perceived patterns of child behavior disorder showed that the APRS compares well with other instruments and supports the strategy of proceeding from the study of normative populations to the study of clinically defined groups.This work was supported in part by a grant from the Department of Health, Canberra.