Sample size determinations for the two rater kappa statistic

This paper gives a method for determining a sample size that will achieve a prespecified bound on confidence interval width for the interrater agreement measure,. The same results can be used when a prespecified power is desired for testing hypotheses about the value of kappa. An example from the literature is used to illustrate the methods proposed here.     

Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.     

A generalized family of coefficients of relational agreement for numerical scales

A family of coefficients of relational agreement for numerical scales is proposed. The theory is a generalization to multiple judges of the Zegers and ten Berge theory of association coefficients for two variables and is based on the premise that the choice of a coefficient depends on the scale type of the variables, defined by the class of admissible transformations. Coefficients of relational agreement that denote agreement with respect to empirically meaningful relationships are derived for absolute, ratio, interval, and additive scales. The proposed theory is compared to intraclass correlation, and it is shown that the coefficient of additivity is identical to one measure of intraclass correlation.The author thanks the Editor and anonymous reviewers for helpful suggestions.     

Teachers' and mothers' assessment of children's behaviors from kindergarten to grade two: Stability and change within and across informants

Consistency between teachers' and parents' ratings of 229 children's prosocial and problem behaviors was investigated from kindergarten through grade two. Intrarater stability was also analyzed. Results indicated moderate but decreasing interrater agreement from kindergarten to grade two for boys on externalizing behaviors. Interrater agreement for the same behavior dimension for girls was moderate and stable, whereas it was low on internalizing behavior problems and prosocial behaviors, for both genders throughout the 3-year period. Moreover, difference between teachers' and parents' ratings of externalizing behaviors increased from kindergarten to grade two. Decreasing level of agreement through time appeared to be related to a reduction of teacher-rated externalizing behaviors not paralleled by a reduction in parent ratings. Analyses were also performed at the item level of the externalization scale to identify which specific behaviors were responsible for the increasing divergence in interrater agreement from kindergarten through grade two. Alternative interpretations were considered in order to account for present and past results. Implications for screening and assessment purposes were discussed.This research was supported by grants from the Social Sciences and Humanities Research Council of Canada and from Québec's Health Research Fund (FRSQ). We wish to thank the authorities and directors of the schools from the Val d'Or School Board as well as the kindergarten, first and second grade teachers, children, and parents for their collaboration. We also thank Lyse Desmarais-Gervais, Daniel Corriveau, Jean Desrosiers and Hélène Boileau for their participation in the data collection or analysis.     

Assessment of incarcerated juvenile delinquents: Agreement across raters and approaches to psychopathology

The purpose of the current study was to examine the agreement across three informants (youth, teacher, caretaker) when rating behaviors of incarcerated juvenile delinquents. Furthermore, the agreement between a dimensional approach (Child Behavior Checklist) and a categorical approach (DSM-III-R) of assessing behavior was examined. Fifty-two delinquent youth were assessed with a structured interview. In addition, the youth, teacher, and caretaker completed the Child Behavior Checklist (CBCL). The results indicated significant correlations across the teacher and caretaker for externalizing types of problems. The youth report form of the CBCL did not correspond to either adult informant. However, the ratings by the teacher and caretaker were related to externalizing types of problems, primarily hyperactivity/inattention, derived from the structured interview with the delinquent. The results suggest that, in the psychological assessment of incarcerated juvenile delinquents, agreement across informants is dependent on multiple factors, including type of behavior assessed and the approach utilized in assessment.This work was supported, in part, by the Juvenile Justice Coordinating Council of Georgia, the Office of Juvenile Justice and Delinquency Prevention, and the University of Georgia's Institute for Behavioral Research.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Frequency,complexity, and clarity as covariates of observer reliability

Factors affecting interobserver agreement (reliability) with a comprehensive coding system in the naturalistic observation of children were examined. Data from 117 pairs of observations on 35 children and their families were examined with respect to reliability and three possible covariates: response frequency, observation complexity, and code definition clarity. Analysis of results strongly supported response frequency as a positive covariate of interobserver agreement. Complexity was found to negatively covary with interobserver agreement. The relationship between code clarity and reliability was in the predicted direction but failed to obtain statistical significance. Implications for observer training and data collection in observational studies are discussed.An earlier version of this article was prepared for presentation at The Association for the Advancement of Behavior Therapy, Atlanta, Georgia, December 1977.     

Hubert's multi-rater kappa revisited

There is a frequent need to measure the degree of agreement among R observers who independently classify n subjects within K nominal or ordinal categories. The most popular methods are usually kappa-type measurements. When R = 2, Cohen's kappa coefficient (weighted or not) is well known. When defined in the ordinal case while assuming quadratic weights, Cohen's kappa has the advantage of coinciding with the intraclass and concordance correlation coefficients. When R > 2, there are more discrepancies because the definition of the kappa coefficient depends on how the phrase ‘an agreement has occurred’ is interpreted. In this paper, Hubert's interpretation, that ‘an agreement occurs if and only if all raters agree on the categorization of an object’, is used, which leads to Hubert's (nominal) and Schuster and Smith's (ordinal) kappa coefficients. Formulae for the large-sample variances for the estimators of all these coefficients are given, allowing the latter to illustrate the different ways of carrying out inference and, with the use of simulation, to select the optimal procedure. In addition, it is shown that Schuster and Smith's kappa coefficient coincides with the intraclass and concordance correlation coefficients if the first coefficient is also defined assuming quadratic weights.     

Issues in Parent-Child Agreement: The Case of Structured Diagnostic Interviews

There are three primary purposes of this review. First, the review distinguishes among three types of reliability and describes the importance of evaluating the reliability of child psychopathology assessment instruments for clinical practice and research. Second, parent-child reliability findings from 5 of the more carefully studied and frequently used Structured (semi and highly) diagnostic interviews (The Schedule for Affective Disorders and Schizophrenia for School-age Children, The Child Assessment Scale, The Anxiety Disorders Interview Schedule for Children. The Diagnostic Interview for Children and Adolescents, and the Diagnostic Interview Schedule for Children) are examined. Finally, this review explores factors that have been implicated in terms of their potential effect on parent-child agreement. In addition, future directions for research and clinical practice within this area are identified and potential resolutions to the conundrum of parent-child discordance are discussed.