Sample size determinations for the two rater kappa statistic

This paper gives a method for determining a sample size that will achieve a prespecified bound on confidence interval width for the interrater agreement measure,. The same results can be used when a prespecified power is desired for testing hypotheses about the value of kappa. An example from the literature is used to illustrate the methods proposed here.     

Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.     

A generalized family of coefficients of relational agreement for numerical scales

A family of coefficients of relational agreement for numerical scales is proposed. The theory is a generalization to multiple judges of the Zegers and ten Berge theory of association coefficients for two variables and is based on the premise that the choice of a coefficient depends on the scale type of the variables, defined by the class of admissible transformations. Coefficients of relational agreement that denote agreement with respect to empirically meaningful relationships are derived for absolute, ratio, interval, and additive scales. The proposed theory is compared to intraclass correlation, and it is shown that the coefficient of additivity is identical to one measure of intraclass correlation.The author thanks the Editor and anonymous reviewers for helpful suggestions.     

Teachers' and mothers' assessment of children's behaviors from kindergarten to grade two: Stability and change within and across informants

Consistency between teachers' and parents' ratings of 229 children's prosocial and problem behaviors was investigated from kindergarten through grade two. Intrarater stability was also analyzed. Results indicated moderate but decreasing interrater agreement from kindergarten to grade two for boys on externalizing behaviors. Interrater agreement for the same behavior dimension for girls was moderate and stable, whereas it was low on internalizing behavior problems and prosocial behaviors, for both genders throughout the 3-year period. Moreover, difference between teachers' and parents' ratings of externalizing behaviors increased from kindergarten to grade two. Decreasing level of agreement through time appeared to be related to a reduction of teacher-rated externalizing behaviors not paralleled by a reduction in parent ratings. Analyses were also performed at the item level of the externalization scale to identify which specific behaviors were responsible for the increasing divergence in interrater agreement from kindergarten through grade two. Alternative interpretations were considered in order to account for present and past results. Implications for screening and assessment purposes were discussed.This research was supported by grants from the Social Sciences and Humanities Research Council of Canada and from Québec's Health Research Fund (FRSQ). We wish to thank the authorities and directors of the schools from the Val d'Or School Board as well as the kindergarten, first and second grade teachers, children, and parents for their collaboration. We also thank Lyse Desmarais-Gervais, Daniel Corriveau, Jean Desrosiers and Hélène Boileau for their participation in the data collection or analysis.     

Assessment of incarcerated juvenile delinquents: Agreement across raters and approaches to psychopathology

The purpose of the current study was to examine the agreement across three informants (youth, teacher, caretaker) when rating behaviors of incarcerated juvenile delinquents. Furthermore, the agreement between a dimensional approach (Child Behavior Checklist) and a categorical approach (DSM-III-R) of assessing behavior was examined. Fifty-two delinquent youth were assessed with a structured interview. In addition, the youth, teacher, and caretaker completed the Child Behavior Checklist (CBCL). The results indicated significant correlations across the teacher and caretaker for externalizing types of problems. The youth report form of the CBCL did not correspond to either adult informant. However, the ratings by the teacher and caretaker were related to externalizing types of problems, primarily hyperactivity/inattention, derived from the structured interview with the delinquent. The results suggest that, in the psychological assessment of incarcerated juvenile delinquents, agreement across informants is dependent on multiple factors, including type of behavior assessed and the approach utilized in assessment.This work was supported, in part, by the Juvenile Justice Coordinating Council of Georgia, the Office of Juvenile Justice and Delinquency Prevention, and the University of Georgia's Institute for Behavioral Research.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Frequency,complexity, and clarity as covariates of observer reliability

Factors affecting interobserver agreement (reliability) with a comprehensive coding system in the naturalistic observation of children were examined. Data from 117 pairs of observations on 35 children and their families were examined with respect to reliability and three possible covariates: response frequency, observation complexity, and code definition clarity. Analysis of results strongly supported response frequency as a positive covariate of interobserver agreement. Complexity was found to negatively covary with interobserver agreement. The relationship between code clarity and reliability was in the predicted direction but failed to obtain statistical significance. Implications for observer training and data collection in observational studies are discussed.An earlier version of this article was prepared for presentation at The Association for the Advancement of Behavior Therapy, Atlanta, Georgia, December 1977.     

Time counts: Bidirectional interaction between time and numbers in human adults

Number is known for influencing time processing, but to what extent time influences number in human adults is unclear. We investigated possible bidirectional interactions (number on time and time on number) using a novel Stroop-like task; participants compared numbers or temporal durations in congruent (larger number presented for longer duration) or incongruent conditions (smaller number presented for longer duration). Time and number tasks were presented in different blocks (Experiment 1) or within the same block of trials with task instructions provided at the offset of the stimuli (Experiment 2). Analyses of response times (RTs) and their distribution revealed that number affected time from early RTs, and time affected number at late RTs – an asymmetry observed only when time and number tasks were presented in separate blocks. Thus, carefully chosen tasks and appropriate data analysis can reveal bidirectionality between time and number, consistent with shared magnitude or decision mechanisms.     

Tapping into the interplay of lexical and number knowledge using fast mapping: A longitudinal eye-tracking study with two-year-olds

Language skills and mathematical competencies are argued to influence each other during development. While a relation between the development of vocabulary size and mathematical skills is already documented in the literature, this study further examines how children’s ability to map a novel word to an unknown object as well as their ability to retain this word from memory may be related to their knowledge of number words. Twenty-five children were tested longitudinally (at 30 and at 36 months of age) using an eye-tracking-based fast mapping task, the Give-a-Number task, and standardized measures of vocabulary. The results reveal that children’s ability to create and retain a mental representation of a novel word was related to number knowledge at 30 months, but not at 36 months while vocabulary size correlated with number knowledge only at 36 months. These results show that even specific mapping processes are initially related to the acquisition of number words and they speak for a parallelism between the development of lexical and number-concept knowledge despite their semantic and syntactic differences.