Sample size determinations for the two rater kappa statistic

This paper gives a method for determining a sample size that will achieve a prespecified bound on confidence interval width for the interrater agreement measure,. The same results can be used when a prespecified power is desired for testing hypotheses about the value of kappa. An example from the literature is used to illustrate the methods proposed here.     

Ordinal consistency and ordinal true scores

This paper argues that test data are ordinal, that latent trait scores are only determined ordinally, and that test data are used largely for ordinal purposes. Therefore it is desirable to develop a test theory based only on ordinal assumptions. A set of ordinal assumptions is presented, including an ordinal version of local independence. From these assumptions it is first shown that the gamma-correlation between two tests is the product of their gamma-correlations with the true latent order. The theory is generalized to allow for heterogeneous tests by defining a weighted average local independence. The tau-correlations between total score and the latent order can be found in both homogeneous and heterogeneous cases, and a system of differential item weighting to maximize the tau-correlation between weighted items and the latent order is provided. Thus a purely ordinal test theory seems possible.Part of this work was done while the author was a Visiting Fellow at Macquarrie University. The paper has benefitted from discussions with Professors Thomas J. Reynolds and Roderick P. McDonald and from the comments of several anonymous reviewers.     

Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.     

A generalized family of coefficients of relational agreement for numerical scales

A family of coefficients of relational agreement for numerical scales is proposed. The theory is a generalization to multiple judges of the Zegers and ten Berge theory of association coefficients for two variables and is based on the premise that the choice of a coefficient depends on the scale type of the variables, defined by the class of admissible transformations. Coefficients of relational agreement that denote agreement with respect to empirically meaningful relationships are derived for absolute, ratio, interval, and additive scales. The proposed theory is compared to intraclass correlation, and it is shown that the coefficient of additivity is identical to one measure of intraclass correlation.The author thanks the Editor and anonymous reviewers for helpful suggestions.     

自由回忆和线索回忆测验中的系列位置效应

本文以96个中国汉字为材料,使用自由回忆和线索回忆测验方法,对20名被试进行6组分测验和最后所有汉字的总测验。发现自由回忆分测验中存在明显的系列位置效应,总测验出现负近因现象;线索回忆分测验也表现出明显的系列位置效应,但最后的总测验没有出现负近因现象。上述结果验证了Bjork和Whitten（1974）提出的顺序或情景理论。     

Isotonic ordinal probabilistic models (ISOP)

The concept of an ordinal instrumental probabilistic comparison is introduced. It relies on an ordinal scale given a priori and on the concept of stochastic dominance. It is used to define a weakly independently ordered system, or isotonic ordinal probabilistic (ISOP) model, which allows the construction of separate sample-free ordinal scales on a set of subjects and a set of items. The ISOP-model is a common nonparametric theoretical structure for unidimensional models for quantitative, ordinal and dichotomous variables.Fundamental theorems on dichotomous and polytomous weakly independently ordered systems are derived. It is shown that the raw score system has the same formal properties as the latent system, and therefore the latter can be tested at the observed empirical level.I wish to thank 3 reviewers and 2 editors who contributed a lot to the readability and precision of the article.     

Teachers' and mothers' assessment of children's behaviors from kindergarten to grade two: Stability and change within and across informants

Consistency between teachers' and parents' ratings of 229 children's prosocial and problem behaviors was investigated from kindergarten through grade two. Intrarater stability was also analyzed. Results indicated moderate but decreasing interrater agreement from kindergarten to grade two for boys on externalizing behaviors. Interrater agreement for the same behavior dimension for girls was moderate and stable, whereas it was low on internalizing behavior problems and prosocial behaviors, for both genders throughout the 3-year period. Moreover, difference between teachers' and parents' ratings of externalizing behaviors increased from kindergarten to grade two. Decreasing level of agreement through time appeared to be related to a reduction of teacher-rated externalizing behaviors not paralleled by a reduction in parent ratings. Analyses were also performed at the item level of the externalization scale to identify which specific behaviors were responsible for the increasing divergence in interrater agreement from kindergarten through grade two. Alternative interpretations were considered in order to account for present and past results. Implications for screening and assessment purposes were discussed.This research was supported by grants from the Social Sciences and Humanities Research Council of Canada and from Québec's Health Research Fund (FRSQ). We wish to thank the authorities and directors of the schools from the Val d'Or School Board as well as the kindergarten, first and second grade teachers, children, and parents for their collaboration. We also thank Lyse Desmarais-Gervais, Daniel Corriveau, Jean Desrosiers and Hélène Boileau for their participation in the data collection or analysis.     

Assessment of incarcerated juvenile delinquents: Agreement across raters and approaches to psychopathology

The purpose of the current study was to examine the agreement across three informants (youth, teacher, caretaker) when rating behaviors of incarcerated juvenile delinquents. Furthermore, the agreement between a dimensional approach (Child Behavior Checklist) and a categorical approach (DSM-III-R) of assessing behavior was examined. Fifty-two delinquent youth were assessed with a structured interview. In addition, the youth, teacher, and caretaker completed the Child Behavior Checklist (CBCL). The results indicated significant correlations across the teacher and caretaker for externalizing types of problems. The youth report form of the CBCL did not correspond to either adult informant. However, the ratings by the teacher and caretaker were related to externalizing types of problems, primarily hyperactivity/inattention, derived from the structured interview with the delinquent. The results suggest that, in the psychological assessment of incarcerated juvenile delinquents, agreement across informants is dependent on multiple factors, including type of behavior assessed and the approach utilized in assessment.This work was supported, in part, by the Juvenile Justice Coordinating Council of Georgia, the Office of Juvenile Justice and Delinquency Prevention, and the University of Georgia's Institute for Behavioral Research.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.