Interobserver Agreement in Behavioral Research: Importance and Calculation

Behavioral researchers have developed a sophisticated methodology to evaluate behavioral change which is dependent upon accurate measurement of behavior. Direct observation of behavior has traditionally been the mainstay of behavioral measurement. Consequently, researchers must attend to the psychometric properties, such as interobserver agreement, of observational measures to ensure reliable and valid measurement. Of the many indices of interobserver agreement, percentage of agreement is the most popular. Its use persists despite repeated admonitions and empirical evidence indicating that it is not the most psychometrically sound statistic to determine interobserver agreement due to its inability to take chance into account. Cohen's (1960) kappa has long been proposed as the more psychometrically sound statistic for assessing interobserver agreement. Kappa is described and computational methods are presented.     

Effects of two levels of procedural fidelity with constant time delay on children's learning

The purpose of this study was to compare the effects of constant time delay delivered with high procedural fidelity to constant time delay with high procedural fidelity on all variables except delivery of the controlling prompt (i.e., on a mean of 44% of the trials, the controlling prompt was not delivered when it should have been provided). Six preschool children with disabilities were taught to identify photographs in two alternating conditions (e.g., high procedural fidelity and low procedural fidelity). An adapted alternating treatments design was used to evaluate the instructional conditions on the effectiveness and efficiency of instruction. In addition, daily measures were taken of the teacher's implementation of each step of the constant time delay procedures which indicated that the two conditions were implemented as planned. The results indicate that both conditions were effective for four children; for three of these, the high procedural fidelity condition resulted in more efficient learning. For the fifth child, the high-fidelity condition resulted in criterion level responding, but the low fidelity condition did not. However, when the high fidelity procedure and trial-by-trial reinforcement were used for the low-fidelity stimuli, these also were acquired. For the sixth child, neither procedure was effective; thus, the high fidelity condition was used alone and resulted in learning. The results are discussed in terms of using the constant time delay procedure and studying the procedural fidelity of other strategies.     

The effect of witnessing consequences on the behavioral recordings of experimental observers

The cueing effects of interviewer praise contingent on a target behavior and expectation of behavior change were examined with six observers. Experiment I investigated the effect of cues in conjunction with expectation. Experiment II assessed the relative contributions of cues and expectation, and Experiment III examined the effect of cues in the absence of expectation. The frequencies of two behaviors, client eye contact and face touching, were held constant throughout a series of videotaped interviews between an "interviewer" and a "client". A within-subjects design was used in each experiment. During baseline conditions, praise did not follow eye contact by the client on the videotape. In all experimental conditions, praise statements from the interviewer followed each occurrence of eye contact with an equal number of praises delivered at random times when there was no eye contact. Three of the six observers dramatically increased their recordings of eye contact during the first experimental phase, but these increases were not replicated in a second praise condition. There were no systematic changes in recorded face touching. Witnessing the delivery of consequences, rather than expectation seemed to be responsible for the effect. This potential threat to the internal validity of studies using observational data may go undetected by interobserver agreement checks.     

Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment

The percentage agreement index has been and continues to be a popular measure of interobserver reliability in applied behavior analysis and child development, as well as in other fields in which behavioral observation techniques are used. An algebraic method and a linear programming method were used to assess chance-corrected reliabilities for a sample of past observations in which the percentage agreement index was used. The results indicated that, had kappa been used instead of percentage agreement, between one-fourth and three-fourth of the reported observations could be judged as unreliable against a lenient criterion and between one-half and three-fourths could be judged as unreliable against a more stringent criterion. It is suggested that the continued use of the percentage agreement index has seriously undermined the reliabilities of past observations and can no longer be justified in future studies.     

Alternate methods and software for calculating interobserver agreement for continuous observation data

Portable electronic data collection devices permit investigators to collect large amounts of observational data in a form ready for computer analysis. These devices are particularly efficient for gathering continuous data on multiple behavior categories. We expect that the increasing availability of these devices will lead to greater use of continuous data collection methods in observational research. This paper addresses the difficulties encountered when calculating traditional interobserver agreement statistics for continuous, multiple-code scoring. Two alternative strategies are described that yield interobserver agreement values based on the exact time of behavior code entries by the primary and secondary observers.Work on this paper was supported in part by NICHD Grants P01HD15051 and R01HD17650 and Office of Special Education and Rehabilitation Services Grant G008302980.     

Effects of video modeling on treatment integrity of behavioral interventions

We examined the effects of individualized video modeling on the accurate implementation of behavioral interventions using a multiple baseline design across 3 teachers. During video modeling, treatment integrity improved above baseline levels; however, teacher performance remained variable. The addition of verbal performance feedback increased treatment integrity to 100% for all participants, and performance was maintained 1 week later. Teachers found video modeling to be more socially acceptable with performance feedback than alone, but rated both positively.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Measures of interobserver agreement: Calculation formulas and distribution effects

Seventeen measures of association for observer reliability (interobserver agreement) are reviewed and computational formulas are given in a common notational system. An empirical comparison of 10 of these measures is made over a range of potential reliability check results. The effects on percentage and correlational measures of occurrence frequency, error frequency, and error distribution are examined. The question of which is the best measure of interobserver agreement is discussed in terms of critical issues to be considered     

Rorschach interrater agreement estimates: An empirical evaluation

A standardized estimation of Rorschach interrater agreement is needed. Percentage agreement, although widely used, is found to be unsuitable. Forty-one protocols from adults in both a normal and a psychiatric sample were scored by two or three scorers, making 85 scoring pairs. Percentage agreement, correlations (phi and Pearson's r ), and kappa were computed on single response, total score, and category level. Percentage agreement shows minimal variation. Even when exceeding 0.80, it can obscure major disagreements. Kappa and correlations both vary in a similar way with level of disagreement. Total score level does not give additional information compared to single score and category levels. Kappa proved to be conservative and reliable and is therefore suggested as a standard estimate.     

A parametric analysis of errors of commission during discrete-trial training

We investigated the effects of systematic changes in levels of treatment integrity by altering errors of commission during error-correction procedures as part of discrete-trial training. We taught 3 students with autism receptive nonsense shapes under 3 treatment integrity conditions (0%, 50%, or 100% errors of commission). Participants exhibited higher levels of performance during perfect implementation (0% errors). For 2 of the 3 participants, performance was low and showed no differentiation in the remaining conditions. Findings suggest that 50% commission errors may be as detrimental as 100% commission errors on teaching outcomes.     

Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions

Assessing treatment fidelity is a core methodological consideration in the study of treatment outcome; it influences both the degree to which changes can be attributed to the intervention and the ability to replicate and disseminate the intervention. Efforts to increase access to evidence-based psychological treatments are receiving unprecedented support; but pressures exist to adapt treatments to service settings, running the risk of compromising fidelity. However, little evidence is available to inform the necessary conditions for the transportation of interventions to service provision settings, and the degree to which fidelity is even evaluated or emphasized in dissemination and implementation programs varies dramatically. Moreover, adaptation is associated with several benefits for dissemination efforts and may address relevant barriers to adoption. A particularly promising strategy for maximizing the benefits of both fidelity and adaptation is the use of transdiagnostic interventions. Such treatments allow for greater flexibility of the pacing and content of treatment, while still providing structure to facilitate testing and replication. Preliminary evidence supports the efficacy of this strategy, which may be particularly conducive to dissemination into service provision settings. At this time, further research is needed to evaluate the relationships among fidelity, adaptation, and outcome, and to determine the potential for transdiagnostic treatments to facilitate dissemination.     

A method for combining occurrence and nonoccurrence interobserver agreement scores

Various statistics have been proposed as standard methods for calculating and reporting interobserver agreement scores. The advantages and disadvantages of each have been discussed in this journal recently but without resolution. A formula is presented that combines separate measures of occurrence and nonoccurrence percentages of agreement, with weight assigned to each measure, varying according to the observed rate of behavior. This formula, which is a modification of a formula proposed by Clement (1976), appears to reduce distortions due to "chance" agreement encountered with very high or low observed rates of behavior while maintaining the mathematical and conceptual simplicity of the conventional method for calculating occurrence and nonoccurrence agreement.     

An analysis of instructor- and tablet-presented conditional discriminations: Fidelity and rapidity

Matching-to-sample arrangements are commonly used to teach conditional discriminations. In these arrangements, instructors must systematically arrange instruction to ensure that a learner's response comes under the intended sources of stimulus control. Given the multitude of instructional considerations, the instructors' procedural fidelity has been a significant concern. Recently, LeBlanc et al. found that brief training and access to enhanced data sheets produced high levels of fidelity with experienced service providers. The current study extended LeBlanc et al. by examining the effects of a similar training on the fidelity and instructional pacing by participants with and without previous experience. The participants' performance was also compared when using a flashcard or binder (i.e., printed) arrays and relative to a tablet-delivered instructional program. High levels of fidelity were observed following training, although pacing was slow. Slight differences in performance were observed across comparison arrays; nevertheless, the tablet-based program outperformed instructors.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Reliability in the context of the experiment: A commentary on two articles by Birkimer and Brown

Two sources of variability must each be considered when examining change in level between two sets of data obtained by human observers; namely, variance within data sets (phases) and variability attributed to each data point (reliability). Birkimer and Brown (1979a, 1979b) have suggested that both chance levels and disagreement bands be considered in examining observer reliability and have made both methods more accessible to researchers. By clarifying and extending Birkimer and Brown's papers, a system is developed using observer agreement to determine the data point variability and thus to check the adequacy of obtained data within the experimental context.     

Moral disagreement and procedural justice: Moral mandates as constraints to voice effects

Procedural voice is a widely used and effective means to reduce or eliminate conflict. Moral disagreements, however, are particularly inflammatory, divisive, and difficult to manage. The current article reports two studies that demonstrated the unique challenge that moral disagreements pose. Specifically, the studies tested the extent to which procedural voice affected justice judgements, group climate, and decision acceptance when people perceived decisions to have moral implications. Results indicated that when people's outcome preferences represent strong moral convictions, outcomes were the primary determinant of perceived fairness and related judgements, irrespective of whether people had voice in the decision‐making process.     

团队权力分布差异对团队冲突的影响：程序公平和合法性的作用

近年来团队权力分布差异与团队冲突间关系受到了学界较大关注, 但相关研究结果仍存在分歧。通过对70个工作团队的调查, 探讨了团队权力分布差异对团队冲突(任务冲突、关系冲突)作用的边界条件。结果发现, 程序公平调节了两者间关系：当程序公平较高时, 团队权力分布差异与任务冲突、关系冲突负相关; 当程序公平较低时, 团队权力分布差异与任务冲突、关系冲突正相关。此外, 被中介的调节模型分析显示, 团队合法性感知中介了程序公平的上述调节作用。     

Police,courts, and corrections: Experiences of procedural injustice among Black adults

Racial disparities and a corresponding lack of trust have been documented within the criminal legal system. In response, criminal legal system actors have sought to strengthen the legitimacy of their agencies. However, legitimizing these agencies can be problematic. Some argue that the current criminal legal system continues the legacy of slavery and Jim Crow as Blacks are disproportionately policed and incarcerated. As a framework, procedural injustice can offer a unique backdrop and interrogate ways in which the criminal legal system engages in delegitimizing actions that provoke noncompliance and enable social control. Using a procedural injustice lens, this study examines how justice-involved Black adults experience mistreatment by justice system actors. Semistructured interviews were conducted with 84 Black adults in Newark and Cleveland. Study findings offer a comprehensive account of how participants experience procedural injustice as arrestees, defendants, and incarcerated persons. More specifically, participant narratives describe deliberately antagonistic, abusive, and dehumanizing treatment by justice-system agents—often depicted as racially motivated. Participant accounts also describe this mistreatment as occurring in a context of coercion and powerlessness and as being institutionally sanctioned. Implications for the preservation of racial hierarchies, research, practice, and community psychology are discussed.