Interobserver Agreement in Behavioral Research: Importance and Calculation

Behavioral researchers have developed a sophisticated methodology to evaluate behavioral change which is dependent upon accurate measurement of behavior. Direct observation of behavior has traditionally been the mainstay of behavioral measurement. Consequently, researchers must attend to the psychometric properties, such as interobserver agreement, of observational measures to ensure reliable and valid measurement. Of the many indices of interobserver agreement, percentage of agreement is the most popular. Its use persists despite repeated admonitions and empirical evidence indicating that it is not the most psychometrically sound statistic to determine interobserver agreement due to its inability to take chance into account. Cohen's (1960) kappa has long been proposed as the more psychometrically sound statistic for assessing interobserver agreement. Kappa is described and computational methods are presented.     

An exploration of the interrater agreement of visual analysis with and without context

Visual analysis is integral to the analysis of single-case experimental design (SCED) data. Previous studies have shown that many factors may influence the interrater agreement (IRA) of visual analysis. One factor that has received little direct attention is the impact of contextual information. In the current study, authors of recently published SCED studies were asked to make judgments regarding functional relations based on published datasets that met criteria for design quality. Respondents were randomly assigned to view graphs with or without contextual information and the degree of interrater agreement was compared. Results revealed that contextual information had no impact on IRA for decisions of a functional relation. IRA was high across both groups for 6 of the 7 datasets examined. Implications and recommendations based on these results are discussed.     

Random effects diagonal metric multidimensional scaling models

By assuming a distribution for the subject weights in a diagonal metric (INDSCAL) multidimensional scaling model, the subject weights become random effects. Including random effects in multidimensional scaling models offers several advantages over traditional diagonal metric models such as those fitted by the INDSCAL, ALSCAL, and other multidimensional scaling programs. Unlike traditional models, the number of parameters does not increase with the number of subjects, and, because the distribution of the subject weights is modeled, the construction of linear models of the subject weights and the testing of those models is immediate. Here we define a random effects diagonal metric multidimensional scaling model, give computational algorithms, describe our experiences with these algorithms, and provide an example illustrating the use of the model and algorithms.We would like to thank J. Douglas Carroll for early consultation of this research, and Robert I. Jennrich for commenting on an earlier draft of this paper and for help on the computational algorithms. James O. Ramsay and Forrest W. Young were instrumental in providing the example data. This work was supported in part by National Institute of Mental Health grant 1 R43 MH57559-01. We would also like to thank the anonymous referees for comments that helped to clarify our work.     

A Polychoric Instrumental Variable (PIV) Estimator for Structural Equation Models with Categorical Variables

This paper presents a new polychoric instrumental variable (PIV) estimator to use in structural equation models (SEMs) with categorical observed variables. The PIV estimator is a generalization of Bollen’s (Psychometrika 61:109–121, 1996) 2SLS/IV estimator for continuous variables to categorical endogenous variables. We derive the PIV estimator and its asymptotic standard errors for the regression coefficients in the latent variable and measurement models. We also provide an estimator of the variance and covariance parameters of the model, asymptotic standard errors for these, and test statistics of overall model fit. We examine this estimator via an empirical study and also via a small simulation study. Our results illustrate the greater robustness of the PIV estimator to structural misspecifications than the system-wide estimators that are commonly applied in SEMs. Kenneth Bollen gratefully acknowledges support from NSF SES 0617276, NIDA 1-RO1-DA13148-01, and DA013148-05A2. Albert Maydeu-Olivares was supported by the Department of Universities, Research and Information Society (DURSI) of the Catalan Government, and by grant BSO2003-08507 from the Spanish Ministry of Science and Technology. We thank Sharon Christ, John Hipp, and Shawn Bauldry for research assistance. The comments of the members of the Carolina Structural Equation Modeling (CSEM) group are greatly appreciated. An earlier version of this paper under a different title was presented by K. Bollen at the Psychometric Society Meetings, June, 2002, Chapel Hill, North Carolina.     

Family Emotional Climate and Sibling Relationship Quality: Influences on Behavioral Problems and Adaptation in Preschool-Aged Children

We examined the impact of family emotional climate and sibling relationship quality on behavioral problems and adaptation in preschool-aged children. Participants were 63 mothers with a preschool-aged child enrolled in a Southern Arizona Head Start Program. Siblings were identified as children closest in age to target child. Mothers of predominantly Mexican descent (95%) participated in home interviews during the Fall and Spring of the year children entered center-based programs. Sibling relationship quality (warmth, agonism/competition) was proposed to predict children’s adjustment (behavioral problems and adaptation) longitudinally. Results indicate that after controlling for child characteristics (temperament, child gender, birth order) and after accounting for family characteristics (family emotional expressiveness, child exposure to interparental conflict, and parental agreement on childrearing), sibling warmth made a significant and unique contribution to child adjustment as reported by mothers and teachers six months later. Findings are consistent with existing research indicating that sibling relationships impact children’s adjustment and shape young children’s lives in meaningful and marked ways. Moreover, these associations were found with an understudied sample of young children of predominantly Mexican descent in low-income families, and thus make an important contribution to knowledge in the field.     

评分者内部一致性的研究和应用

在行为科学的研究和实践中,研究者常常需要将个体层次的评价,整合到群体层次的评价,对于这种自下而上整合模式的一致性问题,国内一些学者常常错误地使用评分者内部信度作为评分者内部一致性的指标。评分者内部一致性和评分者内部信度不仅在理论基础上存在差异,而且在实践中也存在前者很高(或很低),而后者却很低(或很高)的不一致情况。文章阐述了学术界对评分一致性这一问题的提出、争论和取得一致观点的发展脉络,以期学者们对这一问题能够有深入的思索,避免在今后的研究中出现类似的错误。     

Agreement of adolescent ratings with mother ratings and teacher ratings of ADHD symptom groups: A correlated trait-correlated method minus one analysis

This study examined the level of agreement of adolescent ratings with mother ratings, and adolescent ratings with teacher ratings of the inattention (IA) and hyperactivity/impulsivity (HI) symptom groups of ADHD. A total of 214 adolescents provided self-ratings of IA and HI, and their IA and HI were also rated by their mothers and teachers. The correlated trait-correlated method minus one model was applied, with adolescent ratings as the reference method, and the other two ratings as the non-reference methods. The findings indicated no additional variance in adolescent ratings for IA and HI that could not be accounted by mother ratings of IA and HI, respectively. In contrast, there was additional variance in adolescent ratings for IA and HI that could was not accounted by teacher ratings of IA and HI, respectively. The findings suggest that when diagnosing ADHD in adolescents, their reports of their own ADHD behaviors are not needed when mother reports of such behaviors are used.