Interobserver Agreement in Behavioral Research: Importance and Calculation

Behavioral researchers have developed a sophisticated methodology to evaluate behavioral change which is dependent upon accurate measurement of behavior. Direct observation of behavior has traditionally been the mainstay of behavioral measurement. Consequently, researchers must attend to the psychometric properties, such as interobserver agreement, of observational measures to ensure reliable and valid measurement. Of the many indices of interobserver agreement, percentage of agreement is the most popular. Its use persists despite repeated admonitions and empirical evidence indicating that it is not the most psychometrically sound statistic to determine interobserver agreement due to its inability to take chance into account. Cohen's (1960) kappa has long been proposed as the more psychometrically sound statistic for assessing interobserver agreement. Kappa is described and computational methods are presented.     

An exploration of the interrater agreement of visual analysis with and without context

Visual analysis is integral to the analysis of single-case experimental design (SCED) data. Previous studies have shown that many factors may influence the interrater agreement (IRA) of visual analysis. One factor that has received little direct attention is the impact of contextual information. In the current study, authors of recently published SCED studies were asked to make judgments regarding functional relations based on published datasets that met criteria for design quality. Respondents were randomly assigned to view graphs with or without contextual information and the degree of interrater agreement was compared. Results revealed that contextual information had no impact on IRA for decisions of a functional relation. IRA was high across both groups for 6 of the 7 datasets examined. Implications and recommendations based on these results are discussed.     

The Scale for Assessing Emotional Disturbance: Concurrent Validity

The Scale for Assessing Emotional Disturbance (SAED) is a standardized, norm-referenced measure designed to operationalize the federal definition of Emotional Disturbance. It has demonstrated content validity and reliability. The two studies reported here address the SAED's concurrent validity. Samples in each study consisted of students identified with Emotional Disturbance. In both studies, special education teachers rated students on the SAED and a second measure of children's emotional and behavior problems. In the first study the SAED was compared to the Revised Behavior Problem Checklist. Correlations were generally moderate to high for similar subscales and low for dissimilar subscales. In the second study, the SAED was compared to the Teacher Report Form. Again, correlations generally were as expected and compatible with results of the first study. Results indicate that the SAED emotional and behavior problem subscales measure constructs akin to emotional and behavior problems measured by similar collections of item on the Revised Behavior Problem Checklist and Teacher Report Form. Implications of results and recommendations for use of the SAED are discussed.     

HSK主观考试评分的Rasch实验分析

主观评分中存在的不一致性导致主观评分的信度降低。多面Rasch模型基于项目反应理论,可以应用于评分员效应的识别和消除,从而提高主观评分的信度。该文介绍多面Rasch模型的理论和应用框架,设计了基于该模型的HSK主观考试评分质量控制应用框架,利用HSK作文评分数据进行了实验验证。     

Family Emotional Climate and Sibling Relationship Quality: Influences on Behavioral Problems and Adaptation in Preschool-Aged Children

We examined the impact of family emotional climate and sibling relationship quality on behavioral problems and adaptation in preschool-aged children. Participants were 63 mothers with a preschool-aged child enrolled in a Southern Arizona Head Start Program. Siblings were identified as children closest in age to target child. Mothers of predominantly Mexican descent (95%) participated in home interviews during the Fall and Spring of the year children entered center-based programs. Sibling relationship quality (warmth, agonism/competition) was proposed to predict children’s adjustment (behavioral problems and adaptation) longitudinally. Results indicate that after controlling for child characteristics (temperament, child gender, birth order) and after accounting for family characteristics (family emotional expressiveness, child exposure to interparental conflict, and parental agreement on childrearing), sibling warmth made a significant and unique contribution to child adjustment as reported by mothers and teachers six months later. Findings are consistent with existing research indicating that sibling relationships impact children’s adjustment and shape young children’s lives in meaningful and marked ways. Moreover, these associations were found with an understudied sample of young children of predominantly Mexican descent in low-income families, and thus make an important contribution to knowledge in the field.     

评分者内部一致性的研究和应用

在行为科学的研究和实践中,研究者常常需要将个体层次的评价,整合到群体层次的评价,对于这种自下而上整合模式的一致性问题,国内一些学者常常错误地使用评分者内部信度作为评分者内部一致性的指标。评分者内部一致性和评分者内部信度不仅在理论基础上存在差异,而且在实践中也存在前者很高(或很低),而后者却很低(或很高)的不一致情况。文章阐述了学术界对评分一致性这一问题的提出、争论和取得一致观点的发展脉络,以期学者们对这一问题能够有深入的思索,避免在今后的研究中出现类似的错误。