评分者内部一致性的研究和应用

在行为科学的研究和实践中,研究者常常需要将个体层次的评价,整合到群体层次的评价,对于这种自下而上整合模式的一致性问题,国内一些学者常常错误地使用评分者内部信度作为评分者内部一致性的指标。评分者内部一致性和评分者内部信度不仅在理论基础上存在差异,而且在实践中也存在前者很高(或很低),而后者却很低(或很高)的不一致情况。文章阐述了学术界对评分一致性这一问题的提出、争论和取得一致观点的发展脉络,以期学者们对这一问题能够有深入的思索,避免在今后的研究中出现类似的错误。

The Research and Application of Interrater Agreement

Hierarchical data structures are very common in behavioral sciences.A hierarchy consists of lower level units nested within higher level.One of the main questions in multilevel data analysis with which psychologists have been concerned is whether it is appropriate to aggregated data and to use the aggregated measures to make inferences about the higher level units.Some researchers did not notice the great difference between interrater agreement and interrater reliability.This led to the misuse of interrater agreement.The article explained the history of interrater agreement,in the hope that researchers will give deeper consideration to this question,and avoid repeating similar mistakes in the future research.