A probability-based formula for calculating interobserver agreement

Estimates of observer agreement are necessary to assess the acceptability of interval data. A common method for assessing observer agreement, per cent agreement, includes several major weaknesses and varies as a function of the frequency of behavior recorded and the inclusion or exclusion of agreements on nonoccurrences. Also, agreements that might be expected to occur by chance are not taken into account. An alternative method for assessing observer agreement that determines the exact probability that the obtained number of agreements or better would have occurred by chance is presented and explained. Agreements on both occurrences and nonoccurrences of behavior are considered in the calculation of this probability.     

Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: II: A search procedure to locate the greatest lower bound

Finding the greatest lower bound for the reliability of the total score on a test comprisingn non-homogenous items with dispersion matrix Σ _x is equivalent to maximizing the trace of a diagonal matrix Σ _E with elements θ _I , subject to Σ _E and Σ _T =Σ _x − Σ _E being non-negative definite. The casesn=2 andn=3 are solved explicity. A computer search in the space of the θ _i is developed for the general case. When Guttman's λ₄ (maximum split-half coefficient alpha) is not the g.l.b., the maximizing set of θ _i makes the rank of Σ _T less thann − 1. Numerical examples of various bounds are given. Present affiliation of the first author: St. Hild's College of Education, Durham City, England.     

信度系数与观测分数和潜在特质的相关比

在经典真分数模型中,信度系数R=D(T)/D(X)。通常认为,信度系数与项目反应理论没有什么联系。事实上,信度系数恰好等于考生观测分数与潜在特质分数的非线性相关比η_(xθ)~2=1-MD(x|θ)/D(X),据此我们得出估计信度系数的一种新途径,同时讨论了x、θ的相关系数与相关比η_(xθ)~2的关系。     

A Norwegian translation of the Plutchik questionnaire for psychological defense

The reliability and validity of the Norwegian translation of Plutchik's Life Style Index (LSI) have been tested in a material of 704 individuals, 471 men and 233 women. The results showed low internal consistency for most of the subscales (Alpha ranging from 0.43 to 0.68). Several of the scales were also significantly correlated. Factor analyses showed that some of them loaded on common factors, indicating low specificity of the scales. Further, the validity of the test was not confirmed.     

Interobserver Agreement in Behavioral Research: Importance and Calculation

Behavioral researchers have developed a sophisticated methodology to evaluate behavioral change which is dependent upon accurate measurement of behavior. Direct observation of behavior has traditionally been the mainstay of behavioral measurement. Consequently, researchers must attend to the psychometric properties, such as interobserver agreement, of observational measures to ensure reliable and valid measurement. Of the many indices of interobserver agreement, percentage of agreement is the most popular. Its use persists despite repeated admonitions and empirical evidence indicating that it is not the most psychometrically sound statistic to determine interobserver agreement due to its inability to take chance into account. Cohen's (1960) kappa has long been proposed as the more psychometrically sound statistic for assessing interobserver agreement. Kappa is described and computational methods are presented.     

Assessing Personality States: What to Consider when Constructing Personality State Measures

Repeated assessments of personality states in daily diary or experience sampling studies have become a more and more common tool in the psychologist's toolbox. However, and contrary to the widely available literature on personality traits, no best practices for the development of personality state measures exist, and personality state measures have been developed in many different ways. To address this, we first define what a personality state is and discuss important components. On the basis of this, we define what a personality state measure is and suggest a general guideline for the development of such measures. Following the ABC of test construction can then guide the strategy for obtaining validity and reliability evidence: (A) What is the construct being measured? (B) What is the intended purpose of the measure? And (C) What is the targeted population of persons and situations? We then conclude with an example by developing an initial item pool for the assessment of conscientiousness personality states. © 2020 The Authors. European Journal of Personality published by John Wiley & Sons Ltd on behalf of European Association of Personality Psychology     

An exploration of the interrater agreement of visual analysis with and without context

Visual analysis is integral to the analysis of single-case experimental design (SCED) data. Previous studies have shown that many factors may influence the interrater agreement (IRA) of visual analysis. One factor that has received little direct attention is the impact of contextual information. In the current study, authors of recently published SCED studies were asked to make judgments regarding functional relations based on published datasets that met criteria for design quality. Respondents were randomly assigned to view graphs with or without contextual information and the degree of interrater agreement was compared. Results revealed that contextual information had no impact on IRA for decisions of a functional relation. IRA was high across both groups for 6 of the 7 datasets examined. Implications and recommendations based on these results are discussed.     

Bootstrap区间估计在认知诊断模型误设中的应用

认知诊断模型选择是认知诊断评估中重要研究问题之一。在实际应用中实践者并不知道真正拟合数据的模型,通常会用模型拟合指标检验模型与数据的拟合程度。从测量结果质量来看,除保证模型与数据拟合之外,还需要重点评价模型诊断结果的信度和效度等。考虑到以往研究大都采用基于信息量的拟合指标去判定模型与数据的匹配性,本研究提出综合考虑模型拟合指标与信度指标用于模型选择或评价模型误设。考虑实验因素为真实模型或分析模型（DINA模型、G-DINA模型、R-RUM模型）、样本量、题量和属性个数,在五因素（3×3×2×2×2）实验设计条件下,比较Bootstrap区间估计的属性分类一致性信度平均数与标准误和常用的拟合统计量-2LL、AIC、BIC对正确模型的选择率。结果表明：-2LL在题目数量多的情况下表现较好,而AIC、BIC在被试量较大的情况下表现较好,在不同的研究条件下,-2LL、AIC、BIC的模型选择率很不稳定,而用Bootstrap法估计的属性分类一致性信度平均数和标准误在不同研究条件的模型选择率较稳定,总体表现较好。     

罗夏墨迹技术的近期发展与争论

综合系统在罗夏墨迹技术（RIM）的标准化、客观化方面取得了很大的成功,但仍有一些批评者对它的信度、效度、常模与标准化等问题提出了质疑,作者们对这些争论的焦点问题进行了系统的回顾。总结有关的元分析和实验研究,发现RIM的多数核心变量都有较好的评分者信度和跨时间稳定性,只有心理状态变量的信度较低。以往的整体元分析和局部元分析都发现RIM不逊于其他测验。这些结果表明RIM在临床疾病诊断,外显行为预测方面都是有价值的。与其他 测验一样,RIM也有其优势与不足,在应用中应与其他测验互补     

The reliability, stability, and predictive utility of the self-report version of the Antisocial Process Screening Device

The psychometric properties of the self-report version of the Antisocial Processes Screening Device (APSD; Frick & Hare, 2001), a rating scale developed to assess traits associated with the construct of psychopathy in youth, was tested in a sample of 91 non-referred young adolescents with an average age of 13.38 (SD = 1.75) at the initial assessment. The sample was recruited from a large community-wide screening, where youth with conduct problems and youth high on psychopathic traits were over-sampled. The sample was reassessed three times at yearly intervals. The self-report scores on the APSD showed moderate correlations with parent ratings of psychopathic traits, were moderately stable across 1-2 years, and showed significant correlations with measures of antisocial behavior both concurrently and predictively. One major weakness of the self-report ratings was the low internal consistency of the subscales, which were much lower than the internal consistency found on the parent report version of the scale.