An ordinal coefficient of relational agreement for multiple judges

In a recent article, Fagot proposed a generalized family of coefficients of relational agreement for multiple judges, focusing on the concept of empirically meaningful relationships. In this paper an ordinal coefficient of relational agreement, based on ranking data, is presented as a special case of the generalized family. It is shown that the proposed ordinal coefficient encompasses other ordinal coefficients, such as the Kendall coefficient of concordance, the average Spearman rank-order coefficient, and intraclass correlation based on ranks. It is also shown that the Kendall coefficient of concordance, corrected for chance agreement, is equivalent to the ordinal coefficient proposed in this paper.     

The effects of instructions and calculation procedures on observers' accuracy, agreement, and calculation correctness

Although the quality of observational data is generally evaluated by observer agreement, measures of both observer agreement and accuracy were available in the present study. Videotapes with a criterion protocol were coded by 16 observers. All observers calculated agreement scores both on their own and their partner's data and on a contrived data set misrepresented as data collected by other observers. Compared with agreement scores calculated by the experimenter, observers erroneously inflated their own agreement scores and deflated the agreement scores on the contrived data. Half of the observers (n = 8) had been given instructions emphasizing the importance of accuracy during observation while the other half had been given instructions emphasizing interobserver agreement. Accuracy exceeded agreement for the former group, whereas agreement exceeded accuracy for the latter group. The implications are that agreement should be calculated by the experimenter and that the accuracy-agreement relationship can be altered by differential observer instructions.     

A review of the observational data-collection and reliability procedures reported in The Journal of Applied Behavior Analysis

The research published in the Journal of Applied Behavior Analysis (1968 to 1975) was surveyed for three basic elements: data-collection methods, reliability procedures, and reliability scores. Three-quarters of the studies reported observational data. Most of these studies' observational methods were variations of event recording, trial scoring, interval recording, or time-sample recording. Almost all studies reported assessment of observer reliability, usually total or point-by-point percentage agreement scores. About half the agreement scores were consistently above 90%. Less than one-quarter of the studies reported that reliability was assessed at least once per condition.     

Artifact, bias, and complexity of assessment: the ABCs of reliability

Interobserver agreement (also referred to here as "reliability") is influenced by diverse sources of artifact, bias, and complexity of the assessment procedures. The literature on reliability assessment frequently has focused on the different methods of computing reliability and the circumstances under which these methods are appropriate. Yet, the credence accorded estimates of interobserver agreement, computed by any method, presupposes eliminating sources of bias that can spuriously affect agreement. The present paper reviews evidence pertaining to various sources of artifact and bias, as well as characteristics of assessment that influence interpretation of interobserver agreement. These include reactivity of reliability assessment, observer drift, complexity of response codes and behavioral observations, observer expectancies and feedback, and others. Recommendations are provided for eliminating or minimizing the influence of these factors from interobserver agreement.     

Observer reliability as a function of circumstances of assessment

THREE FACTORS CHARACTERISTIC OF EXPERIMENTAL SETTINGS WERE HYPOTHESIZED TO INFLATE ARTIFACTUALLY THE RELIABILITY OF OBSERVATIONAL RECORDINGS: (a) knowledge by observers of when and by whom their reliability is being assessed, (b) the absence of the experimenter or a monitor to prevent cheating, and (c) computation of reliability within- (versus between-) observer group. Three groups of four observers used a standard nine-category observational code for disruptive behavior in recording from videotapes of a classroom for 22 days. Analyses revealed considerable increases in average occurrence reliability as a function of the main effects of each of the experimental factors. The specific increases in reliability associated with each of the 12 combinations of the experimental factors are presented for each category of behavior. The possible role of observer-training procedures and behavioral definitions as determiners of nonartifactual reliability is discussed.     

Testing the significance of interobserver agreement measures in the presence of autocorrelation: The jackknife procedure

Users of interobserver agreement statistics have heretofore ignored the problem of autocorrelation in behavior sequences when testing the statistical significance of agreement measures. Due to autocorrelation traditional reliability tests based on the 2 × 2 contingency-table model (e.g., kappa, phi) are incorrect. Correct tests can be developed by using the bivariate time series as a statistical model. Seen from this perspective, testing the significance of interobserver agreement becomes formally equivalent to testing the significance of the lag-zero cross-correlation between two time series. The robust procedure known as the jackknife is suggested for this purpose.     

Matching models in the analysis of cross-classifications

Inference models motivated by the combinatorial chance literature and the concept of object matching may be used in the analysis of a contingency table if the conditional assumption of fixed row and column totals is imposed. More specifically, by developing a matching reinterpretation for several problems of interest in the prediction analysis of cross-classifications—as defined by Hildebrand, Laing and Rosenthal, appropriate significance tests can be given that may differ from those justified by the more common multinomial models. In the course of the paper the distinction between a degree-1 statistic (based on the relationship between single objects) and a degree-2 statistic (based on the relationship between object pairs) is reviewed in some detail. Also, several specializations are presented to topics of current methodological importance in psychology; for instance, a number of references are made to the measurement of nominal scale response agreement between two raters.Partial support for this research was provided by the National Science Foundation through GSOC-77-28227.     

Gender and Attitudes Towards a Peace Agreement in Situations of Intractable Conflict: The Case of the Jewish-Israeli Society

The question of whether women are more oriented towards peace has been debated in the research literature for several decades but has not been systematically tested in conflict-driven areas. The aim of our article is twofold: (1) to suggest a conceptual framework regarding gender differences in support for peace in intractable conflict and test it in a prominent case study; and (2) to examine whether major events leading to conflict reescalation have differential effects on women and men. Across 145 polls carried out on a monthly basis among nationally representative samples of Jewish-Israelis between 1995 and 2006, we found only marginally higher levels of support for the Oslo Accords among women versus men, while conflict-related worldviews—political ideology and religiosity—had a much larger effect. Furthermore, violent reescalation in the conflict had a stronger effect on reducing men's levels of support for the Oslo Accords than those of women. Robustness analysis of 196 monthly surveys from 2002 until 2018 examining the effect of gender on support for negotiation provided further support for our findings. Overall, our analyses indicate that sex-based differences play a minor role in explaining attitudes towards peace in situations of protracted violent conflicts at different stages, compared to the prominent effects of political ideology and religiosity.     

影响图画命名时间的因素

实验共测量了311张图片的H值、命名一致性百分数、概念一致性百分数、表象一致性、熟悉性、视觉复杂性、词长和图片命名的时间,主要探索影响图片命名潜伏期的因素,结果发现概念一致性、熟悉性、表象一致性、词频是影响图片命名时间的主要因素。并尝试建立了概念一致性、熟悉性、表象一致性和词频对图片命名时间的回归方程,其预测能力为55.5%。根据图片的命名时间对命名的难易程度进行了分级,希望能对利用图片作为实验材料的认知实验提供有价值的信息