Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Considerations in the choice of interobserver reliability estimates

Two types of interobserver reliability values may be needed in treatment studies in which observers constitute the primary data-acquisition system: trial reliability and the reliability of the composite unit or score which is subsequently analyzed, e.g., daily or weekly session totals. Two approaches to determining interobserver reliability are described: percentage agreement and "correlational" measures of reliability. The interpretation of these estimates, factors affecting their magnitude, and the advantages and limitations of each approach are presented.     

EMPLOYING TASK ARRANGEMENTS AND VERBAL CONTINGENCIES TO PROMOTE VERBALIZATIONS BETWEEN RETARDED CHILDREN1

This study investigated the effects of arranging task events for interdependence, to increase the probability of social responding. During task interdependence, the subjects, participating in dyads and a four-person group, obtained task materials (a puzzle piece) from their partner before completing their task (appropriately placing the puzzle piece). The verbal contingency required a verbal request to precede a subject's receiving a task material from his partner. The verbal contingency yoked with task interdependence made task completion contingent on the appropriate verbalization. The findings from two experiments suggested that task interdependence was sufficient to increase partner-directed verbalizations for three of the four subjects. When the verbal contingency was added, all subjects increased their requests and other verbalizations to partner. Applied to a four-person group, the verbal contingency yoked with varying levels of task interdependence correspondingly affected the pattern and level of group communications. The greater the task interdependence, i.e., the more members each subject depended on to complete his task, the more complex the social network of verbal contacts, and the higher the level of both requests and other verbalizations for the group.     

Correlation and prediction in ordinal test theory

Based on the test theory model for ordinal measurements proposed by Schulman and Haden, the present paper considers correlations between tests, attenuation, regressions involving true and observed scores, and prediction of test reliability.The population correlation between tests is shown to be related to the expected sample correlation for samples of sizen ₁ andn ₂. Errors of estimation, measurement and prediction are found to be similar to their counterparts in interval test theory, while attenuation is identical to its counterpart. The bias in estimating population reliability from sample data is compared for Kendall's tau and Spearman's rho.The author wishes to thank the referees for their helpful comments on an earlier draft of this paper, and in particular, for the suggested alternative methods of establishing some of the presented results.     

校园霸凌行为问卷的编制

目的:从贵州省青少年校园霸凌行为的实际出发,以大样本为基础,遵循心理测量学的方法编制《校园霸凌行为问卷》。方法:经理论建模、题库建设编制初测问卷,以3923名学生为被试进行项目分析和探索性因子分析; 以3899名被试进行信度检验及验证性因子分析,形成最终问卷。结果:问卷由7个一阶因子和2个二阶因子组成,共40个题项。问卷及各维度的内部一致性系数在0.7～0.927之间,重测信度在0.291～0.49之间,结构效度良好,与OBVQ-V和MBVS的相关分别为0.646和0.726。结论:本研究探索、发现并整合创新了反映当代校园霸凌治理难点的“恶整霸凌”; 问卷信效度良好,可作为国内青少年校园霸凌行为的有效测量工具。     

Reliability of variable message signs information: A field experiment study in Chongqing

A variable message sign (VMS), which is a key component of intelligent transportation systems, has been frequently used in the management of urban roads and motorways to provide drivers with real-time traffic condition information about a road section or area. Nevertheless, there is a lack of unified regulations for VMS design, which makes it difficult to accurately determine the understandability and legibility of VMS information. In practice, inappropriate designs of VMSs are common, such as overload of VMS information and excessive number of phases, particularly on the urban roads of China. Building on our earlier findings obtained by surveys, in this study, field tests were conducted to assess the reliability of VMS information in terms of the visual perception characteristics of drivers. For a full visual perception, the information displayed on a VMS must be reliable, i.e., it should be easy for drivers to detect, obtain, and understand the messages they require. Two basic theories—information theory and visual perception theory—were introduced to build a quantification model of VMS information, select evaluation indices, and design the experimental process. A total of 24 drivers participated in the field experiments and questionnaires. Five indices—information obtainment rate, periodic validity, driver subjective scoring and the corresponding UI (represents information intelligibility), and CE (represents information legibility)—were analyzed to evaluate their relationships to VMS information reliability. The results confirmed that the amount of information, and the scrolling period considering redundant distance, significantly affect the reliability of VMSs information. For static VMSs, the information obtainment rate and the subjective scoring decrease with increasing amount of information. The recollection accuracy of the drivers significantly declines when the amount of information shown increases to 90 bits, corresponding to an information obtainment rate of less than 0.8 and UI equal to 0. For dynamic VMSs, the information reliability deteriorates as the scrolling period shortens and the number of phases increases. Unreasonable reliability is found when the periodic validity is less than 0.9, i.e., the actual scrolling period is more than 10% less than the calculated one, corresponding to CE equal to 0. The reliability of information was evaluated by combining the subjective scoring of the drivers and a data-based statistical analysis and considering driving safety. Accordingly, it was recommended that 90 bit is the maximum amount of information to be shown on a VMS and that the preferred scrolling period of each phase of a dynamic VMS is 5 s. The results of this study support the objective of providing reliable information to drivers by addressing the problems related to the amount of information and presentation time for VMSs. These findings provide a basis for determining the thresholds for VMS information to promote practical and user-friendly designs of VMSs on urban roads.     

Improved Test–retest Reliability Criteria as Applied to Probability Assessments

Test–retest reliability is a common indicator of response consistency. It is argued that using regression coefficients for detecting systematic response error is less appropriate than testing for shifts in the mean (median) and variance. This procedure is exemplified using probability response data. For this data, shifts in centrality were found to be about 2.5 times more likely than shifts in variability. The shifts in centrality did not favor any particular direction; however, variability tended to decrease over time in early sessions.     

Indicators of Affect Dynamics: Structure,Reliability, and Personality Correlates

Researchers are increasingly interested in the affect dynamics of individuals for describing and explaining personality and psychopathology. Recently, the incremental validity of more complex indicators of affect dynamics (IADs; e.g. autoregression) has been called into question (Dejonckheere et al., 2019), with evidence accumulating that these might convey little unique information beyond mean level and general variability of emotions. Our study extends the evidence for the construct validity of IADs by investigating their redundancy and uniqueness, split-half reliability based on indices from odd-numbered and even-numbered days, and association with big five personality traits. We used three diverse samples that assessed daily and momentary emotions, including community participants, individuals with personality pathology, and their significant others (total N = 1192, total number of occasions = 51 278). Mean and variability of affects had high reliability and distinct nomological patterns to big five personality traits. In contrast, more complex IADs exhibited substantial redundancies with mean level and general variability of emotions. When partialing out these redundancies by using residual variables, some of the more complex IADs had acceptable reliability, but only a few of these showed incremental associations with big five personality traits, indicating that IADs have limited validity using the current assessment practices. © 2020 The Authors. European Journal of Personality published by John Wiley & Sons Ltd on behalf of European Association of Personality Psychology     

简式元认知问卷中文版的心理计量学特性

以大学生为被试,考察简式元认知问卷(BQM)在中国文化背景下的心理计量学特性。采用探索性和验证性因素分析方法对BQM的结构效度进行交叉检验,结果支持了元记忆和元注意的双因子结构; 问卷及其因子的内部一致性和重测信度良好; 项目分析的结果显示,问卷的条目具有良好的分数分布、区分度和内部一致性; BQM分数与负性情绪、执行功能和睡眠质量的关联性也为效标关联效度提供了支持性证据。上述结果表明BQM中文版具有良好的心理计量学特性,可作为青年群体元认知水平的测量工具。