Observer agreement, credibility, and judgment: some considerations in presenting observer agreement data

Graphical and statistical indices employed to represent observer agreement in interval recording are described as "judgmental aids", stimuli to which the researcher and scientific community must respond when viewing observer agreement data. The advantages and limitations of plotting calibrating observer agreement data and reporting conventional statistical aids are discussed in the context of their utility for researchers and research consumers of applied behavior analysis. It is argued that plotting calibrating observer data is a useful supplement to statistical aids for researchers but is of only limited utility for research consumers. Alternatives to conventional per cent agreement statistics for research consumers include reporting special agreement estimates (e.g., per cent occurrence agreement and nonoccurrence agreement) and correlational statistics (e.g., Kappa and Phi).     

A probability-based formula for calculating interobserver agreement

Estimates of observer agreement are necessary to assess the acceptability of interval data. A common method for assessing observer agreement, per cent agreement, includes several major weaknesses and varies as a function of the frequency of behavior recorded and the inclusion or exclusion of agreements on nonoccurrences. Also, agreements that might be expected to occur by chance are not taken into account. An alternative method for assessing observer agreement that determines the exact probability that the obtained number of agreements or better would have occurred by chance is presented and explained. Agreements on both occurrences and nonoccurrences of behavior are considered in the calculation of this probability.     

TEMPORAL AND CLIMATIC VARIABLES IN NATURALISTIC OBSERVATION1

Home-observation data on 5- to 7-yr-old boys collected over 2 yr were examined for systematic variations in rates of desirable and undesirable behaviors associated with several temporal and climatic variables. Significant effects associated with time of day, day of the week, precipitation, and temperature were found. No significant effects on the naturalistic observation data were found for environmental factors associated with lunar phase. It was noted that the correlational nature of the findings did not obviate the necessity for control of the influence of temporal and climatic variables. Several methodological strategies for such control were discussed.     

MSA for a special spearman matrix

Kaiser's measure of sampling adequacy is applied to a special Spearman matrix and a specialq-cluster generalization. The result supports the contention that the measure should be no less than .5 for data to be appropriate for factor analysis.The research reported in this note was supported in part by the Program in Applied Mathematics and Statistics, National Science Foundation, and by the National Institute of Education, Department of Health, Education, and Welfare.     

A bridge between cultures: Linked ethnographic-empirical methodology for culture anchored research

Relatively little research in community psychology has explicitly focused on groups whose world views, values, and practices diverge from mainstream North American and social science culture. Furthermore, when research does include members of subcultural groups, the same research methodologies used to stydy mainstream populations tend to be employed. In the current article, it is proposed that a linked ethnographic—empirical methodology is critical for culturally anchored research of rarely studied subcultures which differ markedly form the dominant culture. The use of a linked ethnographic—empirical methodology in the study of a nontraditional religious subculture is described, and the study's primary findings summarized. Respective and combined contributions of the ethnographic and empirical (i.e., quantitative) research components in seven key areas are discussed: (a) selection of phenomenon to study; (b) generation of hypotheses; (c) culturally anchored measure development; (d) sampling; (e) analyses; (f) interpretation of findings; and (g) social action. Limitations of the linked methodology are discussed. The author thanks Julian Rappaport, Chris Murphy, Robert Serpell, three anonymous reviewers, and the editors of the special issue for their valuable feedback.     

TEACHING PERSONS WITH PROFOUND MULTIPLE HANDICAPS: A REVIEW OF THE EFFECTS OF BEHAVIORAL RESEARCH

The behavioral research on teaching individuals who have profound multiple handicaps is reviewed. The primary focus is on determining the degree to which behavioral research has demonstrated the teaching of meaningful skills to this population. Results of the review indicate that investigations have demonstrated, albeit inconsistently, that behavior change has resulted from contingency management interventions with persons who have profound multiple handicaps. However, there is little evidence that such interventions have resulted in meaningful behavior change according to currently accepted criteria for beneficially affecting the quality of life of persons with serious handicaps. Potential explanations for the lack of such evidence are offered, including the relative lack of research attention given to this issue, the possible ineffectiveness of the components of the technology applied, and possible ineffective application of the potentially effective technology. Suggestions for future research are discussed in terms of developing more effective educational and habilitative services for persons with profound multiple handicaps. In particular, we suggest research on a wider variety of behavioral teaching procedures, providing more comprehensive evaluations of the applications of procedures and developing treatment programs that do not focus solely on traditional skill acquisition.     

ASSESSING STIMULUS CONTROL IN NATURAL SETTINGS: AN ANALYSIS OF STIMULI THAT ACQUIRE CONTROL DURING TRAINING

When a learner is taught a new response, the stimuli that influence its display are often unknown. The presence or absence of these stimuli alters the probability of occurrence of the response. By identifying the stimuli influencing the probability of newly acquired responses, interventionists may program for their generalization more effectively and efficiently. This investigation describes the application of an operant methodology to assess functional relationships between responses and specific stimulus variables. Four young adults with moderate mental retardation were taught to include “please” as part of requests they made in school. Four environmental stimuli, present during training, were assessed for the controlling properties they acquired. Each of the four was assessed prior to and after training by presenting it in isolation (i.e., the other three were varied). If the presence of a single stimulus associated with training did not occasion “please,” then pairs of stimuli were probed. The results revealed that single-stimulus probing occasioned responding by only 1 learner; paired-stimulus probing set the occasion for including “please” by 2 others. Control of the 4th learner's responding was lost before training was introduced, because he began including “please” in his requests during baseline. The implications of these results are discussed in terms of analyzing stimulus control and promoting stimulus generalization.     

EM and beyond

The basic theme of the EM algorithm, to repeatedly use complete-data methods to solve incomplete data problems, is also a theme of several more recent statistical techniques. These techniques—multiple imputation, data augmentation, stochastic relaxation, and sampling importance resampling—combine simulation techniques with complete-data methods to attack problems that are difficult or impossible for EM.A preliminary version of this article was the Keynote Address at the 1987 European Meeting of the Psychometric Society June 24–26, 1987 in Enschede, The Netherlands. The author wishes to thank the editor and reviewers for helpful comments.     

情绪调节灵活性对负性情绪的影响：来自经验取样的证据

情绪调节灵活性是指个体根据不断变化的情境需求灵活部署情绪调节策略的能力。本研究采用经验取样方法,通过拟合个体在日常生活事件(如,未通过考试)和突发公共卫生事件(COVID-19)中的策略使用剖面结构和情境负性程度与策略使用程度的共变关系测量个体的情绪调节灵活性水平,并探讨其对个体后续负性情绪(抑郁和焦虑)的影响。两个独立样本结果表明：单一策略使用偏好(如沉浸偏好和表达抑制偏好)的个体在负性生活事件中和疫情期间经历了更高水平的抑郁和焦虑情绪。此外,当个体随情境负性程度提高使用更多分心策略,而随情境负性程度降低使用更多认知重评策略(意味较高的情绪调节灵活性),其抑郁和焦虑情绪水平更低。以上结果共同证实了情绪调节灵活性有利于减少个体的负性情绪体验。     

A hot new way to measure aggression: Hot sauce allocation

Laboratory experiments investigating aggressive behavior have operationalized and assessed aggression in a variety of ways; however, these measures are often problematic because they do not create a situation in which participants perceive potential for real harm to come to the target, there is a risk of actual harm to the target, or they are too familiar to participants. To overcome these limitations, we developed a new method for measuring aggression, specifically, the amount of hot sauce administered to a target known to dislike spicy foods. We summarize a series of experiments assessing theory‐based hypotheses regarding aggression in which this measure is employed. We then briefly consider the strengths and limitations of this new measure. Aggr. Behav. 25:331–348, 1999. © 1999 Wiley‐Liss, Inc.