Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.     

脑成像在心理学研究领域的价值

现在普遍使用的脑成像技术给心理学研究增加了新的数据和资料。和任何新的方法一样,我们需要决定如何以适当的方式应用这项技术。这项技术如何以现有的方法所不能的方式帮助回答理论问题？这项技术最好是作为因变量还是作为预测变量来使用？它如何与其它感兴趣的心理变量相关？这种新的成像技术有助于我们了解大脑的运作及其与心理学的关系。研究人员需要弄清楚如何利用这项技术提供的信息加深对心理现象的理解。     

Using models to correct data: paleodiversity and the fossil record

Despite an enormous philosophical literature on models in science, surprisingly little has been written about data models and how they are constructed. In this paper, I examine the case of how paleodiversity data models are constructed from the fossil data. In particular, I show how paleontologists are using various model-based techniques to correct the data. Drawing on this research, I argue for the following related theses: first, the ‘purity’ of a data model is not a measure of its epistemic reliability. Instead it is the fidelity of the data that matters. Second, the fidelity of a data model in capturing the signal of interest is a matter of degree. Third, the fidelity of a data model can be improved ‘vicariously’, such as through the use of post hoc model-based correction techniques. And, fourth, data models, like theoretical models, should be assessed as adequate (or inadequate) for particular purposes.

     

The metric‐frequency measure of similarity for ill‐structured data sets,with an application to family therapy

Similarity measures have been studied extensively in many domains, but usually with well‐structured data sets. In many psychological applications, however, such data sets are not available. It often cannot even be predicted how many items will be observed, or what exactly they will entail. This paper introduces a similarity measure, called the metric‐frequency (MF) measure, that can be applied to such data sets. If it is not known beforehand how many items will be observed, then the number of items actually observed in itself carries information. A typical feature of the MF is that it incorporates such information. The primary purpose of our measure is that it should be pragmatic, widely applicable, and tractable, even if data are complex. The MF generalizes Tversky's set‐theoretic measure of similarity to cases where items may be present or absent and at the same time can be numerical as with Shepard's metric measure, but need not be so. As an illustration, we apply the MF to family therapy where it cannot be predicted what issues the clients will raise in therapy sessions. The MF is flexible enough to be applicable to idiographic data.     

A Person‐Centered Examination of Adolescent Religiosity Using Latent Class Analysis

Empirical studies of religion's role in society, especially those focused on individuals and analyzing survey data, conceptualize and measure religiosity as ranging from low to high on a single measure or a summary index of multiple measures. Other concepts, such as “lived religion,” “believing without belonging,” or “fuzzy fidelity” emphasize what scholars have noted for decades: humans are rarely consistently low, medium, or high across dimensions of religiosity including institutional involvement, private practice, salience, or belief. A method with great promise for identifying population patterns in how individuals combine types and levels of belief, practice, and personal religious salience is latent class analysis. In this article, we use data from the first wave of the National Study of Youth and Religion's telephone survey to discuss how to select indicators of religiosity in an informed manner, as well as the implications of the number and types of indicators used for model fit. We identify five latent classes of religiosity among adolescents in the United States and their sociodemographic correlates. Our findings highlight the value of a person‐centered approach to understanding how religion is lived by American adolescents.     

An information theoretic measure for the evaluation of ordinal scale data

This article describes a new measure of dispersion as an indication of consensus and dissention. Building on the generally accepted Shannon entropy, this measure utilizes a probability distribution and the ordered ranking of categories in an ordinal scale distribution to yield a value confined to the unit interval. Unlike other measures that need to be normalized, this measure is always in the interval 0 to 1. The measure is typically applied to the Likert scale to determine degrees of agreement among ordinal-ranked categories when one is dealing with data collection and analysis, although other scales are possible. Using this measure, investigators can easily determine the proximity of ordinal data to consensus (agreement) or dissention. Consensus and dissention are defined relative to the degree of proximity of values constituting a frequency distribution on the ordinal scale measure. The authors identify a set of criteria that a measure must satisfy in order to be an acceptable indicator of consensus and show how the consensus measure satisfies all the criteria.     

Ear Decomposition for Pair Comparison Data

An efficient graph-theoretical decomposition technique is introduced that treats inconsistencies in behavioral data as systematic adaptations rather than random errors. This technique, which is known as ear decomposition, reduces inconsistencies in any binary data set to a basis of directed cycles. Such a basis characterizes the data set in terms of inconsistencies and its size offers an improved measure of internal consistency. In two examples it is illustrated how different implementations of the ear decomposition technique can help to identify choices that are critical for violations of transitivity.     

A cautionary note on using internal cross validation to select the number of clusters

A highly popular method for examining the stability of a data clustering is to split the data into two parts, cluster the observations in Part A, assign the objects in Part B to their nearest centroid in Part A, and then independently cluster the Part B objects. One then examines how close the two partitions are (say, by the Rand measure). Another proposal is to split the data into k parts, and see how their centroids cluster. By means of synthetic data analyses, we demonstrate that these approaches fail to identify the appropriate number of clusters, particularly as sample size becomes large and the variables exhibit higher correlations.The authors express their thanks to the Sol C. Snider Entrepreneurial Center, Wharton School, for support of this project.     

An adaptive temporal-causal network model to analyse extinction of communication over time

The persistence of information communicated between humans is difficult to measure as it is affected by many features. This paper presents an approach to computationally model the cognitive processes of information sharing to describe persistence or extinction of communication in Twitter over time. The adaptive mental network model explains, for example, how an individual can experience information overflow on a topic, and how this affects the sharing of information. Parameter tuning by Simulated Annealing is used to identify characteristics of the network model that fit to empirical data from Twitter. The data collected is related to the independentism in Catalunya, Spain, which is considered a global issue with repercussion in Europe.     

Citation trends of applied journals in behavioral psychology 1981-2000

One variable with which to evaluate scientific journals is how often their articles are cited in the literature. Such data are amenable to longitudinal analysis and can be used as a measure of a journal's impact on research within a discipline. We evaluated multiple citation measures for a number of applied journals in behavioral psychology from 1981 to 2000. The results indicate a relatively consistent impact across these journals, with some evidence of growth.     

One-gun-a-month: Measuring public opinion concerning a gun control initiative

State lawmakers in Virginia recently approved a measure to limit to one the number of handguns a person can purchase within a thirty day period. In the months preceding the law's approval, a survey was conducted to measure the level of public support for the proposed initiative. The results of the survey were provided to lawmakers and other high level government officials in an effort to provide policymakers with objective data for gauging support (or non-support) for the proposal. Past public opinion polls which have measured attitudes concerning gun control reveal differences in the levels of support with regard to such factors as individual gun ownership and region of residence. The following research reveals the sentiment of one State's citizenry toward a specific handgun control measure by focusing on how responses varied across selected sub-groups within the sample.     

JAM-boree: An application of observation oriented modelling to judgements of associative memory

Null hypothesis significance testing is criticised for emphatically focusing on using the appropriate statistic for the data and an overwhelming concern with low p-values. Here, we present a new technique, Observation Oriented Modeling (OOM), as an alternative to traditional techniques in the social sciences. Ten experiments on judgements of associative memory (JAM) were analysed with OOM to show data analysis procedures and the consistency of JAM results across several types of experimental manipulations. In a typical JAM task, participants are asked to rate the frequency of word pairings, such as LOST-FOUND, and are then compared to actual normed associative frequencies to measure how accurately participants can judge word pairs. Three types of JAM tasks are outlined (traditional, paired, and instructional manipulations) to demonstrate how modelling complex hypotheses can be applied through OOM to this type of data that would be conventionally analysed with null hypothesis significance testing.     

Reliability and expected loss: A unifying principle

We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the proportional reduction in loss (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.Bruce Cooil is an Associate Professor of Statistics, and Roland T. Rust is a Professor and area head for Marketing. The authors thank three anonymous reviewers and an Associate Editor for their helpful comments and suggestions. This work was supported in part by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.     

Sequelae of child sexual abuse: a child and parent assessment

We present a case example of a 9-year-old, biracial girl and her mother. We integrate data collected from rating scales (e.g., Child Behavior Checklist; Achenbach & Rescorla, 2001), a free response measure (Thematic Apperception Test; Murray, 1943), and a direct observation measure (Parent-Child Interaction Assessment-II; Holigrocki, Kaminski, & Frieswyk, 1999, 2002) and reveal how a child sexual abuse victim's internal representations and symptoms manifest in both an interpersonal context and in the realm of play. We discuss assessment findings regarding how they provide for an idiographic understanding of the child.     

Measurement of Deliberate Self-Harm: Preliminary Data on the Deliberate Self-Harm Inventory

Deliberate self-harm has recently begun to receive more systematic attention from clinical researchers. However, there remains a general lack of consensus as to how to define and measure this important clinical construct. There is still no standardized, empirically validated measure of deliberate self-harm, making it more difficult for research in this area to advance. The present paper provides an integrative, conceptual definition of deliberate self-harm as well as preliminary psychometric data on a newly developed measure of self-harm, the Deliberate Self-Harm Inventory (DSHI). One hundred and fifty participants from undergraduate psychology courses completed research packets consisting of the DSHI and other measures, and 93 of these participants completed the DSHI again after an interval of 2–4 weeks (M = 3.3 weeks). Preliminary findings indicate that the DSHI has high internal consistency; adequate construct, convergent, and discriminant validity; and adequate test-retest reliability.     

Explicitly modeling the effects of aging on response time

Research into the effects of aging on response time has focused on Brinley plots. Brinley plots are constructed by plotting mean response times for older subjects against those for young subjects for a set of experimental conditions. The typical result is a straight line with a slope greater than 1 and a negative intercept. This linear function has been interpreted as showing that aging leads to a general slowing of cognitive processes. In this article, we show that the slope of the Brinley plot is actually a measure of the relative standard deviations of older versus young subjects’ response times; it is not a measure of general slowing. We examine current models of the effects of aging on mean response time and show how they might be reinterpreted. We also show how a more comprehensive model, Ratcliff’s diffusion model (1978), can account for Brinley plot regularities and, at the same time, provide an account of accuracy rates, the shapes of response time distributions, and the relative speeds of error and correct response times, aspects of the data about which models designed to account for Brinley plots are mute. We conclude by endorsing a research approach that applies explicit models to response time data in aging in order to use the parameters of the model to interpret the effects of aging.     

Distinction Between Well-Standardized Norms and the Psychometric Properties of a Measure

Funderburk and Eyberg (1989) described the psychometric properties of the Sutter-Eyberg Student Behavior Inventory (SESBI), a teacher rating scale of disruptive behaviors, for a sample of 55 preschool children. Additional data on the SESBI are presented for a sample of 60 preschool children. While both studies produced almost identical reliability and validity analyses, the scale score means are statistically and clinically different (i.e., a child in the clinical range in one study would be in the middle of the normal range in the other). These findings are used to emphasize the distinction between well-standardized norms and the psychometric properties of a measure. Suggestion are also made as to how behavioral assessment can more thoroughly attend to both of these properties of a measure.     

Will the most informative object stand? Determining the impact of structural context on informativeness judgements

A fundamental unsolved problem in the cognitive sciences concerns why, how, and to what extent humans judge object stimuli as conveying different amounts of information. Central to this problem is how the notion of informativeness is conceptualised by humans in the first place. In this paper, we investigate this question from the standpoint of how the structure of categories of objects influences informativeness judgements about their members. Results from our two experiments show that the structural or relational context surrounding single-object cues from a categorical stimulus largely determines such informativeness judgements. Moreover, we found that object cues elicit absolute magnitude judgements about their associated concept that are not consistent with the prototype interpretation of the concept. We were able to account for over 90% of the variance in the data from our two judgement experiments with a general theory and measure of information referred to as Representational Information.     

Reliability of the Go/No Go Association Task

The Go/No Go Association Task (GNAT; Nosek & Banaji, 2001) is an implicit measure with broad application in social psychology. It has several conceptual strengths to recommend it over other implicit methods, but the belief that it has poor reliability coupled with the absence of a method for calculating this important psychometric property has hindered its wider acceptance and use. Using data obtained from six GNAT studies covering a wide range of content areas, Study 1 compares the properties of different methods for estimating reliability of the GNAT. Study 2 demonstrates a resampling procedure to investigate how reliability varies as a function of block length. Study 1 shows that with appropriately chosen stimuli the GNAT can be a very reliable measure, while Study 2 indicates that as an empirical rule of thumb 50 to 80 trials per block should yield adequate to very good reliability. However, researchers are urged to calculate their own reliability coefficients, to this end we discuss GNAT design issues and provide procedures for calculating GNAT reliability which we hope will enhance the utility of the GNAT as a measure and promote its use in studying implicit cognition.