The good, the bad, and the aggregate

To evaluate progress and focus goals, scientific disciplines need to identify relations that are robust across many situations. One approach is the literature review, which characterizes generality across studies. Some writers (e.g., Baron & Derenne, 2000) claim that quantitative literature reviews, but not narrative reviews, violate the methodological precepts of behavior analysis by pooling data from nonidentical studies. We argue that it is impossible to assess generality without varying the context in which relationships are studied. Properly chosen data-aggregation strategies can reveal which behavior-environment relations are general and which are procedure dependent. Within behavior analysis, reluctance to conduct quantitative reviews may reflect unsupported assumptions about the consequences of aggregating data across studies. Whether specific data-aggregation techniques help or harm a research program is an empirical issue that cannot be resolved by unstructured discussion. Some examples of how aggregation has been used in identifying behavior-environment relations are examined.     

Quantitative summaries of single-subject studies: What do group comparisons tell us about individual performances?

Kollins, Newland, and Critchfield (1999) responded to our comments about their review by arguing that their quantitative summary was not a meta-analysis and should not be criticized in these terms. We reply that regardless of what they call their review, it included confounding effects that make interpretations of the results problematic. Kollins et al. also argued that unexpected findings of the sort they reported can serve as a spur for further research. We reply that the understanding of findings that deviate from existing knowledge may well require empirical investigation. Such endeavors, however, should begin with an evaluation of the review procedures that suggested the existence of the differences. Finally, we emphasize that quantitative summaries of individual data are, in the end, a form of group comparison. The implications of using group methods to clarify individual data deserve frank recognition in discussions of the outcomes.     

Statistical inference in behavior analysis: Environmental determinants?

Use of inferential statistics should be based on the experimental question, the nature of the design, and the nature of the data. A hallmark of single-subject designs is that such statistics should not be required to determine whether the data answer the experimental question. Yet inferential statistics are being included more often in papers that purport to present data relevant to the behavior of individual organisms. The reasons for this too often seem to be extrinsic to the experimental analysis of behavior. They include lapses in experimental design and social pressure from colleagues who are unfamiliar with single-subject research. Regardless of whether inferential statistics are used, behavior analysts need to be sophisticated about experimental design and inferential statistics. Such sophistication not only will enhance design and analysis of behavioral experiments, but also will make behavior analysts more persuasive in presenting rationales for the use or nonuse of inferential statistics to the larger scientific community.     

Randomization tests in single-subject designs: Illustrative examples

Randomization tests have recently been adapted for use in the analysis of single-subject data. The advantages of these tests lie in their ease of implementation and interpretation as well as their freedom from underlying distributions. Even though numerous articles and books have explicated randomization test procedures, due to the lack of appropriate examples, very little use of these procedures has been made by applied behavior analysts. Data sets reported in a prominent applied behavior journal are used to demonstrate the application of randomization tests to the following three single-subject design models: (a) two-phase random intervention point, (b) multiple phase, and (c) multiple phase with a predicted order of effect size.     

A primer for using multilevel models to meta-analyze single case design data with AB phases

Meta-analytic methods provide a way to synthesize data across treatment evaluation studies. However, these well-accepted methods are infrequent with behavior analytic studies. Multilevel models may be a promising method to meta-analyze single-case data. This technical article provides a primer for how to conduct a multilevel model with single-case designs with AB phases using data from the differential-reinforcement-of-low-rate behavior literature. We provide details, recommendations, and considerations for searching for appropriate studies, organizing the data, and conducting the analyses. All data sets are available to allow the reader to follow along with this primer. The purpose of this technical article is to minimally equip behavior analysts to complete a meta-analysis that will summarize a current state of affairs as it relates to the science of behavior analysis and its practice. Moreover, we aim to demonstrate the value of analyses of this sort for behavior analysis.     

Statistical inference in behavior analysis: Experimental control is better

Statistical inference promises automatic, objective, reliable assessments of data, independent of the skills or biases of the investigator, whereas the single-subject methods favored by behavior analysts often are said to rely too much on the investigator's subjective impressions, particularly in the visual analysis of data. In fact, conventional statistical methods are difficult to apply correctly, even by experts, and the underlying logic of null-hypothesis testing has drawn criticism since its inception. By comparison, single-subject methods foster direct, continuous interaction between investigator and subject and development of strong forms of experimental control that obviate the need for statistical inference. Treatment effects are demonstrated in experimental designs that incorporate replication within and between subjects, and the visual analysis of data is adequate when integrated into such designs. Thus, single-subject methods are ideal for shaping-and maintaining-the kind of experimental practices that will ensure the continued success of behavior analysis.     

Evaluating treatment effects in single-subject behavioral experiments using quality-control charts

Quality-control charts can be particularly useful in identifying treatment effects and patterns of behaviors in single-subject behavior-analytic experiments that cannot be determined by visual inspection of their graphs. Using an example from the behavior analysis literature the quality-control charts identified the presence of treatment effects across phases as well as the presence of trends within and between phases. The ease of their calculations suggest use of them by behavior analysts whenever the effects of particular interventions are questionable.     

The validity and appropriateness of methods, analyses, and conclusions in Rind et al. (1998): A rebuttal of victimological critique from Ondersma et al. (2001) and Dallam et al. (2001).

The authors respond to 2 victimological critiques of their 1998 meta-analysis on child sexual abuse (CSA). S. J. Dallam et al. (2001) claimed that B. Rind, P. Tromovitch, and R. Bauserman (1998) committed numerous methodological and statistical errors, and often miscoded and misinterpreted data. The authors show all these claims to be invalid. To the contrary, they demonstrate frequent bias in Dallam et al.'s criticisms. S. J. Ondersma et al. (2001) claimed that Rind et al.'s study is part of a backlash against psychotherapists, that its suggestions regarding CSA definitions were extrascientific, and that the moral standard is needed to understand CSA scientifically. The authors show their suggestions to have been scientific and argue that it is Ondersma et al.'s issue-framing and moral standard that are extrascientific. This reply supports the original methods, analyses, recommendations, and conclusions of Rind et al.     

Data Smoothing: Prediction of Human Behavior, Detection of Behavioral Patterns, and Monitoring Treatment Effectiveness in Single-Subject Behavioral Studies

Data-smoothing can be particularly useful in predicting human behavior, detecting behavioral patterns, and monitoring treatment effectiveness in highly variable single-subject behavioral experiments that cannot be determined by only visual inspection of their graphs. Using an example from the applied behavior analytic literature, the use of moving-average and exponential data-smoothing aided the detection of the unique behavioral patterns of a child with autism across different treatments. Furthermore, the utility of the data-smoothing procedures to monitor and control the effectiveness of an intervention is discussed. The ease of their calculations suggest use of data-smoothing by behavior analysts whenever the effects of particular interventions are questionable.     

The distinction between positive and negative reinforcement: use with care

It is customary in behavior analysis to distinguish between positive and negative reinforcement in terms of whether the reinforcing event involves onset or offset of a stimulus. In a previous article (Baron & Galizio, 2005), we concluded that a distinction of these terms is not only ambiguous but has little if any functional significance. Here, we respond to commentaries by a group of distinguished behavior analysts about the issues we raised. Although several of the commentators argued for preservation of the distinction, we remain unconvinced that its benefits outweigh its weaknesses. Because this distinction is so deeply embedded in the language of behavior analysis, we hardly expect that it will be abandoned. However, we hope that the terms positive and negative reinforcement will be used with circumspection and with full knowledge of the confusion they can engender.     

Confirmatory Factor Analysis of Arnold et al.’s Parenting Scale Across Race, Age, and Sex

We used confirmatory factor analysis (CFA) to investigate the factor structure of several models of Arnold et al.’s Parenting Scale [Arnold et al. (1993). Psychological Assessment, 5, 137–144] across children from various age groups and races. Participants were parents of children (ages 2–16 years) presenting to four community-based pediatric practices for routine care. Parents completed questionnaires pertaining to various aspects of parenting and child behavior problems. Results indicated that a two-factor revision proposed by Reitman et al. [(2001). Journal of Clinical Child Psychology, 30, 514–524] represented the data well and better than the original three-factor structure and other two-factor models. Results from multigroup CFA analyses indicated that this factor structure did not vary across child sex, child age, and parental race. Results of validity analyses indicated that scores on both factors were related to reports of children’s behavior and parental affect and cognitions. This study was the first to use multigroup CFA procedures to demonstrate that relationships between individual items and factors of the Parenting Scale are similar for parents of children across various age groups.     

The importance of shared environmental influences in explaining the overlap between mother's parenting and sibling relationships: reply to Neale (1999)

This article addresses concerns raised by M. C. Neale (1999) in his commentary on the D. A. Bussell et al. (1999) Nonshared Environment in Adolescent Development (NEAD) study. These concerns fall into two categories: (a) model assumptions and sample design and (b) testing of alternative models. The validity of the assumptions of quantitative genetic models is a concern for all researchers in this area. Discussion of those assumptions in this reply is brief and focuses on those most relevant to the NEAD sample. The two alternative models proposed by Neale were designed to provide alternatives to the large shared environmental effect found in the original report of Bussell et al. Because these alternative models did not provide a better fit, the appropriateness of Bussell et al.'s basic model and the importance of shared environmental influences for explaining the association among family subsystems are supported.     

A topographical map approach to representing treatment efficacy: a focus on positive psychology interventions

A recent meta-analysis by Bolier et al. indicated that positive psychology interventions have overall small to moderate effects on well-being, but results were quite heterogeneous across intervention trials. Such meta-analytic research helps condense information on the efficacy of a broad psychosocial intervention by averaging across many effects; however, such global averages may provide limited navigational guidance for selecting among specific interventions. Here, we introduce a novel method for displaying qualitative and quantitative information on the efficacy of interventions using a topographical map approach. As an initial prototype for demonstrating this method, we mapped 50 positive psychology interventions targeting well-being (as captured in the Bolier et al. [2013] meta-analysis, [Bolier, L., Haverman, M., Westerhof, G. J., Riper, H., Smit, F., & Bohlmeijer, E. (2013). Positive psychology interventions: A meta-analysis of randomized controlled studies. BMC Public Health, 13, 83]). Each intervention domain/subdomain was mapped according to its average effect size (indexed by vertical elevation), number of studies providing effect sizes (indexed by horizontal area), and therapist/client burden (indexed by shading). The geographical placement of intervention domains/subdomains was determined by their conceptual proximity, allowing viewers to gauge the general conceptual “direction” in which promising intervention effects can be found. The resulting graphical displays revealed several prominent features of the well-being intervention “landscape,” such as more strongly and uniformly positive effects of future-focused interventions (including, goal-pursuit and optimism training) compared to past/present-focused ones.     

Selective Literature Review and Selective Data Analyses: Implications for the (Re)Analysis of Public Access Research Data

This article addresses concerns raised by Pashler et al. (2016/this issue) about the effect size and data of Study 3 of Chatterjee et al. (2013). The analysis of the original data of Study 3, corrected for coding errors, shows that cash priming is associated with cost-related words (Cohen’s d = 0.49) and effect sizes of all three studies are within the range reported in the relevant literature. While this article underscores the importance of public access to research data to strengthen research enterprise, it alerts researchers that the use of selective literature and partial analysis can lead to misleading inferences.     

Psychometric analysis of the systemizing quotient (SQ) scale

The psychometric properties of the systemizing quotient (SQ) developed by Baron‐Cohen (2003) are investigated in three studies. Furthermore, we examine the notion that the ability to systemize should be independent of intelligence. In Studies 1 and 2, confirmatory factor analyses are used to examine the factor structure of the SQ. Study 3 examines the relationship between systemizing, mental rotation and intelligence. Studies 1 and 2 indicate that the SQ does not possess a unifactorial structure but is best considered as four related factors; Study 3 found that SQ was not related to intelligence, although mental rotation was. A four factor structure using fewer items was a better fit for the data than either the original version of the SQ or Wakabayashi et al.'s (2006) revised version. Overall these results support Baron‐Cohen's view that SQ is not related to intelligence. Although mental rotation is correlated to SQ, it is not the main determinant of SQ. The problems of self‐report measures are discussed along with the difficulties related to measuring systemizing.     

Comparing instructor-led,video-model,and no-instruction control tutorials for creating single-subject graphs in Microsoft Excel: A systematic replication and extension

Visual inspection of single-subject data is the primary method for behavior analysts to interpret the effect of an independent variable on a dependent variable; however, there is no consensus on the most suitable method for teaching graph construction for single-subject designs. We systematically replicated and extended Tyner and Fienup (2015) using a repeated-measures between-subjects design to compare the effects of instructor-led, video-model, and no-instruction control tutorials on the graphing performance of 81 master's students with some reported Microsoft Excel experience. Our mixed-design analysis revealed a statistically significant main effect of pretest, tutorial, and posttest submissions for each tutorial group and a nonsignificant main effect of tutorial group. Tutorial group significantly interacted with submissions, suggesting that both instructor-led and video-model tutorials may be superior to providing graduate students with a written list of graphing conventions (i.e., control condition). Finally, training influenced performance on an untrained graph type (multielement) for all tutorial groups.     

Regression toward the mean or heuristic processing in detecting deception? Reply to Elaad (2010)

Masip, et al. (2009) conducted a study in which observers had to make truth-lie judgments at the beginning, middle, or end of a series of videotaped statements. They found a decline in truth judgments over time and explained this finding in terms of information processing mode. Recently, Elaad (2010) challenged this explanation and contended that the decline could be a result of regression toward the mean. In the present paper, it is argued that because Masip, et al. took multiple Moment 1 judgments over time and then averaged across judgments, regression toward the mean was extremely unlikely. Furthermore, the decrease in truth judgments was found under several separate conditions; this cannot be explained by random fluctuations alone. Finally, Masip, et al.'s data were re-analyzed adjusting for the effects of regression toward the mean. The outcomes of these analyses were the same as those reported in the original article.     

The analyst's participation: a new look

Over the past fifteen years or so, advocates of a relational theory of psychoanalytic process have developed a compelling challenge to the classical approach to clinical work. Their critique of a fixed "standard technique," applicable across the board to all analyzable patients, has been particularly effective. The new approach opens the possibility of tailoring technique to individual analysands, negotiating the best way of working within each unique analytic dyad. But despite the openness of relational theory, many of the most influential clinical vignettes in the recent literature emphasize the analyst's risk-taking, engaging patients in a highly personal way that breaks the traditional analytic frame. Various implications of the tendency of relational analysts to emphasize this sort of intervention are discussed, and questions raised about the way this may affect how relational thinking is received.     

Assessment of the functions of vocal behavior in children with developmental disabilities: a replication

Although experimental analysis methodologies have been useful for identifying the function of a wide variety of target behaviors (e.g., Iwata, Dorsey, Slifer, Bauman, & Richman, 1982/1994), only recently have such procedures been applied to verbal operants (Lerman et al., 2005). In the current study, we conducted a systematic replication of the methodology developed by Lerman et al. Participants were 4 children who had been diagnosed with developmental disabilities and who engaged in limited vocal behavior. The function of vocal behavior was assessed by exposing target vocal responses to experimental analyses. Results showed that experimental analyses were generally useful for identifying the functions of vocal behavior across all participants.     

CREATING SINGLE-SUBJECT DESIGN GRAPHS WITH MICROSOFT EXCEL™

Because single-subject design methodology is the cornerstone of behavior-analytic research, behavior analysts are often engaged in the practice of graph production. There are a variety of computer software packages on the market that can be used to create graphs; however, not all of them are suitable for producing single-subject design line graphs. Microsoft Excel 97 for Windows 95/NT and MacOS operating systems contains a variety of features that are conducive to producing reversal, multiple baseline, and multielement graphs (among others) that conform to many of the technical recommendations for publication in the Journal of Applied Behavior Analysis. We present detailed instructions on creating these graphs using Microsoft Excel.