Randomization tests as alternative analysis methods for behavior‐analytic data

Randomization statistics offer alternatives to many of the statistical methods commonly used in behavior analysis and the psychological sciences, more generally. These methods are more flexible than conventional parametric and nonparametric statistical techniques in that they make no assumptions about the underlying distribution of outcome variables, are relatively robust when applied to small‐n data sets, and are generally applicable to between‐groups, within‐subjects, mixed, and single‐case research designs. In the present article, we first will provide a historical overview of randomization methods. Next, we will discuss the properties of randomization statistics that may make them particularly well suited for analysis of behavior‐analytic data. We will introduce readers to the major assumptions that undergird randomization methods, as well as some practical and computational considerations for their application. Finally, we will demonstrate how randomization statistics may be calculated for mixed and single‐case research designs. Throughout, we will direct readers toward resources that they may find useful in developing randomization tests for their own data.     

Analysis of categorical moderators in mixed‐effects meta‐analysis: Consequences of using pooled versus separate estimates of the residual between‐studies variances

Subgroup analyses allow us to examine the influence of a categorical moderator on the effect size in meta‐analysis. We conducted a simulation study using a dichotomous moderator, and compared the impact of pooled versus separate estimates of the residual between‐studies variance on the statistical performance of the Q _{B (P)} and Q _{B (S)} tests for subgroup analyses assuming a mixed‐effects model. Our results suggested that similar performance can be expected as long as there are at least 20 studies and these are approximately balanced across categories. Conversely, when subgroups were unbalanced, the practical consequences of having heterogeneous residual between‐studies variances were more evident, with both tests leading to the wrong statistical conclusion more often than in the conditions with balanced subgroups. A pooled estimate should be preferred for most scenarios, unless the residual between‐studies variances are clearly different and there are enough studies in each category to obtain precise separate estimates.     

Relationships between event‐related potentials and behavioral and scholastic measures of reading ability: A large‐scale,cross‐sectional study

In the cognitive, computational, neuropsychological, and educational literatures, it is established that children approach text in unique ways, and that even adult readers can differ in the strategies they bring to reading. In the developmental event‐related potential (ERP) literature, however, children with differing degrees of reading ability are, the majority of the time, placed in monolithic groups such as ‘normal’ and ‘dyslexic’ (e.g. Araújo et al., 2012) and analyzed only at the group level. This is likely done due to methodological concerns – such as low sample size or a lack of statistical power – that can make it difficult to perform analysis at the individual level. Here, we collected ERPs and behavior from > 100 children in grades pre‐K–7, as they read unconnected text silently to themselves. This large sample, combined with the statistical power of the Linear Mixed Effects Regression (LMER) technique, enables us to address individual differences in ERP component effects due to reading ability at an unprecedented level of detail. Results indicate that it is possible to predict reading‐related report card scores from ERP component amplitudes – especially that of the N250, a component pertaining to sublexical processing (including phonological decoding). Results also reveal relationships between behavioral measures of reading ability and ERP component effects that have previously been elusive, such as the relationship between vocabulary and N400 mean amplitude (cf. Henderson et al., 2011). We conclude that it is possible to meaningfully examine reading‐related ERP effects at the single subject level in developing readers, and that this type of analysis can provide novel insights into both behavior and scholastic achievement.     

A Monte Carlo method for comparing generalized estimating equations to conventional statistical techniques for discounting data

Discounting is the process by which outcomes lose value. Much of discounting research has focused on differences in the degree of discounting across various groups. This research has relied heavily on conventional null hypothesis significance tests that are familiar to psychologists, such as t‐tests and ANOVAs. As discounting research questions have become more complex by simultaneously focusing on within‐subject and between‐group differences, conventional statistical testing is often not appropriate for the obtained data. Generalized estimating equations (GEE) are one type of mixed‐effects model that are designed to handle autocorrelated data, such as within‐subject repeated‐measures data, and are therefore more appropriate for discounting data. To determine if GEE provides similar results as conventional statistical tests, we compared the techniques across 2,000 simulated data sets. The data sets were created using a Monte Carlo method based on an existing data set. Across the simulated data sets, the GEE and the conventional statistical tests generally provided similar patterns of results. As the GEE and more conventional statistical tests provide the same pattern of result, we suggest researchers use the GEE because it was designed to handle data that has the structure that is typical of discounting data.     

Properties of bootstrap tests for N‐of‐1 studies

N‐of‐1 study designs involve the collection and analysis of repeated measures data from an individual not using an intervention and using an intervention. This study explores the use of semi‐parametric and parametric bootstrap tests in the analysis of N‐of‐1 studies under a single time series framework in the presence of autocorrelation. When the Type I error rates of bootstrap tests are compared to Wald tests, our results show that the bootstrap tests have more desirable properties. We compare the results for normally distributed errors with those for contaminated normally distributed errors and find that, except when there is relatively large autocorrelation, there is little difference between the power of the parametric and semi‐parametric bootstrap tests. We also experiment with two intervention designs: ABAB and AB, and show the ABAB design has more power. The results provide guidelines for designing N‐of‐1 studies, in the sense of how many observations and how many intervention changes are needed to achieve a certain level of power and which test should be performed.     

Comparing self‐directed methods for training staff to create graphs using Graphpad Prism

An important skill for behavior analysts is creating graphs that clearly convey outcomes and conform to publication conventions. GraphPad Prism is software designed for creating scientific graphs, but no prior research has empirically evaluated training graphing skills using Prism. Two effective training methods are enhanced written instructions (EWI) and video modeling with voiceover instructions (VMVO), but no single‐subject studies have compared the effects of these methods. In this study, we compared the efficacy and social validity of EWI and VMVO for training staff to create graphs using Prism. A single‐subject design was employed to compare the effects of the methods on the individual performance of 11 graduate students. EWI and VMVO were both found to be effective, and more participants chose to use EWI.     

CREATING SINGLE‐SUBJECT DESIGN GRAPHS IN MICROSOFT EXCEL™ 2007

Over 10 years have passed since the publication of Carr and Burkholder's (1998) technical article on how to construct single‐subject graphs using Microsoft Excel. Over the course of the past decade, the Excel program has undergone a series of revisions that make the Carr and Burkholder paper somewhat difficult to follow with newer versions. The present article provides task analyses for constructing various types of commonly used single‐subject design graphs in Microsoft Excel 2007. The task analyses were evaluated using a between‐subjects design that compared the graphing skills of 22 behavior‐analytic graduate students using Excel 2007 and either the Carr and Burkholder or newly developed task analyses. Results indicate that the new task analyses yielded more accurate and faster graph construction than the Carr and Burkholder instructions.     

The Power to Detect and Predict Individual Differences in Intra-Individual Variability Using the Mixed-Effects Location-Scale Model

Our goal is to provide empirical scientists with practical tools and advice with which to test hypotheses related to individual differences in intra-individual variability using the mixed-effects location-scale model. To that end, we evaluate Type I error rates and power to detect and predict individual differences in intra-individual variability using this model and provide empirically-based guidelines for building scale models that include random and/or systematically-varying fixed effects. We also provide two power simulation programs that allow researchers to conduct a priori empirical power analyses. Our results aligned with statistical power theory, in that, greater power was observed for designs with more individuals, more repeated occasions, greater proportions of variance available to be explained, and larger effect sizes. In addition, our results indicated that Type I error rates were acceptable in situations when individual differences in intra-individual variability were not initially detectable as well as when the scale-model individual-level predictor explained all initially detectable individual differences in intra-individual variability. We conclude our paper by providing study design and model building advice for those interested in using the mixed-effects location-scale model in practice.     

Modeling Lag‐2 Revisits to Understand Trade‐Offs in Mixed Control of Fixation Termination During Visual Search

An important question about eye‐movement behavior is when the decision is made to terminate a fixation and program the following saccade. Different approaches have found converging evidence in favor of a mixed‐control account, in which there is some overlap between processing information at fixation and planning the following saccade. We examined one interesting instance of mixed control in visual search: lag‐2 revisits, during which observers fixate a stimulus, move to a different stimulus, and then revisit the first stimulus on the next fixation. Results show that the probability of lag‐2 revisits occurring increased with the number of target‐similar stimuli, and revisits were preceded by a brief fixation on the intervening distractor stimulus. We developed the Efficient Visual Sampling (EVS) computational model to simulate our findings (fixation durations and fixation locations) and to provide insight into mixed control of fixations and the perceptual, cognitive, and motor processes that produce lag‐2 revisits.     

Randomization procedures in single‐case intervention research contexts: (Some of) “the rest of the story”

Following up on articles recently published in this journal, the present contribution tells (some of) “the rest of the story” about the value of randomization in single‐case intervention research investigations. Invoking principles of internal, statistical‐conclusion, and external validity, we begin by emphasizing the critical distinction between design randomization and analysis randomization, along with the necessary correspondence between the two. Four different types of single‐case design‐and‐analysis randomization are then discussed. The persistent negative influence of serially dependent single‐case outcome observations is highlighted, accompanied by examples of inappropriate applications of parametric and nonparametric tests that have appeared in the literature. We conclude by presenting valid applications of single‐case randomization procedures in various single‐case intervention contexts, with specific reference to a freely available Excel‐based software package that can be accessed to incorporate the present randomization schemes into a wide variety of single‐case intervention designs and analyses.     

Multiple test batteries as predictors for pilot performance: A meta‐analytic investigation

Test Batteries (TBs) have a long history of use in pilot selection. The extent to which TBs predict future pilot performance has important implications. The existing pilot‐related psychometric meta‐analyses have focused primarily on scores of individual ability tests, rather than the combined scores composited from multiple ability tests. The objective of this study was to investigate the predictive validity of TBs' composite scores for several criteria of pilot performance. Informed by the Cattell–Horn–Carroll theory, we proposed a classification scheme of six categories representing the most common composite scores in selection assessment: Acquired Knowledge, Perceptual Processing, Motor Abilities, Controlled Attention, General Ability, and Work Sample. For overall pilot performance, based on 267 correlations from 118 independent samples, results showed that the six categories of TBs are valid predictors (Mean_r = .10–.34), and at least five of them have validity that is likely to generalize across selection contexts.     

Predictors of help‐seeking behavior: Distinction between help‐seeking intentions and help‐seeking behavior

The overall goals of this research were to: (a) examine whether help‐seeking intentions, subjective needs, depressive symptoms, and social support can predict actual help‐seeking behavior; and (b) clarify the moderating effects of social support on help‐seeking behavior using a longitudinal design. University students (N = 370) completed questionnaires that measured social support, subjective needs, depressive symptoms, and help‐seeking intentions during Time1, and questionnaires that measured actual help‐seeking behavior during Time2. Only subjective needs showed a positive effect on both help‐seeking intentions and actual help‐seeking behavior. Although depressive symptoms had a negative effect on help‐seeking intentions, they had a positive effect on actual help‐seeking behavior. Moreover, social support had a positive effect on help‐seeking intentions, and moderated the influence of subjective needs on actual help‐seeking behavior. Simple slope analysis indicated that subjective needs did not facilitate help‐seeking behavior among those with low levels of social support.     

Personality and team performance: a meta‐analysis

Using a meta‐analytical procedure, the relationship between team composition in terms of the Big‐Five personality traits (trait elevation and variability) and team performance were researched. The number of teams upon which analyses were performed ranged from 106 to 527. For the total sample, significant effects were found for elevation in agreeableness (ρ = 0.24) and conscientiousness (ρ = 0.20), and for variability in agreeableness (ρ = ?0.12) and conscientiousness (ρ = ?0.24). Moderation by type of team was tested for professional teams versus student teams. Moderation results for agreeableness and conscientiousness were in line with the total sample results. However, student and professional teams differed in effects for emotional stability and openness to experience. Based on these results, suggestions for future team composition research are presented. Copyright © 2006 John Wiley & Sons, Ltd.     

Simple Co‐Occurrence Statistics Reproducibly Predict Association Ratings

What determines human ratings of association? We planned this paper as a test for association strength (AS) that is derived from the log likelihood that two words co‐occur significantly more often together in sentences than is expected from their single word frequencies. We also investigated the moderately correlated interactions of word frequency, emotional valence, arousal, and imageability of both words (r's ≤ .3). In three studies, linear mixed effects models revealed that AS and valence reproducibly account for variance in the human ratings. To understand further correlated predictors, we conducted a hierarchical cluster analysis and examined the predictors of four clusters in competitive analyses: Only AS and word2vec skip‐gram cosine distances reproducibly accounted for variance in all three studies. The other predictors of the first cluster (number of common associates, (positive) point‐wise mutual information, and word2vec CBOW cosine) did not reproducibly explain further variance. The same was true for the second cluster (word frequency and arousal); the third cluster (emotional valence and imageability); and the fourth cluster (consisting of joint frequency only). Finally, we discuss emotional valence as an important dimension of semantic space. Our results suggest that a simple definition of syntagmatic word contiguity (AS) and a paradigmatic measure of semantic similarity (skip‐gram cosine) provide the most general performance‐independent explanation of association ratings.     

Same‐gender and same‐race bias in assessment center ratings: A rating error approach to understanding subgroup differences

This study investigated leniency and similar‐to‐me bias as mechanisms underlying demographic subgroup differences among assessees in assessors’ initial dimension ratings from three assessment center (AC) simulation exercises used as part of high‐stakes promotional testing. It examined whether even small individual‐level effects can accumulate (i.e., “trickle‐up”) to produce larger subgroup‐level differences. Individual‐level analyses were conducted using cross‐classified multilevel modeling and conducted separately for each exercise. Results demonstrated weak evidence of leniency toward White assessees and similar‐to‐me bias among non‐White assessee–assessor pairs. Similar leniency was found toward female assessees, but no statistically significant effects were found for assessee or assessor gender or assessee–assessor gender similarity. Using traditional d effect size estimates, weak individual level assessee effects translated into small but consistent subgroup differences favoring White and female assessees. Generally small but less consistent subgroup differences indicated that non‐White and male assessors gave higher ratings. Moreover, analyses of overall promotion decisions indicate the absence of adverse impact. Findings from this AC provide some support for the “trickle‐up” effect, but the effect on subgroup differences is trivial. The results counter recent reviews of AC studies suggesting larger than previously assumed subgroup differences. Consequently, the findings demonstrate the importance of following established best practices when developing and implementing the AC method for selection purposes to minimize subgroup differences.     

Reducing Alcohol‐Related Consequences Among High School Seniors: Efficacy of a Brief,Web‐Based Intervention

The authors examined the efficacy of a brief, web‐based personalized feedback intervention on reducing alcohol‐related consequences among high school seniors (N = 105) using a group‐randomized controlled design. Results of repeated measures mixed‐models analyses indicated significant intervention effects over time for alcohol‐related consequences at 30‐day and 6‐month follow‐up assessments. Drinking risk status moderated intervention effects such that results were only significant for high‐risk drinkers (i.e., students reporting initiation of heavy episodic drinking at baseline).     

Replicability and randomization test logic in behavior analysis

Randomization tests are a class of nonparametric statistics that determine the significance of treatment effects. Unlike parametric statistics, randomization tests do not assume a random sample, or make any of the distributional assumptions that often preclude statistical inferences about single‐case data. A feature that randomization tests share with parametric statistics, however, is the derivation of a p‐value. P‐values are notoriously misinterpreted and are partly responsible for the putative “replication crisis.” Behavior analysts might question the utility of adding such a controversial index of statistical significance to their methods, so it is the aim of this paper to describe the randomization test logic and its potentially beneficial consequences. In doing so, this paper will: (1) address the replication crisis as a behavior analyst views it, (2) differentiate the problematic p‐values of parametric statistics from the, arguably, more useful p‐values of randomization tests, and (3) review the logic of randomization tests and their unique fit within the behavior analytic tradition of studying behavioral processes that cut across species.     

Identification With All Humanity: The role of self‐definition and self‐investment

Identification With All Humanity (IWAH) relates to higher levels of concern and supportive behavior toward the disadvantaged, stronger endorsement of human rights, and stronger responses in favor of global harmony. So far, IWAH has been conceptualized as a one‐dimensional construct describing the degree with which one identifies with all humans as a superordinate ingroup. However, recent group identification models suggest a multi‐dimensional model to provide a more differentiated approach toward the understanding of the highest level of social identification. Using principal axis (Study 1) and confirmatory (Study 2) factor analyses, we suggest that IWAH sub‐divides into two dimensions—global self‐definition and global self‐investment. Study 2 revealed that global self‐investment was a stronger predictor for both convergent measures (e.g., social dominance orientation and authoritarianism) and behavioral intentions than global self‐definition. Finally, in Study 3, we manipulated IWAH to test its causal effect on donation behavior. Participants in the experimental condition, compared with the control condition, showed higher global self‐investment, which in turn predicted greater giving to global charity. These findings suggest that two dimensions with different behavioral outcomes underlie IWAH.     

Repeated,within‐session resurgence

Resurgence is a reliable, transient effect that only occasionally is replicated more than once within a single experiment or subject. In the present experiments, within‐session resurgence was generated repeatedly by dividing individual sessions into three phases (Training, Alternative‐Reinforcement, and Resurgence‐Test). In Experiments 1 and 2, resurgence reliably occurred in most of the 22‐30 daily sessions when responding was reinforced on, respectively, fixed‐ and variable‐interval schedules. Resurgence magnitude and duration did decrease across replications for some subjects, but not for others. To examine the utility of the procedure in studying the effects of an independent variable on resurgence, in Experiment 3 the effects of rich and lean baseline and alternative reinforcement rates on resurgence were compared. The target response was eliminated more rapidly, resurgence occurred more often, and usually was greater following rich alternative reinforcement rates. Resurgence was of greater magnitude when the baseline reinforcement rate was relatively lean compared to the alternative reinforcement rate. These experiments provide a reliable method for generating resurgence within individual sessions, instead of across multiple‐session conditions, that can be repeated over many successive sessions.     

A Comparison of General and Work‐specific Personality Measures as Predictors of Organizational Citizenship Behavior

The current study compared general and work‐specific measures of personality as predictors of organizational citizenship behavior (OCB). Consistent with the literature on frame‐of‐reference effects in personality assessment, two of the Five Factor Model dimensions – agreeableness and conscientiousness – were significantly related to OCB. Use of a frame of reference that is conceptually relevant to the criterion led to increased validity as a result of the decrement in between‐subject variability and within‐subject inconsistency. Results indicated that work‐specific personality yielded significant incremental relationships with OCB even after general personality is controlled. Finally, regression analyses found that the incremental variance predicted by work‐specific personality decreased as the degree of between‐subject variability and within‐subject inconsistency increased. Overall, the results suggest that there are benefits to considering the work‐specific measure of personality in the prediction of OCB.