Estimating and testing autocorrelation with small samples: A comparison of the c-statistic to a modified estimator

Huitema and McKean (Psychological Bulletin, 110, 291–304, 1991) recently showed, in a Monte-Carlo study, that five conventional estimators of first-order autocorrelation perform poorly for small (< 50) sample sizes. They suggested a modified estimator and a test for autocorrelation. We examine an estimator not considered by Huitema and McKean: the C-statistic (Young, Annals of Mathematical Statistics, 12, 293–300, 1941). A Monte-Carlo study of the small sample properties of the C-statistic shows that it performs as well or better than the modified estimator suggested by Huitema and McKean (1991). The C-statistic is also shown to be closely related to the d-statistic of the widely used Durbin-Watson test.     

Bayesian estimates of autocorrelations in single-case designs

Researchers in the single-case design tradition have debated the size and importance of the observed autocorrelations in those designs. All of the past estimates of the autocorrelation in that literature have taken the observed autocorrelation estimates as the data to be used in the debate. However, estimates of the autocorrelation are subject to great sampling error when the design has a small number of time points, as is typically the situation in single-case designs. Thus, a given observed autocorrelation may greatly over- or underestimate the corresponding population parameter. This article presents Bayesian estimates of the autocorrelation that greatly reduce the role of sampling error, as compared to past estimators. Simpler empirical Bayes estimates are presented first, in order to illustrate the fundamental notions of autocorrelation sampling error and shrinkage, followed by fully Bayesian estimates, and the difference between the two is explained. Scripts to do the analyses are available as supplemental materials. The analyses are illustrated using two examples from the single-case design literature. Bayesian estimation warrants wider use, not only in debates about the size of autocorrelations, but also in statistical methods that require an independent estimate of the autocorrelation to analyze the data.     

Reconsidering the use of the general linear model with single-case data

Using a low point estimate of autocorrelation to justify analyzing single-case data with the general linear model (GLM) is questioned. Monte Carlo methods are used to examine the degree to which bias in the estimate of autocorrelation depends on the complexity of the linear model used to describe the data. A method is then illustrated for determining the range of autocorrelation parameters that could reasonably have led to the observed autocorrelation. The argument for using a GLM analysis can be strengthened when the GLM analysis functions appropriately across the range of plausible autocorrelations. For situations in which the GLM analysis does not function appropriately across this range, a method is provided for adjusting the confidence intervals to ensure adequate coverage probabilities for specified levels of autocorrelation.     

The Origins of Religious Homophily in a Medium and Large School

How religion influences social interactions, and how social interactions influence religion, are fundamental questions to the sociology of religion. We address these processes and build on Cheadle and Schwadel’s (Soc Sci Res 41:1198–1212, 2012) analysis of selection and influence in religion-based social tie homogeneity (i.e. network–religion autocorrelation) in small schools by analyzing networks from larger schools, by focusing on differences across schools, and by testing different operationalizations of social influence. Using two waves of full network data from the National Longitudinal Study of Adolescent Health and dynamic longitudinal network SIENA models, we find (1) that both selection and influence impact network–religion autocorrelation; (2) that the factors influencing network–religion autocorrelation vary across school contexts; and (3) that religious influence is proportional to the number of friends in an adolescent’s network, which means influence reflects both the size of an individual’s network and the consistency of religion among members of the network. We conclude by addressing potential reasons for differences across school contexts and by discussing the theoretical logic behind the total similarity effect that best operationalized religious influence.     

Sequential effects in successive ratio estimation

Sequential effects are examined in four successive ratio estimation (RE) experiments. The procedure in successive RE is identical to that for magnitude estimation (ME), but the task in successive RE is to estimate the ratio of the current to the previous sensation magnitude, and not the separate magnitudes of the sensations. A positive stimulus context effect was found in successive RE for several continua, in agreement with results previously found for ME. The residual autocorrelation for successive RE was zero in many cases, but in some cases negative autocorrelation was found, which is in contrast to the positive autocorrelation that is typically found for ME and other magnitude scaling procedures. It is shown that, when the role of perceptual error is recognized, negative autocorrelation is predicted by a classical model of ratio estimation. Some aspects of response bias are also discussed.     

Testing autocorrelation and partial autocorrelation: Asymptotic methods versus resampling techniques

Autocorrelation and partial autocorrelation, which provide a mathematical tool to understand repeating patterns in time series data, are often used to facilitate the identification of model orders of time series models (e.g., moving average and autoregressive models). Asymptotic methods for testing autocorrelation and partial autocorrelation such as the 1/T approximation method and the Bartlett's formula method may fail in finite samples and are vulnerable to non-normality. Resampling techniques such as the moving block bootstrap and the surrogate data method are competitive alternatives. In this study, we use a Monte Carlo simulation study and a real data example to compare asymptotic methods with the aforementioned resampling techniques. For each resampling technique, we consider both the percentile method and the bias-corrected and accelerated method for interval construction. Simulation results show that the surrogate data method with percentile intervals yields better performance than the other methods. An R package pautocorr is used to carry out tests evaluated in this study.     

Abstract: A Meta-Analysis of the Autocorrelation in Single Case Designs

Single case design (SCD) experiments in the behavioral sciences utilize just one participant from whom data is collected over time. This design permits causal inferences to be made regarding various intervention effects, often in clinical or educational settings, and is especially valuable when between-participant designs are not feasible or when interest lies in the effects of an individualized treatment. Regression techniques are the most common quantitative practice for analyzing time series data and provide parameter estimates for both treatment and trend effects. However, the presence of serially correlated residuals, known as autocorrelation, can severely bias inferences made regarding these parameter estimates. Despite the severity of the issue, few researchers test or correct for the autocorrelation in their analyses.

Shadish and Sullivan (in press) recently conducted a meta-analysis of over 100 studies in order to assess the prevalence of the autocorrelation in the SCD literature. Although they found that the meta-analytic weighted average of the autocorrelation was close to zero, the distribution of autocorrelations was found to be highly heterogeneous. Using the same set of SCDs, the current study investigates various factors that may be related to the variation in autocorrelation estimates (e.g., study and outcome characteristics). Multiple moderator variables were coded for each study and then used in a metaregression in order to estimate the impact these predictor variables have on the autocorrelation.

This current study investigates the autocorrelation using a multilevel meta-analytic framework. Although meta-analyses involve nested data structures (e.g., effect sizes nested within studies nested within journals), there are few instances of meta-analysts utilizing multilevel frameworks with more than two levels. This is likely attributable to the fact that very few software packages allow for meta-analyses to be conducted with more than two levels and those that do allow this provide sparse documentation on how to implement these models. The proposed presentation discusses methods for carrying out a multilevel meta-analysis. The presentation also discusses the findings from the metaregression on the autocorrelation and the implications these findings have on SCDs.     

Stripes within words affect reading

In a series of eight studies it is shown that the first peak in the horizontal autocorrelation of the image of a word (which captures the similarity in shape between the neighbouring strokes of letters) determines (i) the appearance of the words as striped; (ii) the speed with which the words are read, both aloud and silently; and (iii) the speed with which a paragraph of text can be searched. By subtly distorting the horizontal dimension of text, and thereby reducing the first peak in the horizontal autocorrelation, it is shown that the speed of word recognition can be increased. The increase in speed is greater in poor readers.     

An experimentally confirmed statistical model on arm movement

The purpose of this research work was to develop a methodology to model arm movement in normal subjects and neurologically impaired individuals through the application of a statistical modelling method. Thirteen subjects with Parkinson's disease and 29 normal controls were recruited to participate in an arm motor task. An infrared optoelectronic kinematic movement analysis system was employed to record arm movement at 50 times per second. This study identified the modified extended Freundlich model as one that could be used to describe this task. Results showed that this model fit the data well and that it has a good correspondence between the observed and the predicted data. However, verification of the model showed that the residuals contained a sizeable autocorrelation factor. The Cochrane and Orcutt method was applied to remove this factor, which improved the fit of the model. Results showed that Parkinson's disease subjects had a higher autocorrelation coefficient than the normal subjects for this task. A positive correlation (r(s) = 0.72, p < 0.001) was found between the Langton-Hewer stage and the autocorrelation coefficient of PD subjects. This finding suggests that if autocorrelation is positively correlated with disease progression, clinicians in their clinical practice might use the autocorrelation value as a useful indicator to quantify the progression of a subjects' disease. Significant differences in model parameters were seen between normal and Parkinson's disease subjects. The use of such a model to represent and quantify movement patterns provides an important base for future study.     

Is Human Behavior Autocorrelated? An Empirical Analysis

The current review and analysis investigated the presence of serial dependency (or autocorrelation) in single-subject applied behavior-analytic research. While well researched, few studies have controlled for the number of data points that appeared in the time-series and, thus, the negative bias of the r coefficient, and the power to detect true serial dependency effects. Therefore, all baseline graphs that appeared in the Journal of Applied Behavior Analysis (JABA) between 1968 and 1993 that provided more than 30 data points were examined for the presence of serial dependency (N = 103). Results indicated that 12% of the baseline graphs provided a significant lag-1 autocorrelation, and over 83% of them had coefficient values less than or equal to (±.25). The distribution of the lag-1 autocorrelation coefficients had a mean of .10. Subsequent distributions of partial autocorrelations at lags two through seven had smaller means indicating that as the distance between observations increases (i.e., the lag), serial dependency decreased. Although serial dependency did not appear to be a common property of the single-subject behavioral experiments, it is recommended that, whenever statistical analyses are contemplated, its presence should always be examined. Alternatives for coping with the presence of significant levels of serial dependency were discussed in terms of: (a) using alternative statistical procedures (e.g., ARIMA models, randomization tests, the Shewhart quality-control charts); (b) correcting statistics of traditional parametric procedures (e.g., t, F); or (c) using the autocorrelation coefficient as an indicator and estimate of reliable intervention effects.     

Inferential Precision in Single-Case Time-Series Data Streams: How Well Does the EM Procedure Perform When Missing Observations Occur in Autocorrelated Data?

The case-based time-series design is a viable methodology for treatment outcome research. However, the literature has not fully addressed the problem of missing observations with such autocorrelated data streams. Mainly, to what extent do missing observations compromise inference when observations are not independent? Do the available missing data replacement procedures preserve inferential integrity? Does the extent of autocorrelation matter? We use Monte Carlo simulation modeling of a single-subject intervention study to address these questions. We find power sensitivity to be within acceptable limits across four proportions of missing observations (10%, 20%, 30%, and 40%) when missing data are replaced using the Expectation-Maximization Algorithm, more commonly known as the EM Procedure (Dempster, Laird, &; Rubin, 1977). This applies to data streams with lag-1 autocorrelation estimates under 0.80. As autocorrelation estimates approach 0.80, the replacement procedure yields an unacceptable power profile. The implications of these findings and directions for future research are discussed.     

Evaluating effects in short time series: alternative models of analysis

In the applied context, short time-series designs are suitable to evaluate a treatment effect. These designs present serious problems given autocorrelation among data and the small number of observations involved. This paper describes analytic procedures that have been applied to data from short time series, and an alternative which is a new version of the generalized least squares method to simplify estimation of the error covariance matrix. Using the results of a simulation study and assuming a stationary first-order autoregressive model, it is proposed that the original observations and the design matrix be transformed by means of the square root or Cholesky factor of the inverse of the covariance matrix. This provides a solution to the problem of estimating the parameters of the error covariance matrix. Finally, the results of the simulation study obtained using the proposed generalized least squares method are compared with those obtained by the ordinary least squares approach. The probability of Type I error associated with the proposed method is close to the nominal value for all values of rho1 and n investigated, especially for positive values of rho1. The proposed generalized least squares method corrects the effect of autocorrelation on the test's power.     

Testing the significance of interobserver agreement measures in the presence of autocorrelation: The jackknife procedure

Users of interobserver agreement statistics have heretofore ignored the problem of autocorrelation in behavior sequences when testing the statistical significance of agreement measures. Due to autocorrelation traditional reliability tests based on the 2 × 2 contingency-table model (e.g., kappa, phi) are incorrect. Correct tests can be developed by using the bivariate time series as a statistical model. Seen from this perspective, testing the significance of interobserver agreement becomes formally equivalent to testing the significance of the lag-zero cross-correlation between two time series. The robust procedure known as the jackknife is suggested for this purpose.     

Detrended windowed (lag one) autocorrelation: a new method for distinguishing between event-based and emergent timing

The aim of this study was to test different methods for distinguishing between two known timing processes involved in human rhythmic behaviours. We examined the implementation of two approaches used in the literature: the high-frequency slope of the power spectrum and the lag one value of the autocorrelation function, ACF(1). We developed another method based on the Wing and Kristofferson (1973a) model and the predicted negative ACF(1) for event-based series: the detrended windowed (lag one) autocorrelation (DWA). We compared the reliability and performance of these three methods on simulation and experimental series. DWA gave the best results, and guidelines are given for its appropriate use for identifying underlying timing processes.     

Small samples do not cause greater accuracy--but clear data may cause small samples: comment on Fiedler and Kareev (2006)

Fiedler and Kareev (2006) have claimed that taking a small sample of information (as opposed to a large one) can, in certain specific situations, lead to greater accuracy--beyond that gained by avoiding fatigue or overload. Specifically, they have argued that the propensity of small samples to provide more extreme evidence is sufficient to create an accuracy advantage in situations of high caution and uncertainty. However, a close examination of Fiedler and Kareev's experimental results does not reveal any strong reason to conclude that small samples can cause greater accuracy. We argue that the negative correlation between sample size and accuracy that they reported (found only for the second half of Experiment 1) is also consistent with mental fatigue and that their data in general are consistent with the causal structure opposite to the one they suggest: Rather than small samples causing clear data, early clear data may cause participants to stop sampling. More importantly, Experiment 2 provides unequivocal evidence that large samples result in greater accuracy; Fiedler and Kareev only found a small sample advantage here when they artificially reduced the data set. Finally, we examine the model that Fiedler and Kareev used; they surmised that decision makers operate with a fixed threshold independent of sample size. We discuss evidence for an alternative (better performing) model that incorporates a dynamic threshold that lowers with sample size. We conclude that there is no evidence currently to suggest that humans benefit from taking a small sample, other than as a tactic for avoiding fatigue, overload, and/or opportunity cost-that is, there is no accuracy advantage inherent to small samples.     

A Mixed Model to Disentangle Variance and Serial Autocorrelation in Affective Instability Using Ecological Momentary Assessment Data

Affective instability, the tendency to experience emotions that fluctuate frequently and intensively over time, is a core feature of several mental disorders including borderline personality disorder. Currently, affect is often measured with Ecological Momentary Assessment protocols, which yield the possibility to quantify the instability of affect over time. A number of linear mixed models are proposed to examine (diagnostic) group differences in affective instability. The models contribute to the existing literature by estimating simultaneously both the variance and serial dependency component of affective instability when observations are unequally spaced in time with the serial autocorrelation (or emotional inertia) declining as a function of the time interval between observations. In addition, the models can eliminate systematic trends, take between subject differences into account and test for (diagnostic) group differences in serial autocorrelation, short-term as well as long-term affective variability. The usefulness of the models is illustrated in a study on diagnostic group differences in affective instability in the domain of eating disorders. Limitations of the model are that they pertain to group (and not individual) differences and do not focus explicitly on circadian rhythms or cycles in affect.     

Poisson Multilevel Models with Small Samples

Recent methodological studies have investigated the properties of multilevel models with small samples. Previous work has primarily focused on continuous outcomes and little attention has been paid to count outcomes. The estimation of count outcome models can be difficult because the likelihood has no closed-form solution, meaning that approximation methods are required. Although adaptive Gaussian quadrature (AGQ) is generally seen as the gold standard, its comparative performance has been investigated with larger samples. AGQ approximates the full likelihood, a function that is known to produce biased estimates with small samples with continuous outcomes. Conversely, penalized quasi-likelihood (PQL) is considered to be a less desirable approximation; however, it can approximate the restricted likelihood function, a function that is known to perform well with smaller samples with continuous outcomes. The goal of this paper is to compare the small sample bias of full likelihood methods to the linearization bias of PQL with restricted likelihood. Simulation results indicate that the linearization bias of PQL is preferable to the finite sample bias of AGQ with smaller samples.     

Increased reliability for single-case research results: is the bootstrap the answer?

There is need for objective and reliable single-case research (SCR) results in the movement toward evidence-based interventions (EBI), for inclusion in meta-analyses, and for funding accountability in clinical contexts. Yet SCR deals with data that often do not conform to parametric data assumptions and that yield results of low reliability. A resampling technique, the bootstrap, largely bypasses statistical assumptions and usually yields more reliable results. This study answers questions about the extent of need for the bootstrap in SCR and its impact on effect size reliability. The bootstrap was applied in Allison et al. mean shift analyses (Faith, Allison, & Gorman, 1997) to data from 166 published AB graphs. Results showed the bootstrap improved reliability of 88% of the analyses and reduced reliability of only 3%. The reliability improvement was large enough to be practically useful. The bootstrap was paired with a method for cleansing data of autocorrelation, which also proved effective. Pending replication, the findings encourage broad application within SCR of both the bootstrap and autocorrelation cleansing.     

Visual and auditory choice reaction times

Choice reaction times are measured for three values of a priori signal probability with three well-practiced observers. Two sets of data are taken with the only difference being the modality of the reaction signal. In one set of conditions it is auditory, in the other, visual. The auditory reaction times are faster than the visual and in addition several other differences are noted. The latency of the errors and correct responses are nearly equal for the auditory data. Error latencies are nearly 30% faster for the visual data. Non-stationary effects, autocorrelation between successive latencies and non-homogeneous distribution of errors, are clearly evident in the visual data, but are small or non-existent in the auditory data. The data are compared with several models of the choice reaction time process but none of the models is completely adequate.     

An application of a dynamic model of judgment to magnitude production

A dynamic model of judgment, together with a model of stimulus context effects, is applied to magnitude production (MP) and magnitude estimation (ME) experiments. Participants' responses in MP were correlated across trials, as is typically found for ME. The magnitudeof the autocorrelation, however, was small, which suggests that participants in MP tend to rely more heavily on a long-term frame of reference. Second, a stimulus context effect found for ME did not appear for MP, most likely because of the different nature of the task (i.e., intermediate values of the stimulus were heard while the participant produced a response). A fit of an earlier regression model, on the other hand, suggests that the number presented on the previous trial in MP has a large contrastive effect on the current response.The present model offers a different view of this result, in that it shows that a negative coefficient for the earlier model is consistent with a positive judgmental effect. The regression effect noted by Stevens and Greenbaum (1966), which is a value of the estimated ME exponent that is smaller than the inverse of the estimated MP exponent, was also found; it i s shown that the effect did not arise from bias in estimation.