Analyzing single-case data with visually guided randomization tests

The valid use of a randomization test with single-case data typically involves the choice of a test statistic prior to gathering data. In situations in which it is difficult to anticipate the form of the effect, the early specification of the test statistic can be problematic. The purpose of this paper is to demonstrate methods that will allow single-case researchers to conduct valid randomization tests in situations in which they wish to delay the specification of the test statistic until the data have been observed. A concrete example is given to motivate the use, explicate the logic, and demonstrate the conduct of this method.     

Empirical bayes estimation of coefficients in the general linear model from data of deficient rank

Empirical Bayes methods are shown to provide a practical alternative to standard least squares methods in fitting high dimensional models to sparse data. An example concerning prediction bias in educational testing is presented as an illustration.The authors would like to thank the referees for several useful comments.The analysis of the data discussed in this report was part of a study funded jointly by the Graduate Management Admission Council and Educational Testing Service.     

Sequential meta-analysis of single-case experimental data

In this article, sequential meta-analysis is presented as a method for determining the sufficiency of cumulative knowledge in single-case research synthesis. Sufficiency addresses the question of whether there is enough cumulative knowledge on a topic to yield convincing statistical evidence. The method combines cumulative meta-analysis of single-case experimental data with formal sequential testing. After describing the underlying statistical techniques, a strategy for conducting a sequential single-case meta-analysis is illustrated using a real meta-analytic database. The sequential methodology may serve as a valuable tool for behavioral researchers to guide them in making optimal use of limited resources.     

A general model for the analysis of multilevel data

A general model is developed for the analysis of multivariate multilevel data structures. Special cases of the model include repeated measures designs, multiple matrix samples, multilevel latent variable models, multiple time series, and variance and covariance component models.We would like to acknowledge the helpful comments of Ruth Silver. We also wish to thank the referees for helping to clarify the paper. This work was partly carried out with research funds provided by the Economic and Social Research Council (U.K.).     

A general approach to categorical data analysis with missing data,using generalized linear models with composite links

A general approach for analyzing categorical data when there are missing data is described and illustrated. The method is based on generalized linear models with composite links. The approach can be used (among other applications) to fill in contingency tables with supplementary margins, fit loglinear models when data are missing, fit latent class models (without or with missing data on observed variables), fit models with fused cells (including many models from genetics), and to fill in tables or fit models to data when variables are more finely categorized for some cases than others. Both Newton-like and EM methods are easy to implement for parameter estimation.The author thanks the editor, the reviewers, Laurie Hopp Rindskopf, and Clifford Clogg for comments and suggestions that substantially improved the paper.     

Hierarchical linear models for the quantitative integration of effect sizes in single-case research

In this article, the calculation of effect size measures in single-case research and the use of hierarchical linear models for combining these measures are discussed. Special attention is given to meta-analyses that take into account a possible linear trend in the data. We show that effect size measures that have been proposed for this situation appear to be systematically affected by the duration of the experiment and fail to distinguish between effects on level and slope. To avoid these flaws, we propose to perform a multivariate meta-analysis on the standardized ordinary least squares regression coefficients from the study-specific regression equations describing the response variable.     

The impact of ordinate scaling on the visual analysis of single-case data

Visual analysis is the primary method for detecting the presence of treatment effects in graphically displayed single-case data and it is often referred to as the “gold standard.” Although researchers have developed standards for the application of visual analysis (e.g., Horner et al., 2005), over- and underestimation of effect size magnitude is not uncommon among analysts. Several characteristics have been identified as potential contributors to these errors; however, researchers have largely focused on characteristics of the data itself (e.g., autocorrelation), paying less attention to characteristics of the graphic display which are largely in control of the analyst (e.g., ordinate scaling). The current study investigated the impact that differences in ordinate scaling, a graphic display characteristic, had on experts' accuracy in judgments regarding the magnitude of effect present in single-case percentage data. 32 participants were asked to evaluate eight ABAB data sets (2 each presenting null, small, moderate, and large effects) along with three iterations of each (32 graphs in total) in which only the ordinate scale was manipulated. Results suggest that raters are less accurate in their detection of treatment effects as the ordinate scale is constricted. Additionally, raters were more likely to overestimate the size of a treatment effect when the ordinate scale was constricted.     

Regularization and a general linear model for event-related potential estimation

The usual event-related potential (ERP) estimation is the average across epochs time-locked on stimuli of interest. These stimuli are repeated several times to improve the signal-to-noise ratio (SNR) and only one evoked potential is estimated inside the temporal window of interest. Consequently, the average estimation does not take into account other neural responses within the same epoch that are due to short inter stimuli intervals. These adjacent neural responses may overlap and distort the evoked potential of interest. This overlapping process is a significant issue for the eye fixation-related potential (EFRP) technique in which the epochs are time-locked on the ocular fixations. The inter fixation intervals are not experimentally controlled and can be shorter than the neural response’s latency. To begin, the Tikhonov regularization, applied to the classical average estimation, was introduced to improve the SNR for a given number of trials. The generalized cross validation was chosen to obtain the optimal value of the ridge parameter. Then, to deal with the issue of overlapping, the general linear model (GLM), was used to extract all neural responses inside an epoch. Finally, the regularization was also applied to it. The models (the classical average and the GLM with and without regularization) were compared on both simulated data and real datasets from a visual scene exploration in co-registration with an eye-tracker, and from a P300 Speller experiment. The regularization was found to improve the estimation by average for a given number of trials. The GLM was more robust and efficient, its efficiency actually reinforced by the regularization.     

Reconsidering evidence for the suppression model of the octave illusion

The octave illusion is elicited by a sequence of tones presented to each ear that continuously alternate in frequency by one octave, but with high and low frequencies always in different ears. The percept for most listeners is a high pitch in one ear, alternating with a low pitch in the other ear. The influentialsuppression model of the illusion proposed by Deutsch and Roll (1976) carries three postulates: first, that listeners perceive only the pitch of the tones presented to their dominant ear; second, that this pitch is heard in whichever ear received the higher frequency tone; and third, that this apparent dissociation betweenwhat andwhere mechanisms arises from sequential interactions between the tones. In the present article, we reappraise evidence for the suppression model and demonstrate (1) the incompatibility of the theory with the existing literature on pitch perception, sound localization, and ear dominance and (2) methodological limitations in studies that have claimed to provide support for the suppression model. We conclude by proposing an alternative theory of the octave illusion that is based on established principles of fusion, rather than suppression, between ears.     

A general diagnostic model applied to language testing data

Probabilistic models with one or more latent variables are designed to report on a corresponding number of skills or cognitive attributes. Multidimensional skill profiles offer additional information beyond what a single test score can provide, if the reported skills can be identified and distinguished reliably. Many recent approaches to skill profile models are limited to dichotomous data and have made use of computationally intensive estimation methods such as Markov chain Monte Carlo, since standard maximum likelihood (ML) estimation techniques were deemed infeasible. This paper presents a general diagnostic model (GDM) that can be estimated with standard ML techniques and applies to polytomous response variables as well as to skills with two or more proficiency levels. The paper uses one member of a larger class of diagnostic models, a compensatory diagnostic model for dichotomous and partial credit data. Many well‐known models, such as univariate and multivariate versions of the Rasch model and the two‐parameter logistic item response theory model, the generalized partial credit model, as well as a variety of skill profile models, are special cases of this GDM. In addition to an introduction to this model, the paper presents a parameter recovery study using simulated data and an application to real data from the field test for TOEFL^® Internet‐based testing.     

A general linear model for estimating effect size in the presence of publication bias

When the process of publication favors studies with smallp-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailedp-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.Authors' note: The contributions of the authors are considered equal, and the order of authorship was chosen to be reverse-alphabetical.     

The impact of baseline trend control on visual analysis of single-case data

The impact of baseline trend control on visual analyses of AB intervention graphs was examined with simulated data at various values of baseline trend, autocorrelation, and effect size. Participants included 202 undergraduate students with minimal training in visual analysis and 10 graduate students and faculty with more training and experience in visual analysis. In general, results were similar across both groups of participants. Without statistical adjustments to correct for baseline trend, Type I errors greatly increased as baseline trend increased. With corrections for baseline trend, fewer Type I errors were made. As trend increased, participants made fewer Type II errors on the unadjusted graphs as compared to the graphs with baseline trend control. The greater Type II error rate on adjusted graphs could be an artifact of study design (i.e., participants did not know if baseline trend control had been applied), and the impact of MASAJ on Type II errors needs to be explored in detail prior to more widespread use of the method. Implications for future use of baseline trend control techniques by educational professionals are discussed.     

A reformulation of the general Euclidean model for the external analysis of preference data

This paper discusses least squares methods for fitting a reformulation of the general Euclidean model for the external analysis of preference data. The reformulated subject weights refer to a common set of reference vectors for all subjects and hence are comparable across subjects. If the rotation of the stimulus space is fixed, the subject weight estimates in the model are uniquely determined. Weight estimates can be guaranteed nonnegative. While the reformulation is a metric model for single stimulus data, the paper briefly discusses extensions to nonmetric, pairwise, and logistic models. The reformulated model is less general than Carroll's earlier formulation.The author is grateful to Christopher J. Nachtsheim for his helpful suggestions.     

Evaluating single-case research data for systematic review: A commentary for the special issue

The purpose of this commentary is to provide observation on the statistical procedures described throughout this special section from the perspective of researchers with experience in conducting systematic reviews and meta-analyses of single-case research to address issues of evidence-based practice. It is our position that both visual and statistical analyses are complimentary methods for evaluating single-case research data for these purposes. Given the recent developments regarding the use of single-case research to inform evidence-based practice and policy, the developments described in the present issue will be contextualized within the need for a widely accepted process for data evaluation to assist with extending the impact of single-case research. The commentary will, therefore, begin with providing an overview of the conceptual underpinnings of a systematic review of single-case research and will be followed by a discussion of several features that are essential to the development of a conceptually sound and widely used statistical procedure for single-case research. The commentary will conclude with recommendations and guidelines for the use of both visual and statistical analyses within primary research reports and recommendations for future research.     

Using the linear mixed model to analyze nonnormal data distributions in longitudinal designs

Using a Monte Carlo simulation and the Kenward–Roger (KR) correction for degrees of freedom, in this article we analyzed the application of the linear mixed model (LMM) to a mixed repeated measures design. The LMM was first used to select the covariance structure with three types of data distribution: normal, exponential, and log-normal. This showed that, with homogeneous between-groups covariance and when the distribution was normal, the covariance structure with the best fit was the unstructured population matrix. However, with heterogeneous between-groups covariance and when the pairing between covariance matrices and group sizes was null, the best fit was shown by the between-subjects heterogeneous unstructured population matrix, which was the case for all of the distributions analyzed. By contrast, with positive or negative pairings, the within-subjects and between-subjects heterogeneous first-order autoregressive structure produced the best fit. In the second stage of the study, the robustness of the LMM was tested. This showed that the KR method provided adequate control of Type I error rates for the time effect with normally distributed data. However, as skewness increased—as occurs, for example, in the log-normal distribution—the robustness of KR was null, especially when the assumption of sphericity was violated. As regards the influence of kurtosis, the analysis showed that the degree of robustness increased in line with the amount of kurtosis.     

A general Bayesian multilevel multidimensional IRT model for locally dependent data

Many item response theory (IRT) models take a multidimensional perspective to deal with sources that induce local item dependence (LID), with these models often making an orthogonal assumption about the dimensional structure of the data. One reason for this assumption is because of the indeterminacy issue in estimating the correlations among the dimensions in structures often specified to deal with sources of LID (e.g., bifactor and two-tier structures), and the assumption usually goes untested. Unfortunately, the mere fact that assessing these correlations is a challenge for some estimation methods does not mean that data seen in practice support such orthogonal structure. In this paper, a Bayesian multilevel multidimensional IRT model for locally dependent data is presented. This model can test whether item response data violate the orthogonal assumption that many IRT models make about the dimensional structure of the data when addressing sources of LID, and this test is carried out at the dimensional level while accounting for sampling clusters. Simulations show that the model presented is effective at carrying out this task. The utility of the model is also illustrated on an empirical data set.