Bootstrap scree tests: A Monte Carlo simulation and applications to published data

A non‐parametric procedure for Cattell's scree test is proposed, using the bootstrap method. Bentler and Yuan developed parametric tests for the linear trend of scree eigenvalues in principal component analysis. The proposed method is for cases where parametric assumptions are not realistic. We define the break in the scree trend in several ways, based on linear slopes defined with two or three consecutive eigenvalues, or all eigenvalues after the k largest. The resulting scree test statistics are evaluated under various data conditions, among which Gorsuch and Nelson's bootstrap CNG performs best and is reasonably consistent and efficient under leptokurtic and skewed conditions. We also examine the bias‐corrected and accelerated bootstrap method for these statistics, and the bias correction is found to be too unstable to be useful. Using seven published data sets which Bentler and Yuan analysed, we compare the bootstrap approach to the scree test with the parametric linear trend test.     

Properties of bootstrap tests for N‐of‐1 studies

N‐of‐1 study designs involve the collection and analysis of repeated measures data from an individual not using an intervention and using an intervention. This study explores the use of semi‐parametric and parametric bootstrap tests in the analysis of N‐of‐1 studies under a single time series framework in the presence of autocorrelation. When the Type I error rates of bootstrap tests are compared to Wald tests, our results show that the bootstrap tests have more desirable properties. We compare the results for normally distributed errors with those for contaminated normally distributed errors and find that, except when there is relatively large autocorrelation, there is little difference between the power of the parametric and semi‐parametric bootstrap tests. We also experiment with two intervention designs: ABAB and AB, and show the ABAB design has more power. The results provide guidelines for designing N‐of‐1 studies, in the sense of how many observations and how many intervention changes are needed to achieve a certain level of power and which test should be performed.     

The big none: No evidence for a general factor of personality in chimpanzees, orangutans, or rhesus macaques

We examined whether a general factor of personality (GFP) was present in chimpanzees, orangutans, or rhesus macaques. We used confirmatory factor analysis (CFA) to model correlations among first-order factors as arising from a GFP. We then conducted principal axis factor analyses (PFA) of the first-order factors to extract a single higher-order factor and then to extract two oblique higher-order factors. The CFA model fit was poor for chimpanzees and orangutans, but not rhesus macaques. The single higher-order factors extracted via PFA did not resemble the GFP in all three species. The oblique higher-order factors extracted via PFA were only weakly correlated in all three species. These results do not support the existence of a GFP in nonhuman primates.     

Effect of non-normality on test statistics for one-way independent groups designs

The data obtained from one‐way independent groups designs is typically non‐normal in form and rarely equally variable across treatment populations (i.e. population variances are heterogeneous). Consequently, the classical test statistic that is used to assess statistical significance (i.e. the analysis of variance F test) typically provides invalid results (e.g. too many Type I errors, reduced power). For this reason, there has been considerable interest in finding a test statistic that is appropriate under conditions of non‐normality and variance heterogeneity. Previously recommended procedures for analysing such data include the James test, the Welch test applied either to the usual least squares estimators of central tendency and variability, or the Welch test with robust estimators (i.e. trimmed means and Winsorized variances). A new statistic proposed by Krishnamoorthy, Lu, and Mathew, intended to deal with heterogeneous variances, though not non‐normality, uses a parametric bootstrap procedure. In their investigation of the parametric bootstrap test, the authors examined its operating characteristics under limited conditions and did not compare it to the Welch test based on robust estimators. Thus, we investigated how the parametric bootstrap procedure and a modified parametric bootstrap procedure based on trimmed means perform relative to previously recommended procedures when data are non‐normal and heterogeneous. The results indicated that the tests based on trimmed means offer the best Type I error control and power when variances are unequal and at least some of the distribution shapes are non‐normal.     

Controlling the Type I error rate by using the nonparametric bootstrap when comparing means

Of the several tests for comparing population means, the best known are the ANOVA, Welch, Brown–Forsythe, and James tests. Each performs appropriately only in certain conditions, and none performs well in every setting. Researchers, therefore, have to select the appropriate procedure and run the risk of making a bad selection and, consequently, of erroneous conclusions. It would be desirable to have a test that performs well in any situation and so obviate preliminary analysis of data. We assess and compare several tests for equality of means in a simulation study, including non‐parametric bootstrap techniques, finding that the bootstrap ANOVA and bootstrap Brown–Forsythe tests exhibit a similar and exceptionally good behaviour.     

Type I errors and power of the parametric bootstrap goodness‐of‐fit test: Full and limited information

In sparse tables for categorical data well‐known goodness‐of‐fit statistics are not chi‐square distributed. A consequence is that model selection becomes a problem. It has been suggested that a way out of this problem is the use of the parametric bootstrap. In this paper, the parametric bootstrap goodness‐of‐fit test is studied by means of an extensive simulation study; the Type I error rates and power of this test are studied under several conditions of sparseness. In the presence of sparseness, models were used that were likely to violate the regularity conditions. Besides bootstrapping the goodness‐of‐fit usually used (full information statistics), corrected versions of these statistics and a limited information statistic are bootstrapped. These bootstrap tests were also compared to an asymptotic test using limited information. Results indicate that bootstrapping the usual statistics fails because these tests are too liberal, and that bootstrapping or asymptotically testing the limited information statistic works better with respect to Type I error and outperforms the other statistics by far in terms of statistical power. The properties of all tests are illustrated using categorical Markov models.     

Parametric and nonparametric bootstrap methods for meta-analysis

In a meta-analysis, the unknown parameters are often estimated using maximum likelihood, and inferences are based on asymptotic theory. It is assumed that, conditional on study characteristics included in the model, the between-study distribution and the sampling distributions of the effect sizes are normal. In practice, however, samples are finite, and the normality assumption may be violated, possibly resulting in biased estimates and inappropriate standard errors. In this article, we propose two parametric and two nonparametric bootstrap methods that can be used to adjust the results of maximum likelihood estimation in meta-analysis and illustrate them with empirical data. A simulation study, with raw data drawn from normal distributions, reveals that the parametric bootstrap methods and one of the nonparametric methods are generally superior to the ordinary maximum likelihood approach but suffer from a bias/precision tradeoff. We recommend using one of these bootstrap methods, but without applying the bias correction.     

Estimating Standard Errors in Exploratory Factor Analysis

The article describes 6 issues influencing standard errors in exploratory factor analysis and reviews 7 methods of computing standard errors for rotated factor loadings and factor correlations. These 7 methods are the augmented information method, the nonparametric bootstrap method, the infinitesimal jackknife method, the method using the asymptotic distributions of unrotated factor loadings, the sandwich method, the parametric bootstrap method, and the jackknife method. Standard error estimates are illustrated using a personality study with 537 men and an intelligence study with 145 children.     

Unidimensional factor models imply weaker partial correlations than zero-order correlations

In this paper we present a new implication of the unidimensional factor model. We prove that the partial correlation between two observed variables that load on one factor given any subset of other observed variables that load on this factor lies between zero and the zero-order correlation between these two observed variables. We implement this result in an empirical bootstrap test that rejects the unidimensional factor model when partial correlations are identified that are either stronger than the zero-order correlation or have a different sign than the zero-order correlation. We demonstrate the use of the test in an empirical data example with data consisting of fourteen items that measure extraversion.     

Detecting Outliers in Factor Analysis Using the Forward Search Algorithm

Dynamic factor analysis summarizes changes in scores on a battery of manifest variables over repeated measurements in terms of a time series in a substantially smaller number of latent factors. Algebraic formulae for standard errors of parameter estimates are more difficult to obtain than in the usual intersubject factor analysis because of the interdependence of successive observations. Bootstrap methods can fill this need, however. The standard bootstrap of individual timepoints is not appropriate because it destroys their order in time and consequently gives incorrect standard error estimates. Two bootstrap procedures that are appropriate for dynamic factor analysis are described. The moving block bootstrap breaks down the original time series into blocks and draws samples of blocks instead of individual timepoints. A parametric bootstrap is essentially a Monte Carlo study in which the population parameters are taken to be estimates obtained from the available sample. These bootstrap procedures are demonstrated using 103 days of affective mood self-ratings from a pregnant woman, 90 days of personality self-ratings from a psychology freshman, and a simulation study.     

The Dependability of the Exposure Effect

A bootstrap factor analysis was used to test the validity of the Love Relationships Scale by combining the data from earlier studies (Borrello &; Thompson, 1987; Thompson &; Borrello, 1987a, 1987b) with data from an additional 51 subjects recruited for this study. The results provide insight into the structure underlying perceptions of love relationships and suggest that a g-factor phenomenon involving obsessive thoughts dominates the experience of love. The utility of bootstrap analysis in validity studies is discussed.     

Bootstrapping to test for nonzero population correlation coefficients using univariate sampling

This article proposes 2 new approaches to test a nonzero population correlation (rho): the hypothesis-imposed univariate sampling bootstrap (HI) and the observed-imposed univariate sampling bootstrap (OI). The authors simulated correlated populations with various combinations of normal and skewed variates. With alpha set=.05, N> or =10, and rho< or =0.4, empirical Type I error rates of the parametric r and the conventional bivariate sampling bootstrap reached .168 and .081, respectively, whereas the largest error rates of the HI and the OI were .079 and .062. On the basis of these results, the authors suggest that the OI is preferable in alpha control to parametric approaches if the researcher believes the population is nonnormal and wishes to test for nonzero rhos of moderate size.     

Application of the bootstrap methods in factor analysis

A Monte Carlo experiment is conducted to investigate the performance of the bootstrap methods in normal theory maximum likelihood factor analysis both when the distributional assumption is satisfied and unsatisfied. The parameters and their functions of interest include unrotated loadings, analytically rotated loadings, and unique variances. The results reveal that (a) bootstrap bias estimation performs sometimes poorly for factor loadings and nonstandardized unique variances; (b) bootstrap variance estimation performs well even when the distributional assumption is violated; (c) bootstrap confidence intervals based on the Studentized statistics are recommended; (d) if structural hypothesis about the population covariance matrix is taken into account then the bootstrap distribution of the normal theory likelihood ratio test statistic is close to the corresponding sampling distribution with slightly heavier right tail.This study was carried out in part under the ISM cooperative research program (91-ISM · CRP-85, 92-ISM · CRP-102). The authors would like to thank the editor and three reviewers for their helpful comments and suggestions which improved the quality of this paper considerably.     

Variance estimation in factor analysis: An application of the bootstrap

Sampling variability of the estimates of factor loadings is neglected in modern factor analysis. Such investigations are generally normal theory based and asymptotic in nature. The bootstrap, a computer-based methodology, is described and then applied to demonstrate how the sampling variability of the estimates of factor loadings can be estimated for a given set of data. The issue of the number of factors to be retained in a factor model is also addressed. The bootstrap is shown to be an effective data-analytic tool for computing various statistics of interest which are otherwise intractable.     

Bootstrap approach to inference and power analysis based on three test statistics for covariance structure models

We study several aspects of bootstrap inference for covariance structure models based on three test statistics, including Type I error, power and sample‐size determination. Specifically, we discuss conditions for a test statistic to achieve a more accurate level of Type I error, both in theory and in practice. Details on power analysis and sample‐size determination are given. For data sets with heavy tails, we propose applying a bootstrap methodology to a transformed sample by a downweighting procedure. One of the key conditions for safe bootstrap inference is generally satisfied by the transformed sample but may not be satisfied by the original sample with heavy tails. Several data sets illustrate that, by combining downweighting and bootstrapping, a researcher may find a nearly optimal procedure for evaluating various aspects of covariance structure models. A rule for handling non‐convergence problems in bootstrap replications is proposed.     

A robust method for the analysis of experiments with ordered treatment levels.

A robust approach for the analysis of experiments with ordered treatment levels is presented as an alternative to existing approaches such as the parametric Abelson-Tukey test for monotone alternatives and the nonparametric Terpstra-Jonckheere test. The method integrates the familiar Spearman rank-order correlation with bootstrap routines to provide magnitude of association measures, p values, and confidence intervals for magnitude of association measures. The advantages of this method relative to five alternative approaches are pointed out.     

Increased reliability for single-case research results: is the bootstrap the answer?

There is need for objective and reliable single-case research (SCR) results in the movement toward evidence-based interventions (EBI), for inclusion in meta-analyses, and for funding accountability in clinical contexts. Yet SCR deals with data that often do not conform to parametric data assumptions and that yield results of low reliability. A resampling technique, the bootstrap, largely bypasses statistical assumptions and usually yields more reliable results. This study answers questions about the extent of need for the bootstrap in SCR and its impact on effect size reliability. The bootstrap was applied in Allison et al. mean shift analyses (Faith, Allison, & Gorman, 1997) to data from 166 published AB graphs. Results showed the bootstrap improved reliability of 88% of the analyses and reduced reliability of only 3%. The reliability improvement was large enough to be practically useful. The bootstrap was paired with a method for cleansing data of autocorrelation, which also proved effective. Pending replication, the findings encourage broad application within SCR of both the bootstrap and autocorrelation cleansing.     

Assessing model mimicry using the parametric bootstrap

We present a general sampling procedure to quantify model mimicry, defined as the ability of a model to account for data generated by a competing model. This sampling procedure, called the parametric bootstrap cross-fitting method (PBCM; cf. Williams (J. R. Statist. Soc. B 32 (1970) 350; Biometrics 26 (1970) 23)), generates distributions of differences in goodness-of-fit expected under each of the competing models. In the data informed version of the PBCM, the generating models have specific parameter values obtained by fitting the experimental data under consideration. The data informed difference distributions can be compared to the observed difference in goodness-of-fit to allow a quantification of model adequacy. In the data uninformed version of the PBCM, the generating models have a relatively broad range of parameter values based on prior knowledge. Application of both the data informed and the data uninformed PBCM is illustrated with several examples.     

Bootstrap Confidence Intervals for Ordinary Least Squares Factor Loadings and Correlations in Exploratory Factor Analysis

This article is concerned with using the bootstrap to assign confidence intervals for rotated factor loadings and factor correlations in ordinary least squares exploratory factor analysis. Coverage performances of SE-based intervals, percentile intervals, bias-corrected percentile intervals, bias-corrected accelerated percentile intervals, and hybrid intervals are explored using simulation studies involving different sample sizes, perfect and imperfect models, and normal and elliptical data. The bootstrap confidence intervals are also illustrated using a personality data set of 537 Chinese men. The results suggest that the bootstrap is an effective method for assigning confidence intervals at moderately large sample sizes.