Random‐effects meta‐analysis of correlations: Monte Carlo evaluation of mean estimators

Several authors have cautioned against using Fisher's z‐transformation in random‐effects meta‐analysis of correlations, which seems to perform poorly in some situations, especially with substantial inter‐study heterogeneity. Attributing this performance largely to the direct z‐to‐r transformation (DZRT) of Fisher z results (e.g. point estimate of mean correlation), in a previous paper Hafdahl (2009) proposed point and interval estimators of the mean Pearson r correlation that instead use an integral z‐to‐r transformation (IZRT). The present Monte Carlo study of these IZRT Fisher z estimators includes comparisons with their DZRT counterparts and with estimators based on Pearson r correlations. The IZRT point estimator was usually more accurate and efficient than its DZRT counterpart and comparable to the two Pearson r point estimators – better in some conditions but worse in others. Coverage probability for the IZRT confidence intervals (CIs) was often near nominal, much better than for the DZRT CIs, and comparable to coverage for the Pearson r CIs; every approach's CI fell markedly below nominal in some conditions. The IZRT estimators contradict warnings about Fisher z estimators' poor performance. Recommendations for practising research synthesists are offered, and an Appendix provides computing code to implement the IZRT as in the real‐data example.     

Bootstrap scree tests: A Monte Carlo simulation and applications to published data

A non‐parametric procedure for Cattell's scree test is proposed, using the bootstrap method. Bentler and Yuan developed parametric tests for the linear trend of scree eigenvalues in principal component analysis. The proposed method is for cases where parametric assumptions are not realistic. We define the break in the scree trend in several ways, based on linear slopes defined with two or three consecutive eigenvalues, or all eigenvalues after the k largest. The resulting scree test statistics are evaluated under various data conditions, among which Gorsuch and Nelson's bootstrap CNG performs best and is reasonably consistent and efficient under leptokurtic and skewed conditions. We also examine the bias‐corrected and accelerated bootstrap method for these statistics, and the bias correction is found to be too unstable to be useful. Using seven published data sets which Bentler and Yuan analysed, we compare the bootstrap approach to the scree test with the parametric linear trend test.     

Monte Carlo tests of the Rasch model based on scalability coefficients

For item responses fitting the Rasch model, the assumptions underlying the Mokken model of double monotonicity are met. This makes non‐parametric item response theory a natural starting‐point for Rasch item analysis. This paper studies scalability coefficients based on Loevinger's H coefficient that summarizes the number of Guttman errors in the data matrix. These coefficients are shown to yield efficient tests of the Rasch model using p‐values computed using Markov chain Monte Carlo methods. The power of the tests of unequal item discrimination, and their ability to distinguish between local dependence and unequal item discrimination, are discussed. The methods are illustrated and motivated using a simulation study and a real data example.     

A Monte Carlo study comparing PIV,ULS and DWLS in the estimation of dichotomous confirmatory factor analysis

We conducted a Monte Carlo study to investigate the performance of the polychoric instrumental variable estimator (PIV) in comparison to unweighted least squares (ULS) and diagonally weighted least squares (DWLS) in the estimation of a confirmatory factor analysis model with dichotomous indicators. The simulation involved 144 conditions (1,000 replications per condition) that were defined by a combination of (a) two types of latent factor models, (b) four sample sizes (100, 250, 500, 1,000), (c) three factor loadings (low, moderate, strong), (d) three levels of non‐normality (normal, moderately, and extremely non‐normal), and (e) whether the factor model was correctly specified or misspecified. The results showed that when the model was correctly specified, PIV produced estimates that were as accurate as ULS and DWLS. Furthermore, the simulation showed that PIV was more robust to structural misspecifications than ULS and DWLS.     

Multiway contingency tables: Monte Carlo resampling probability values for the chi-squared and likelihood-ratio tests

Monte Carlo resampling methods to obtain probability values for chi-squared and likelihood-ratio test statistics for multiway contingency tables are presented. A resampling algorithm provides random arrangements of cell frequencies in a multiway contingency table, given fixed marginal frequency totals. Probability values are obtained from the proportion of resampled test statistic values equal to or greater than the observed test statistic value.     

Testing for robustness in Monte Carlo studies

Monte Carlo studies provide the information needed to help researchers select appropriate analytical procedures under design conditions in which the underlying assumptions of the procedures are not met. In Monte Carlo studies, the 2 errors that one could commit involve (a) concluding that a statistical procedure is robust when it is not or (b) concluding that it is not robust when it is. In previous attempts to apply standard statistical design principles to Monte Carlo studies, the less severe of these errors has been wrongly designated the Type I error. In this article, a method is presented for controlling the appropriate Type I error rate; the determination of the number of iterations required in a Monte Carlo study to achieve desired power is described; and a confidence interval for a test's true Type I error rate is derived. A robustness criterion is also proposed that is a compromise between W. G. Cochran's (1952) and J. V. Bradley's (1978) criteria.     

TouchStat: A Monte Carlo program for calculating sequential touching probabilities

A growing number of child cognition researchers are using an object-manipulation, sequential-touching paradigm to explore young children’s conceptual abilities. Within this paradigm, it is essential to distinguish children’s intracategory touching sequences from those expected by chance. The sequentialtouching approach is congruent with a permutation testing model of statistical inference and is best modeled by sampled permutations as derived from Monte Carlo procedures. In this article, we introduce a computer program for generating Monte Carlo sequential-touching simulations. TouchStat permits users to determine their own specifications to simulate sequential touching to category exemplars across a wide range of task parameters. We also present Monte Carlo chance probabilities for the standard two-category, four-exemplar task, with sequences up to 30 touches long. Finally, we consider broader applications of the TouchStat program.     

The conditional power of randomization tests for single-case effect sizes in designs with randomized treatment order: A Monte Carlo simulation study

The conditional power (CP) of the randomization test (RT) was investigated in a simulation study in which three different single-case effect size (ES) measures were used as the test statistics: the mean difference (MD), the percentage of nonoverlapping data (PND), and the nonoverlap of all pairs (NAP). Furthermore, we studied the effect of the experimental design on the RT’s CP for three different single-case designs with rapid treatment alternation: the completely randomized design (CRD), the randomized block design (RBD), and the restricted randomized alternation design (RRAD). As a third goal, we evaluated the CP of the RT for three types of simulated data: data generated from a standard normal distribution, data generated from a uniform distribution, and data generated from a first-order autoregressive Gaussian process. The results showed that the MD and NAP perform very similarly in terms of CP, whereas the PND performs substantially worse. Furthermore, the RRAD yielded marginally higher power in the RT, followed by the CRD and then the RBD. Finally, the power of the RT was almost unaffected by the type of the simulated data. On the basis of the results of the simulation study, we recommend at least 20 measurement occasions for single-case designs with a randomized treatment order that are to be evaluated with an RT using a 5% significance level. Furthermore, we do not recommend use of the PND, because of its low power in the RT.     

A Monte Carlo simulation study of the reliability of intraindividual variability

Recent research has seen intraindividual variability become a useful technique to incorporate trial-to-trial variability into many types of psychological studies. Intraindividual variability, as measured by individual standard deviations (ISDs), has shown unique prediction to several types of positive and negative outcomes (Ram, Rabbit, Stollery, & Nesselroade, 2005). One unanswered question regarding measuring intraindividual variability is its reliability and the conditions under which optimal reliability is achieved. Monte Carlo simulation studies were conducted to determine the reliability of the ISD as compared with the intraindividual mean. The results indicate that ISDs generally have poor reliability and are sensitive to insufficient measurement occasions, poor test reliability, and unfavorable amounts and distributions of variability in the population. Secondary analysis of psychological data shows that use of individual standard deviations in unfavorable conditions leads to a marked reduction in statistical power, although careful adherence to underlying statistical assumptions allows their use as a basic research tool. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Monte Carlo Evaluation of Two-Level Logistic Regression for Assessing Person Fit

Person fit is the degree to which an item response model fits for individual examinees. Reise (2000) Reise, S. P. 2000. Using multilevel logistic regression to evaluate person-fit in IRT models. Multivariate Behavioral Research, 35: 543–568. [Taylor & Francis Online], [Web of Science ®] , [Google Scholar] described how two-level logistic regression can be used to detect heterogeneity in person fit, evaluate potential predictors of person fit heterogeneity, and identify potentially aberrant individuals. The method has apparently never been applied to empirical data or evaluated in a simulation study. The present research applies Reise's method to empirical data obtained from university undergraduates measured on the Fear of Negative Evaluation scale. Additionally, Reise's method is evaluated under conditions varying according to the type of aberrancy, level of test reliability, and scale length. Statistical power to detect aberrancy was highly dependent on manipulated variables, and some results were affected by bias in model parameters that was due to the aberrant responders. Nevertheless, Reise's method generally performed well and detected aberrant individuals either as well as, or better than, the well-established l _z person-fit statistic.     

A Markov Chain Monte Carlo Approach to Confirmatory Item Factor Analysis

Item factor analysis has a rich tradition in both the structural equation modeling and item response theory frameworks. The goal of this paper is to demonstrate a novel combination of various Markov chain Monte Carlo (MCMC) estimation routines to estimate parameters of a wide variety of confirmatory item factor analysis models. Further, I show that these methods can be implemented in a flexible way which requires minimal technical sophistication on the part of the end user. After providing an overview of item factor analysis and MCMC, results from several examples (simulated and real) will be discussed. The bulk of these examples focus on models that are problematic for current “gold-standard” estimators. The results demonstrate that it is possible to obtain accurate parameter estimates using MCMC in a relatively user-friendly package.     

Comments on Monte Carlo simulations of void-lattice formation

We study the coalescence of neighbouring voids along close-packed directions in recent computer simulation studies of void-lattice formation. The stability against coalescence of a developing void lattice is found to depend very much on the detailed geometry of the local void distribution. The possibility of void coalescence as an artifact, caused by the incorrect assumption of homogeneous void nucleation in these simulations, is suggested and discussed.     

An elastic-energy-relaxation/Monte Carlo hybrid scheme of computer simulation for solids

Abstract

We describe a new method of computer simulation which combines the methods of lattice statics and Monte Carlo importance-sampling. The method is illustrated with a calculation of the zero-Kelvin or configurational chemical potential in the linear deformable lattice gas.     

A Monte Carlo study of the Hoyt reliability coefficient in the Bower-Trabasso model

Monte Carlo procedures are used to study the sampling distribution of the Hoyt reliability coefficient. Estimates of mean, variance, and skewness are made for the case of the Bower-Trabasso concept identification model. Given the Bower-Trabasso assumptions, the Hoyt coefficient of a particular concept identification experiment is shown to be statistically unlikely.     

Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model

Numerous ways to meta-analyze single-case data have been proposed in the literature; however, consensus has not been reached on the most appropriate method. One method that has been proposed involves multilevel modeling. For this study, we used Monte Carlo methods to examine the appropriateness of Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach for the meta-analysis of single-case data. Specifically, we examined the fixed effects (e.g., the overall average treatment effect) and the variance components (e.g., the between-person within-study variance in the treatment effect) in a three-level multilevel model (repeated observations nested within individuals, nested within studies). More specifically, bias of the point estimates, confidence interval coverage rates, and interval widths were examined as a function of the number of primary studies per meta-analysis, the modal number of participants per primary study, the modal series length per primary study, the level of autocorrelation, and the variances of the error terms. The degree to which the findings of this study are supportive of using Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach to meta-analyzing single-case data depends on the particular parameter of interest. Estimates of the average treatment effect tended to be unbiased and produced confidence intervals that tended to overcover, but did come close to the nominal level as Level-3 sample size increased. Conversely, estimates of the variance in the treatment effect tended to be biased, and the confidence intervals for those estimates were inaccurate.     

SPSS and SAS programs for comparing Pearson correlations and OLS regression coefficients

Several procedures that use summary data to test hypotheses about Pearson correlations and ordinary least squares regression coefficients have been described in various books and articles. To our knowledge, however, no single resource describes all of the most common tests. Furthermore, many of these tests have not yet been implemented in popular statistical software packages such as SPSS and SAS. In this article, we describe all of the most common tests and provide SPSS and SAS programs to perform them. When they are applicable, our code also computes 100 × (1 ? α)% confidence intervals corresponding to the tests. For testing hypotheses about independent regression coefficients, we demonstrate one method that uses summary data and another that uses raw data (i.e., Potthoff analysis). When the raw data are available, the latter method is preferred, because use of summary data entails some loss of precision due to rounding.     

A Monte Carlo Investigation of the Likelihood Ratio Test for Number of Classes in Latent Class Analysis

When using latent class analysis to explore multivariate categorical data an important question is -- how many classes are appropriate for this data? An obvious candidate to answer the question is the likelihood ratio test of c[SUB0] against c[SUB1] classes. In this paper this test is investigated by Monte Carlo methods; results confirm that the usually assumed null distribution is inappropriate.     

Monte Carlo simulations and the chi-square test of independence

A Monte Carlo program for sampling 2 by 2 contingency tables from a user-specified population is discussed. Applications include computer-assisted instruction (CAI) of statistics, evaluation of actual vs nominal Type I error rates of the chi-square test of independence when expected frequencies are less than 10, and estimation of the power of the chi-square test.