Calculating and graphing within-subject confidence intervals for ANOVA

The psychological and statistical literature contains several proposals for calculating and plotting confidence intervals (CIs) for within-subjects (repeated measures) ANOVA designs. A key distinction is between intervals supporting inference about patterns of means (and differences between pairs of means, in particular) and those supporting inferences about individual means. In this report, it is argued that CIs for the former are best accomplished by adapting intervals proposed by Cousineau (Tutorials in Quantitative Methods for Psychology, 1, 42–45, 2005) and Morey (Tutorials in Quantitative Methods for Psychology, 4, 61–64, 2008) so that nonoverlapping CIs for individual means correspond to a confidence for their difference that does not include zero. CIs for the latter can be accomplished by fitting a multilevel model. In situations in which both types of inference are of interest, the use of a two-tiered CI is recommended. Free, open-source, cross-platform software for such interval estimates and plots (and for some common alternatives) is provided in the form of R functions for one-way within-subjects and two-way mixed ANOVA designs. These functions provide an easy-to-use solution to the difficult problem of calculating and displaying within-subjects CIs.     

Accuracy in parameter estimation for ANCOVA and ANOVA contrasts: sample size planning via narrow confidence intervals

Contrasts of means are often of interest because they describe the effect size among multiple treatments. High-quality inference of population effect sizes can be achieved through narrow confidence intervals (CIs). Given the close relation between CI width and sample size, we propose two methods to plan the sample size for an ANCOVA or ANOVA study, so that a sufficiently narrow CI for the population (standardized or unstandardized) contrast of interest will be obtained. The standard method plans the sample size so that the expected CI width is sufficiently small. Since CI width is a random variable, the expected width being sufficiently small does not guarantee that the width obtained in a particular study will be sufficiently small. An extended procedure ensures with some specified, high degree of assurance (e.g., 90% of the time) that the CI observed in a particular study will be sufficiently narrow. We also discuss the rationale and usefulness of two different ways to standardize an ANCOVA contrast, and compare three types of standardized contrast in the ANCOVA/ANOVA context. All of the methods we propose have been implemented in the freely available MBESS package in R so that they can be easily applied by researchers.     

Estimating confidence intervals for eigenvalues in exploratory factor analysis

Exploratory factor analysis (EFA) has become a common procedure in educational and psychological research. In the course of performing an EFA, researchers often base the decision of how many factors to retain on the eigenvalues for the factors. However, many researchers do not realize that eigenvalues, like all sample statistics, are subject to sampling error, which means that confidence intervals (CIs) can be estimated for each eigenvalue. In the present article, we demonstrate two methods of estimating CIs for eigenvalues: one based on the mathematical properties of the central limit theorem, and the other based on bootstrapping. References to appropriate SAS and SPSS syntax are included. Supplemental materials for this article may be downloaded from http://brm.psychonomic-journals.org/content/supplemental.     

RMediation: An R package for mediation analysis confidence intervals

This article describes the RMediation package,which offers various methods for building confidence intervals (CIs) for mediated effects. The mediated effect is the product of two regression coefficients. The distribution-of-the-product method has the best statistical performance of existing methods for building CIs for the mediated effect. RMediation produces CIs using methods based on the distribution of product, Monte Carlo simulations, and an asymptotic normal distribution. Furthermore, RMediation generates percentiles, quantiles, and the plot of the distribution and CI for the mediated effect. An existing program, called PRODCLIN, published in Behavior Research Methods, has been widely cited and used by researchers to build accurate CIs. PRODCLIN has several limitations: The program is somewhat cumbersome to access and yields no result for several cases. RMediation described herein is based on the widely available R software, includes several capabilities not available in PRODCLIN, and provides accurate results that PRODCLIN could not.     

On confidence intervals for within-subjects designs

Confidence intervals (CIs) for means are frequently advocated as alternatives to null hypothesis significance testing (NHST), for which a common theme in the debate is that conclusions from CIs and NHST should be mutually consistent. The authors examined a class of CIs for which the conclusions are said to be inconsistent with NHST in within-subjects designs and a class for which the conclusions are said to be consistent. The difference between them is a difference in models. In particular, the main issue is that the class for which the conclusions are said to be consistent derives from fixed-effects models with subjects fixed, not mixed models with subjects random. Offered is mixed model methodology that has been popularized in the statistical literature and statistical software procedures. Generalizations to different classes of within-subjects designs are explored, and comments on the future direction of the debate on NHST are offered.     

Bayesian analogy with relational transformations

How can humans acquire relational representations that enable analogical inference and other forms of high-level reasoning? Using comparative relations as a model domain, we explore the possibility that bottom-up learning mechanisms applied to objects coded as feature vectors can yield representations of relations sufficient to solve analogy problems. We introduce Bayesian analogy with relational transformations (BART) and apply the model to the task of learning first-order comparative relations (e.g., larger, smaller, fiercer, meeker) from a set of animal pairs. Inputs are coded by vectors of continuous-valued features, based either on human magnitude ratings, normed feature ratings (De Deyne et al., 2008), or outputs of the topics model (Griffiths, Steyvers, & Tenenbaum, 2007). Bootstrapping from empirical priors, the model is able to induce first-order relations represented as probabilistic weight distributions, even when given positive examples only. These learned representations allow classification of novel instantiations of the relations and yield a symbolic distance effect of the sort obtained with both humans and other primates. BART then transforms its learned weight distributions by importance-guided mapping, thereby placing distinct dimensions into correspondence. These transformed representations allow BART to reliably solve 4-term analogies (e.g., larger:smaller::fiercer:meeker), a type of reasoning that is arguably specific to humans. Our results provide a proof-of-concept that structured analogies can be solved with representations induced from unstructured feature vectors by mechanisms that operate in a largely bottom-up fashion. We discuss potential implications for algorithmic and neural models of relational thinking, as well as for the evolution of abstract thought.     

Large-sample confidence intervals for validity and reliability coefficients

Large-sample confidence intervals (CI) for reliability, validity, and unattenuated validity are presented. The CI for unattenuated validity is based on the Bonferroni inequality, which relies on one CI for test-retest reliability and one for validity. Covered are four reliability-validity situations: (a) both estimates were from random samples; (b) reliability was from a random sample but validity was from a selected sample; (c) validity was from a random sample but reliability was from a selected sample; and (d) both estimates were from selected samples. All CIs were evaluated by using a simulation. CIs on reliability, validity, or unattenuated validity are accurate as long as selection ratio is at least 20% and selected sample size is 100 or larger. When selection ratio is less than 20%, estimators tend to underestimate their parameters.     

Bootstrap confidence intervals for principal covariates regression

Principal covariate regression (PCOVR) is a method for regressing a set of criterion variables with respect to a set of predictor variables when the latter are many in number and/or collinear. This is done by extracting a limited number of components that simultaneously synthesize the predictor variables and predict the criterion ones. So far, no procedure has been offered for estimating statistical uncertainties of the obtained PCOVR parameter estimates. The present paper shows how this goal can be achieved, conditionally on the model specification, by means of the bootstrap approach. Four strategies for estimating bootstrap confidence intervals are derived and their statistical behaviour in terms of coverage is assessed by means of a simulation experiment. Such strategies are distinguished by the use of the varimax and quartimin procedures and by the use of Procrustes rotations of bootstrap solutions towards the sample solution. In general, the four strategies showed appropriate statistical behaviour, with coverage tending to the desired level for increasing sample sizes. The main exception involved strategies based on the quartimin procedure in cases characterized by complex underlying structures of the components. The appropriateness of the statistical behaviour was higher when the proper number of components were extracted.     

Revisiting confidence intervals for repeated measures designs

Loftus and Masson (1994) proposed a method for computing confidence intervals (CIs) in repeated measures (RM) designs and later proposed that RM CIs for factorial designs should be based on number of observations rather than number of participants (Masson & Loftus, 2003). However, determining the correct number of observations for a particular effect can be complicated, given that its value depends on the relation between the effect and the overall design. To address this, we recently defined a general number-of-observations principle, explained why it obtains, and provided step-by-step instructions for constructing CIs for various effect types (Jarmasz & Hollands, 2009). In this note, we provide a brief summary of our approach.     

Implications of statistical power for confidence intervals

The statistical power of a hypothesis test is closely related to the precision of the accompanying confidence interval. In the case of a z-test, the width of the confidence interval is a function of statistical power for the planned study. If minimum effect size is used in power analysis, the width of the confidence interval is the minimum effect size times a multiplicative factor φ. The index φ, or the precision-to-effect ratio, is a function of the computed statistical power. In the case of a t-test, statistical power affects the probability of achieving a certain width of confidence interval, which is equivalent to the probability of obtaining a certain value of φ. To consider estimate precision in conjunction with statistical power, we can choose a sample size to obtain a desired probability of achieving a short width conditional on the rejection of the null hypothesis.     

Design and power analysis: n and confidence intervals of means

In this study, we analyzed the validity of the conventional 80% power. The minimal sample size and power needed to guarantee non-overlapping (1-alpha)% confidence intervals for population means were calculated. Several simulations indicate that the minimal power for two means (m = 2) to have non-overlapping CIs is .80, for (1-alpha) set to 95%. The minimal power becomes .86 for 99% CIs and .75 for 90% CIs. When multiple means are considered, the required minimal power increases considerably. This increase is even higher when the population means do not increase monotonically. Therefore, the often adopted criterion of a minimal power equal to .80 is not always adequate. Hence, to guarantee that the limits of the CIs do not overlap, most situations require a direct calculation of the minimum number of observations that should enter in a study.     

Four bootstrap confidence intervals for the binomial-error model

Confidence intervals for the mean function of the true proportion score ( _x ), where andx respectively denote the true proportion and observed test scores, can be approximated by the Efron, Bayesian, and parametric empirical Bayes (PEB) bootstrap procedures. The similarity of results yielded by all the bootstrap methods suggests the following: the unidentifiability problem of the prior distributiong() can be bypassed with respect to the construction of confidence intervals for the mean function, and a beta distribution forg() is a reasonable assumption for the test scores in compliance with a negative hypergeometric distribution. The PEB bootstrap, which reflects the construction of Morris intervals, is introduced for computing predictive confidence bands for x. It is noted that the effect of test reliability on the precision of interval estimates varies with the two types of confidence statements concerned.The Authors are indebted to the Editor and anonymous reviewers for constructive suggestions and comments. The authors wish to thank Min-Te Chao and Cheng-Der Fuh for some useful suggestions at earlier stages of writing this paper.     

Approximate confidence intervals for a robust scale parameter

A recent paper by Wainer and Thissen has renewed the interest in Gini's mean difference,G, by pointing out its robust characteristics. This note presents distribution-free asymptotic confidence intervals for its population value,γ, in the one sample case and for the difference Δ=(γ ₁?γ ₂) in the two sample situations. Both procedures are based on a technique of jackknifingU-statistics developed by Arvesen.     

Constructing second‐order accurate confidence intervals for communalities in factor analysis

In an effort to find accurate alternatives to the usual confidence intervals based on normal approximations, this paper compares four methods of generating second‐order accurate confidence intervals for non‐standardized and standardized communalities in exploratory factor analysis under the normality assumption. The methods to generate the intervals employ, respectively, the Cornish–Fisher expansion and the approximate bootstrap confidence (ABC), and the bootstrap‐t and the bias‐corrected and accelerated bootstrap (BC_a). The former two are analytical and the latter two are numerical. Explicit expressions of the asymptotic bias and skewness of the communality estimators, used in the analytical methods, are derived. A Monte Carlo experiment reveals that the performance of central intervals based on normal approximations is a consequence of imbalance of miscoverage on the left‐ and right‐hand sides. The second‐order accurate intervals do not require symmetry around the point estimates of the usual intervals and achieve better balance, even when the sample size is not large. The behaviours of the second‐order accurate intervals were similar to each other, particularly for large sample sizes, and no method performed consistently better than the others.     

Problematic standard errors and confidence intervals for skewness and kurtosis

Many statistics packages print skewness and kurtosis statistics with estimates of their standard errors. The function most often used for the standard errors (e.g., in SPSS) assumes that the data are drawn from a normal distribution, an unlikely situation. Some textbooks suggest that if the statistic is more than about 2 standard errors from the hypothesized value (i.e., an approximate value for the critical value from the t distribution for moderate or large sample sizes when α = 5%), the hypothesized value can be rejected. This is an inappropriate practice unless the standard error estimate is accurate and the sampling distribution is approximately normal. We show distributions where the traditional standard errors provided by the function underestimate the actual values, often being 5 times too small, and distributions where the function overestimates the true values. Bootstrap standard errors and confidence intervals are more accurate than the traditional approach, although still imperfect. The reasons for this are discussed. We recommend that if you are using skewness and kurtosis statistics based on the 3^rd and 4^th moments, bootstrapping should be used to calculate standard errors and confidence intervals, rather than using the traditional standard. Software in the freeware R for this article provides these estimates.     

On approximate confidence intervals for measures of concordance

The use ofU-statistics based on rank correlation coefficients in estimating the strength of concordance among a group of rankers is examined for cases where the null hypothesis of random rankings is not tenable. The studentizedU-statistics is asymptotically distribution-free, and the Student-t approximation is used for small and moderate sized samples. An approximate confidence interval is constructed for the strength of concordance. Monte Carlo results indicate that the Student-t approximation can be improved by estimating the degrees of freedom.Research partially supported on ONR Contract N00014-82-K-0207.     

Goodness-of-fit and confidence intervals of approximate models

If the model for the data are strictly speaking incorrect, then how can one test whether the model fits? Standard goodness-of-fit (GOF) tests rely on strictly correct or incorrect models. But in practice the correct model is not assumed to be available. It would still be of interest to determine how good or how bad the approximation is. But how can this be achieved? If it is determined that a model is a good approximation and hence a good explanation of the data, how can reliable confidence intervals be constructed? In this paper, an attempt is made to answer the above questions. Several GOF tests and methods of constructing confidence intervals are evaluated both in a simulation and with real data from the internet-based daily news memory test.     

Bootstrap confidence intervals in multi‐level simultaneous component analysis

Multi‐level simultaneous component analysis (MLSCA) was designed for the exploratory analysis of hierarchically ordered data. MLSCA specifies a component model for each level in the data, where appropriate constraints express possible similarities between groups of objects at a certain level, yielding four MLSCA variants. The present paper discusses different bootstrap strategies for estimating confidence intervals (CIs) on the individual parameters. In selecting a proper strategy, the main issues to address are the resampling scheme and the non‐uniqueness of the parameters. The resampling scheme depends on which level(s) in the hierarchy are considered random, and which fixed. The degree of non‐uniqueness depends on the MLSCA variant, and, in two variants, the extent to which the user exploits the transformational freedom. A comparative simulation study examines the quality of bootstrap CIs of different MLSCA parameters. Generally, the quality of bootstrap CIs appears to be good, provided the sample sizes are sufficient at each level that is considered to be random. The latter implies that if more than a single level is considered random, the total number of observations necessary to obtain reliable inferential information increases dramatically. An empirical example illustrates the use of bootstrap CIs in MLSCA.