BootES: An R package for bootstrap confidence intervals on effect sizes

Bootstrap Effect Sizes (bootES; Gerlanc & Kirby, 2012) is a free, open-source software package for R (R Development Core Team, 2012), which is a language and environment for statistical computing. BootES computes both unstandardized and standardized effect sizes (such as Cohen’s d, Hedges’s g, and Pearson’s r) and makes easily available for the first time the computation of their bootstrap confidence intervals (CIs). In this article, we illustrate how to use bootES to find effect sizes for contrasts in between-subjects, within-subjects, and mixed factorial designs and to find bootstrap CIs for correlations and differences between correlations. An appendix gives a brief introduction to R that will allow readers to use bootES without having prior knowledge of R.     

A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of freedom heteroscedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. The authors describe a nonparametric bootstrap methodology that can provide improved Type I error control. In addition, the authors indicate how researchers can set robust confidence intervals around a robust effect size parameter estimate. In an online supplement, the authors use several examples to illustrate the application of an SAS program to implement these statistical methods.     

The statistical recommendations of the American Psychological Association Publication Manual: Effect sizes,confidence intervals,and meta‐analysis

Estimation based on effect sizes, confidence intervals, and meta‐analysis usually provides a more informative analysis of empirical results than does statistical significance testing, which has long been the conventional choice in psychology. The sixth edition of the American Psychological Association Publication Manual now recommends that psychologists should, wherever possible, use estimation and base their interpretation of research results on point and interval estimates. We outline the Manual's recommendations and suggest how they can be put into practice: adopt an estimation framework, starting with the formulation of research aims as ‘How much?’ or ‘To what extent?’ questions. Calculate from your data effect size estimates and confidence intervals to answer those questions, then interpret. Wherever appropriate, use meta‐analysis to integrate evidence over studies. The Manual's recommendations can assist psychologists improve they way they do their statistics and help build a more quantitative and cumulative discipline.     

Design and power analysis: n and confidence intervals of means

In this study, we analyzed the validity of the conventional 80% power. The minimal sample size and power needed to guarantee non-overlapping (1-alpha)% confidence intervals for population means were calculated. Several simulations indicate that the minimal power for two means (m = 2) to have non-overlapping CIs is .80, for (1-alpha) set to 95%. The minimal power becomes .86 for 99% CIs and .75 for 90% CIs. When multiple means are considered, the required minimal power increases considerably. This increase is even higher when the population means do not increase monotonically. Therefore, the often adopted criterion of a minimal power equal to .80 is not always adequate. Hence, to guarantee that the limits of the CIs do not overlap, most situations require a direct calculation of the minimum number of observations that should enter in a study.     

Continued misinterpretation of confidence intervals: response to Miller and Ulrich

Miller and Ulrich (2015) critique our claim (Hoekstra et al., Psychonomic Bulletin & Review, 21(5), 1157–1164, 2014), based on a survey given to researchers and students, of widespread misunderstanding of confidence intervals (CIs). They suggest that survey respondents may have interpreted the statements in the survey that we deemed incorrect in an idiosyncratic, but correct, way, thus calling into question the conclusion that the results indicate that respondents could not properly interpret CIs. Their alternative interpretations, while correct, cannot be deemed acceptable renderings of the questions in the survey due to the well-known reference class problem. Moreover, there is no support in the data for their contention that participants may have had their alternative interpretations in mind. Finally, their alternative interpretations are merely trivial restatements of the definition of a confidence interval, and have no implications for the location of a parameter.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling

The psychometric function relates an observer’s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper’s principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one’s choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

The meaning and suitability of various effect sizes for structured Rater x Ratee designs

Four types of analysis are commonly applied to data from structured Rater x Ratee designs. These types are characterized by the unit of analysis, which is either raters or ratees, and by the design used, which is either between-units or within-unit design. The 4 types of analysis are quite different, and therefore they give rise to effect sizes that differ in their substantive interpretations. In most cases, effect sizes based on between-ratee analysis have the least ambiguous meaning and will best serve the aims of meta-analysts and primary researchers. Effect sizes that arise from within-unit designs confound the strength of an effect with its homogeneity. Nonetheless, the authors identify how a range of effect-size types such as these serve the aims of meta-analysis appropriately.     

Interpreting confidence intervals: A comment on Hoekstra,Morey, Rouder,and Wagenmakers (2014)

Hoekstra, Morey, Rouder, and Wagenmakers (Psychonomic Bulletin & Review 21(5), 1157–1164 2014) reported the results of a questionnaire designed to assess students’ and researchers’ understanding of confidence intervals (CIs). They interpreted their results as evidence that these groups “have no reliable knowledge about the correct interpretation of CIs” (Hoekstra et al. Psychonomic Bulletin & Review 21(5), 1157–1164 2014, p. 1161). We argue that their data do not substantiate this conclusion and that their report includes misleading suggestions about the correct interpretations of confidence intervals.     

VALCOR: A program for estimating standard error, confidence intervals, and probability of corrected validity

VALCOR is a Turbo-Basic program that corrects the observed (uncorrected) validity coefficients for criterion and predictor unreliability and range restriction in the predictor. Furthermore, using the formulas for the standard error of functions of correlations derived by Bobko and Rieck (1980), the program provides an estimation of the standard error, the confidence intervals, and the probability of the corrected validity coefficients. In this way, the probability and the boundaries of the corrected validity coefficients may be reported together with the probability of the uncorrected validity coefficients. The results are presented on the computer screen and may be saved in an external file.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling.

The psychometric function relates an observer's performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper's principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one's choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

To test or not to test: A clinical dilemma

This paper argues that clinicians are sometimes justified in not testing diagnoses or in not subjecting them to a full battery of tests. In deciding whether to conduct a test, a clinician may consider and weigh several different factors, including her confidence in her initial diagnosis, the specificity and sensitivity of the test, the consequences of making a false diagnosis, the pain, harm, and inconvenience caused by the test, and the costs of the test to the patient and society. This view suggests that diagnoses are fundamentally different from scientific hypotheses in that they are not always subjected to the same evidential standards.     

Headache triggers: To avoid or not to avoid,that is the question

Abstract

Traditional clinical advice in the management of headaches is to avoid trigger factors. There is a danger, however, that avoidance of triggers results in a sensitisation process whereby tolerance for the triggers decreases, in a manner analogous to increments in anxiety arising from avoidance of anxiety-eliciting stimuli. Reported here are six single-case experiments in which the aim was to desensitise headache sufferers to an experimentally validated trigger, namely “visual disturbance”. The results demonstrated that repeated, prolonged exposure to a headache trigger led to desensitisation with participants experiencing less visual disturbance, less negative affect and less head pain in response to the trigger. These findings have theoretical significance as they speak to the issue of the aetiology of chronic headache, and practical significance as they suggest that a key aspect of current management may contribute to the disorder becoming worse rather than better.