Point-biserial correlation: Interval estimation,hypothesis testing,meta-analysis,and sample size determination

The point-biserial correlation is a commonly used measure of effect size in two-group designs. New estimators of point-biserial correlation are derived from different forms of a standardized mean difference. Point-biserial correlations are defined for designs with either fixed or random group sample sizes and can accommodate unequal variances. Confidence intervals and standard errors for the point-biserial correlation estimators are derived from the sampling distributions for pooled-variance and separate-variance versions of a standardized mean difference. The proposed point-biserial confidence intervals can be used to conduct directional two-sided tests, equivalence tests, directional non-equivalence tests, and non-inferiority tests. A confidence interval for an average point-biserial correlation in meta-analysis applications performs substantially better than the currently used methods. Sample size formulas for estimating a point-biserial correlation with desired precision and testing a point-biserial correlation with desired power are proposed. R functions are provided that can be used to compute the proposed confidence intervals and sample size formulas.     

Is the meta-analysis of correlation coefficients accurate when population correlations vary?

One conceptualization of meta-analysis is that studies within the meta-analysis are sampled from populations with mean effect sizes that vary (random-effects models). The consequences of not applying such models and the comparison of different methods have been hotly debated. A Monte Carlo study compared the efficacy of Hedges and Vevea's random-effects methods of meta-analysis with Hunter and Schmidt's, over a wide range of conditions, as the variability in population correlations increases. (a) The Hunter-Schmidt method produced estimates of the average correlation with the least error, although estimates from both methods were very accurate; (b) confidence intervals from Hunter and Schmidt's method were always slightly too narrow but became more accurate than those from Hedges and Vevea's method as the number of studies included in the meta-analysis, the size of the true correlation, and the variability of correlations increased; and (c) the study weights did not explain the differences between the methods.     

A primer on the understanding of meta-analysis

ObjectivesThe purpose of the present paper is to provide a primer on the understanding of meta-analysis.Design and methodAfter presenting the rationale behind meta-analysis, the present paper defines statistical artifacts of sampling error and measurement.FindingsExamples show that statistical artifacts influence the correlation coefficient. The paper also explains the notions of confidence intervals and credibility intervals and how correlations corrected for sampling error and measurement error are calculated.ConclusionsThe paper concludes by explaining the notion of second-order sampling error and moderator meta-analysis.     

认知诊断属性分类一致性信度区间估计三种方法

摘要：引入了三种可以估计认知诊断属性分类一致性信度置信区间的方法：Bootstrap法、平行测验法和平行测验配对法。用模拟研究验证和比较了这三种方法的表现,结果发现,平行测验法和Bootstrap法在被试量比较少、题目数量比较少的情况下,估计的标准误和置信区间较接近,但是随着被试量的增加,Bootstrap法的估计精度提高较快,在被试量大和题目数量较多时基本接近平行测验配对法的结果。Bootstrap法的所需时间最少,平行测验配对法计算过程复杂且用时较长,推荐用Bootstrap法估计认知诊断属性分类一致性信度的置信区间。     

A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of freedom heteroscedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. The authors describe a nonparametric bootstrap methodology that can provide improved Type I error control. In addition, the authors indicate how researchers can set robust confidence intervals around a robust effect size parameter estimate. In an online supplement, the authors use several examples to illustrate the application of an SAS program to implement these statistical methods.     

Confidence intervals for standardized linear contrasts of means

Most psychology journals now require authors to report a sample value of effect size along with hypothesis testing results. The sample effect size value can be misleading because it contains sampling error. Authors often incorrectly interpret the sample effect size as if it were the population effect size. A simple solution to this problem is to report a confidence interval for the population value of the effect size. Standardized linear contrasts of means are useful measures of effect size in a wide variety of research applications. New confidence intervals for standardized linear contrasts of means are developed and may be applied to between-subjects designs, within-subjects designs, or mixed designs. The proposed confidence interval methods are easy to compute, do not require equal population variances, and perform better than the currently available methods when the population variances are not equal.     

Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model

Numerous ways to meta-analyze single-case data have been proposed in the literature; however, consensus has not been reached on the most appropriate method. One method that has been proposed involves multilevel modeling. For this study, we used Monte Carlo methods to examine the appropriateness of Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach for the meta-analysis of single-case data. Specifically, we examined the fixed effects (e.g., the overall average treatment effect) and the variance components (e.g., the between-person within-study variance in the treatment effect) in a three-level multilevel model (repeated observations nested within individuals, nested within studies). More specifically, bias of the point estimates, confidence interval coverage rates, and interval widths were examined as a function of the number of primary studies per meta-analysis, the modal number of participants per primary study, the modal series length per primary study, the level of autocorrelation, and the variances of the error terms. The degree to which the findings of this study are supportive of using Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach to meta-analyzing single-case data depends on the particular parameter of interest. Estimates of the average treatment effect tended to be unbiased and produced confidence intervals that tended to overcover, but did come close to the nominal level as Level-3 sample size increased. Conversely, estimates of the variance in the treatment effect tended to be biased, and the confidence intervals for those estimates were inaccurate.     

An alternative to Cohen's standardized mean difference effect size: a robust parameter and confidence interval in the two independent groups case

The authors argue that a robust version of Cohen's effect size constructed by replacing population means with 20% trimmed means and the population standard deviation with the square root of a 20% Winsorized variance is a better measure of population separation than is Cohen's effect size. The authors investigated coverage probability for confidence intervals for the new effect size measure. The confidence intervals were constructed by using the noncentral t distribution and the percentile bootstrap. Over the range of distributions and effect sizes investigated in the study, coverage probability was better for the percentile bootstrap confidence interval.     

Constructing Confidence Intervals for Effect Size Measures of an Indirect Effect

Confidence intervals for an effect size can provide the information about the magnitude of an effect and its precision as well as the binary decision about the existence of an effect. In this study, the performances of five different methods for constructing confidence intervals for ratio effect size measures of an indirect effect were compared in terms of power, coverage rates, Type I error rates, and widths of confidence intervals. The five methods include the percentile bootstrap method, the bias-corrected and accelerated (BCa) bootstrap method, the delta method, the Fieller method, and the Monte Carlo method. The results were discussed with respect to the adequacy of the distributional assumptions and the nature of ratio quantity. The confidence intervals from the five methods showed similar results for samples of more than 500, whereas, for samples of less than 500, the confidence intervals were sufficiently narrow to convey the information about the population effect sizes only when the effect sizes of regression coefficients defining the indirect effect are large.     

Bootstrap standard error and confidence intervals for the correlations corrected for indirect range restriction

The standard Pearson correlation coefficient, r, is a biased estimator of the population correlation coefficient, ρ(XY) , when predictor X and criterion Y are indirectly range-restricted by a third variable Z (or S). Two correction algorithms, Thorndike's (1949) Case III, and Schmidt, Oh, and Le's (2006) Case IV, have been proposed to correct for the bias. However, to our knowledge, the two algorithms did not provide a procedure to estimate the associated standard error and confidence intervals. This paper suggests using the bootstrap procedure as an alternative. Two Monte Carlo simulations were conducted to systematically evaluate the empirical performance of the proposed bootstrap procedure. The results indicated that the bootstrap standard error and confidence intervals were generally accurate across simulation conditions (e.g., selection ratio, sample size). The proposed bootstrap procedure can provide a useful alternative for the estimation of the standard error and confidence intervals for the correlation corrected for indirect range restriction.     

差数显著性t检验与元分析的对比研究

利用计算机构造被试总体、模拟实验研究程序进行抽样研究,探讨显著性ｔ检验方法与元分析方法在检验实验结果数据方面的差异。在模拟实验过程中,ｔ验受到显著性水平、样本容量和总体效果大小的影响,因此最终影响了统计推断的可靠性,建议：在进行显著性检验过程中,应对统计检验能力进行估计;元分析方法以样本为元素对总体进行推断,因此具有很高的准确性和可靠性,它将很有可能成为今后心理学研究的重要统计工具。     

An inferential confidence interval method of establishing statistical equivalence that corrects Tryon's (2001) reduction factor

Evidence of group matching frequently takes the form of a nonsignificant test of statistical difference. Theoretical hypotheses of no difference are also tested in this way. These practices are flawed in that null hypothesis statistical testing provides evidence against the null hypothesis and failing to reject H-sub-0 is not evidence supportive of it. Tests of statistical equivalence are needed. This article corrects the inferential confidence interval (ICI) reduction factor introduced by W. W. Tryon (2001) and uses it to extend his discussion of statistical equivalence. This method is shown to be algebraically equivalent with D. J. Schuirmann's (1987) use of 2 one-sided t tests, a highly regarded and accepted method of testing for statistical equivalence. The ICI method provides an intuitive graphic method for inferring statistical difference as well as equivalence. Trivial difference occurs when a test of difference and a test of equivalence are both passed. Statistical indeterminacy results when both tests are failed. Hybrid confidence intervals are introduced that impose ICI limits on standard confidence intervals. These intervals are recommended as replacements for error bars because they facilitate inferences.     

Using confidence intervals in within-subject designs

We argue that to best comprehend many data sets, plotting judiciously selected sample statistics with associated confidence intervals can usefully supplement, or even replace, standard hypothesis-testing procedures. We note that most social science statistics textbooks limit discussion of confidence intervals to their use in between-subject designs. Our central purpose in this article is to describe how to compute an analogous confidence interval that can be used in within-subject designs. This confidence interval rests on the reasoning that because between-subject variance typically plays no role in statistical analyses of within-subject designs, it can legitimately be ignored; hence, an appropriate confidence interval can be based on the standard within-subject error term—that is, on the variability due to the subject × condition interaction. Computation of such a confidence interval is simple and is embodied in Equation 2 on p. 482 of this article. This confidence interval has two useful properties. First, it is based on the same error term as is the corresponding analysis of variance, and hence leads to comparable conclusions. Second, it is related by a known factor (√2) to a confidence interval of the difference between sample means; accordingly, it can be used to infer the faith one can put in some pattern of sample means as a reflection of the underlying pattern of population means. These two properties correspond to analogous properties of the more widely used between-subject confidence interval.     

Covariances between regression coefficient estimates in a single mediator model

This study presents formulae for the covariances between parameter estimates in a single mediator model. These covariances are necessary to build confidence intervals (CI) for effect size measures in mediation studies. We first analytically derived the covariances between the parameter estimates in a single mediator model. Using the derived covariances, we computed the multivariate‐delta standard errors, and built the 95% CIs for the effect size measures. A simulation study evaluated the accuracy of the standard errors as well as the Type I error, power, and coverage of the CIs using various parameter values and sample sizes. Finally, we presented a numerical example and a SAS MACRO that calculates the CIs for the effect size measures.     

Standard errors and confidence intervals for correlations corrected for indirect range restriction: A simulation study comparing analytic and bootstrap methods

A frequent topic of psychological research is the estimation of the correlation between two variables from a sample that underwent a selection process based on a third variable. Due to indirect range restriction, the sample correlation is a biased estimator of the population correlation, and a correction formula is used. In the past, bootstrap standard error and confidence intervals for the corrected correlations were examined with normal data. The present study proposes a large-sample estimate (an analytic method) for the standard error, and a corresponding confidence interval for the corrected correlation. Monte Carlo simulation studies involving both normal and non-normal data were conducted to examine the empirical performance of the bootstrap and analytic methods. Results indicated that with both normal and non-normal data, the bootstrap standard error and confidence interval were generally accurate across simulation conditions (restricted sample size, selection ratio, and population correlations) and outperformed estimates of the analytic method. However, with certain combinations of distribution type and model conditions, the analytic method has an advantage, offering reasonable estimates of the standard error and confidence interval without resorting to the bootstrap procedure's computer-intensive approach. We provide SAS code for the simulation studies.     

Sample size planning for composite reliability coefficients: accuracy in parameter estimation via narrow confidence intervals

Composite measures play an important role in psychology and related disciplines. Composite measures almost always have error. Correspondingly, it is important to understand the reliability of the scores from any particular composite measure. However, the point estimates of the reliability of composite measures are fallible and thus all such point estimates should be accompanied by a confidence interval. When confidence intervals are wide, there is much uncertainty in the population value of the reliability coefficient. Given the importance of reporting confidence intervals for estimates of reliability, coupled with the undesirability of wide confidence intervals, we develop methods that allow researchers to plan sample size in order to obtain narrow confidence intervals for population reliability coefficients. We first discuss composite reliability coefficients and then provide a discussion on confidence interval formation for the corresponding population value. Using the accuracy in parameter estimation approach, we develop two methods to obtain accurate estimates of reliability by planning sample size. The first method provides a way to plan sample size so that the expected confidence interval width for the population reliability coefficient is sufficiently narrow. The second method ensures that the confidence interval width will be sufficiently narrow with some desired degree of assurance (e.g., 99% assurance that the 95% confidence interval for the population reliability coefficient will be less than W units wide). The effectiveness of our methods was verified with Monte Carlo simulation studies. We demonstrate how to easily implement the methods with easy-to-use and freely available software.     

The value of RCT evidence depends on the quality of statistical analysis

The authors examined statistical practices in 193 randomized controlled trials (RCTs) of psychological therapies published in prominent psychology and psychiatry journals during 1999-2003. Statistical significance tests were used in 99% of RCTs, 84% discussed clinical significance, but only 46% considered-even minimally-statistical power, 31% interpreted effect size and only 2% interpreted confidence intervals. In a second study, 42 respondents to an email survey of the authors of RCTs analyzed in the first study indicated they consider it very important to know the magnitude and clinical importance of the effect, in addition to whether a treatment effect exists. The present authors conclude that published RCTs focus on statistical significance tests ("Is there an effect or difference?"), and neglect other important questions: "How large is the effect?" and "Is the effect clinically important?" They advocate improved statistical reporting of RCTs especially by reporting and interpreting clinical significance, effect sizes and confidence intervals.     

元回归中效应量的最小个数需求：基于统计功效和估计精度

元回归模型被广泛应用于调节变量的识别。从元分析技术的原理谈起, 介绍了元回归模型, 然后采用蒙特卡洛模拟, 基于统计功效和估计精度探究效应量个数对元回归模型参数估计的影响, 从而确立效应量的最小个数需求。主要研究结果为：(1) Wald-type z检验方法在元回归中易犯I类错误; (2)为达到参数估计要求, 元回归至少需要20个效应量; (3)纳入合适的调节变量能降低对效应量的个数需求。基于研究结果, 提出以下建议：(1)研究者应慎重使用Wald-type z检验方法和CMA软件; (2)研究者至少需要20个效应量, 且应当根据实际情况进一步增加效应量个数; (3)研究者应当积极探索合适的调节变量; (4)未来审稿人可参考最小效应量个数需求对元回归研究进行质量评估。     

单维测验合成信度元分析

元分析是根据现有研究对感兴趣的主题得出比较准确和有代表性结论的一种重要方法,在心理、教育、管理、医学等社会科学研究中得到广泛应用。信度是衡量测验质量的重要指标,用合成信度能比较准确的估计测验信度。未见有文献提供合成信度元分析方法。本研究在比较对参数进行元分析的三种模型优劣的基础上,在变化系数模型下推出合成信度元分析点估计及区间估计的方法;以区间覆盖率为衡量指标,模拟研究表明本研究提出的合成信度元分析区间估计的方法得当;举例说明如何对单维测验的合成信度进行元分析。