Bayes factor design analysis: Planning for compelling evidence

A sizeable literature exists on the use of frequentist power analysis in the null-hypothesis significance testing (NHST) paradigm to facilitate the design of informative experiments. In contrast, there is almost no literature that discusses the design of experiments when Bayes factors (BFs) are used as a measure of evidence. Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness. We elaborate on three possible BF designs, (a) a fixed-n design, (b) an open-ended Sequential Bayes Factor (SBF) design, where researchers can test after each participant and can stop data collection whenever there is strong evidence for either \(\mathcal {H}_{1}\) or \(\mathcal {H}_{0}\), and (c) a modified SBF design that defines a maximal sample size where data collection is stopped regardless of the current state of evidence. We demonstrate how the properties of each design (i.e., expected strength of evidence, expected sample size, expected probability of misleading evidence, expected probability of weak evidence) can be evaluated using Monte Carlo simulations and equip researchers with the necessary information to compute their own Bayesian design analyses.     

Prediction in future samples studied in terms of the gain from selection

The gain from selection (GS) is defined as the standardized average performance of a group of subjects selected in a future sample using a regression equation derived on an earlier sample. Expressions for the expected value, density, and distribution function (DF) of GS are derived and studied in terms of sample size, number of predictors, and the prior distribution assigned to the population multiple correlation. The DF of GS is further used to determine how large sample sizes must be so that with probability .90 (.95), the expected GS will be within 90 percent of its maximum possible value. An approximately unbiased estimator of the expected GS is also derived.     

2PL模型的两种马尔可夫蒙特卡洛缺失数据处理方法比较

马尔科夫蒙特卡洛（MCMC）是项目反应理论中处理缺失数据的一种典型方法。文章通过模拟研究比较了在不同被试人数,项目数,缺失比例下两种MCMC方法（M-H within Gibbs和DA-T Gibbs）参数估计的精确性,并结合了实证研究。研究结果表明,两种方法是有差异的,项目参数估计均受被试人数影响很大,受缺失比例影响相对更小。在样本较大缺失比例较小时,M-H within Gibbs参数估计的均方误差（RMSE）相对略小,随着样本数的减少或缺失比例的增加,DA-T Gibbs方法逐渐优于M-H within Gibbs方法     

Nearly unbiased estimators for the three‐parameter Weibull distribution with greater efficiency than the iterative likelihood method

The maximum likelihood estimation (MLE) method is the most commonly used method to estimate the parameters of the three‐parameter Weibull distribution. However, it returns biased estimates. In this paper, we show how to calculate weights which cancel the biases contained in the MLE equations. The exact weights can be computed when the population parameters are known and the expected weights when they are not. Two of the three weights' expected values are dependent only on the sample size, whereas the third also depends on the population shape parameters. Monte Carlo simulations demonstrate the practicability of the weighted MLE method. When compared with the iterative MLE technique, the bias is reduced by a factor of 7 (irrespective of the sample size) and the variability of the parameter estimates is also reduced by a factor of 7 for very small sample sizes, but this gain disappears for large sample sizes.     

Automatic Bayes Factors for Testing Equality- and Inequality-Constrained Hypotheses on Variances

In comparing characteristics of independent populations, researchers frequently expect a certain structure of the population variances. These expectations can be formulated as hypotheses with equality and/or inequality constraints on the variances. In this article, we consider the Bayes factor for testing such (in)equality-constrained hypotheses on variances. Application of Bayes factors requires specification of a prior under every hypothesis to be tested. However, specifying subjective priors for variances based on prior information is a difficult task. We therefore consider so-called automatic or default Bayes factors. These methods avoid the need for the user to specify priors by using information from the sample data. We present three automatic Bayes factors for testing variances. The first is a Bayes factor with equal priors on all variances, where the priors are specified automatically using a small share of the information in the sample data. The second is the fractional Bayes factor, where a fraction of the likelihood is used for automatic prior specification. The third is an adjustment of the fractional Bayes factor such that the parsimony of inequality-constrained hypotheses is properly taken into account. The Bayes factors are evaluated by investigating different properties such as information consistency and large sample consistency. Based on this evaluation, it is concluded that the adjusted fractional Bayes factor is generally recommendable for testing equality- and inequality-constrained hypotheses on variances.     

Comparison of a two-stage and three-stage interim-analysis procedure.

A statistical model for combining p values from multiple tests of significance is used to define rejection and acceptance regions for two-stage and three-stage sampling plans. Type I error rates, power, frequencies of early termination decisions, and expected sample sizes are compared. Both the two-stage and three-stage procedures provide appropriate protection against Type I errors. The two-stage sampling plan with its single interim analysis entails minimal loss in power and provides substantial reduction in expected sample size as compared with a conventional single end-of-study test of significance for which power is in the adequate range. The three-stage sampling plan with its two interim analyses introduces somewhat greater reduction in power, but it compensates with greater reduction in expected sample size. Either interim-analysis strategy is more efficient than a single end-of-study analysis in terms of power per unit of sample size.     

Sample size determination for at test given at value from a previous study: A FORTRAN 77 program

When uncertain about the magnitude of an effect, researchers commonly substitute in the standard sample-size-determination formula an estimate of effect size derived from a previous experiment. A problem with this approach is that the traditional sample-size-determination formula was not designed to deal with the uncertainty inherent in an effect-size estimate. Consequently, estimate-substitution in the traditional sample-size-determination formula can lead to a substantial loss of power. A method of sample-size determination designed to handle uncertainty in effect-size estimates is described. The procedure uses thet value and sample size from a previous study, which might be a pilot study or a related study in the same area, to establish a distribution of probable effect sizes. The sample size to be employed in the new study is that which supplies an expected power of the desired amount over the distribution of probable effect sizes. A FORTRAN 77 program is presented that permits swift calculation of sample size for a variety oft tests, including independentt tests, relatedt tests,t tests of correlation coefficients, andt tests of multiple regressionb coefficients.     

A hierarchical bayesian statistical framework for response time distributions

This paper provides a statistical framework for estimating higher-order characteristics of the response time distribution, such as the scale (variability) and shape. Consideration of these higher order characteristics often provides for more rigorous theory development in cognitive and perceptual psychology (e.g., Luce, 1986). RT distribution for a single participant depends on certain participant characteristics, which in turn can be thought of as arising from a distribution of latent variables. The present work focuses on the three-parameter Weibull distribution, with parameters for shape, scale, and shift (initial value). Bayesian estimation in a hierarchical framework is conceptually straightforward. Parameter estimates, both for participant quantities and population parameters, are obtained through Markov Chain Monte Carlo methods. The methods are illustrated with an application to response time data in an absolute identification task. The behavior of the Bayes estimates are compared to maximum likelihood (ML) estimates through Monte Carlo simulations. For small sample size, there is an occasional tendency for the ML estimates to be unreasonably extreme. In contrast, by borrowing strength across participants, Bayes estimation shrinks extreme estimates. The results are that the Bayes estimators are more accurate than the corresponding ML estimators.We are grateful to Michael Stadler who allowed us use of his data. This research is supported by (a) National Science Foundation Grant SES-0095919 to J. Rouder, D. Sun, and P. Speckman, (b) University of Missouri Research Board Grant 00-77 to J. Rouder, (c) National Science Foundation grant DMS-9972598 to Sun and Speckman, and (d) a grant from the Missouri Department of Conservation to D. Sun.     

Structural Equation Modeling with Small Samples: Test Statistics

Structural equation modeling is a well-known technique for studying relationships among multivariate data. In practice, high dimensional nonnormal data with small to medium sample sizes are very common, and large sample theory, on which almost all modeling statistics are based, cannot be invoked for model evaluation with test statistics. The most natural method for nonnormal data, the asymptotically distribution free procedure, is not defined when the sample size is less than the number of nonduplicated elements in the sample covariance. Since normal theory maximum likelihood estimation remains defined for intermediate to small sample size, it may be invoked but with the probable consequence of distorted performance in model evaluation. This article studies the small sample behavior of several test statistics that are based on maximum likelihood estimator, but are designed to perform better with nonnormal data. We aim to identify statistics that work reasonably well for a range of small sample sizes and distribution conditions. Monte Carlo results indicate that Yuan and Bentler's recently proposed F-statistic performs satisfactorily.     

Logistic Approximation to the Normal: The KL Rationale

A rationale is proposed for approximating the normal distribution with a logistic distribution using a scaling constant based on minimizing the Kullback–Leibler (KL) information, that is, the expected amount of information available in a sample to distinguish between two competing distributions using a likelihood ratio (LR) test, assuming one of them is true. The new constant 1.749, computed assuming the normal distribution is true, yields an approximation that is an improvement in fit of the tails of the distribution as compared to the minimax constant of 1.702, widely used in item response theory (IRT). The minimax constant is by definition marginally better in its overall maximal error. It is argued that the KL constant is more statistically appropriate for use in IRT. The author would like to thank Sebastian Schreiber for his generous assistance with this project.     

Bayes factors for testing inequality constrained hypotheses: Issues with prior specification

Several issues are discussed when testing inequality constrained hypotheses using a Bayesian approach. First, the complexity (or size) of the inequality constrained parameter spaces can be ignored. This is the case when using the posterior probability that the inequality constraints of a hypothesis hold, Bayes factors based on non‐informative improper priors, and partial Bayes factors based on posterior priors. Second, the Bayes factor may not be invariant for linear one‐to‐one transformations of the data. This can be observed when using balanced priors which are centred on the boundary of the constrained parameter space with a diagonal covariance structure. Third, the information paradox can be observed. When testing inequality constrained hypotheses, the information paradox occurs when the Bayes factor of an inequality constrained hypothesis against its complement converges to a constant as the evidence for the first hypothesis accumulates while keeping the sample size fixed. This paradox occurs when using Zellner's g prior as a result of too much prior shrinkage. Therefore, two new methods are proposed that avoid these issues. First, partial Bayes factors are proposed based on transformed minimal training samples. These training samples result in posterior priors that are centred on the boundary of the constrained parameter space with the same covariance structure as in the sample. Second, a g prior approach is proposed by letting g go to infinity. This is possible because the Jeffreys–Lindley paradox is not an issue when testing inequality constrained hypotheses. A simulation study indicated that the Bayes factor based on this g prior approach converges fastest to the true inequality constrained hypothesis.     

A FORTRAN 77 program for sample-size determination in replication attempts when effect size is uncertain

A method of sample-size determination for use in attempts to replicate experiments is described. It is appropriate in situations where there is uncertainty about the magnitude of the effect under investigation. The procedure uses information supplied by the original experiment to establish a distribution of probable effect sizes. The sample size to be used in a replication study is that which provides an expected power of the desired amount over the distribution of probable effect sizes A FORTRAN 77 program is presented that permits rapid calculation of sample size in replication attempts employing comparisons of means, correlation coefficients, or proportions.     

Determining the sample size for a replication attempt: A short and simple microcomputer program

Replication studies frequently fail to detect genuine effects because too few subjects are employed to yield an acceptable level of power. To remedy this situation, a method of sample size determination in replication attempts is described that uses information supplied by the original experiment to establish a distribution of probable effect sizes. The sample size to be employed is that which supplies an expected power of the desired amount over the distribution of probable effect sizes. The method may be used in replication attempts involving the comparison of means, the comparison of correlation coefficients, and the comparison of proportions. The widely available equation-solving program EUREKA provides a rapid means of executing the method on a microcomputer. Only ten lines are required to represent the method as a set of equations in EUREKA’s language. Such an equation file is readily modified, so that even inexperienced users find it a straightforward means of obtaining the sample size for a variety of designs.     

Estimation of the Coefficient of Variation with Minimum Risk: A Sequential Method for Minimizing Sampling Error and Study Cost

The coefficient of variation is an effect size measure with many potential uses in psychology and related disciplines. We propose a general theory for a sequential estimation of the population coefficient of variation that considers both the sampling error and the study cost, importantly without specific distributional assumptions. Fixed sample size planning methods, commonly used in psychology and related fields, cannot simultaneously minimize both the sampling error and the study cost. The sequential procedure we develop is the first sequential sampling procedure developed for estimating the coefficient of variation. We first present a method of planning a pilot sample size after the research goals are specified by the researcher. Then, after collecting a sample size as large as the estimated pilot sample size, a check is performed to assess whether the conditions necessary to stop the data collection have been satisfied. If not an additional observation is collected and the check is performed again. This process continues, sequentially, until a stopping rule involving a risk function is satisfied. Our method ensures that the sampling error and the study costs are considered simultaneously so that the cost is not higher than necessary for the tolerable sampling error. We also demonstrate a variety of properties of the distribution of the final sample size for five different distributions under a variety of conditions with a Monte Carlo simulation study. In addition, we provide freely available functions via the MBESS package in R to implement the methods discussed.     

The robustness of estimates of total indirect effects in covariance structure models estimated by maximum

The large sample distribution of total indirect effects in covariance structure models in well known. Using Monte Carlo methods, this study examines the applicability of the large sample theory to maximum likelihood estimates oftotal indirect effects in sample sizes of 50, 100, 200, 400, and 800. Two models are studied. Model 1 is a recursive model with observable variables and Model 2 is a nonrecursive model with latent variables. For the large sample theory to apply, the results suggest that sample szes of 200 or more and 400 or more are required for models such as Model 1 and Model 2, respectively.For helpful comments on a previous draft of this paper, we are grateful to Gerhard Arminger, Clifford C. Clogg, and several anonymous reviewers.     

A Bayesian Power Analysis Procedure Considering Uncertainty in Effect Size Estimates from a Meta-analysis

In conventional frequentist power analysis, one often uses an effect size estimate, treats it as if it were the true value, and ignores uncertainty in the effect size estimate for the analysis. The resulting sample sizes can vary dramatically depending on the chosen effect size value. To resolve the problem, we propose a hybrid Bayesian power analysis procedure that models uncertainty in the effect size estimates from a meta-analysis. We use observed effect sizes and prior distributions to obtain the posterior distribution of the effect size and model parameters. Then, we simulate effect sizes from the obtained posterior distribution. For each simulated effect size, we obtain a power value. With an estimated power distribution for a given sample size, we can estimate the probability of reaching a power level or higher and the expected power. With a range of planned sample sizes, we can generate a power assurance curve. Both the conventional frequentist and our Bayesian procedures were applied to conduct prospective power analyses for two meta-analysis examples (testing standardized mean differences in example 1 and Pearson's correlations in example 2). The advantages of our proposed procedure are demonstrated and discussed.     

An investigation of hierarchical Bayes procedures in item response theory

Hierarchical Bayes procedures for the two-parameter logistic item response model were compared for estimating item and ability parameters. Simulated data sets were analyzed via two joint and two marginal Bayesian estimation procedures. The marginal Bayesian estimation procedures yielded consistently smaller root mean square differences than the joint Bayesian estimation procedures for item and ability estimates. As the sample size and test length increased, the four Bayes procedures yielded essentially the same result.The authors wish to thank the Editor and anonymous reviewers for their insightful comments and suggestions.     

Understanding the effects of sample size on the variability of the mean

In the first three experiments, we attempted to learn more about subjects' understanding of the importance of sample size by systematically changing aspects of the problems we gave to subjects. In a fourth study, understanding of the effects of sample size was tested as subjects went through a computerassisted training procedure that dealt with random sampling and the sampling distribution of the mean. Subjects used sample size information more appropriately for problems that were stated in terms of the accuracy of the sample average or the center of the sampling distribution than for problems stated in terms of the tails of the sampling distribution. Apparently, people understand that the means of larger samples are more likely to resemble the population mean but not the implications of this fact for the variability of the mean. The fourth experiment showed that although instruction about the sampling distribution of the mean led to better understanding of the effects of sample size, subjects were still unable to make correct inferences about the variability of the mean. The appreciation that people have for some aspects of the law of large numbers does not seem to result from an in-depth understanding of the relation between sample size and variability.     

The Impact of the Safe Conversations Workshop With Diverse,Low-Income Couples

The purpose of this study was to investigate the effectiveness of the Safe Conversations workshop with low income, racial and ethnic minority couples. A sample of 156 individuals (N = 156) participated in a relationship workshop developed by Harville Hendrix and Helen LaKelly Hunt which utilized the theory and practice of Imago Relationship Therapy (IRT). Participants were administered the Dyadic Adjustment Scale before and after the workshop. Findings indicated statistical significance in their overall DAS score and a large effect size in one of four relationship subscales: Consensus. Both males and females were found to derive benefits from the workshop at equal rates.