Bayes factor approaches for testing interval null hypotheses

Psychological theories are statements of constraint. The role of hypothesis testing in psychology is to test whether specific theoretical constraints hold in data. Bayesian statistics is well suited to the task of finding supporting evidence for constraint, because it allows for comparing evidence for 2 hypotheses against each another. One issue in hypothesis testing is that constraints may hold only approximately rather than exactly, and the reason for small deviations may be trivial or uninteresting. In the large-sample limit, these uninteresting, small deviations lead to the rejection of a useful constraint. In this article, we develop several Bayes factor 1-sample tests for the assessment of approximate equality and ordinal constraints. In these tests, the null hypothesis covers a small interval of non-0 but negligible effect sizes around 0. These Bayes factors are alternatives to previously developed Bayes factors, which do not allow for interval null hypotheses, and may especially prove useful to researchers who use statistical equivalence testing. To facilitate adoption of these Bayes factor tests, we provide easy-to-use software.     

Bayes factor design analysis: Planning for compelling evidence

A sizeable literature exists on the use of frequentist power analysis in the null-hypothesis significance testing (NHST) paradigm to facilitate the design of informative experiments. In contrast, there is almost no literature that discusses the design of experiments when Bayes factors (BFs) are used as a measure of evidence. Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness. We elaborate on three possible BF designs, (a) a fixed-n design, (b) an open-ended Sequential Bayes Factor (SBF) design, where researchers can test after each participant and can stop data collection whenever there is strong evidence for either \(\mathcal {H}_{1}\) or \(\mathcal {H}_{0}\), and (c) a modified SBF design that defines a maximal sample size where data collection is stopped regardless of the current state of evidence. We demonstrate how the properties of each design (i.e., expected strength of evidence, expected sample size, expected probability of misleading evidence, expected probability of weak evidence) can be evaluated using Monte Carlo simulations and equip researchers with the necessary information to compute their own Bayesian design analyses.     

Bayes factors: Prior sensitivity and model generalizability

Model selection is a central issue in mathematical psychology. One useful criterion for model selection is generalizability; that is, the chosen model should yield the best predictions for future data. Some researchers in psychology have proposed that the Bayes factor can be used for assessing model generalizability. An alternative method, known as the generalization criterion, has also been proposed for the same purpose. We argue that these two methods address different levels of model generalizability (local and global), and will often produce divergent conclusions. We illustrate this divergence by applying the Bayes factor and the generalization criterion to a comparison of retention functions. The application of alternative model selection criteria will also be demonstrated within the framework of model generalizability.     

Bayes factors for the linear ballistic accumulator model of decision-making

Evidence accumulation models of decision-making have led to advances in several different areas of psychology. These models provide a way to integrate response time and accuracy data, and to describe performance in terms of latent cognitive processes. Testing important psychological hypotheses using cognitive models requires a method to make inferences about different versions of the models which assume different parameters to cause observed effects. The task of model-based inference using noisy data is difficult, and has proven especially problematic with current model selection methods based on parameter estimation. We provide a method for computing Bayes factors through Monte-Carlo integration for the linear ballistic accumulator (LBA; Brown and Heathcote, 2008), a widely used evidence accumulation model. Bayes factors are used frequently for inference with simpler statistical models, and they do not require parameter estimation. In order to overcome the computational burden of estimating Bayes factors via brute force integration, we exploit general purpose graphical processing units; we provide free code for this. This approach allows estimation of Bayes factors via Monte-Carlo integration within a practical time frame. We demonstrate the method using both simulated and real data. We investigate the stability of the Monte-Carlo approximation, and the LBA’s inferential properties, in simulation studies.     

FACAIC: Model selection algorithm for the orthogonal factor model using AIC and CAIC

This paper describes the authors' FORTRAN algorithm FACAIC for choosing the number of factors for an orthogonal factor model using Akaike's Information Criterion. FACAIC utilizes the IMSL subroutine OFCOMM.The authors dedicate this algorithm to Professor Hirotugu Akaike in appreciation of his pioneering work on AIC which was originally intended for the factor analysis and other statistical model identification problems.     

A tutorial on Bayes factor estimation with the product space method

The Bayes factor is an intuitive and principled model selection tool from Bayesian statistics. The Bayes factor quantifies the relative likelihood of the observed data under two competing models, and as such, it measures the evidence that the data provides for one model versus the other. Unfortunately, computation of the Bayes factor often requires sampling-based procedures that are not trivial to implement. In this tutorial, we explain and illustrate the use of one such procedure, known as the product space method (Carlin & Chib, 1995). This is a transdimensional Markov chain Monte Carlo method requiring the construction of a “supermodel” encompassing the models under consideration. A model index measures the proportion of times that either model is visited to account for the observed data. This proportion can then be transformed to yield a Bayes factor. We discuss the theory behind the product space method and illustrate, by means of applied examples from psychological research, how the method can be implemented in practice.     

Cooperative selection of movements: The optimal selection model

How one selects a movement when faced with alternative ways of doing a task is a central problem in human motor control. Moving the fingertip a short distance can be achieved with any of an infinite number of combinations of knuckle, wrist, elbow, shoulder, and hip movements. The question therefore arises: how is a unique combination chosen? In our model, choice is achieved by consideration of the similarity between the task requirements and the optimal biomechanical performance of each limb segment. Two variants of the model account for the movements that are selected when subjects freely oscillate the fingertip and when they tap against an obstacle. An important feature of both is that the impulse of collision with an obstacle (as in drumming with the hand or tapping with the finger) is assumed to be controlled in part by aiming for a point beyond the surface being struck. Thus, a force-related control variable may be represented and controlled spatially.     

Bayesian nonparametric model selection and model testing

This article examines a Bayesian nonparametric approach to model selection and model testing, which is based on concepts from Bayesian decision theory and information theory. The approach can be used to evaluate the predictive-utility of any model that is either probabilistic or deterministic, with that model analyzed under either the Bayesian or classical-frequentist approach to statistical inference. Conditional on an observed set of data, generated from some unknown true sampling density, the approach identifies the “best” model as the one that predicts a sampling density that explains the most information about the true density. Furthermore, in the approach, the decision is to reject a model when it does not explain enough information about the true density (according to a straightforward calibration of the Kullback-Leibler divergence measure). The posterior estimate of the true density is based on a Bayesian nonparametric prior that can give positive support to the entire space of sampling densities (defined on some sample space). This article also discusses the theoretical and practical advantages of the Bayesian nonparametric approach over all other types of model selection procedures, and over any model testing procedure that depends on interpreting a p-value. Finally, the Bayesian nonparametric approach is illustrated on four real data sets, in the comparison and testing of order-constrained models, cognitive models, models of choice-behavior, and a test of a general psychometric model.     

Stepwise variable selection in factor analysis

It is very important to choose appropriate variables to be analyzed in multivariate analysis when there are many observed variables such as those in a questionnaire. What is actually done in scale construction with factor analysis is nothing but variable selection.In this paper, we take several goodness-of-fit statistics as measures of variable selection and develop backward elimination and forward selection procedures in exploratory factor analysis. Once factor analysis is done for a certain numberp of observed variables (thep-variable model is labeled the current model), simple formulas for predicted fit measures such as chi-square, GFI, CFI, IFI and RMSEA, developed in the field of the structural equation modeling, are provided for all models obtained by adding an external variable (so that the number of variables isp + 1) and for those by deleting an internal variable (so that the number isp – 1), provided that the number of factors is held constant.A programSEFA (Stepwise variable selection in Exploratory Factor Analysis) is developed to actually obtain a list of the fit measures for all such models. The list is very useful in determining which variable should be dropped from the current model to improve the fit of the current model. It is also useful in finding a suitable variable that may be added to the current model. A model with more appropriate variables makes more stable inference in general.The criteria traditionally often used for variable selection is magnitude of communalities. This criteria gives a different choice of variables and does not improve fit of the model in most cases.The URL of the programSEFA is http://koko15.hus.osaka-u.ac.jp/~harada/factor/stepwise/.     

Prior sensitivity in theory testing: An apologia for the Bayes factor

A commonly voiced concern with the Bayes factor is that, unlike many other Bayesian and non-Bayesian quantitative measures of model evaluation, it is highly sensitive to the parameter prior. This paper argues that, when dealing with psychological models that are quantitatively instantiated theories, being sensitive to the prior is an attractive feature of a model evaluation measure. This assertion follows from the observation that in psychological models parameters are not completely unknown, but correspond to psychological variables about which theory often exists. This theory can be formally captured in the prior range and prior distribution of the parameters, indicating which parameter values are allowed, likely, unlikely and forbidden. Because the prior is a vehicle for expressing psychological theory, it should, like the model equation, be considered as an integral part of the model. It is argued that the combined practice of building models using informative priors, and evaluating models using prior sensitive measures advances knowledge.     

贝叶斯因子及其在JASP中的实现

统计推断在科学研究中起到关键作用, 然而当前科研中最常用的经典统计方法——零假设检验(Null hypothesis significance test, NHST)却因难以理解而被部分研究者误用或滥用。有研究者提出使用贝叶斯因子(Bayes factor)作为一种替代和(或)补充的统计方法。贝叶斯因子是贝叶斯统计中用来进行模型比较和假设检验的重要方法, 其可以解读为对零假设H₀或者备择假设H₁的支持程度。其与NHST相比有如下优势：同时考虑H₀和H₁并可以用来支持H₀、不“严重”地倾向于反对H₀、可以监控证据强度的变化以及不受抽样计划的影响。目前, 贝叶斯因子能够很便捷地通过开放的统计软件JASP实现, 本文以贝叶斯t检验进行示范。贝叶斯因子的使用对心理学研究者来说具有重要的意义, 但使用时需要注意先验分布选择的合理性以及保持数据分析过程的透明与公开。     

The effects of selection in factor analysis

Factorial results are affected by selection of subjects and by selection of tests. It is shown that the addition of one or more tests which are linear combinations of tests already in a battery causes the addition of one or more incidental factors. If the given test battery reveals a simple structure, the addition of tests which are linear combinations of the given tests leaves the structure unaffected unless the number of incidental factors is so large that the common factors become indeterminate.     

The FeatureGate model of visual selection

The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate. Received: 4 July 1997 / Accepted: 23 July 1998     

A quantitative model of optimal data selection in Wason's selection task.

The optimal data selection model proposed by Oaksford and Chater (1994) successfully formalized Wason's selection task (Wason, 1966). The model, however, involved some questionable assumptions and was also not sufficient as a model of the task because it could not provide quantitative predictions of the card selection frequencies. In this paper, the model was revised to provide quantitative fits to the data. The model can predict the selection frequencies of cards based on a selection tendency function (STF), or conversely, it enables the estimation of subjective probabilities from data. Past experimental data were first re-analysed based on the model. In Experiment 1, the superiority of the revised model was shown. However, when the relationship between antecedent and consequent was forced to deviate from the biconditional form, the model was not supported. In Experiment 2, it was shown that sufficient emphasis on probabilistic information can affect participants' performance. A detailed experimental method to sort participants by probabilistic strategies was introduced. Here, the model was supported by a subgroup of participants who used the probabilistic strategy. Finally, the results were discussed from the viewpoint of adaptive rationality.     

AIC model selection using Akaike weights

The Akaike information criterion (AIC; Akaike, 1973) is a popular method for comparing the adequacy of multiple, possibly nonnested models. Current practice in cognitive psychology is to accept a single model on the basis of only the “raw” AIC values, making it difficult to unambiguously interpret the observed AIC differences in terms of a continuous measure such as probability. Here we demonstrate that AIC values can be easily transformed to so-called Akaike weights (e.g., Akaike, 1978, 1979; Bozdogan, 1987; Burnham & Anderson, 2002), which can be directly interpreted as conditional probabilities for each model. We show by example how these Akaike weights can greatly facilitate the interpretation of the results of AIC model comparison procedures.     

Flexibility versus generalizability in model selection

Which quantitative method should be used to choose among competing mathematical models of cognition? Massaro, Cohen, Campbell, and Rodriguez (2001) favor root mean squared deviation (RMSD), choosing the model that provides the best fit to the data. Their simulation results appear to legitimize its use for comparing two models of information integration because it performed just as well as Bayesian model selection (BMS), which had previously been shown by Myung and Pitt (1997) to be a superior alternative selection method because it considers a model's complexity in addition to its fit. In the present study, after contrasting the theoretical approaches to model selection espoused by Massaro et al. and Myung and Pitt, we discuss the cause of the inconsistencies by expanding on the simulations of Massaro et al. Findings demonstrate that the results from model recovery simulations can be misleading if they are not interpreted relative to the data on which they were evaluated, and that BMS is a more robust selection method.     

Stochastic search item selection for factor analytic models

In this paper we implement a Markov chain Monte Carlo algorithm based on the stochastic search variable selection method of George and McCulloch (1993) for identifying promising subsets of manifest variables (items) for factor analysis models. The suggested algorithm is constructed by embedding in the usual factor analysis model a normal mixture prior for the model loadings with latent indicators used to identify not only which manifest variables should be included in the model but also how each manifest variable is associated with each factor. We further extend the suggested algorithm to allow for factor selection. We also develop a detailed procedure for the specification of the prior parameters values based on the practical significance of factor loadings using ideas from the original work of George and McCulloch (1993). A straightforward Gibbs sampler is used to simulate from the joint posterior distribution of all unknown parameters and the subset of variables with the highest posterior probability is selected. The proposed method is illustrated using real and simulated data sets.     

A conditional logit model of collegiate major selection

The choice of a major by a college student is viewed as depending upon the quantity of various outputs (such as income and status) each major provides and the taste structure of the student for these outputs. Tradeoffs in outputs occur so that high status may overcome low-income potential in the occupation into which the major provides entry. Different taste structures among students mean tradeoffs occur at different rates, so that taste structues are best associated with student major selection in a probability sense. Changes in the probability of selecting a collegiate major are associated with changes in student taste structures via a conditional logit model. The taste structure of a student is hypothesized to depend upon the environment in which the student was raised and the environment in which the student now lives. The relationship between environment and taste is also best stated in a probability sense and a second logit model is used to describe this relationship. Estimation of the two logit models, using survey data, attempts to quantify the hypothesized route from environment to taste structure to major selection for entry into an occupation. An example of this route is the finding that women students place more importance on obtaining high income than do men students and the greater the importance of high income, the lower the probability of selecting an education major, the greater the probability of selecting a humanities or social science major. Other results of estimation are presented and discussed.