Towards end-to-end likelihood-free inference with convolutional neural networks

Complex simulator-based models with non-standard sampling distributions require sophisticated design choices for reliable approximate parameter inference. We introduce a fast, end-to-end approach for approximate Bayesian computation (ABC) based on fully convolutional neural networks. The method enables users of ABC to derive simultaneously the posterior mean and variance of multidimensional posterior distributions directly from raw simulated data. Once trained on simulated data, the convolutional neural network is able to map real data samples of variable size to the first two posterior moments of the relevant parameter's distributions. Thus, in contrast to other machine learning approaches to ABC, our approach allows us to generate reusable models that can be applied by different researchers employing the same model. We verify the utility of our method on two common statistical models (i.e., a multivariate normal distribution and a multiple regression scenario), for which the posterior parameter distributions can be derived analytically. We then apply our method to recover the parameters of the leaky competing accumulator (LCA) model and we reference our results to the current state-of-the-art technique, which is the probability density estimation (PDA). Results show that our method exhibits a lower approximation error compared with other machine learning approaches to ABC. It also performs similarly to PDA in recovering the parameters of the LCA model.     

Simple use of BIC to Assess Model Selection Uncertainty: An Illustration using Mediation and Moderation Models

Abstract

The Bayesian information criterion (BIC) has been used sometimes in SEM, even adopting a frequentist approach. Using simple mediation and moderation models as examples, we form posterior probability distribution via using BIC, which we call the BIC posterior, to assess model selection uncertainty of a finite number of models. This is simple but rarely used. The posterior probability distribution can be used to form a credibility set of models and to incorporate prior probabilities for model comparisons and selections. This was validated by a large scale simulation and results showed that the approximation via the BIC posterior is very good except when both the sample sizes and magnitude of parameters are small. We applied the BIC posterior to a real data set, and it has the advantages of flexibility in incorporating prior, addressing overfitting problems, and giving a full picture of posterior distribution to assess model selection uncertainty.     

Model inference or model selection: discussion of Klugkist, Laudy, and Hoijtink (2005)

I. Klugkist, O. Laudy, and H. Hoijtink (2005) presented a Bayesian approach to analysis of variance models with inequality constraints. Constraints may play 2 distinct roles in data analysis. They may represent prior information that allows more precise inferences regarding parameter values, or they may describe a theory to be judged against the data. In the latter case, the authors emphasized the use of Bayes factors and posterior model probabilities to select the best theory. One difficulty is that interpretation of the posterior model probabilities depends on which other theories are included in the comparison. The posterior distribution of the parameters under an unconstrained model allows one to quantify the support provided by the data for inequality constraints without requiring the model selection framework.     

Stochastic search item selection for factor analytic models

In this paper we implement a Markov chain Monte Carlo algorithm based on the stochastic search variable selection method of George and McCulloch (1993) for identifying promising subsets of manifest variables (items) for factor analysis models. The suggested algorithm is constructed by embedding in the usual factor analysis model a normal mixture prior for the model loadings with latent indicators used to identify not only which manifest variables should be included in the model but also how each manifest variable is associated with each factor. We further extend the suggested algorithm to allow for factor selection. We also develop a detailed procedure for the specification of the prior parameters values based on the practical significance of factor loadings using ideas from the original work of George and McCulloch (1993). A straightforward Gibbs sampler is used to simulate from the joint posterior distribution of all unknown parameters and the subset of variables with the highest posterior probability is selected. The proposed method is illustrated using real and simulated data sets.     

Coherent psychometric modelling with Bayesian nonparametrics

In this paper we argue that model selection, as commonly practised in psychometrics, violates certain principles of coherence. On the other hand, we show that Bayesian nonparametrics provides a coherent basis for model selection, through the use of a ‘nonparametric’ prior distribution that has a large support on the space of sampling distributions. We illustrate model selection under the Bayesian nonparametric approach, through the analysis of real questionnaire data. Also, we present ways to use the Bayesian nonparametric framework to define very flexible psychometric models, through the specification of a nonparametric prior distribution that supports all distribution functions for the inverse link, including the standard logistic distribution functions. The Bayesian nonparametric approach provides a coherent method for model selection that can be applied to any statistical model, including psychometric models. Moreover, under a ‘non‐informative’ choice of nonparametric prior, the Bayesian nonparametric approach is easy to apply, and selects the model that maximizes the log likelihood. Thus, under this choice of prior, the approach can be extended to non‐Bayesian settings where the parameters of the competing models are estimated by likelihood maximization, and it can be used with any psychometric software package that routinely reports the model log likelihood.     

A tutorial on approximate Bayesian computation

This tutorial explains the foundation of approximate Bayesian computation (ABC), an approach to Bayesian inference that does not require the specification of a likelihood function, and hence that can be used to estimate posterior distributions of parameters for simulation-based models. We discuss briefly the philosophy of Bayesian inference and then present several algorithms for ABC. We then apply these algorithms in a number of examples. For most of these examples, the posterior distributions are known, and so we can compare the estimated posteriors derived from ABC to the true posteriors and verify that the algorithms recover the true posteriors accurately. We also consider a popular simulation-based model of recognition memory (REM) for which the true posteriors are unknown. We conclude with a number of recommendations for applying ABC methods to solve real-world problems.     

A possible basis for conservatism in signal detection and probabilistic categorization tasks

It is commonly claimed that conservative placement of the criterion in signal detection is due to the form of the utility function of money, to conservatism in the estimation of prior probabilities, or to probability matching tendencies. This article shows how conservatism could be caused by a systematic misconception of the shape of the underlying distributions. An experiment is described in which subjects were asked to make posterior probability judgments after performing numerical analogues of signal detection. The posterior probability judgments were radical, i.e., high posterior probabilities were overestimated and low posterior probabilities were underestimated; if this pattern of radical probability estimation reflects the subjects’ understanding of the underlying distributions, it would account for conservative criterion placement.     

Using priors to formalize theory: Optimal attention and the generalized context model

Formal models in psychology are used to make theoretical ideas precise and allow them to be evaluated quantitatively against data. We focus on one important??but under-used and incorrectly maligned??method for building theoretical assumptions into formal models, offered by the Bayesian statistical approach. This method involves capturing theoretical assumptions about the psychological variables in models by placing informative prior distributions on the parameters representing those variables. We demonstrate this approach of casting basic theoretical assumptions in an informative prior by considering a case study that involves the generalized context model (GCM) of category learning. We capture existing theorizing about the optimal allocation of attention in an informative prior distribution to yield a model that is higher in psychological content and lower in complexity than the standard implementation. We also highlight that formalizing psychological theory within an informative prior distribution allows standard Bayesian model selection methods to be applied without concerns about the sensitivity of results to the prior. We then use Bayesian model selection to test the theoretical assumptions about optimal allocation formalized in the prior. We argue that the general approach of using psychological theory to guide the specification of informative prior distributions is widely applicable and should be routinely used in psychological modeling.     

Bayesian eggs and Bayesian omelettes: reply to Stern (2005)

In this response to Stern's (2005) discussion of Klugkist, Laudy, and Hoijtink (2005), model inference based on posterior probabilities on the parameter space is discussed. Furthermore, the authors respond to Stern's example in which all possible orderings are included via a short discussion of exploratory versus theory-based modeling. Finally, the authors show that the Bayesian approach is flexible and can deal with many types of constraints. This is illustrated using a model with constraints on the differences between means.     

Bayesian model selection of informative hypotheses for repeated measurements

When analyzing repeated measurements data, researchers often have expectations about the relations between the measurement means. The expectations can often be formalized using equality and inequality constraints between (i) the measurement means over time, (ii) the measurement means between groups, (iii) the means adjusted for time-invariant covariates, and (iv) the means adjusted for time-varying covariates. The result is a set of informative hypotheses. In this paper, the Bayes factor is used to determine which hypothesis receives most support from the data. A pivotal element in the Bayesian framework is the specification of the prior. To avoid subjective prior specification, training data in combination with restrictions on the measurement means are used to obtain so-called constrained posterior priors. A simulation study and an empirical example from developmental psychology show that this prior results in Bayes factors with desirable properties.     

Inferential processes in the forced compliance situation: A Bayesian analysis

The attribution made by an observer (O) to an actor in the forced compliance situation was regarded as a probability revision process which can be described by a Bayesian inference model. Os' perceptions of the forced compliance situation were analyzed in terms of the input components into the Bayesian model: prior probabilities of the relevant attitudes and the diagnostic values of the behaviors which the actor may choose. In order to test propositions made by attribution theory about such perceptions (Kelley, 1967;Messick, 1971), Os viewed actors under conditions of Low Inducement (LI) and High Inducement (HI). Before observing the actor's decision, Os estimated the prior probabilities of the relevant attitudes and the conditional probabilities of compliance and refusal given each of the attitudes. After observing the actor's decision, Os estimated the posterior probabilities of the attitudes. As expected, in the LI condition, compared to the HI condition, compliance was seen as less probable and more diagnostic about the actor's attitudes, and the posterior probability of the corresponding attitude was higher. Contrary to expectations, within both conditions, compliance, compared to refusal, was seen as less diagnostic and more probable.     

BRIEF REPORT: THE EQUIVALENCE OF TWO MULTIVARIATE CLASSlFICATION SCHEMES

The equivalence of two multivariate classification schemes is shown when the sizes of the samples drawn from the populations to which assignment is required are identical. One scheme is based on posterior probabilities determined from a Bayesian density function; the second scheme is based on likelihood ratio discriminated scores. Both of these procedures involve prior probabilities; if estimates of these priors are obtained from the identical sample sizes, the equivalence follows.     

Bayesian inference for graphical factor analysis models

We generalize factor analysis models by allowing the concentration matrix of the residuals to have nonzero off-diagonal elements. The resulting model is named graphical factor analysis model. Allowing a structure of associations gives information about the correlation left unexplained by the unobserved variables, which can be used both in the confirmatory and exploratory context. We first present a sufficient condition for global identifiability of this class of models with a generic number of factors, thereby extending the results in Stanghellini (1997) and Vicard (2000). We then consider the issue of model comparison and show that fast local computations are possible for this purpose, if the conditional independence graphs on the residuals are restricted to be decomposable and a Bayesian approach is adopted. To achieve this aim, we propose a new reversible jump MCMC method to approximate the posterior probabilities of the considered models. We then study the evolution of political democracy in 75 developing countries based on eight measures of democracy in two different years. We acknowledge support from M.U.R.S.T. of Italy and from the European Science Foundation H.S.S.S. Network. We are grateful to the referees and the Editor for many useful suggestions and comments which led to a substantial improvement of the paper. We also thank Nanny Wermuth for stimulating discussions and Kenneth A. Bollen for kindly providing us with the data-set.     

Marginal distributions for the estimation of proportions inm groups

A Bayesian Model II approach to the estimation of proportions inm groups (discussed by Novick, Lewis, and Jackson) is extended to obtain posterior marginal distributions for the proportions. It is anticipated that these will be useful in applications (such as Individually Prescribed Instruction) where decisions are to be made separately for each proportion, rather than jointly for the set of proportions. In addition, the approach is extended to allow greater use of prior information than previously and the specification of this prior information is discussed.We are grateful to a reviewer for suggestions that made possible a more concise and complete presentation of our work.     

A generalized,likelihood-free method for posterior estimation

Recent advancements in Bayesian modeling have allowed for likelihood-free posterior estimation. Such estimation techniques are crucial to the understanding of simulation-based models, whose likelihood functions may be difficult or even impossible to derive. However, current approaches are limited by their dependence on sufficient statistics and/or tolerance thresholds. In this article, we provide a new approach that requires no summary statistics, error terms, or thresholds and is generalizable to all models in psychology that can be simulated. We use our algorithm to fit a variety of cognitive models with known likelihood functions to ensure the accuracy of our approach. We then apply our method to two real-world examples to illustrate the types of complex problems our method solves. In the first example, we fit an error-correcting criterion model of signal detection, whose criterion dynamically adjusts after every trial. We then fit two models of choice response time to experimental data: the linear ballistic accumulator model, which has a known likelihood, and the leaky competing accumulator model, whose likelihood is intractable. The estimated posterior distributions of the two models allow for direct parameter interpretation and model comparison by means of conventional Bayesian statistics—a feat that was not previously possible.     

Bayesian Model Comparison for the Order Restricted RC Association Model

Association models constitute an attractive alternative to the usual log-linear models for modeling the dependence between classification variables. They impose special structure on the underlying association by assigning scores on the levels of each classification variable, which can be fixed or parametric. Under the general row-column (RC) association model, both row and column scores are unknown parameters without any restriction concerning their ordinality. However, when the classification variables are ordinal, order restrictions on the scores arise naturally. Under such restrictions, we adopt an alternative parameterization and draw inferences about the equality of adjacent scores using the Bayesian approach. To achieve that, we have constructed a reversible jump Markov chain Monte Carlo algorithm for moving across models of different dimension and estimate accurately the posterior model probabilities which can be used either for model comparison or for model averaging. The proposed methodology is evaluated through a simulation study and illustrated using actual datasets.     

贝叶斯多组比较——渐近测量不变性

在行为科学研究领域中,检验测量工具的测量不变性是进行群体差异比较的前提。目前,多组验证性因子分析(多组CFA)方法被广泛用于检验测量不变性,但是它对跨组等值的限制过于严格,在实际应用中常常存在大量局限。贝叶斯渐近测量不变性方法基于贝叶斯思想的优良特性,放宽了传统多组CFA方法对跨组差异的严格限制,避免了传统方法的问题,具有较高的应用价值。文章详细介绍了贝叶斯渐近测量不变性方法的原理及优势,同时通过实例展示了渐近测量不变性方法在Mplus软件中的具体分析过程。     

Modeling behavior in a clinically diagnostic sequential risk-taking task

This article models the cognitive processes underlying learning and sequential choice in a risk-taking task for the purposes of understanding how they occur in this moderately complex environment and how behavior in it relates to self-reported real-world risk taking. The best stochastic model assumes that participants incorrectly treat outcome probabilities as stationary, update probabilities in a Bayesian fashion, evaluate choice policies prior to rather than during responding, and maintain constant response sensitivity. The model parameter associated with subjective value of gains correlates well with external risk taking. Both the overall approach, which can be expanded as the basic paradigm is varied, and the specific results provide direction for theories of risky choice and for understanding risk taking as a public health problem.     

A visualization technique for Bayesian reasoning

We tested a method for solving Bayesian reasoning problems in terms of spatial relations as opposed to mathematical equations. Participants completed Bayesian problems in which they were given a prior probability and two conditional probabilities and were asked to report the posterior odds. After a pretraining phase in which participants completed problems with no instruction or external support, participants watched a video describing a visualization technique that used the length of bars to represent the probabilities provided in the problem. Participants then completed more problems with a chance to implement the technique by clicking interactive bars on the computer screen. Performance improved dramatically from the pretraining phase to the interactive‐bar phase. Participants maintained improved performance in transfer phases in which they were required to implement the visualization technique with either pencil‐and‐paper or no external medium. Accuracy levels for participants using the visualization technique were very similar to participants trained to solve the Bayes theorem equation. The results showed no evidence of learning across problems in the pretraining phase or for control participants who did not receive training, so the improved performance of participants using the visualization method could be uniquely attributed to the method itself. A classroom sample demonstrated that these benefits extend to instructional settings. The results show that people can quickly learn to perform Bayesian reasoning without using mathematical equations. We discuss ways that a spatial solution method can enhance classroom instruction on Bayesian inference and help students apply Bayesian reasoning in everyday settings.     

Fitting the psychometric function

A constrained generalized maximum likelihood routine for fitting psychometric functions is proposed, which determines optimum values for the complete parameter set--that is, threshold and slope--as well as for guessing and lapsing probability. The constraints are realized by Bayesian prior distributions for each of these parameters. The fit itself results from maximizing the posterior distribution of the parameter values by a multidimensional simplex method. We present results from extensive Monte Carlo simulations by which we can approximate bias and variability of the estimated parameters of simulated psychometric functions. Furthermore, we have tested the routine with data gathered in real sessions of psychophysical experimenting.