An introduction to Bayesian hierarchical models with an application in the theory of signal detection

Although many nonlinear models of cognition have been proposed in the past 50 years, there has been little consideration of corresponding statistical techniques for their analysis. In analyses with nonlinear models, unmodeled variability from the selection of items or participants may lead to asymptotically biased estimation. This asymptotic bias, in turn, renders inference problematic. We show, for example, that a signal detection analysis of recognition memory data leads to asymptotic underestimation of sensitivity. To eliminate asymptotic bias, we advocate hierarchical models in which participant variability, item variability, and measurement error are modeled simultaneously. By accounting for multiple sources of variability, hierarchical models yield consistent and accurate estimates of participant and item effects in recognition memory. This article is written in tutorial format; we provide an introduction to Bayesian statistics, hierarchical modeling, and Markov chain Monte Carlo computational techniques.     

Bayesian inference and the classical test theory model: Reliability and true scores

A general one-way analysis of variance components with unequal replication numbers is used to provide unbiased estimates of the true and error score variance of classical test theory. The inadequacy of the ANOVA theory is noted and the foundations for a Bayesian approach are detailed. The choice of prior distribution is discussed and a justification for the Tiao-Tan prior is found in the particular context of the “n-split” technique. The posterior distributions of reliability, error score variance, observed score variance and true score variance are presented with some extensions of the original work of Tiao and Tan. Special attention is given to simple approximations that are available in important cases and also to the problems that arise when the ANOVA estimate of true score variance is negative. Bayesian methods derived by Box and Tiao and by Lindley are studied numerically in relation to the problem of estimating true score. Each is found to be useful and the advantages and disadvantages of each are discussed and related to the classical test-theoretic methods. Finally, some general relationships between Bayesian inference and classical test theory are discussed. Supported in part by the National Institute of Child Health and Human Development under Research Grant 1 PO1 HDO1762. Reproduction, translation, use or disposal by or for the United States Government is permitted.     

Bayesian inference for psychology. Part II: Example applications with JASP

Bayesian hypothesis testing presents an attractive alternative to p value hypothesis testing. Part I of this series outlined several advantages of Bayesian hypothesis testing, including the ability to quantify evidence and the ability to monitor and update this evidence as data come in, without the need to know the intention with which the data were collected. Despite these and other practical advantages, Bayesian hypothesis tests are still reported relatively rarely. An important impediment to the widespread adoption of Bayesian tests is arguably the lack of user-friendly software for the run-of-the-mill statistical problems that confront psychologists for the analysis of almost every experiment: the t-test, ANOVA, correlation, regression, and contingency tables. In Part II of this series we introduce JASP (http://www.jasp-stats.org), an open-source, cross-platform, user-friendly graphical software package that allows users to carry out Bayesian hypothesis tests for standard statistical problems. JASP is based in part on the Bayesian analyses implemented in Morey and Rouder’s BayesFactor package for R. Armed with JASP, the practical advantages of Bayesian hypothesis testing are only a mouse click away.     

BasicBayes: A tutor system for simple Bayesian inference

To date, attempts to teach Bayesian inference to nonexperts have not met with much success. BasicBayes, the computerized tutor presented here, is an attempt to change this state of affairs. BasicBayes is based on a novel theoretical framework about Bayesian reasoning recently introduced by Gigerenzer and Hoffrage (1995). This framework focuses on the connection between “cognitive algorithms” and “information formats.” BasicBayes teaches people how to translate Bayesian text problems into frequency formats, which have been shown to entail computationally simpler cognitive algorithms than those entailed by probability formats. The components and mode of functioning of BasicBayes are described in detail. Empirical evidence demonstrates the effectiveness of BasicBayes in teaching people simple Bayesian inference. Because of its flexible system architecture, BasicBayes can also be used as a research tool.     

Threshold nonlinearities and signal detection theory

The review summarizes Kulikowski's contributions to the mechanisms involved in visual detection and discrimination. His findings provide strong evidence for the existence of 'barriers' which block weak activity in the visual system. It is proposed that the properties of these barriers may underlie near-optimal detection in conditions of signal uncertainty.     

Calculation of signal detection theory measures

Signal detection theory (SDT) may be applied to any area of psychology in which two different types of stimuli must be discriminated. We describe several of these areas and the advantages that can be realized through the application of SDT. Three of the most popular tasks used to study discriminability are then discussed, together with the measures that SDT prescribes for quantifying performance in these tasks. Mathematical formulae for the measures are presented, as are methods for calculating the measures with lookup tables, computer software specifically developed for SDT applications, and general purpose computer software (including spreadsheets and statistical analysis software).     

Bayesian inference for graphical factor analysis models

We generalize factor analysis models by allowing the concentration matrix of the residuals to have nonzero off-diagonal elements. The resulting model is named graphical factor analysis model. Allowing a structure of associations gives information about the correlation left unexplained by the unobserved variables, which can be used both in the confirmatory and exploratory context. We first present a sufficient condition for global identifiability of this class of models with a generic number of factors, thereby extending the results in Stanghellini (1997) and Vicard (2000). We then consider the issue of model comparison and show that fast local computations are possible for this purpose, if the conditional independence graphs on the residuals are restricted to be decomposable and a Bayesian approach is adopted. To achieve this aim, we propose a new reversible jump MCMC method to approximate the posterior probabilities of the considered models. We then study the evolution of political democracy in 75 developing countries based on eight measures of democracy in two different years. We acknowledge support from M.U.R.S.T. of Italy and from the European Science Foundation H.S.S.S. Network. We are grateful to the referees and the Editor for many useful suggestions and comments which led to a substantial improvement of the paper. We also thank Nanny Wermuth for stimulating discussions and Kenneth A. Bollen for kindly providing us with the data-set.     

Bayesian inference for the information gain model

One of the most popular paradigms to use for studying human reasoning involves the Wason card selection task. In this task, the participant is presented with four cards and a conditional rule (e.g., “If there is an A on one side of the card, there is always a 2 on the other side”). Participants are asked which cards should be turned to verify whether or not the rule holds. In this simple task, participants consistently provide answers that are incorrect according to formal logic. To account for these errors, several models have been proposed, one of the most prominent being the information gain model (Oaksford & Chater, Psychological Review, 101, 608–631, 1994). This model is based on the assumption that people independently select cards based on the expected information gain of turning a particular card. In this article, we present two estimation methods to fit the information gain model: a maximum likelihood procedure (programmed in R) and a Bayesian procedure (programmed in WinBUGS). We compare the two procedures and illustrate the flexibility of the Bayesian hierarchical procedure by applying it to data from a meta-analysis of the Wason task (Oaksford & Chater, Psychological Review, 101, 608–631, 1994). We also show that the goodness of fit of the information gain model can be assessed by inspecting the posterior predictives of the model. These Bayesian procedures make it easy to apply the information gain model to empirical data. Supplemental materials may be downloaded along with this article from .     

Source memory for unrecognized items: Predictions from multivariate signal detection theory

We report three experiments investigating source memory for words that were called "new" on a recognition test. In each experiment, participants could accurately specify the source of words that they failed to recognize. Results also demonstrated that source memory for unrecognized items varied with the bias to respond "old" in recognition decisions: Participants displayed unrecognized source memory when they were told that 25% of the recognition test words were old (promoting conservative responding) but not when they were told that 75% of the test words were old (promoting liberal responding). Our results were successfully predicted by a multivariate signal detection approach to recognition/source memory.     

Decision noise: An explanation for observed violations of signal detection theory

In signal detection theory (SDT), responses are governed by perceptual noise and a flexible decision criterion. Recent criticisms of SDT (see, e.g., Balakrishnan, 1999) have identified violations of its assumptions, and researchers have suggested that SDT fundamentally misrepresents perceptual and decision processes. We hypothesize that, instead, these violations of SDT stem from decision noise: the inability to use deterministic response criteria. In order to investigate this hypothesis, we present a simple extension of SDT—the decision noise model—with which we demonstrate that shifts in a decision criterion can be masked by decision noise. In addition, we propose a new statistic that can help identify whether the violations of SDT stem from perceptual or from decision processes. The results of a stimulus classification experiment—together with model fits to past experiments—show that decision noise substantially affects performance. These findings suggest that decision noise is important across a wide range of tasks and needs to be better understood in order to accurately measure perceptual processes.     

Bayesian inference using WBDev: A tutorial for social scientists

Over the last decade, the popularity of Bayesian data analysis in the empirical sciences has greatly increased. This is partly due to the availability of WinBUGS, a free and flexible statistical software package that comes with an array of predefined functions and distributions, allowing users to build complex models with ease. For many applications in the psychological sciences, however, it is highly desirable to be able to define one’s own distributions and functions. This functionality is available through the WinBUGS Development Interface (WBDev). This tutorial illustrates the use of WBDev by means of concrete examples, featuring the expectancyvalence model for risky behavior in decision making, and the shifted Wald distribution of response times in speeded choice.     

Assessment of shock discrimination in rats with signal detection theory

Nociception in six rats was evaluated with signal detection theory (TSD) techniques using a modified “yes-no” paradigm. A .1-sec, .075-mA shock was the discriminative stimulus for a choice to one side of a T-maze, and a .1-sec, .25-mA shock (capable of reliably eliciting a “flinch”) was the discriminative stimulus for the opposite choice. Response bias was manipulated through differential reinforcement for the two choices, and signal sensitivity was manipulated by varying the intensity of the higher intensity shock signal. Individual detection ROCs appeared linear with slopes of one when plotted on double-probability scales. Indices of signal sensitivity and response bias were independent, as predicted by TSD. Morphine sulfate (4 mg/kg) significantly reduced d’ but did not affect β, suggesting that, if appropriate parameters are used, TSD analyses may have value in assessing nociception in rats.     

Source monitoring and multivariate signal detection theory, with a model for selection

Participants in source monitoring studies, in addition to determining whether an item is old or new, also discriminate the source of the item, such as whether the item was presented in a male or female voice. This article shows how to apply multivariate signal detection theory (SDT) to source monitoring. An interesting aspect of one version of the source monitoring procedure, from the perspective of multivariate SDT, is that it involves a type of selection, in that a discrimination response is observed only if the detection decision is that an item is old. If the selection is ignored, then the estimate of the discrimination parameter can be biased; the nature and magnitude of the bias are illustrated. A bivariate signal detection model that recognizes selection is presented and its application is illustrated. The approach to source monitoring via multivariate SDT provides new results that are informative about underlying psychological processes.     

Process dissociation and mixture signal detection theory

The process dissociation procedure was developed in an attempt to separate different processes involved in memory tasks. The procedure naturally lends itself to a formulation within a class of mixture signal detection models. The dual process model is shown to be a special case. The mixture signal detection model is applied to data from a widely analyzed study. The results suggest that a process other than recollection may be involved in the process dissociation procedure.     

Bayesian inference with Stan: A tutorial on adding custom distributions

When evaluating cognitive models based on fits to observed data (or, really, any model that has free parameters), parameter estimation is critically important. Traditional techniques like hill climbing by minimizing or maximizing a fit statistic often result in point estimates. Bayesian approaches instead estimate parameters as posterior probability distributions, and thus naturally account for the uncertainty associated with parameter estimation; Bayesian approaches also offer powerful and principled methods for model comparison. Although software applications such as WinBUGS (Lunn, Thomas, Best, & Spiegelhalter, Statistics and Computing, 10, 325–337, 2000) and JAGS (Plummer, 2003) provide “turnkey”-style packages for Bayesian inference, they can be inefficient when dealing with models whose parameters are correlated, which is often the case for cognitive models, and they can impose significant technical barriers to adding custom distributions, which is often necessary when implementing cognitive models within a Bayesian framework. A recently developed software package called Stan (Stan Development Team, 2015) can solve both problems, as well as provide a turnkey solution to Bayesian inference. We present a tutorial on how to use Stan and how to add custom distributions to it, with an example using the linear ballistic accumulator model (Brown & Heathcote, Cognitive Psychology, 57, 153–178. doi: 10.1016/j.cogpsych.2007.12.002, 2008).     

Functions for traditional and multilevel approaches to signal detection theory

In the present article, functions written in the freeware R are presented that calculate several measures from traditional signal detection theory for each individual in a sample, along with summary statistics for the sample. Bias-corrected and accelerated bootstrap confidence intervals are also produced. Arguments are made for using an alternative approach—multilevel generalized linear models—and a function is presented for it. These functions are part of the R package sdtalt, which is available on the Comprehensive R Archive Network. Recent data from memory recognition studies are used to illustrate these functions.     

Background beliefs in Bayesian inference

We report five experiments in which the role of background beliefs in social judgments of posterior probability was investigated. From a Bayesian perspective, people should combine prior probabilities (or base rates) and diagnostic evidence with equal weighting, although previous research shows that base rates are often underweighted. These experiments were designed so that either piece of information was supplied either by personal beliefs or by presented statistics, and regression analyses were performed on individual participants to assess the relative influence of information. We found that both prior probabilities and diagnostic information significantly influenced judgments, whether supplied by beliefs or by statistical information, but that belief-based information tended to dominate the judgments made.