Research report — Beguiled by Bayes: a re-analysis of Mossman and Hart's estimates of malingering

Bayes' theorem has been successfully applied to clinical decision making in circumstances where the prevalence rate is both stable and well established. An attempt by Mossman and Hart (1996) to apply Bayes' theorem to malingering is critically examined. When marked fluctuations in the base rates of malingering (both within and across settings) are considered, this model appears to have little clinical utility. For the sake of illustration, Mossman and Hart data were reanalyzed, taking into account the pronounced variation in base rates. Within a 90% confidence level, the estimates of malingering for many scores extended the full range from 0% to 100%. © 1998 John Wiley & Sons, Ltd.     

SIGNIFICANCE TESTS HAVE THEIR PLACE

Abstract— Null-hypothesis significance tests (NHST), properly used, tell us whether we have sufficient evidence to be confident of the sign of the population effect—but only if we abandon two-valued logic in favor of Kaiser's (1960) three-alternative hypothesis tests Confidence intervals provide a useful addition to NHSTs, and can be used to provide the same sign-determination function as NHST However, when so used, confidence intervals are subject to exactly the same Type I, II, and III error rates as NHST In addition, NHSTs provide two pieces of information about our data—maximum probability of a Type III error and probability of a successful exact replication—that confidence intervals do not The proposed alternative to NHST is just as susceptible to misinterpretation as is NHST The problem of bias due to censoring of data collection or publication can be handled by providing archives for all methodologically sound data sets, but reserving interpretations and conclusions for statistically significant results.     

Children can solve Bayesian problems: the role of representation in mental computation

Can children reason the Bayesian way? We argue that the answer to this question depends on how numbers are represented, because a representation can do part of the computation. We test, for the first time, whether Bayesian reasoning can be elicited in children by means of natural frequencies. We show that when information was presented to fourth, fifth, and sixth graders in terms of probabilities, their ability to estimate the Bayesian posterior probability was zero. Yet when the same information was presented in natural frequencies, Bayesian reasoning showed a steady increase from fourth to sixth grade, reaching an average level of 19, 39, and 53%, respectively, in two studies. Sixth graders' performance with natural frequencies matched the performance of adults with probabilities. But this general increase was accompanied by striking individual differences. More than half of the sixth graders solved most or all problems, whereas one third could not solve a single one. An analysis of the children's responses provides evidence for the use of three non-Bayesian strategies. These follow an overlapping wave model of development and continue to be observed in the minds of adults. More so than adults' probabilistic reasoning, children's reasoning depends on a proper representation of information.     

MorePower 6.0 for ANOVA with relational confidence intervals and Bayesian analysis

MorePower 6.0 is a flexible freeware statistical calculator that computes sample size, effect size, and power statistics for factorial ANOVA designs. It also calculates relational confidence intervals for ANOVA effects based on formulas from Jarmasz and Hollands (Canadian Journal of Experimental Psychology 63:124–138, 2009), as well as Bayesian posterior probabilities for the null and alternative hypotheses based on formulas in Masson (Behavior Research Methods 43:679–690, 2011). The program is unique in affording direct comparison of these three approaches to the interpretation of ANOVA tests. Its high numerical precision and ability to work with complex ANOVA designs could facilitate researchers’ attention to issues of statistical power, Bayesian analysis, and the use of confidence intervals for data interpretation. MorePower 6.0 is available at https://wiki.usask.ca/pages/viewpageattachments.action?pageId=420413544.     

Within-subject consistency and between-subject variability in Bayesian reasoning strategies

It is well known that people tend to perform poorly when asked to determine a posterior probability on the basis of a base rate, true positive rate, and false positive rate. The present experiments assessed the extent to which individual participants nevertheless adopt consistent strategies in these Bayesian reasoning problems, and investigated the nature of these strategies. In two experiments, one laboratory-based and one internet-based, each participant completed 36 problems with factorially manipulated probabilities. Many participants applied consistent strategies involving use of only one of the three probabilities provided in the problem, or additive combination of two of the probabilities. There was, however, substantial variability across participants in which probabilities were taken into account. In the laboratory experiment, participants’ eye movements were tracked as they read the problems. There was evidence of a relationship between information use and attention to a source of information. Participants’ self-assessments of their performance, however, revealed little confidence that the strategies they applied were actually correct. These results suggest that the hypothesis of base rate neglect actually underestimates people’s difficulty with Bayesian reasoning, but also suggest that participants are aware of their ignorance.     

Confirmation and reduction: a Bayesian account

Various scientific theories stand in a reductive relation to each other. In a recent article, we have argued that a generalized version of the Nagel-Schaffner model (GNS) is the right account of this relation. In this article, we present a Bayesian analysis of how GNS impacts on confirmation. We formalize the relation between the reducing and the reduced theory before and after the reduction using Bayesian networks, and thereby show that, post-reduction, the two theories are confirmatory of each other. We then ask when a purported reduction should be accepted on epistemic grounds. To do so, we compare the prior and posterior probabilities of the conjunction of both theories before and after the reduction and ask how well each is confirmed by the available evidence.     

HMMTree: A computer program for latent-class hierarchical multinomial processing tree models

Latent-class hierarchical multinomial models are an important extension of the widely used family of multinomial processing tree models, in that they allow for testing the parameter homogeneity assumption and provide a framework for modeling parameter heterogeneity. In this article, the computer program HMMTree is introduced as a means of implementing latent-class hierarchical multinomial processing tree models. HMMTree computes parameter estimates, confidence intervals, and goodness-of-fit statistics for such models, as well as the Fisher information, expected category means and variances, and posterior probabilities for class membership. A brief guide to using the program is provided.     

Statistical Data and Mathematical Propositions

Statistical tests of the primality of some numbers look similar to statistical tests of many nonmathematical, clearly empirical propositions. Yet interpretations of probability prima facie appear to preclude the possibility of statistical tests of mathematical propositions. For example, it is hard to understand how the statement that n is prime could have a frequentist probability other than 0 or 1. On the other hand, subjectivist approaches appear to be saddled with ‘coherence’ constraints on rational probabilities that require rational agents to assign extremal probabilities to logical and mathematical propositions. In the light of these problems, many philosophers have come to think that there must be some way to generalize a Bayesian statistical account. In this article I propose that a classical frequentist approach should be reconsidered. I conclude that we can give a conditional justification of statistical testing of at least some mathematical hypotheses: if statistical tests provide us with reasons to believe or bet on empirical hypotheses in the standard situations, then they also provide us with reasons to believe or bet on mathematical hypotheses in the structurally similar mathematical cases.     

Bayesian data analysis for newcomers

This article explains the foundational concepts of Bayesian data analysis using virtually no mathematical notation. Bayesian ideas already match your intuitions from everyday reasoning and from traditional data analysis. Simple examples of Bayesian data analysis are presented that illustrate how the information delivered by a Bayesian analysis can be directly interpreted. Bayesian approaches to null-value assessment are discussed. The article clarifies misconceptions about Bayesian methods that newcomers might have acquired elsewhere. We discuss prior distributions and explain how they are not a liability but an important asset. We discuss the relation of Bayesian data analysis to Bayesian models of mind, and we briefly discuss what methodological problems Bayesian data analysis is not meant to solve. After you have read this article, you should have a clear sense of how Bayesian data analysis works and the sort of information it delivers, and why that information is so intuitive and useful for drawing conclusions from data.     

Facts and Possibilities: A Model‐Based Theory of Sentential Reasoning

This article presents a fundamental advance in the theory of mental models as an explanation of reasoning about facts, possibilities, and probabilities. It postulates that the meanings of compound assertions, such as conditionals (if) and disjunctions (or), unlike those in logic, refer to conjunctions of epistemic possibilities that hold in default of information to the contrary. Various factors such as general knowledge can modulate these interpretations. New information can always override sentential inferences; that is, reasoning in daily life is defeasible (or nonmonotonic). The theory is a dual process one: It distinguishes between intuitive inferences (based on system 1) and deliberative inferences (based on system 2). The article describes a computer implementation of the theory, including its two systems of reasoning, and it shows how the program simulates crucial predictions that evidence corroborates. It concludes with a discussion of how the theory contrasts with those based on logic or on probabilities.     

Frequency versus probability formats in statistical word problems

Three experiments examined people's ability to incorporate base rate information when judging posterior probabilities. Specifically, we tested the (Cosmides, L., & Tooby, J. (1996). Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature on judgement under uncertainty. Cognition, 58, 1-73) conclusion that people's reasoning appears to follow Bayesian principles when they are presented with information in a frequency format, but not when information is presented as one case probabilities. First, we found that frequency formats were not generally associated with better performance than probability formats unless they were presented in a manner which facilitated construction of a set inclusion mental model. Second, we demonstrated that the use of frequency information may promote biases in the weighting of information. When participants are asked to express their judgements in frequency rather than probability format, they were more likely to produce the base rate as their answer, ignoring diagnostic evidence.     

On the interpretation of below-chance responding in forced-choice tests

Two-alternative, forced-choice tests are commonly used to assess cooperation in examinations of neurocognitive functioning. Most commercially available tests do not primarily depend on comparing the total correct responses to the number expected by guessing. Nevertheless, the tests afford an opportunity to make stronger judgments about the cooperation of test-takers when the test score is lower than the range of scores expected for guessing. Unfortunately, many researchers and clinicians make serious errors in communicating what is "guessing" and what is "worse than guessing" (or malingering). This article describes proper methods of evaluating total correct responses on a forced-choice test.     

Retrospective Frequency Formats Promote Consistent Experience‐Based Bayesian Judgments

On the basis of their experiences with pregnant patients in their practice, obstetrician/gynecologists estimated the posterior probability of Down syndrome given a positive screening result. They also estimated the base rate of Down syndrome in their practice, along with the hit and false alarm rates for the screening test; for each subject, these numbers were combined to calculate a posterior probability to which the initial estimated posterior probability could be compared. Physicians gave highly consistent estimates when asked to think about their past experiences in terms of event frequencies. However, those told to respond using single event probabilities or to use past experiences to predict prospective frequencies gave inconsistent Bayesian estimates. Thus, when making Bayesian judgments based on real life experience, natural frequency formats only lead to better judgments, compared with single event probability formats, if people think retrospectively, not when using past experiences to make prospective predictions. Copyright © 2011 John Wiley & Sons, Ltd.     

Bayesian nonparametric model selection and model testing

This article examines a Bayesian nonparametric approach to model selection and model testing, which is based on concepts from Bayesian decision theory and information theory. The approach can be used to evaluate the predictive-utility of any model that is either probabilistic or deterministic, with that model analyzed under either the Bayesian or classical-frequentist approach to statistical inference. Conditional on an observed set of data, generated from some unknown true sampling density, the approach identifies the “best” model as the one that predicts a sampling density that explains the most information about the true density. Furthermore, in the approach, the decision is to reject a model when it does not explain enough information about the true density (according to a straightforward calibration of the Kullback-Leibler divergence measure). The posterior estimate of the true density is based on a Bayesian nonparametric prior that can give positive support to the entire space of sampling densities (defined on some sample space). This article also discusses the theoretical and practical advantages of the Bayesian nonparametric approach over all other types of model selection procedures, and over any model testing procedure that depends on interpreting a p-value. Finally, the Bayesian nonparametric approach is illustrated on four real data sets, in the comparison and testing of order-constrained models, cognitive models, models of choice-behavior, and a test of a general psychometric model.     

Incorporating prior theory in covariance structure analysis: A bayesian approach

A Bayesian approach to the testing of competing covariance structures is developed. The method provides approximate posterior probablities for each model under consideration without prior specification of individual parameter distributions. The method is based on ayesian updating using cross-validated pseudo-likelihoods. Given that the observed variables are the samefor all competing models, the approximate posterior probabilities may be obtained easily from the chi square values and other known constants, using only a hand calculator. The approach is illustrated using and example which illustrates how the prior probabilities can alter the results concerning which model specification is preferred.     

Understanding trust and confidence: two paradigms and their significance for health and social care

Trusting agents characteristically anticipate beneficial outcomes, under conditions of uncertainty, in their engagement with others. However, debates about trust incorporate different interpretations of risk, uncertainty, calculation, affect, morality and motivation in explaining when trust is appropriate and how it operates. This article argues that discussions about trust have produced a concept without coherent boundaries and with little operational value. Two paradigms are identified, which distinguish the characteristics of trust and confidence. It is argued that a reliance on confidence in human affairs makes trust redundant and that this has undesirable moral consequences. Discussion is illustrated by the UK Government's 'modernisation' policy in health and social care, which privileges confidence in systems over trust in moral agents.     

The Bayesian New Statistics: Hypothesis testing,estimation, meta-analysis,and power analysis from a Bayesian perspective

In the practice of data analysis, there is a conceptual distinction between hypothesis testing, on the one hand, and estimation with quantified uncertainty on the other. Among frequentists in psychology, a shift of emphasis from hypothesis testing to estimation has been dubbed “the New Statistics” (Cumming 2014). A second conceptual distinction is between frequentist methods and Bayesian methods. Our main goal in this article is to explain how Bayesian methods achieve the goals of the New Statistics better than frequentist methods. The article reviews frequentist and Bayesian approaches to hypothesis testing and to estimation with confidence or credible intervals. The article also describes Bayesian approaches to meta-analysis, randomized controlled trials, and power analysis.     

Model inference or model selection: discussion of Klugkist, Laudy, and Hoijtink (2005)

I. Klugkist, O. Laudy, and H. Hoijtink (2005) presented a Bayesian approach to analysis of variance models with inequality constraints. Constraints may play 2 distinct roles in data analysis. They may represent prior information that allows more precise inferences regarding parameter values, or they may describe a theory to be judged against the data. In the latter case, the authors emphasized the use of Bayes factors and posterior model probabilities to select the best theory. One difficulty is that interpretation of the posterior model probabilities depends on which other theories are included in the comparison. The posterior distribution of the parameters under an unconstrained model allows one to quantify the support provided by the data for inequality constraints without requiring the model selection framework.     

A Two-Step Bayesian Approach for Propensity Score Analysis: Simulations and Case Study

A two-step Bayesian propensity score approach is introduced that incorporates prior information in the propensity score equation and outcome equation without the problems associated with simultaneous Bayesian propensity score approaches. The corresponding variance estimators are also provided. The two-step Bayesian propensity score is provided for three methods of implementation: propensity score stratification, weighting, and optimal full matching. Three simulation studies and one case study are presented to elaborate the proposed two-step Bayesian propensity score approach. Results of the simulation studies reveal that greater precision in the propensity score equation yields better recovery of the frequentist-based treatment effect. A slight advantage is shown for the Bayesian approach in small samples. Results also reveal that greater precision around the wrong treatment effect can lead to seriously distorted results. However, greater precision around the correct treatment effect parameter yields quite good results, with slight improvement seen with greater precision in the propensity score equation. A comparison of coverage rates for the conventional frequentist approach and proposed Bayesian approach is also provided. The case study reveals that credible intervals are wider than frequentist confidence intervals when priors are non-informative.     

Simple use of BIC to Assess Model Selection Uncertainty: An Illustration using Mediation and Moderation Models

Abstract

The Bayesian information criterion (BIC) has been used sometimes in SEM, even adopting a frequentist approach. Using simple mediation and moderation models as examples, we form posterior probability distribution via using BIC, which we call the BIC posterior, to assess model selection uncertainty of a finite number of models. This is simple but rarely used. The posterior probability distribution can be used to form a credibility set of models and to incorporate prior probabilities for model comparisons and selections. This was validated by a large scale simulation and results showed that the approximation via the BIC posterior is very good except when both the sample sizes and magnitude of parameters are small. We applied the BIC posterior to a real data set, and it has the advantages of flexibility in incorporating prior, addressing overfitting problems, and giving a full picture of posterior distribution to assess model selection uncertainty.