Background beliefs and evidence interpretation

In this paper we argue that it is often adaptive to use one's background beliefs when interpreting information that, from a normative point of view, is incomplete. In both of the experiments reported here participants were presented with an item possessing two features and were asked to judge, in the light of some evidence concerning the features, to which of two categories it was more likely that the item belonged. It was found that when participants received evidence relevant to just one of these hypothesised categories (i.e. evidence that did not form a Bayesian likelihood ratio) they used their background beliefs to interpret this information. In Experiment 2, on the other hand, participants behaved in a broadly Bayesian manner when the evidence they received constituted a completed likelihood ratio. We discuss the circumstances under which participants, when making their judgements, consider the alternative hypothesis. We conclude with a discussion of the implications of our results for an understanding of hypothesis testing, belief revision, and categorisation.     

Relevance-driven information search in "pseudodiagnostic" reasoning

When faced with two competing hypotheses, people sometimes prefer to look at multiple sources of information in support of one hypothesis rather than to establish the diagnostic value of a single piece of information for the two hypotheses. This is termed pseudodiagnostic reasoning and has often been understood to reflect, among other things, poor information search strategies. Past research suggests that diagnostic reasoning may be more easily fostered when participants seek data to help in the selection of one of two competing courses of action as opposed to situations where they seek data to help infer which of two competing hypotheses is true. In the experiment reported here, we provide the first empirical evidence demonstrating that manipulating the relevance of the feature for which participants initially receive information determines whether they will make a nominally diagnostic or pseudodiagnostic selection. The discussion of these findings focuses on implications for the ability to engage in diagnostic hypothesis testing.     

Relevance-driven information search in “pseudodiagnostic” reasoning

When faced with two competing hypotheses, people sometimes prefer to look at multiple sources of information in support of one hypothesis rather than to establish the diagnostic value of a single piece of information for the two hypotheses. This is termed pseudodiagnostic reasoning and has often been understood to reflect, among other things, poor information search strategies. Past research suggests that diagnostic reasoning may be more easily fostered when participants seek data to help in the selection of one of two competing courses of action as opposed to situations where they seek data to help infer which of two competing hypotheses is true. In the experiment reported here, we provide the first empirical evidence demonstrating that manipulating the relevance of the feature for which participants initially receive information determines whether they will make a nominally diagnostic or pseudodiagnostic selection. The discussion of these findings focuses on implications for the ability to engage in diagnostic hypothesis testing.     

Bayesian inference for psychology. Part II: Example applications with JASP

Bayesian hypothesis testing presents an attractive alternative to p value hypothesis testing. Part I of this series outlined several advantages of Bayesian hypothesis testing, including the ability to quantify evidence and the ability to monitor and update this evidence as data come in, without the need to know the intention with which the data were collected. Despite these and other practical advantages, Bayesian hypothesis tests are still reported relatively rarely. An important impediment to the widespread adoption of Bayesian tests is arguably the lack of user-friendly software for the run-of-the-mill statistical problems that confront psychologists for the analysis of almost every experiment: the t-test, ANOVA, correlation, regression, and contingency tables. In Part II of this series we introduce JASP (http://www.jasp-stats.org), an open-source, cross-platform, user-friendly graphical software package that allows users to carry out Bayesian hypothesis tests for standard statistical problems. JASP is based in part on the Bayesian analyses implemented in Morey and Rouder’s BayesFactor package for R. Armed with JASP, the practical advantages of Bayesian hypothesis testing are only a mouse click away.     

Seeking Confirmation Is Rational for Deterministic Hypotheses

The tendency to test outcomes that are predicted by our current theory (the confirmation bias) is one of the best‐known biases of human decision making. We prove that the confirmation bias is an optimal strategy for testing hypotheses when those hypotheses are deterministic, each making a single prediction about the next event in a sequence. Our proof applies for two normative standards commonly used for evaluating hypothesis testing: maximizing expected information gain and maximizing the probability of falsifying the current hypothesis. This analysis rests on two assumptions: (a) that people predict the next event in a sequence in a way that is consistent with Bayesian inference; and (b) when testing hypotheses, people test the hypothesis to which they assign highest posterior probability. We present four behavioral experiments that support these assumptions, showing that a simple Bayesian model can capture people's predictions about numerical sequences (Experiments 1 and 2), and that we can alter the hypotheses that people choose to test by manipulating the prior probability of those hypotheses (Experiments 3 and 4).     

The Bayesian New Statistics: Hypothesis testing,estimation, meta-analysis,and power analysis from a Bayesian perspective

In the practice of data analysis, there is a conceptual distinction between hypothesis testing, on the one hand, and estimation with quantified uncertainty on the other. Among frequentists in psychology, a shift of emphasis from hypothesis testing to estimation has been dubbed “the New Statistics” (Cumming 2014). A second conceptual distinction is between frequentist methods and Bayesian methods. Our main goal in this article is to explain how Bayesian methods achieve the goals of the New Statistics better than frequentist methods. The article reviews frequentist and Bayesian approaches to hypothesis testing and to estimation with confidence or credible intervals. The article also describes Bayesian approaches to meta-analysis, randomized controlled trials, and power analysis.     

Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications

Bayesian parameter estimation and Bayesian hypothesis testing present attractive alternatives to classical inference using confidence intervals and p values. In part I of this series we outline ten prominent advantages of the Bayesian approach. Many of these advantages translate to concrete opportunities for pragmatic researchers. For instance, Bayesian hypothesis testing allows researchers to quantify evidence and monitor its progression as data come in, without needing to know the intention with which the data were collected. We end by countering several objections to Bayesian hypothesis testing. Part II of this series discusses JASP, a free and open source software program that makes it easy to conduct Bayesian estimation and testing for a range of popular statistical scenarios (Wagenmakers et al. this issue).     

Hierarchical Bayesian estimation and hypothesis testing for delay discounting tasks

A state-of-the-art data analysis procedure is presented to conduct hierarchical Bayesian inference and hypothesis testing on delay discounting data. The delay discounting task is a key experimental paradigm used across a wide range of disciplines from economics, cognitive science, and neuroscience, all of which seek to understand how humans or animals trade off the immediacy verses the magnitude of a reward. Bayesian estimation allows rich inferences to be drawn, along with measures of confidence, based upon limited and noisy behavioural data. Hierarchical modelling allows more precise inferences to be made, thus using sometimes expensive or difficult to obtain data in the most efficient way. The proposed probabilistic generative model describes how participants compare the present subjective value of reward choices on a trial-to-trial basis, estimates participant- and group-level parameters. We infer discount rate as a function of reward size, allowing the magnitude effect to be measured. Demonstrations are provided to show how this analysis approach can aid hypothesis testing. The analysis is demonstrated on data from the popular 27-item monetary choice questionnaire (Kirby, Psychonomic Bulletin & Review, 16(3), 457–462 2009), but will accept data from a range of protocols, including adaptive procedures. The software is made freely available to researchers.     

Expertise and error in diagnostic reasoning

An investigation is presented in which a computer simulation model (DIAGNOSER) is used to develop and test predictions for behavior of subjects in a task of medical diagnosis. The first experiment employed a process-tracing methodology in order to compare hypothesis generation and evaluation behavior of DIAGNOSER with individuals at different levels of expertise (students, trainees, experts). A second experiment performed with only DIAGNOSER identified conditions under which errors in reasoning in the first experiment could be related to interpretation of specific data items. Predictions derived from DIAGNOSER's performance were tested in a third experiment with a new sample of subjects. Data from the three experiments indicated that (1) form of diagnostic reasoning was similar for all subjects trained in medicine and for the simulation model, (2) substance of diagnostic reasoning employed by the simulation model was parable with that of the more expert subjects, and (3) errors in subjects' reasoning were attributable to deficiencies in disease knowledge and the interpretation of specific patient data cues predicted by the simulation model.     

Insensitivity and oversensitivity to answer diagnosticity in hypothesis testing

Two experiments examined how people perceive the diagnosticity of different answers (“yes” and “no”) to the same question. We manipulated whether the “yes” and the “no” answers conveyed the same amount of information or not, as well as the presentation format of the probabilities of the features inquired about. In Experiment 1, participants were presented with only the percentages of occurrence of the features, which most straightforwardly apply to the diagnosticity of “yes” answers. In Experiment 2, participants received in addition the percentages of the absence of features, which serve to assess the diagnosticity of “no” answers. Consistent with previous studies, we found that participants underestimated the difference in the diagnosticity conveyed by different answers to the same question. However, participants' insensitivity was greater when the normative (Bayesian) diagnosticity of the “no” answer was higher than that of the “yes” answer. We also found oversensitivity to answer diagnosticity, whereby participants valued as differentially diagnostic two answers that were normatively equal in terms of their diagnosticity. Presenting to participants the percentages of occurrence of the features inquired about together with their complements increased their sensitivity to the diagnosticity of answers. We discuss the implications of these findings for confirmation bias in hypothesis testing.     

Verbalization of Decision Strategies in Multiple‐Cue Probabilistic Inference

In multiple‐cue probabilistic inference, people choose between alternatives based on several cues, each of which is differentially associated with an alternative's overall value. Various strategies have been proposed for probabilistic inference (e.g., weighted additive, tally, and take‐the‐best). These strategies differ in how many cue values they require to enact and in how they weight each cue. Do decision makers actually use any of these strategies? Ways to investigate this question include analyzing people's choices and the cues that they reveal. However, different strategies often predict the same decisions, and search behavior says nothing about whether or how people use the information that they acquire. In this research, we attempt to elucidate which strategies participants use in a multiple‐cue probabilistic inference task by examining verbal protocols, a high‐density source of process data. The promise of verbal data is in their utility for testing detailed information processing models. To that end, we apply protocol analysis in conjunction with computational simulations. We find converging evidence across outcome measures, search measures, and verbal reports that most participants use simplifying heuristics, namely take‐the‐best. Copyright © 2015 John Wiley & Sons, Ltd.     

Process Tracing as an Effective Epistemic Complement

In the last decades philosophers of science and social scientists promoted the view that knowledge of mechanisms might help causal inference considerably in the social sciences. Mechanisms, however, can only assist causal inference effectively if scientists have a means to identify them correctly. Some scholars suggested that process-tracing might be a helpful strategy in this respect. Shared criteria to assess its performance, however, are not available yet; furthermore, the criteria proposed so far tie the validity of process-tracing findings to the specific kind of evidence it uses. In this paper I shall propose a criterion to assess process-tracing performance in cases in which favorable epistemic circumstances do not occur and the existing criteria thus fail to apply. The criterion I propose does not double as a condition for validity. Rather, it aims to assess whether the mechanism process-tracing outlines constitutes admissible evidence for the hypothesis at hand. It will be argued that only if this requirement is fulfilled process-tracing can be used as an effective complement for causal inference.     

Inferring causal networks from observations and interventions

Information about the structure of a causal system can come in the form of observational data—random samples of the system's autonomous behavior—or interventional data—samples conditioned on the particular values of one or more variables that have been experimentally manipulated. Here we study people's ability to infer causal structure from both observation and intervention, and to choose informative interventions on the basis of observational data. In three causal inference tasks, participants were to some degree capable of distinguishing between competing causal hypotheses on the basis of purely observational data. Performance improved substantially when participants were allowed to observe the effects of interventions that they performed on the systems. We develop computational models of how people infer causal structure from data and how they plan intervention experiments, based on the representational framework of causal graphical models and the inferential principles of optimal Bayesian decision‐making and maximizing expected information gain. These analyses suggest that people can make rational causal inferences, subject to psychologically reasonable representational assumptions and computationally reasonable processing constraints.     

The frequentist implications of optional stopping on Bayesian hypothesis tests

Null hypothesis significance testing (NHST) is the most commonly used statistical methodology in psychology. The probability of achieving a value as extreme or more extreme than the statistic obtained from the data is evaluated, and if it is low enough, the null hypothesis is rejected. However, because common experimental practice often clashes with the assumptions underlying NHST, these calculated probabilities are often incorrect. Most commonly, experimenters use tests that assume that sample sizes are fixed in advance of data collection but then use the data to determine when to stop; in the limit, experimenters can use data monitoring to guarantee that the null hypothesis will be rejected. Bayesian hypothesis testing (BHT) provides a solution to these ills because the stopping rule used is irrelevant to the calculation of a Bayes factor. In addition, there are strong mathematical guarantees on the frequentist properties of BHT that are comforting for researchers concerned that stopping rules could influence the Bayes factors produced. Here, we show that these guaranteed bounds have limited scope and often do not apply in psychological research. Specifically, we quantitatively demonstrate the impact of optional stopping on the resulting Bayes factors in two common situations: (1) when the truth is a combination of the hypotheses, such as in a heterogeneous population, and (2) when a hypothesis is composite—taking multiple parameter values—such as the alternative hypothesis in a t-test. We found that, for these situations, while the Bayesian interpretation remains correct regardless of the stopping rule used, the choice of stopping rule can, in some situations, greatly increase the chance of experimenters finding evidence in the direction they desire. We suggest ways to control these frequentist implications of stopping rules on BHT.     

Within-subject consistency and between-subject variability in Bayesian reasoning strategies

It is well known that people tend to perform poorly when asked to determine a posterior probability on the basis of a base rate, true positive rate, and false positive rate. The present experiments assessed the extent to which individual participants nevertheless adopt consistent strategies in these Bayesian reasoning problems, and investigated the nature of these strategies. In two experiments, one laboratory-based and one internet-based, each participant completed 36 problems with factorially manipulated probabilities. Many participants applied consistent strategies involving use of only one of the three probabilities provided in the problem, or additive combination of two of the probabilities. There was, however, substantial variability across participants in which probabilities were taken into account. In the laboratory experiment, participants’ eye movements were tracked as they read the problems. There was evidence of a relationship between information use and attention to a source of information. Participants’ self-assessments of their performance, however, revealed little confidence that the strategies they applied were actually correct. These results suggest that the hypothesis of base rate neglect actually underestimates people’s difficulty with Bayesian reasoning, but also suggest that participants are aware of their ignorance.     

Mood and positive testing in social interaction

Why, how and when does mood influence positive testing, that is, the selection of matching questions, when people actively search for information about others they meet? In four experiments, we demonstrated that happy mood increased positive testing compared to sad mood. Experiment 1 showed that happy participants were more strongly motivated to get along and smooth the interaction to come than sad ones. In addition, evidence was provided by a mediation analysis that happy mood increased the preference for positive testing because of such an improved motivation to get along. Furthermore, Experiment 2 showed that happy participants' preference for positive testing vanished when cognitive resources were limited. The preference for positive testing appeared under happy mood only when the context made salient the goal to get along (Experiments 3 and 4). Together, these results suggest that positive testing in a social‐hypothesis testing paradigm may have social values. Copyright © 2009 John Wiley & Sons, Ltd.     

Strategies of clinical hypothesis testing

This research examines whether Psychology students, when they test clinical hypotheses, follow either confirmatory or disconfirmatory reasoning strategies. Two hundred and six psychology students, divided in four groups, participated. One group received information about the probability that the hypothesis was correct by means of verbal labels, and another group, by means of numerical expressions. An additional group received the information that getting a precise diagnosis was clinically important. In a last group, diagnostic tests allowed them to increase certainty about the hypothesis. Results show a partial use of confirmatory strategies because, although participants did not seek confirming information, they indeed avoided collecting disconfirming information. When the information increased certainty about the hypothesis, confirmatory strategies became more likely. Neither the increase in the task importance nor the numerical expression of the likelihood that the hypothesis was correct seemed to affect the testing strategy used.     

Improving the Accuracy of Group Judgment: A Process Intervention Combining Group Facilitation,Social Judgment Analysis,and Information Technology

Interacting groups fail to make judgments as accurate as those of their most capable members due to problems associated with both interaction processes and cognitive processing. Group process techniques and decision analytic tools have been used with groups to combat these problems. While such techniques and tools do improve the quality of group judgment, they have not enabled groups to make judgments more accurate than those of their most capable members on tasks that evoke a great deal of systematic bias. A new intervention procedure that integrates group facilitation, social judgment analysis, and information technology was developed to overcome more fully the problems typically associated with interaction processes and cognitive processing. The intervention was evaluated by testing the hypothesis that groups using this new procedure can establish judgment policies for cognitive conflict tasks that are more accurate than the ones produced by any of their members. An experiment involving 16 four- and five-member groups was conducted to compare the accuracy of group judgments with the accuracy of the judgments of the most capable group member. A total of 96 participants (48 males and 48 females) completed the individual part of the task; 71 of these participants worked in groups. Results indicated that the process intervention enabled small, interacting groups to perform significantly better than their most capable members on two cognitive conflict tasks (p < .05). The findings suggest that Group Decision Support Systems that integrate facilitation, social judgment analysis, and information technology should be used to improve the accuracy of group judgment.     

Information Seeking in a Non-Hypothesis Testing Task

The information gain model (Oaksford and Chater, Psychological Review 101, 608–631, 1994) advocates that participants attempt to achieve a larger expected information gain when they have to test an if-then rule or hypothesis. However, acquisition of larger expected information gain could also be operational when participants do not have to test a hypothesis. This study devised a new task to investigate whether participants would seek larger expected information gain when they were not required to test a hypothesis. The task required participants to select one out of two balance scales for weighing coins in order to detect an underweight coin. We discovered that participants more frequently selected the balance scale that provided smaller expected information gain. This finding suggests that the preference for larger expected information gain may not apply to non-hypothesis testing settings.     

Effects of perspective and belief on analytic reasoning in a scientific reasoning task

The purpose of these studies was to test the hypothesis that changing perspectives from one's own to another's promotes the engagement of analytic processing and, in turn, reduces the impact of beliefs. In two experiments participants evaluated research vignettes containing belief-consistent and belief-inconsistent conclusions, and indicated whether the data supported a correlation between two variables. Consistent with our hypothesis, the tendency to endorse correlations consistent with prior belief was reduced when participants evaluated the data from the researcher's perspective relative to their own. We also administered the Actively Open Minded Thinking (AOT) scale (Stanovich & West, 2007, 2008), which did not predict belief effects on our task. We did however observe that the AOT was reliably associated with different response strategies: high AOT scorers were more inclined to choose ambiguous response options, such as “no conclusion is warranted”, whereas low scorers evinced a preference for more determinate options (e.g., there is no relationship between the two variables). We interpret our findings in the context of dual process theories of reasoning and from a Bayesian perspective.