Heuristics and biases in diagnostic reasoning: II. Congruence,information, and certainty

Six experiments were carried out to examine possible heuristics and biases in the evaluation of yes-or-no questions for the purpose of hypothesis testing. In some experiments, the prior probability of the hypotheses and the conditional probabilities of the answers given each hypothesis were elicited from the subjects; in other experiments, they were provided. We found the following biases (systematic departures from a normative model), and interviews and justifications suggested that each was the result of a corresponding heuristic: Congruence bias. Subjects overvalued questions that have a high probability of a positive result given the most likely hypothesis. This bias was apparently reduced when alternative hypotheses or probabilities of negative results are explicitly stated. Information bias. Subjects evaluated questions as worth asking even when there is no answer that can change the hypothesis that will be accepted as a basis for action. Certainty bias. Subjects overvalued questions that have the potential to establish, or rule out, one or more hypotheses with 100% probability. These heuristics are explained in terms of the idea that people fail to consider certain arguments against the use of questions that seem initially worth asking, specifically, that a question may not distinguish likely hypotheses or that no answer can change the hypothesis accepted as a basis for action.     

Sensitivity to alternatives in social hypothesis-testing

Past research has documented a hypothesis-testing strategy wherein evidence is sought to the extent that it is probable under the hypothesis. This strategy may yield nondiagnostic information and even biased confirmation of the hypothesis if the simultaneous probability of the evidence under the alternatives is disregarded. The results of three experiments demonstrated that hypothesis-testers were in fact sensitive to the probability of the evidence under the alternatives. In the first experiment, subjects tested a hypothesis under which two kinds of personal features, A-features and B-features, were highly probable. Subjects could test their hypothesis by selecting questions from a list of questions about A-features and B-features. The rerults showed that subjects' questions depended on the probability of the features under the alternative. Specifically, when the hypothesis shared A-features with the alternative, subjects preferred questions about B-features, but when the hypothesis shared B-features with the alternative, subjects preferred questions about A-features. Experiment 2 extended these findings to self-generated questions about a broader range of hypotheses and alternatives. Experiment 3 found that subjects who were provided with a specific alternative asked more diagnostic questions than subjects who were not provided with a specific alternative. Together, these results suggest that the process of generating and evaluating alternatives plays a crucial role in social hypothesis-testing and categorization.     

Information-gathering strategies in hypothesis-testing

Three experiments were conducted to investigate the informational factors affecting the kinds of questions people ask for testing a hypothesis about another's personality. In Experiment 1, subjects formulated by themselves questions to test either the hypothesis that the interviewee is polite or the hypothesis that he is impolite. The boundary of the hypothesis was set either at an extreme or at an intermediate point on the trait dimension. Experiment 2 also varied the location of the boundary, but the hypotheses concerned extroversion-introversion and subjects chose questions from a predetermined list of questions that asked either about extroverted features or introverted features and that were either low or high in diagnostic value. Both studies found that subjects preferred to ask about features that are consistent with the hypothesis only when the boundary was extreme. In contrast, diagnostic features were preferred in all conditions. Experiment 3 showed that subjects' judgment of the diagnosticity of the various kinds of questions for discriminating at different boundaries paralleled subjects' preferences among these questions. These results were interpreted as providing support for the diagnostic strategy in social information gathering.     

The questions lay interviewers ask

Two studies were conducted to determine the kinds of questions people formulate for testing a hypothesis about another's personality. One group of subjects tested the hypothesis that the interviewee was an extrovert; another group tested the hypothesis that the interviewee was an introvert. Still another group was not provided with a hypothesis; instead, this group was asked to discriminate between an extroverted and an introverted interviewee. All subjects were free to formulate any kind of questions they wished. Analysis of the content of the questions performed by independent judges revealed that the great majority of the questions formulated by all of the groups either presented a choice between extroverted and introverted features or were open-ended. The percentage of questions about features that were consistent with the hypothesis was not significantly greater than the percentage of questions about features that were inconsistent with the hypothesis. Furthermore, in all of the groups there were virtually no biased questions–questions that already assume that the hypothesis is true. Finally, the questions asked by subjects who entertained either the extrovert hypothesis or the introvert hypothesis were as diagnostic (as rated by independent judges) as were the questions asked by subjects whose task was to discriminate between extroverts and introverts. The results were shown to be consistent with the diagnosing strategy but not with the confirmatory strategy of information-gathering.     

Information relevance, working memory, and the consideration of alternatives

People routinely focus on one hypothesis and avoid consideration of alternative hypotheses on problems requiring decisions between possible states of the world--for example, on the “pseudodiagnosticity” task (Doherty, Mynatt, Tweney, & Schiavo, 1979). In order to account for behaviour on such “inference” problems, it is proposed that people can hold in working memory, and operate upon, but one alternative at a time, and that they have a bias to test the hypothesis they think true. In addition to being an ex post facto explanation of data selection in inference tasks, this conceptualization predicts that there are situations in which people will consider alternatives. These are:

1. “action” problems, where the alternatives are possible courses of action;

2. “inference” problems, in which evidence favours an alternative hypothesis.

Experiment 1 tested the first prediction. Subjects were given action or inference problems, each with two alternatives and two items of data relevant to each alternative. They received probabilistic information about the relation between one datum and one alternative and picked one value from among the other three possible pairs of such relations. Two findings emerged; (1) a strong tendency to select information about only one alternative with inferences; and (2) a strong tendency, compared to inferences, to select information about both alternatives with actions.

Experiment 2 tested the second prediction. It was predicted that data suggesting that one alternative was incorrect would lead many subjects to consider, and select information about, the other alternative. For actions, it was predicted that this manipulation would have no effect. Again the data were as predicted.     

Increased sensitivity to differentially diagnostic answers using familiar materials: Implications for confirmation bias

Researchers have recently pointed out that neither biased testing nor biased evaluation of hypotheses necessitates confirmation bias--defined here as systematic overconfidence in a focal hypothesis--but certain testing/evaluation combinations do. One such combination is (1) a tendency to ask about features that are either very likely or very unlikely under the focal hypothesis (extremity bias) and (2) a tendency to treat confirming and disconfirming answers as more similar in terms of their diagnosticity (or informativeness) than they really are. However, in previous research showing the second tendency, materials that are highly abstract and unfamiliar have been used. Two experiments demonstrated that using familiar materials led participants to distinguish much better between the differential diagnosticity of confirming and disconfirming answers. The conditions under which confirmation bias is a serious concern might be quite limited.     

Use of current explanations in multicausal abductive reasoning

In multicausal abductive tasks a person must explain some findings by assembling a composite hypothesis that consists of one or more elementary hypotheses. If there are n elementary hypotheses, there can be up to 2ⁿ composite hypotheses. To constrain the search for hypotheses to explain a new observation, people sometimes use their current explanation—the previous evidence and their present composite hypothesis of that evidence; however, it is unclear when and how the current explanation is used. In addition, although a person's current explanation can narrow the search for a hypothesis, it can also blind the problem solver to alternative, possibly better, explanations. This paper describes a model of multicausal abductive reasoning that makes two predictions regarding the use of the current explanation. The first prediction is that the current explanation is not used to explain new evidence if there is a simple (i.e., nondisjunctive, concrete) hypothesis to account for that evidence. The second prediction is that the current explanation is used when attempting to discriminate among several alternative hypotheses for new evidence. These hypotheses were tested in three experiments. The results are consistent with the second prediction: the current explanation is used when discriminating among alternative hypotheses. However, the first prediction—that the current explanation is not used when a simple hypothesis can account for new data—received only limited support. Participants used the current explanation to constrain their interpretation of new data in 46.5% of all trials. This suggests that context-independent strategies compete with context-dependent ones—an interpretation that is consistent with recent work on strategy selection during problem solving.     

THE INTERVIEWER AS HYPOTHESIS TESTER: THE EFFECTS OF IMPRESSIONS OF AN APPLICANT ON INTERVIEWER QUESTIONING STRATEGY

Recent findings in the impression formation literature suggest that individuals seek evidence to confirm initial hypotheses, or preconceptions, which they form about other people prior to interaction, and that seeking confirmatory evidence makes it likely that a hypothesis will be confirmed (Snyder and Swann, 1978). Snyder and Swann suggest that the employment interview is one context in which this process can be expected to operate. Four studies are reported which examine the generalizability of these findings to the employment interview. Consistent use of confirmatory hypothesis testing strategies was not found when experienced interviewers, rather than college students, were used as subjects, nor when the study was set specifically in an employment interview setting, nor when hypotheses about characteristics other than those examined by Snyder and Swann were studied.     

Satisficing in hypothesis generation

Research in hypothesis generation suggests that people might act as satisficers and be less likely to generate plausible alternative hypotheses when they already have a hypothesis that accounts for the data in hand. Three experiments simulated scientific hypothesis development. In all 3, participants who had been given a hypothesis consistent with available data generated proportionally fewer of the simplest alternative hypotheses than participants given no such satisficing hypothesis. Furthermore, participant satisficing occurred regardless of whether the provided hypothesis was generated a priori or post hoc and despite high incentives for completeness. Limitations and implications of these findings are discussed for hypothesis development and the practice of taking post hoc hypotheses suggested by one's results and presenting them as a priori hypotheses.     

Confirming expectations in asymmetric and symmetric social hypothesis testing

This article examines individuals' expectations in a social hypothesis testing task. Participants selected questions from a list to investigate the presence of personality traits in a target individual. They also identified the responses that they expected to receive and the likelihood of the expected responses. The results of two studies indicated that when people asked questions inquiring about the hypothesized traits that did not entail strong a priori beliefs, they expected to find evidence confirming the hypothesis under investigation. These confirming expectations were more pronounced for symmetric questions, in which the diagnosticity and frequency of the expected evidence did not conflict. When the search for information was asymmetric, confirming expectations were diminished, likely as a consequence of either the rareness or low diagnosticity of the hypothesis-confirming outcome. We also discuss the implications of these findings for confirmation bias.     

Why it doesn’t matter whether the virtues are truth-conducive

A potential explanation of a fact is a hypothesis such that, if it were true, it would explain the fact in question. Let’s suppose that we become aware of a fact and some potential explanations thereof. Let’s also suppose that we would like to believe the truth. Given this aim, we can ask two questions. First, is it likely that one of these hypotheses is true? Second, given an affirmative answer to the first question, which one is it likely to be? Inference to the best explanation (IBE) offers answers to both questions. To the first, it says ‘Yes’—assuming that at least one of the hypotheses would, if true, provide a satisfactory explanation of the fact under consideration. To the second, it says that the hypothesis most likely to be true is the one that scores best on the explanatory virtues: conservatism, modesty, simplicity, generality, and predictive power. Many philosophers have argued against IBE’s answer to the first question. I am interested in an objection to its answer to the second. Many philosophers seem to think that it is unsustainable: they seem to think that even if we assume that one of the competing hypotheses is true, we should not think that IBE will help us to identify it. Or, more carefully, if these philosophers are doing what they appear to be doing—namely, offering critiques of IBE that don’t depend on assumptions about the field of competing hypotheses—then their claim is that IBE will not help us to identify the truth. I believe that this is mistaken: the argument for believing it assumes a model of IBE that we have no reason to accept.     

Causal diversity effects in information seeking

When trying to determine the root cause of an observed effect, people may seek out information with which to test a candidate hypothesis. In two studies, we investigated how knowledge of causal structure influences this information-seeking process. Specifically, we asked whether people would choose to test for pieces of evidence that were far apart or close together in the learned causal structure of a disease category. In parallel with findings showing people’s tendency to select diverse evidence in argument testing (López, 1995), our participants tested for evidence distantly located within the causal structure. Simultaneously, they rated the probability of occurrence of such diverse evidence as comparatively low. These findings suggest that rather than seeking out information most likely to confirm the hypothesis, people seek out evidence that they believe will most strongly support the hypothesis if present but that they also believe is relatively unlikely to be present (that is, might disconfirm the hypothesis).     

Seeking Confirmation Is Rational for Deterministic Hypotheses

The tendency to test outcomes that are predicted by our current theory (the confirmation bias) is one of the best‐known biases of human decision making. We prove that the confirmation bias is an optimal strategy for testing hypotheses when those hypotheses are deterministic, each making a single prediction about the next event in a sequence. Our proof applies for two normative standards commonly used for evaluating hypothesis testing: maximizing expected information gain and maximizing the probability of falsifying the current hypothesis. This analysis rests on two assumptions: (a) that people predict the next event in a sequence in a way that is consistent with Bayesian inference; and (b) when testing hypotheses, people test the hypothesis to which they assign highest posterior probability. We present four behavioral experiments that support these assumptions, showing that a simple Bayesian model can capture people's predictions about numerical sequences (Experiments 1 and 2), and that we can alter the hypotheses that people choose to test by manipulating the prior probability of those hypotheses (Experiments 3 and 4).     

Do people turn to visible reminders of group identity under stress?

How do people cope with stress? Research suggests that people have a number of strategies, including turning to the groups to which they belong, increasing feelings of identification and affiliation. We examine a novel extension of this strategy: adopting visible reminders of one's group identity. In two experiments (Ns = 103 and 194), we explore whether students are more likely to use an artifact that displays their university identity in situations of high, compared to low, evaluative stress. In Experiment 1, students were more likely to choose a university‐branded pen (vs. an identical unbranded pen) to complete exam questions when their performance would be evaluated versus not. In Experiment 2, we found the tendency to use a university‐branded pen in the face of evaluative stress emerged only among people who found the evaluation personally relevant. In addition, we present converging results from two observational studies suggesting that students are more likely to wear clothing that signals their university identity on exam days compared to control days. These findings suggest that people may turn to visible reminders of group membership when facing evaluative stress. Although we found no consistent evidence for a psychological mechanism underlying this effect, we speculate about theoretically relevant possibilities.     

Rotter's I. E.-Scale and Socio-Political Action-Taking: The balance of 20 years of research

Besides Rotter's hypothesis that internals are more likely to participate in sociopolitical action than externals, in this paper the hypothesis is formulated that externals are more likely to participate in socio-political action than internals. Both hypotheses can be justified from a value-expectancy theoretical point of view. These hypotheses are called the efficacy hypothesis and the power-formation hypothesis respectively. The assumption is made that the power-formation hypothesis holds true for people who are convinced that they are powerless, on objective and for ideological grounds, and that the efficacy hypothesis holds true for people who do not have that conviction. Conditions under which a relationship between I. E. and socio-political action-taking can be expected were derived from both hypotheses. Thirty-one studies on I. E. and action-taking were reviewed. Five studies confirmed the efficacy hypothesis, four studies confirmed the power-formation hypothesis. Nineteen studies revealed no relationship between I. E. scores and action-taking. The explanations that various authors give for the absence of a relationship were examined. It was investigated whether explanations that held true for studies other than the ones for which they were formulated would fit into the theoretical framework developed in this article. This appeared to be the case. The implications for further research are discussed.     

Are people's estimates of what other people know influenced by what they themselves know?

An exploratory experiment was done to investigate how estimates of what other people know are influenced by what one knows oneself. Three hypotheses were of interest: (1) that people are more likely to impute a bit of knowledge to other people if they have it themselves than if they do not, (2) that people are likely to overestimate the commonality of their own knowledge, and (3) that people who are unusually knowledgeable with respect to a particular topic are likely to bias their estimates of what other people know in the direction of their personal knowledge store. Subjects answered questions selected from the set for which Nelson and Narens (1980) have provided norms. They also estimated the percentage of other people who would be likely to know the answers to these questions. The results supported the first two hypotheses, but were inconclusive with respect to the third.     

Test of the analysis of competing hypotheses in legal decision‐making

The analysis of competing hypotheses (ACH) has been suggested to be a method that can protect against confirmation bias in the context of intelligence analysis. In the current study, we aimed to determine whether ACH could counter confirmation bias in the reasoning with evidence in the context of criminal law proceedings. Law students (N = 191) received information about the ACH method or general information about biases. They were given a case vignette with a main suspect and a list of 24 questions, 6 of which they could ask about the case. Half of the questions related to incriminating information, whereas the other half related to exonerating information. Contrary to our expectations, participants in both conditions favoured questions relating to exonerating information and rated the exonerating evidence as being more important for their decision. Despite the lack of bias observed, it seems participants failed to properly apply the ACH method.     

Seeing is believing: Priors,trust, and base rate neglect

Kahneman and Tversky (1973) described an effect they called ‘insensitivity to prior probability of outcomes’, later dubbed base rate neglect, which describes people’s tendency to underweight prior information in favor of new data. As probability theory requires that prior probabilities be taken into account, via Bayes’ theorem, the fact that most people fail to do so has been taken as evidence of human irrationality and, by others, of a mismatch between our cognitive processes and the questions being asked (Cosmides & Tooby, 1996). In contrast to both views, we suggest that simplistic Bayesian updating using base rates is not necessarily rational. To that end, we present experiments in which base rate neglect is often the right strategy, and show that people’s base rate usage varies systematically as a function of the extent to which the data that make up a base rate are perceived as trustworthy.     

The effects of interviewers' initial impressions on information gathering

M. Synder and W. B. Swann, Jr. (1978, Journal of Personality and Social Psychology, 36, 1202–1212) suggest that people adopt information seeking strategies that confirm their hypotheses. To investigate whether or not these findings can be extended to employment interviews, an experiment was conducted examining the effects of interviewers' initial impressions and their decision task on the questions they formulate to assess applicants for a sales position. Twenty-six persons with varied interviewing experience reviewed the paper credentials of three applicants and then stated the questions they would ask each one. In separate sessions, ninety-two persons took the role of interviewee and responded to these questions. Contrary to the findings of P. Sackett (1982, Personnel Psychology, 35, 789–803) and T. McDonald and M. D. Hakel (1985, Personnel Psychology, 38, 321–334), the questions that interviewers asked of an applicant with poor credentials were biased in a more negative direction than the questions that were asked of moderately and highly qualified applicants. Consistent with these studies, however, no evidence was found for the strong confirmatory biases revealed in the Snyder and Swann (1978) research.