Reply to William R. Ferrell's paper "Calibration of sensory and cognitive judgments: A single model for both"

William R. Ferrell's paper is a reply to our article "Calibration of sensory and cognitive judgments: Two different accounts", 1993, 34, 135–148 in this journal. Ferrell's critique centres around three issues: (1) The model of sensory discrimination proposed by Björkman et al. (1993) and discussed in Winman and Juslin (1993) is not new, (2) this model does not predict underconfidence, (3) there is no need for separate models of calibration of sensory and cognitive judgments. Below we respond to each of the three issues raised by Ferrell.     

A model for realism of confidence judgments: Implications for underconfidence in sensory discrimination

In a recent issue of this journal, Björkman, Juslin, and Winman (1993) presented a model of the calibration of subjective confidence judgments for sensory discrimination which they called “subjective distance theory.” They proposed that there was a robust underconfidence bias in such judgments, that the model predicted such a bias, and that two different models were needed for the calibration of subjective confidence for cognitive judgments and for sensory ones. This paper addresses issues they raised. It points out that they have not presented a new model, but rather a portion of a more general one, the “decision-variable partition model” originally proposed in Ferrell and McGoey (1980). This paper explores properties of the model and shows, contrary to Björkman, Juslin, and Winman’s hypotheses, that the model does not predict under-confidence, that the “hard-easy effect” can be observed with sensory discriminations, and that the model fits not only sensory, but also cognitive judgments.     

Realism of confidence in sensory discrimination

Is there a common and general basis for confidence in human judgment? Recently, we found that the properties of confidence judgments in the sensory domain mirror those previously established in the cognitive domain; notably, we found underconfidence on easy sensory judgments and overconfidence on hard sensory judgments. In contrast, data from the Uppsala laboratory in Sweden suggest that sensory judgments are unique; they found a pervasive underconfidence bias, with overconfidence being evident only on very hard sensory judgments. Olsson and Winman (1996) attempted to resolve the debate on the basis of methodological issues related to features of the stimulus display in a visual discrimination task. A reanalysis of the data reported in Baranski and Petrusic (1994), together with the findings of a new experiment that controlled stimulus display characteristics, supports the position that the difference between the Canadian and the Swedish data is real and, thus, may reflect cross-national differences in confidence in sensory discrimination.     

Calibration of sensory and cognitive judgments: A single model for both

In a recent issue of this journal, Winman and Juslin (34 , 135–148, 1993) present a model of the calibration of subjective probability judgments for sensory discrimination tasks. They claim that the model predicts a pervasive underconfidence bias observed in such tasks, and present evidence from a training experiment that they interpret as supporting the notion that different models are needed to describe judgment of confidence in sensory and in cognitive tasks. The model is actually part of the more comprehensive decision variable partition model of subjective probability calibration that was originally proposed in Ferrell and McGoey (Organizational Behavior and Human Performance, 26 , 32–53, 1980). The characteristics of the model are described and it is demonstrated that the model does not predict underconfidence, that it is fully compatible with the overconfidence frequently found in calibration studies with cognitive tasks, and that it well represents experimental results from such studies. It is concluded that only a single model is needed for both types of task.     

Underconfidence in sensory discrimination: The interaction between experimental setting and response strategies

In a recent issue of this journal, Baranski and Petrusic (1994) presented empirical data revealing overconfidence in sensory discrimination. In this paper, we propose an explanation of Baranski and Petrusic’s results, based on an idiosyncrasy in the experimental setting that misleads subjects who are using an unwarranted symmetry assumption. Experiment 1 showed that when this hypothesis is controlled for, a large underconfidence bias is obtained with Baranski and Petrusic’s procedure. The results of Experiment 2 confirmed that overconfidence is difficult to obtain in subject-controlled sensory discrimination tasks, even for a very low proportion of correct responses. The different results obtained in sensory and cognitive tasks suggest that one should not uncritically draw parallels between confidence in sensory and cognitive judgments.     

Overconfidence: It Depends on How, What, and Whom You Ask

Many studies have reported that the confidence people have in their judgments exceeds their accuracy and that overconfidence increases with the difficulty of the task. However, some common analyses confound systematic psychological effects with statistical effects that are inevitable if judgments are imperfect. We present three experiments using new methods to separate systematic effects from the statistically inevitable. We still find systematic differences between confidence and accuracy, including an overall bias toward overconfidence. However, these effects vary greatly with the type of judgment. There is little general overconfidence with two-choice questions and pronounced overconfidence with subjective confidence intervals. Over- and underconfidence also vary systematically with the domain of questions asked, but not as a function of difficulty. We also find stable individual differences. Determining why some people, some domains, and some types of judgments are more prone to overconfidence will be important to understanding how confidence judgments are made.     

The role of individual differences in the accuracy of confidence judgments

Generally, self-assessment of accuracy in the cognitive domain produces overconfidence, whereas self-assessment of visual perceptual judgments results in underconfidence. Despite contrary empirical evidence, in models attempting to explain those phenomena, individual differences have often been disregarded. The authors report on 2 studies in which that shortcoming was addressed. In Experiment 1, participants (N= 520) completed a large number of cognitive-ability tests. Results indicated that individual differences provide a meaningful source of overconfidence and that a metacognitive trait might mediate that effect. In further analysis, there was only a relatively small correlation between test accuracy and confidence bias. In Experiment 2 (N = 107 participants), both perceptual and cognitive ability tests were included, along with measures of personality. Results again indicated the presence of a confidence factor that transcended the nature of the testing vehicle. Furthermore, a small relationship was found between that factor and some self-reported personality measures. Thus, personality traits and cognitive ability appeared to play only a small role in determining the accuracy of self-assessment. Collectively, the present results suggest that there are multiple causes of miscalibration, which current models of over- and underconfidence fail to encompass.     

The Overconfidence Phenomenon as a Consequence of Informal Experimenter-Guided Selection of Almanac Items

The paper argues for an ecological approach to realism of confidence in general knowledge. It is stressed that choice of answer to almanac items and confidence judgments derive from knowledge structures formed by adaptation to a natural environment. People are regarded as well-calibrated to their natural environments and the overconfidence phenomenon is seen as a consequence of the procedures involved in the creation of "traditional" general knowledge items, rather than as the result of a cognitive bias. An experiment is reported showing that when items are informally selected by human "selectors," instructed to select items that differentiate between more and less knowledgeable subjects, we observe poor calibration and the usual overconfidence phenomenon. However, when the item selection is "debiased" and the objects to be compared in the almanac items are selected randomly from a natural environment the overconfidence phenomenon disappears and we observe good calibration, and good resolution. These results support an ecological approach to realism of confidence and suggest that general procedures for debiasing overconfidence may be unwarranted.     

Outcome Feedback Effects on Under- and Overconfident Judgments (General Knowledge Tasks)

Effects of outcome feedback on overconfident and underconfident judgments were investigated. It was hypothesized that outcome feedback had a twofold effect on confidence judgments, corrective and confidence increasing, the latter effect operating only when the items to be judged are related and/or nonindependent. The proposed hypothesis implies two predictions which were tested in two experiments: (1) If the items are nonindependent and related, outcome feedback will reduce bias and improve calibration of underconfident judgments; by contrast, the feedback will neither reduce bias nor improve calibration in the case of overconfident judgments. (2) If, on the other hand, the items are independent and unrelated, outcome feedback will reduce underconfidence as well as overconfidence bias and improve calibration in both cases. The obtained results provided strong support for the first prediction, while the second prediction was not supported. Although the differences between with- and without-feedback groups on the dependent variables were in the expected direction, they were nonsignificant.     

The Role of Individual Differences in the Accuracy of Confidence Judgments

Generally, self-assessment of accuracy in the cognitive domain produces overconfidence, whereas self-assessment of visual perceptual judgments results in under-confidence. Despite contrary empirical evidence, in models attempting to explain those phenomena, individual differences have often been disregarded. The authors report on 2 studies in which that shortcoming was addressed. In Experiment 1, participants (N = 520) completed a large number of cognitive-ability tests. Results indicated that individual differences provide a meaningful source of overconfidence and that a metacognitive trait might mediate that effect. In further analysis, there was only a relatively small correlation between test accuracy and confidence bias. In Experiment 2 (N = 107 participants), both perceptual and cognitive ability tests were included, along with measures of personality. Results again indicated the presence of a confidence factor that transcended the nature of the testing vehicle. Furthermore, a small relationship was found between that factor and some self-reported personality measures. Thus, personality traits and cognitive ability appeared to play only a small role in determining the accuracy of self-assessment. Collectively, the present results suggest that there are multiple causes of miscalibration, which current models of over- and underconfidence fail to encompass.     

Should observed overconfidence be dismissed as a statistical artifact? Critique of Erev, Wallsten, and Budescu (1994)

I. Erev, T. S. Wallsten, and D. V. Budescu (1994) showed that the same probability judgment data can reveal both apparent overconfidence and underconfidence, depending on how the data are analyzed. To explain this seeming paradox, I. Erev et al. proposed a general model of judgment in which overt responses are related to underlying "true judgments" that are perturbed by error. A central conclusion of their work is that observed over- and underconfidence can be split into two components: (a) "true" over- and underconfidence and (b) "artifactual" over- and underconfidence due to error in judgment. It is argued in the present article that decomposing over- and underconfidence into true and artifactual components is inappropriate. The mistake stems from giving primacy to ambiguously defined model constructions (true judgments) over observed data.     

Realism of confidence in sensory discrimination: The underconfidence phenomenon

This paper documents a very pervasive underconfidence bias in the area of sensory discrimination. In order to account for this phenomenon, a subjective distance theory of confidence in sensory discrimination is proposed. This theory, based on the law of comparative judgment and the assumption of confidence as an increasing function of the perceived distance between stimuli, predicts underconfidence—that is, that people should perform better than they express in their confidence assessments. Due to the fixed sensitivity of the sensory system, this underconfidence bias is practically impossible to avoid. The results of Experiment 1 confirmed the prediction of underconfidence with the help of present-day calibration methods and indkated-a-good quantitative fit of the theory. The results of Experiment 2 showed that prolonged experience of outcome feedback (160 trials) had no effect on underconfidence. It is concluded that the subjective distance theory provides a better explanation of the underconfidence phenomenon than-do previous accounts in terms of subconscious processes.     

Confidence and decision type under matched stimulus conditions: overconfidence in perceptual but not conceptual decisions

Within the domain of metacognition, there is disagreement whether different processes underlie evaluations of confidence in perceptual versus conceptual decisions. The relationship between confidence and accuracy for perceptual and conceptual decisions was compared using newly created stimuli that could be used to elicit either decision type. Based on theories of Brunswikian and Thurstonian uncertainties, significant underconfidence for perceptual decisions and overconfidence for conceptual decisions were predicted. Three within‐subjects experiments did not support this hypothesis. Participants showed significant overconfidence for perceptual decisions and no overconfidence for conceptual decisions. In addition, significant hard‐easy effects were consistently found for both decision types. Incorporating our findings with past results reveals that both over‐ and underconfidence are attainable on perceptual tasks. This conclusion, in addition to the common presence of hard‐easy effects and significant across‐task correlations in over/underconfidence, suggests that confidence judgments for the two decision types may depend on largely shared processes. Possible contributions to confidence and over/underconfidence are explored, focusing on response time factors and participants' knowledge bases. Copyright © 2007 John Wiley & Sons, Ltd.     

The domain specificity and generality of overconfidence: Individual differences in performance estimation bias

One hundred twenty-three college students performed a knowledge assessment task and a game of motor skill in which they had to predict their performance before each block of trials. There was a bias in the direction of overconfidence on both tasks, even though the latter involved the motor domain, did not require the use of numeric probabilities, and allowed predictions to be made by using an aggregate judgment made in a frequentist mode. An analysis of individual differences indicated that there was considerable domain specificity in confidence judgments. However, participants who persevered in showing overconfidence in the motor task—despite previous feedback revealing their overconfident performance predictions—were significantly more overconfident in the knowledge calibration task than were participants who moderated their motor performance predictions so as to remove their bias toward overconfidence. The latter finding is consistent with explanations of overconfidence effects that implicate mechanisms with some degree of domain generality.     

Determinants of Overconfidence and Miscalibration: The Roles of Random Error and Ecological Structure

Previous authors have attributed findings of overconfidence to psychological bias or to experimental designs unrepresentative of the environment. This paper provides evidence for an alternative explanation. A model is presented in which reported confidence is a function of the validity of information used by the subject, and a random error component. The model predicts greater overconfidence for question sets in which informational cues are less valid. This result corresponds to the well-known hard/easy effect. The model also predicts that unrepresentative design (Gigerenzer, Hoffrage, & Kleinbölting, 1991) is sufficient but not necessary for overconfidence to occur. These predictions are tested, and results provide support for the model. Subjects are overconfident according to usual measures such as calibration, even though on average they report the diagnosticity of informational cues correctly. Furthermore, overconfidence is greater for harder sets of questions, even when those sets are representative of the environment. A post hoc analysis reveals some intriguing individual differences among subjects. Some people appear to have a true psychological bias toward reporting high levelsof confidence, whereas others have a psychological bias in the direction of underconfidence.     

Depressive cognition: a test of depressive realism versus negativity using general knowledge questions

This research is an examination of whether cognition in depressed individuals incorporates a realistic view of the world or a general tendency toward negativity. Participants provided two types of probability judgments of the likelihood that they correctly answered general knowledge questions: the probability that they correctly answered each of the individual questions and an aggregate judgment, after completing the questionnaire, of the percentage of all the questions they thought they had correctly answered. These tasks generally elicit overconfidence and accuracy in nondepressives. In accord with theories of both depressive realism and general negativity, in their item-by-item assessments of their answers to the individual questions, depressed participants demonstrated less overconfidence than nondepressed participants. In accord with the theory of general negativity but not with the theory of depressive realism, however, depressed participants demonstrated underconfidence in their aggregate judgments. The implications of these findings on theories of depressive cognition are discussed.     

Confidence in judgment

Experiments have shown that, generally, people are overconfident about the correctness of their answers to questions. Cognitive psychologists have attributed this to biases in the way people generate and handle evidence for and against their views. The overconfidence phenomenon and cognitive psychologists' accounts of its origins have recently given rise to three debates. Firstly, ecological psychologists have proposed that overconfidence is an artefact that has arisen because experimenters have used question material not representative of the natural environment. However, it now appears that some overconfidence remains even after this problem has been remedied. Secondly, it has been proposed that overconfidence is an artefactual regression effect that arises because judgments contain an inherently random component. However, those claiming this appear to use the term overconfidence to refer to a phenomenon quite different from the one that the cognitive psychologists set out to explain. Finally, a debate has arisen about the status of perceptual judgments. Some claim that these evince only underconfidence and must, therefore, depend on mechanisms fundamentally different from those subserving other types of judgment. Others have obtained overconfidence with perceptual judgments and argue that a unitary theory is more appropriate. At present, however, no single theory provides an adequate account of the many diverse factors that influence confidence in judgment.     

Gender Differences in the Self-Assessment of Accuracy on Cognitive Tasks

Generally, self-assessment of accuracy in the cognitive domain produces overconfidence, whereas self-assessment of the accuracy of visual perceptual judgments produces underconfidence. The possible effect of gender differences on these robust findings appears to be underinvestigated. In this paper, I report two studies that take a step toward redressing this shortcoming. In Study 1, a group of young adults (N =185) were presented with a test of General Knowledge and a visual perceptual task. The results indicated the typical over/underconfidence phenomena noted above, but, when analyzed by sex, indicated statistically significant differences; men were more confident than women on both tasks. In Study 2, participants (N =303) with a wider age range completed 4 tests of cognitive ability, which were drawn from the theory of fluid and crystallized intelligence. Results indicated that the tendency for men to express higher levels of confidence than women in the accuracy of their work appears to remain constant across the life-span. These findings are discussed in relation to self-concept and gender stereotyping.     

Depressive Cognition: A Test of Depressive Realism Versus Negativity Using General Knowledge Questions

This research is an examination of whether cognition in depressed individuals incorporates a realistic view of the world or a general tendency toward negativity. Participants provided two types of probability judgments of the likelihood that they correctly answered general knowledge questions: the probability that they correctly answered each of the individual questions and an aggregate judgment, after completing the questionnaire, of the percentage of all the questions they thought they had correctly answered. These tasks generally elicit overconfidence and accuracy in nondepressives. In accord with theories of both depressive realism and general negativity, in their item-by-item assessments of their answers to the individual questions, depressed participants demonstrated less overconfidence than nondepressed participants. In accord with the theory of general negativity but not with the theory of depressive realism, however, depressed participants demonstrated underconfidence in their aggregate judgments. The implications of these findings on theories of depressive cognition are discussed.     

Overconfidence in Probability and Frequency Judgments: A Critical Examination

The overconfidence observed in calibration studies has recently been questioned on both psychological and methodological grounds. In the first part of the article we discuss these issues and argue that overconfidence cannot be explained as a selection bias, and that it is not eliminated by random sampling of questions. In the second part of the article, we compare probability judgments for single events with judgments of relative frequency. Subjects received a target individual's personality profile and then predicted the target's responses to a series of binary questions. One group predicted the responses of an individual target, while a second group estimated the relative frequency of responses among all target subjects who shared a given personality profile. Judgments of confidence and estimates of relative frequency were practically indistinguishable; both exhibited substantial overconfidence and were highly correlated with independent judgments of representativeness.