An explanation of the hard-easy effect in studies of realism of confidence in one's general knowledge

Abstract

Recent ecological approaches to realism of confidence in general knowledge (Gigerenzer, Hoffrage & Kleinbolting, 1991; Juslin, in press) argue that people are well-calibrated to their natural environments. Both the overconfidence phenomenon and the hard-easy effect are explained as consequences of informal experimenter-guided selection of almanac items, selection that changes the validity of the cues used by the subjects for selection of answers to the items. The paper presents the ecological approach and reports an experiment showing that: (1) when the objects of judgement are selected randomly from a natural environment, people are well-calibrated; (2) when more and less difficult item samples are created by selecting items with more and less familiar contents, i.e. in a way that does not affect the validity of the cues, no hard-easy effect is observed, and people are well-calibrated both for hard and easy item samples. These results, predicted by the ecological approach, provide further support for the ecological approach to realism of confidence.     

Realism in confidence judgements of performance based on implicit learning

In two experiments we employed calibration methods to investigate the realism of participants' confidence ratings of their own classification performance based on knowledge acquired after training on an artificial grammar. In Experiment 1 participants showed good realism (but overconfidence) for grammatical strings but very poor realism for non-grammatical strings. Method of training (string repetition in writing or mere exposure) did not affect the realism. Furthermore, the participants underestimated their overall performance. In Experiment 2, using a more complex grammar and controlling for two types of associative chunk-strength, participants showed good realism (but still overconfidence) for both letter and symbol strings, irrespective of grammaticality. Together, these experiments show that implicit learning can give rise to knowledge products that are associated with fairly realistic meta-knowledge. It is argued that both the zero-correlation criterion and the guessing criterion are misplaced when used to define implicit knowledge; two reasons being that confidence judgements may be affected both by implicit knowledge and by inferences.     

Calibration of sensory and cognitive judgments: Two different accounts

The purpose of this study was to test predictions of two recent theories of realism of confidence. Ecological approaches to realism of confidence in one's general knowledge (Gigerenzer et al. , 1991; Juslin, in press a ; Björkman, in press) predict good calibration or, in the case of poor cognitive adjustment, overconfidence, within the cognitive domain. The subjective distance theory of confidence in sensory discriminations (Björkman et al. , 1992) predicts a pervasive underconfidence bias for sensory discriminations. Empirical data are reported showing that: (a) Calibration for sensory judgments is considerably poorer than calibration for well adapted cognitive judgements, a difference that can be entirely traced to underconfidence in the sensory domain. (b) While an initial overconfidence bias in the cognitive domain is removed by outcome feedback, the bias observed in sensory discriminations is unaffected even by a prolonged feedback session. It is suggested that the nature of confidence in sensory discriminations is different from the nature of confidence in cognitive judgments.     

Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect

Two robust phenomena in research on confidence in one's general knowledge are the overconfidence phenomenon and the hard-easy effect. In this article, the authors propose that the hard-easy effect has been interpreted with insufficient attention to the scale-end effects, the linear dependency, and the regression effects in data and that the continued adherence to the idea of a "cognitive overconfidence bias" is mediated by selective attention to particular data sets. A quantitative review of studies with 2-alternative general knowledge items demonstrates that, contrary to widespread belief, there is (a) very little support for a cognitive-processing bias in these data; (b) a difference between representative and selected item samples that is not reducible to the difference in difficulty; and (c) near elimination of the hard-easy effect when there is control for scale-end effects and linear dependency.     

Ecological and person‐oriented aspects of metacognitive processes in test‐taking

This study examines critical aspects of both the ecological and the person‐oriented accounts of observed biases in confidence judgements on tests of cognitive abilities. These biases reflect metacognitive processes involved in test‐taking. According to the ecological approach, poor realism of confidence judgements is due to the nature of the items included in general knowledge tests (test‐driven biases). The person‐oriented approach, however, argues that biases in confidence judgements may be due to a general self‐monitoring trait. The present study employed the ‘de‐biasing’ procedure proposed by Juslin ( 1994 ) for the selection of general knowledge test items, and used a newly developed geographical knowledge test suitable for the Australian population. Two other cognitive tests (Raven's Progressive Matrices and Line Length) were administered in order to determine whether there is a consistency in confidence ratings across diverse tasks. Statistical procedures traditional to both approaches‐calibration curves and factor analysis ‐ were employed. The results, with minor qualifications, support both perspectives. The study found a separate confidence factor, indicative of a self‐monitoring trait. Two other potential metacognitive factors (i.e. ‘expectation’ and ‘evaluation’, corresponding to self‐assessment/planning and self‐evaluation) could not be separated from accuracy and speed measures. Copyright © 2001 John Wiley & Sons, Ltd.     

Internal Cue Theory: Calibration and Resolution of Confidence in General Knowledge

A theory is proposed in which beliefs in the form of internal cue validities mediate the processing of ecological cue validities in the assessment of confidence. The conditions necessary for perfect calibration are specified: (a) correspondence between ecological and internal validity, (b) perfect translation of internal validity into a confidence assessment, and (c) consistent utilization of cues. Process errors are then added to these conditions to investigate how calibration is affected by error variance of confidence assessments. To accomplish this, the calibration score (C) is decomposed into three additive parts: D² = bias, i.e., the squared difference between mean confidence and proportion correct; R² = resolution, i.e., the squared difference between the standard deviations of confidence and proportion correct; L = linearity, i.e., how closely the calibration curve follows a linear function. In the equation C = D² + R² + L, R² (resolution) reflects the subject′s ability to discriminate cue validities. Selection of items is a critical factor in studies of confidence. Informal selection with a tendency to avoid easy items results in overconfidence. Internal cue theory predicts both that overconfidence should disappear (in accordance with previous research) and that resolution should improve when item selection is made representative of the natural environment. Both predictions are confirmed by data from published studies on confidence in general knowledge. It is noteworthy that resolution is still poor and accounts for the major portion of miscalibration under representative item selection.     

CONFIANCE EXCESSIVE: CONNAISSANCES ET CROYANCES

It had already been found (Lamarche 1988; Oskamp 1965) that the overconfidence shown in answering a question increased with the number of new informations given. The objective of our research was to verify if there was a relation between that phenomenon, obtained with knowledge questions, and the evolution of confidence, in the same circumstances, when belief questions are involved. On the one hand, we presented to our subjects knowledge questions which were repeated with a growing number of clues; on the other hand, the same subjects were asked to form an impression of a person based on a list of traits presented sequentially. Using a calibration index, we were able to observe anew the growing overconfidence with knowledge questions. However, in the case of beliefs, the confidence level did not grow in the same way. We explain that result by our choice of traits. Nevertheless, there was a relation between the increase of confidence and the decrease of calibration. We think this could be the result of a lack of involvement with the belief questionnaire which is less so when answered first.     

Actor-observer differences in realism in confidence and frequency judgments

Taking a social psychological approach to metacognitive judgments, this study analyzed the difference in realism (validity) in confidence and frequency judgments (i.e., estimates of overall accuracy) between one's own and another person's answers to general knowledge questions. Experiment 1 showed that when judging their own answers, compared with another's answers, the participants exhibited higher overconfidence, better ability to discriminate correct from incorrect answers, lower accuracy, and lower confidence. However, the overconfidence effect could be attributable to the lowest level of confidence. Furthermore, when heeding additional information about another's answers the participants showed higher confidence and better discrimination ability. The overconfidence effect of Experiment 1 was not found in Experiment 2. However, the results of Experiment 2 were consistent with Experiment 1 in terms of discrimination ability, confidence, and accuracy. Finally, in both experiments the participants gave lower frequency judgments of their own overall accuracy compared with their frequency judgments of another person's overall accuracy.     

Increased realism in eyewitness confidence judgements: the effect of dyadic collaboration

This study investigated to what extent, and under what circumstances, pair collaboration influences the realism in eyewitness confidence in event memory. The participants first saw a short film clip and then confidence rated their answers to questions on its content. A condition (the Individual–Pair condition) where individual effort preceded pair collaboration showed better calibration compared with a condition (the Simple Pair condition) where no individual effort took place. Furthermore, within the Individual–Pair condition, better calibration, and lower overconfidence, were found in the pair phase compared with the individual phase. The eyewitnesses in the Individual–Pair condition made more realistic judgements of the total number of questions answered correctly. In a control experiment no effect on realism in confidence was found when individuals performed the same task twice. The improved realism in the Individual–Pair condition may partly be explained in terms of the increased accuracy and lowered confidence found for such items where the pair members' had given different answers in the individual phase, and by a risky shift effect for such items where they had given the same answer. Copyright © 2003 John Wiley & Sons, Ltd.     

Confidence judgments in studies of individual differences

This paper examines psychometric properties of scores derived from calibration curves (overconfidence, calibration, resolution, and slope) and an analogue of overconfidence that is based on a posttest estimate of the proportion of correctly solved items. Four tests from the theory of fluid and crystallized intelligence were used, and two of these tests employed both sequential and simultaneous methods of item presentation. The results indicate that the overconfidence score not only has the highest reliability, but is the only score with a reliability normally considered adequate for use in individual differences research. There is some, albeit weak, difference in subjects' level of overconfidence between sequential and simultaneous methods of item presentation. Correlational evidence confirms our previous findings that overconfidence scores from perceptual and ‘knowledge’ tasks define the same factor. In agreement with the results of Gigerenzer, Hoffrage and Kleinbolting (1991), subjects' post-test estimates of their performance showed lower levels of overconfidence than did the traditional measures based on subjects' confidence judgment responses to individual items. Also, after controlling for the actual test performances, the post-test performance estimates and average confidence ratings were only slightly positively correlated, suggesting that different psychological processes may underlie these two measures. Finally, our results suggest that average confidence over all items in the test may be a more useful measure in individual differences research than scores derived from calibration curves.     

The role of short-term memory capacity and task experience for overconfidence in judgment under uncertainty

Research with general knowledge items demonstrates extreme overconfidence when people estimate confidence intervals for unknown quantities, but close to zero overconfidence when the same intervals are assessed by probability judgment. In 3 experiments, the authors investigated if the overconfidence specific to confidence intervals derives from limited task experience or from short-term memory limitations. As predicted by the naive sampling model (P. Juslin, A. Winman, & P. Hansson, 2007), overconfidence with probability judgment is rapidly reduced by additional task experience, whereas overconfidence with intuitive confidence intervals is minimally affected even by extensive task experience. In contrast to the minor bias with probability judgment, the extreme overconfidence bias with intuitive confidence intervals is correlated with short-term memory capacity. The proposed interpretation is that increased task experience is not sufficient to cure the overconfidence with confidence intervals because it stems from short-term memory limitations.     

Realism in Confidence Judgments as a Function of Working in Dyads or Alone

This study compares the realism of confidence judgments made by individuals and pairs of their answers to general knowledge questions, using a within-subjects design. In the initial Control condition, subjects first answered 30 questions and then confidence rated the chosen answer. For the next 30 questions (Condition 2), they wrote down an argument for the chosen answer and then confidence-rated their answer. In the third condition, the 40 subjects were divided into 20 pairs who then answered and confidence-rated the same 30 questions as in Condition 2. Pair members were asked to collaborate on all parts of the task. The results showed that overconfidence decreased in the Pair condition compared with the Single conditions. Analysis of the interaction in the Pair condition showed a higher overconfidence in those instances where one pair member dominated the interaction totally. Other analyses of the interaction in the Pair condition also supported the importance of argumentation between the pair members for realistic confidence judgments. A comparison with previous research suggests that the fact that the subjects first attempted the questions individually may have increased the proportion correct answers and limited the increase in confidence in the Pair condition. A control study, checking for the effect of subjects answering the same questions twice, found no effect of repetition for any of the three calibration measures used.     

Confidence in judgment

Experiments have shown that, generally, people are overconfident about the correctness of their answers to questions. Cognitive psychologists have attributed this to biases in the way people generate and handle evidence for and against their views. The overconfidence phenomenon and cognitive psychologists' accounts of its origins have recently given rise to three debates. Firstly, ecological psychologists have proposed that overconfidence is an artefact that has arisen because experimenters have used question material not representative of the natural environment. However, it now appears that some overconfidence remains even after this problem has been remedied. Secondly, it has been proposed that overconfidence is an artefactual regression effect that arises because judgments contain an inherently random component. However, those claiming this appear to use the term overconfidence to refer to a phenomenon quite different from the one that the cognitive psychologists set out to explain. Finally, a debate has arisen about the status of perceptual judgments. Some claim that these evince only underconfidence and must, therefore, depend on mechanisms fundamentally different from those subserving other types of judgment. Others have obtained overconfidence with perceptual judgments and argue that a unitary theory is more appropriate. At present, however, no single theory provides an adequate account of the many diverse factors that influence confidence in judgment.     

The Importance of Item Selection in "Knew-It-All-Along" Studies of General Knowledge

This study was designed to investigate the effects of item sampling on hindsight bias in experiments using general knowledge material. The results show that the use of random versus traditional experimenter-selected item samples can have different effects on hindsight bias. In a within-subjects study almost twice as many items in a random sample were connected with a reversed effect rather than with a traditional hindsight bias. The same items that resulted in overconfidence in foresight lead to a higher degree of hindsight bias than others. The results suggest that earlier findings of unusually large hindsight effects with general knowledge tasks may be explained by the selection of items used. No hindsight effect was found on confidence scores in a within-subjects design, but was obtained in a between-subjects design. Results suggest that the use of a within-subjects design itself can moderate hindsight bias by familiarizing subjects with the task. The study shows the importance of two conditions for decreasing the hindsight bias: (1) The use of randomly sampled items, and (2) The use of a within-subjects procedure. When these conditions were met, the "knew-it-all-along effect" was completely eliminated.     

The misunderstood limits of folk science: an illusion of explanatory depth

People feel they understand complex phenomena with far greater precision, coherence, and depth than they really do; they are subject to an illusion-an illusion of explanatory depth. The illusion is far stronger for explanatory knowledge than many other kinds of knowledge, such as that for facts, procedures or narratives. The illusion for explanatory knowledge is most robust where the environment supports real-time explanations with visible mechanisms. We demonstrate the illusion of depth with explanatory knowledge in Studies 1-6. Then we show differences in overconfidence about knowledge across different knowledge domains in Studies 7-10. Finally, we explore the mechanisms behind the initial confidence and behind overconfidence in Studies 11 and 12. Implications for the roles of intuitive theories in models of concepts and cognition are discussed.     

Depressive cognition: a test of depressive realism versus negativity using general knowledge questions

This research is an examination of whether cognition in depressed individuals incorporates a realistic view of the world or a general tendency toward negativity. Participants provided two types of probability judgments of the likelihood that they correctly answered general knowledge questions: the probability that they correctly answered each of the individual questions and an aggregate judgment, after completing the questionnaire, of the percentage of all the questions they thought they had correctly answered. These tasks generally elicit overconfidence and accuracy in nondepressives. In accord with theories of both depressive realism and general negativity, in their item-by-item assessments of their answers to the individual questions, depressed participants demonstrated less overconfidence than nondepressed participants. In accord with the theory of general negativity but not with the theory of depressive realism, however, depressed participants demonstrated underconfidence in their aggregate judgments. The implications of these findings on theories of depressive cognition are discussed.     

Depressive Cognition: A Test of Depressive Realism Versus Negativity Using General Knowledge Questions

This research is an examination of whether cognition in depressed individuals incorporates a realistic view of the world or a general tendency toward negativity. Participants provided two types of probability judgments of the likelihood that they correctly answered general knowledge questions: the probability that they correctly answered each of the individual questions and an aggregate judgment, after completing the questionnaire, of the percentage of all the questions they thought they had correctly answered. These tasks generally elicit overconfidence and accuracy in nondepressives. In accord with theories of both depressive realism and general negativity, in their item-by-item assessments of their answers to the individual questions, depressed participants demonstrated less overconfidence than nondepressed participants. In accord with the theory of general negativity but not with the theory of depressive realism, however, depressed participants demonstrated underconfidence in their aggregate judgments. The implications of these findings on theories of depressive cognition are discussed.     

Mood and realism of confidence judgements of one''s own answers to general knowledge questions

This study examines the effect of a depressed mood on the realism of subjects' confidence judgements of the correctness of answers to general knowledge questions. Research conducted on how mood influences cognitive processes gives reason to expect that a depressed mood might increase the realism of individuals' confidence ratings. Sixty subjects were divided into three conditions, two of which were given mood induction, one condition into an elated-happy mood and one condition into a depressed-sad mood. As evidenced by subjects' responses to mood scales only the depressed condition was affected by the mood induction. All subjects answered 93 general knowledge questions and rated their confidence in the correctness of the answer given. Subjects were instructed to think aloud when answering the last 31 questions. The conditions did not differ with respect to the proportion of questions answered correctly, mean level of confidence, nor with respect to three measures of the realism in subjects' confidence ratings (calibration, over/underconfidence and resolution). The results were the same when questions answered with and without think aloud instructions were analysed separately.     

Confidence–Accuracy Calibration with General Knowledge and Eyewitness Memory Cued Recall Questions

Summary: The confidence–accuracy relationship has primarily been studied through recognition tests and correlation analysis. However, cued recall is more ecological from a forensic perspective. Moreover, there may be more informative ways of analysing the confidence–accuracy relationship than correlations. In the present study, participants viewed a video of a bank robbery and were asked cued recall questions covering general knowledge and the video itself. Confidence ratings were collected, and correlations, calibration and discrimination measures were calculated. All measures indicated a strong confidence–accuracy relationship that was better for general knowledge than eyewitness memory questions. However, there were no differences in confidence ratings for correct answers, suggesting that the differences could be limited to the evaluation of incorrect answers. We concluded that confidence may be a good marker for accuracy with cued recall, but that further research using ecological tests and more informative data analysis techniques is needed. Copyright © 2011 John Wiley & Sons, Ltd.     

The domain specificity and generality of overconfidence: Individual differences in performance estimation bias

One hundred twenty-three college students performed a knowledge assessment task and a game of motor skill in which they had to predict their performance before each block of trials. There was a bias in the direction of overconfidence on both tasks, even though the latter involved the motor domain, did not require the use of numeric probabilities, and allowed predictions to be made by using an aggregate judgment made in a frequentist mode. An analysis of individual differences indicated that there was considerable domain specificity in confidence judgments. However, participants who persevered in showing overconfidence in the motor task—despite previous feedback revealing their overconfident performance predictions—were significantly more overconfident in the knowledge calibration task than were participants who moderated their motor performance predictions so as to remove their bias toward overconfidence. The latter finding is consistent with explanations of overconfidence effects that implicate mechanisms with some degree of domain generality.