Unskilled, unaware, or both? The better-than-average heuristic and statistical regression predict errors in estimates of own performance

People who score low on a performance test overestimate their own performance relative to others, whereas high scorers slightly underestimate their own performance. J. Kruger and D. Dunning (1999) attributed these asymmetric errors to differences in metacognitive skill. A replication study showed no evidence for mediation effects for any of several candidate variables. Asymmetric errors were expected because of statistical regression and the general better-than-average (BTA) heuristic. Consistent with this parsimonious model, errors were no longer asymmetric when either regression or the BTA effect was statistically removed. In fact, high rather than low performers were more error prone in that they were more likely to neglect their own estimates of the performance of others when predicting how they themselves performed relative to the group.     

Structuring decision problems and the ‘bias heuristic’

This paper examines our understanding of the decomposition of immediate acts when structuring decision problems. Seven different types of uncertainties are identified, and four of these are shown to be taken explicitly into account in models within the province of decision theory, described in terms of four interlocking systems interfaced with semantic memory (a core act-event system, and systems buffering utilities, probabilities and events, respectively). Requisite decision modeling is shown to require that the remaining three types of uncertainty (procedural uncertainty; how the decision maker will feel about subsequent acts; agency for changing subsequent states of the world) are also resolved. Methods for ‘fixing’ structure are discussed in terms of aiming at a common understanding about the ‘small world’ in which a decision problem is located. Difficulties in resolving uncertainties in doing this are described. An alternative approach, common in studies invoking ‘behavioural decision theory’ is contrasted: imposing structure, assuming common understanding. The latter approach is shown to involve (i) the ‘naturalisation’ of the small world in which the decision problem is located, and (ii) the utilisation of normative models as ‘ideal types’, leading to the use of the ‘bias’ argument in discussing subjects' performance in decision tasks. Using this argument reflexively, the operation of the ‘bias heuristic’ is identified in a survey of published papers referencing this approach to the study of decision making. Effects identified are: availability of tasks, subjects and explanations; representativeness of findings; and anchoring and adjustment of explanations. Implications for practice are discussed throughout the paper.     

Perceptual access reasoning: developmental stage or system 1 heuristic?

In contrast with the two dominant views in Theory of Mind development, the Perceptual Access Reasoning hypothesis of Fabricius and colleagues is that children don’t understand the mental state of belief until around 6 years of age. Evidence for this includes data that many children ages 4 and 5, who pass the standard 2-location false belief task, nonetheless fail the true belief task, and often fail a 3-location false belief task by choosing the irrelevant option. These findings can be explained by the PAR hypothesis but pose challenges for the two dominant views. I argue against an alternate hypothesis which is proposed by Anika Fiebich in a recent paper. According to Fiebich, PAR is not a distinct transitional stage in children’s theory of mind development, but is a fast and frugal System 1 heuristic which fades once children become fluent in social reasoning. However, I point out a number of problems with Fiebich’s proposal and argue for the superiority of the PAR hypothesis. I also present five reasons to be skeptical about the findings of Perner and Horn which purportedly show that 4- and 5-year-olds can pass the 3-location false belief task when suitably modified. This is a further difficulty for Fiebich’s proposal, since she relies on these findings in her fluency theory. Finally, I sketch a dual systems theory of mind account based upon the PAR hypothesis which is different from Fiebich’s.     

Preference for oddity: uniqueness heuristic or hierarchical choice process?

Traditional economic theories assume decision makers in multialternative choice tasks "assign" a value to each option and then express rational preferences. Here, I report an apparent violation of such rationality in gray jays (Perisoreus canadensis). I tested the jays' preference in a quaternary choice task where three options were the same color and the fourth option was a different color. All options offered an identical food reward and so the strictly rational expectation was that subjects would choose the odd-colored option in 25% of choices. In clear disagreement, every subject chose the odd option more frequently than expected. I speculate as to how this surprising preference for oddity might have been ecologically rational: by using a unique-choice heuristic, the jays might have been able to bypass a deliberative phase of the decision process and devote more attention to scanning for predators. Alternatively, it is conceivable that the jays did not prefer oddity per se. Instead, they might have used a hierarchical process, assigning options to color categories and then choosing between categories. If so, their behavior matches expectation after all (on average, subjects chose the odd option 50% of the time). It should be straightforward to test these competing hypotheses. The current results can be viewed as a new example of how simple mechanisms sometimes produce economically puzzling yet ecologically rational decision making.     

Criterion knowledge: A moderator of using the recognition heuristic?

The recognition heuristic (RH) claims that people base inferences on recognition only. This has been questioned by several studies which found that additional knowledge was influential. However, in some of these studies, participants' additional knowledge might have encompassed criterion knowledge thus rendering any inferential strategy superfluous. The present study was therefore designed to test the effect of criterion knowledge on use or non‐use of the RH. Eighty‐one participants made pair‐wise comparisons with respect to the size of Belgian cities and also provided estimates of the cities' actual size. We found that relative criterion knowledge (i.e., knowledge about the relative position of an object on the criterion dimension) did indeed play some role, but its exclusion left the main critical findings intact, nonetheless. We thus conclude that previous studies conducted in the paradigm of natural recognition should not be generally refuted by the argument of participants possessing criterion knowledge. Copyright © 2009 John Wiley & Sons, Ltd.     

Is comprehension of problem solutions resistant to misleading heuristic cues?

Previous studies in the domain of metacomprehension judgments have primarily used expository texts. When these texts include illustrations, even uninformative ones, people were found to judge that they understand their content better. The present study aimed to delineate the metacognitive processes involved in understanding problem solutions — a text type often perceived as allowing reliable judgments regarding understanding, and was not previously considered from a metacognitive perspective. Undergraduate students faced difficult problems. They then studied solution explanations with or without uninformative illustrations and provided judgments of comprehension (JCOMPs). Learning was assessed by application to near-transfer problems in an open-book test format. As expected, JCOMPs were polarized — they tended to reflect good or poor understanding. Yet, JCOMPs were higher for the illustrated solutions and even high certainty did not ensure resistance to this effect. Moreover, success in the transfer problems was lower in the presence of illustrations, demonstrating a bias stronger than that found with expository texts. Previous studies have suggested that weak learners are especially prone to being misled by superficial cues. In the present study, matching the difficulty of the task to the ability of the target population revealed that even highly able participants were not immune to misleading cues. The study extends previous findings regarding potential detrimental effects of illustrations and highlights aspects of the metacomprehension process that have not been considered before.     

When is enough, enough? A comment on Grodzinsky and Marek's "Algorithmic and heuristic processes revisited"

In this comment I argue that although Y. Grodzinsky & A. Marek's (1983, Brain and Language, 33, 216-225) criticism of the conclusions reached by A. Caramazza and E. B. Zurif (1976, Brain and Language, 3, 572-582) in their paper on comprehension disorders in so-called agrammatic patients is not entirely without technical merit, its impact on the claims made by Caramazza and Zurif is inconsequential. I show that there are deeper theoretical and methodological reasons which undermine the claims made by Caramazza and Zurif and the only superficially different proposal of Grodzinsky and Marek.     

What is behind the priority heuristic? A mathematical analysis and comment on Brandstätter, Gigerenzer, and Hertwig (2006)

Comments on the article by E. Brandst?tter, G. Gigerenzer, and R. Hertwig. The authors discuss the priority heuristic, a recent model for decisions under risk. They reanalyze the experimental validity of this approach and discuss how these results compare with cumulative prospect theory, the currently most established model in behavioral economics. They also discuss how general models for decisions under risk based on a heuristic approach can be understood mathematically to gain some insight in their limitations. They finally consider whether the priority heuristic model can lead to some understanding of the decision process of individuals or whether it is better seen as an as-if model.     

Who will win Wimbledon? The recognition heuristic in predicting sports events

Goldstein and Gigerenzer ( 2002 ) described the recognition heuristic as a fast, frugal, and effective decision strategy. However, most studies concerning the recognition heuristic have been conducted in static domains, that is, in domains where it could plausibly be argued that relevant variables stay relatively constant. Yet the question is whether the heuristic would also work in dynamic environments where the quality of the actors rises and falls, such as in sports. We tested performance of the recognition heuristic in a dynamic environment and used it to predict the outcomes of tennis matches in Wimbledon 2003. Recognition data of amateur tennis players and laypeople was used to build recognition rankings. These rankings correlated with official rankings and led to at least as good predictions. Simulations of individual choices showed high recognition validities of both amateurs (0.73) and laypeople (0.67). In a second study the recognition heuristic correctly predicted 90% of actual individual choices. Overall, the recognition heuristic may be effectively generalized to dynamic environments. Copyright © 2006 John Wiley & Sons, Ltd.     

Negations in syllogistic reasoning: Evidence for a heuristic–analytic conflict

Research indicates that cognitive age differences can be influenced by metacognitive factors. This research has generally focused on simple memory tasks. Age differences in working memory (WM) performance are pronounced, but are typically attributed to basic cognitive deficits rather than metacognitive factors. However, WM performance can be influenced by strategic behaviour that might be driven by metacognitive monitoring. In the current project, we attempted to connect these lines of research by examining age differences in metacognitive WM monitoring and strategies. In Experiment 1, younger and older adult participants completed a computerized operation span task in conditions that either required or did not require monitoring reports. Participants in the monitoring condition predicted and postdicted global performance for each block and rated their responses following each trial within a block. In Experiment 2, participants also reported their trial-level strategic approach. In contrast to the age equivalence typically found for simple memory monitoring, results demonstrated age differences in WM monitoring accuracy. Overall age differences in strategy use were not found, but using effective strategies benefited older adults' performance more than younger adults'. Furthermore, age-related differences in the WM task appear to be mediated by the accuracy of performance monitoring.     

Empirical tests of a fast-and-frugal heuristic: Not everyone “takes-the-best”

The fast-and-frugal heuristics approach to decision making under uncertainty advocated by Gigerenzer and colleagues (e.g., Gigerenzer & Goldstein, 1996) has achieved great popularity despite a relative lack of empirical validation. We report two experiments that examine the use of one particular heuristic—“take-the-best” (TTB). In both experiments the majority of participants adopted frugal strategies, but only one-third (33%) behaved in a manner completely consistent with TTB’s search, stopping and decision rules. Furthermore, a significant proportion of participants in both experiments adopted a non-frugal strategy in which they accumulated more information than was predicted by TTB’s stopping rule. The results provide an insight into the conditions under which different heuristics are used, and question the predictive power of the fast-and-frugal approach.     

Fairness heuristic theory is an empirical framework: a reply to Arnadóttir

In this article on fairness heuristic theory, we point out some important flaws in Arnadóttir's (2002) claim that fairness heuristic theory is "not empirical," by which Arnadóttir meant that theory's predictions are knowable a priori, and are not contingent upon circumstances. To this end, we demonstrate that empirically testing effects predicted by fairness heuristic theory was and is important because this showed that the theory's propositions are not necessarily knowable a priori and are contingent upon circumstances. This implies that, according to Arnadóttir's definition, fairness heuristic theory clearly is an empirical framework. It would have been helpful if Arnadóttir had studied the fairness literature more thoroughly (as this would have easily revealed fairness heuristic theory to be not knowable a priori and to be contingent upon circumstances) and also if she had pointed out which of our studies fail to follow her line of reasoning. Our reply was written not as an attempt to defend fairness heuristic theory as we applaud, indeed are honored by, attempts to scrutinize our work in progress. Our only aim here was to point at some important flaws in the Arnadóttir article, because we think these will hamper rather than advance the science of psychology of justice.     

Inducing an availability heuristic on the Wason selection task overrides the matching bias

A long history of examining reasoning using the Wason selection task has revealed that many respondents are biased towards choices that match the items expressed in the rule. One reason for this particular heuristic may be that no better information is immediately available, and thus matching items win over the competing choice simply via recognition. In two experiments, we sought to examine whether a stronger memory trace could override the matching bias. We created rules from common, easily recognisable nursery rhymes and varied the degree to which the presented rules matched the commonly known rhymes. The components of the remembered rhyme had a strong influence on the participants’ selections, suggesting that a strong memory trace can override the usual matching bias. We provide an interpretation of these results in light of answer fluency, mental models, and probabilistic estimates.     

The effect of feedback on children’s metacognitive judgments: a heuristic account

In three experiments, we investigated whether the feedback effect on the accuracy of children’s metacognitive judgments results from an improvement in monitoring processes or the use of the Anchoring-and-Adjustment heuristic. Experiment 1 revealed that adding feedback increased the accuracy of young children’s (aged 4, 6, and 8 years) memory prediction. In Experiment 2, the influence of an external anchor on children’s metacognitive judgment was established. Finally, in Experiment 3, two memory tasks that differed in terms of difficulty were administered. Participants were randomly assigned to an anchoring (high/low/no anchor) and a feedback (feedback/no feedback) condition. Results demonstrated that children in the feedback condition adjusted their predictions toward the feedback, regardless of the task’s difficulty. These findings are consistent with the hypothesis that external information provided by feedback is used as an anchor for judgment. This interpretation is strengthened by the correlation found between the two scores computed to assess participants’ susceptibility to anchoring and feedback effects, which indicates that children who are more sensitive to the anchoring effect are also more sensitive to the feedback effect.     

Does interaction matter? Testing whether a confidence heuristic can replace interaction in collective decision-making

In a range of contexts, individuals arrive at collective decisions by sharing confidence in their judgements. This tendency to evaluate the reliability of information by the confidence with which it is expressed has been termed the ‘confidence heuristic’. We tested two ways of implementing the confidence heuristic in the context of a collective perceptual decision-making task: either directly, by opting for the judgement made with higher confidence, or indirectly, by opting for the faster judgement, exploiting an inverse correlation between confidence and reaction time. We found that the success of these heuristics depends on how similar individuals are in terms of the reliability of their judgements and, more importantly, that for dissimilar individuals such heuristics are dramatically inferior to interaction. Interaction allows individuals to alleviate, but not fully resolve, differences in the reliability of their judgements. We discuss the implications of these findings for models of confidence and collective decision-making.     

Use of the recognition heuristic depends on the domain’s recognition validity,not on the recognition validity of selected sets of objects

According to the recognition-heuristic theory, decision makers solve paired comparisons in which one object is recognized and the other not by recognition alone, inferring that recognized objects have higher criterion values than unrecognized ones. However, success—and thus usefulness—of this heuristic depends on the validity of recognition as a cue, and adaptive decision making, in turn, requires that decision makers are sensitive to it. To this end, decision makers could base their evaluation of the recognition validity either on the selected set of objects (the set’s recognition validity), or on the underlying domain from which the objects were drawn (the domain’s recognition validity). In two experiments, we manipulated the recognition validity both in the selected set of objects and between domains from which the sets were drawn. The results clearly show that use of the recognition heuristic depends on the domain’s recognition validity, not on the set’s recognition validity. In other words, participants treat all sets as roughly representative of the underlying domain and adjust their decision strategy adaptively (only) with respect to the more general environment rather than the specific items they are faced with.     

In conflict with ourselves? An investigation of heuristic and analytic processes in decision making

Many theorists propose two types of processing: heuristic and analytic. In conflict tasks, in which these processing types lead to opposing responses, giving the analytic response may require bothdetection andresolution of the conflict. The ratio bias task, in which people tend to treat larger numbered ratios (e.g., 20/100) as indicating a higher likelihood of winning than do equivalent smaller numbered ratios (e.g., 2/10), is considered to induce such a conflict. Experiment 1 showed response time differences associated with conflict detection, resolution, and the amount of conflict induced. The conflict detection and resolution effects were replicated in Experiment 2 and were not affected by decreasing the influence of the heuristic response or decreasing the capacity to make the analytic response. The results are consistent with dual-process accounts, but a single-process account in which quantitative, rather than qualitative, differences in processing are assumed fares equally well in explaining the data.     

Automatic–heuristic and executive–analytic processing during reasoning: Chronometric and dual-task considerations

Human reasoning has been shown to overly rely on intuitive, heuristic processing instead of a more demanding analytic inference process. Four experiments tested the central claim of current dual-process theories that analytic operations involve time-consuming executive processing whereas the heuristic system would operate automatically. Participants solved conjunction fallacy problems and indicative and deontic selection tasks. Experiment 1 established that making correct analytic inferences demanded more processing time than did making heuristic inferences. Experiment 2 showed that burdening the executive resources with an attention-demanding secondary task decreased correct, analytic responding and boosted the rate of conjunction fallacies and indicative matching card selections. Results were replicated in Experiments 3 and 4 with a different secondary-task procedure. Involvement of executive resources for the deontic selection task was less clear. Findings validate basic processing assumptions of the dual-process framework and complete the correlational research programme of K. E. Stanovich and R. F. West (2000).