The Importance of Item Selection in "Knew-It-All-Along" Studies of General Knowledge

This study was designed to investigate the effects of item sampling on hindsight bias in experiments using general knowledge material. The results show that the use of random versus traditional experimenter-selected item samples can have different effects on hindsight bias. In a within-subjects study almost twice as many items in a random sample were connected with a reversed effect rather than with a traditional hindsight bias. The same items that resulted in overconfidence in foresight lead to a higher degree of hindsight bias than others. The results suggest that earlier findings of unusually large hindsight effects with general knowledge tasks may be explained by the selection of items used. No hindsight effect was found on confidence scores in a within-subjects design, but was obtained in a between-subjects design. Results suggest that the use of a within-subjects design itself can moderate hindsight bias by familiarizing subjects with the task. The study shows the importance of two conditions for decreasing the hindsight bias: (1) The use of randomly sampled items, and (2) The use of a within-subjects procedure. When these conditions were met, the "knew-it-all-along effect" was completely eliminated.