Question order bias in retrospective evaluations of item and associative recognition

Previous research has demonstrated a retrospective memory bias in metacognitive judgments regarding performance on general knowledge questions: Test-takers rate their own performance more optimistically when tests begin with easy questions than when tests begin with hard questions. An anchoring heuristic has been proposed to explain this finding, in which experience with the early questions constrains global performance evaluations of the test. In the current study we report on two experiments using tasks of item recognition and associative recognition to investigate the generality of question order bias. As predicted by an anchoring explanation, participants’ estimates of performance were higher for item recognition tests beginning with easy items. However, the effect was reversed in the associative recognition task: Participants’ estimates of performance were higher for tests beginning with hard items. Specific recollections, if present, may have a greater impact on test performance perception than more general global impressions.     

Effects of orienting task,spacing of repetitions,and list context on judgments of frequency

Subjects were given an unexpected frequency judgment test following a list of words in which items were presented either two, three, and five times or three, five, and seven times, with a spacing of 0, 2, 16, or 32 items between repetitions. During list presentation, they either rated the imagery value of each word or made continuous frequency estimates. Postlist frequency judgments of words presented three and five times were higher for the list containing words of Frequency 7, and judgments were also higher following the imagery rating task. Continuous judgments were unaffected by the list context and showed different effects of spacing than postlist judgments. The results provide support for the operation of response bias factors in the frequency judgment task and are relevant to theoretical interpretations of the spacing effect.     

Actor-observer differences in realism in confidence and frequency judgments

Taking a social psychological approach to metacognitive judgments, this study analyzed the difference in realism (validity) in confidence and frequency judgments (i.e., estimates of overall accuracy) between one's own and another person's answers to general knowledge questions. Experiment 1 showed that when judging their own answers, compared with another's answers, the participants exhibited higher overconfidence, better ability to discriminate correct from incorrect answers, lower accuracy, and lower confidence. However, the overconfidence effect could be attributable to the lowest level of confidence. Furthermore, when heeding additional information about another's answers the participants showed higher confidence and better discrimination ability. The overconfidence effect of Experiment 1 was not found in Experiment 2. However, the results of Experiment 2 were consistent with Experiment 1 in terms of discrimination ability, confidence, and accuracy. Finally, in both experiments the participants gave lower frequency judgments of their own overall accuracy compared with their frequency judgments of another person's overall accuracy.     

Instability of individual differences in the association between confidence judgments and memory performance

There are large individual differences in the degree of association between the accuracy of memories and subjective confidence in those memories. Are these differences stable within the same test, and between alternate forms of a test? In Experiment 1, college students were tested on 3 recognition memory tasks, then retested 2 weeks later on alternate forms of the same tasks. The relationship between confidence judgments and recognition performance displayed low split-half stability and low alternate-forms stability. A second experiment with elderly adults replicated these findings. In a third experiment, college students recalled answers to general knowledge questions and rated confidence in the correctness of each answer. Individual differences in the association between confidence and recall performance were not stable across the odd- and even-numbered items on the test. These data indicate the need for the development of procedures that will produce stable estimates of individuals’ metacognitive accuracy.     

Realism of confidence in sensory discrimination

Is there a common and general basis for confidence in human judgment? Recently, we found that the properties of confidence judgments in the sensory domain mirror those previously established in the cognitive domain; notably, we found underconfidence on easy sensory judgments and overconfidence on hard sensory judgments. In contrast, data from the Uppsala laboratory in Sweden suggest that sensory judgments are unique; they found a pervasive underconfidence bias, with overconfidence being evident only on very hard sensory judgments. Olsson and Winman (1996) attempted to resolve the debate on the basis of methodological issues related to features of the stimulus display in a visual discrimination task. A reanalysis of the data reported in Baranski and Petrusic (1994), together with the findings of a new experiment that controlled stimulus display characteristics, supports the position that the difference between the Canadian and the Swedish data is real and, thus, may reflect cross-national differences in confidence in sensory discrimination.     

The Effects of Subject-Defined Categories on Judgmental Accuracy in Confidence Assessment Tasks

The accuracy of confidence judgments can be determined using measures of discrimination and calibration. The present paper utilizes a new assessment methodology that decomposes the confidence assessment task, allowing us to investigate discrimination and calibration skills in greater depth than has been done in previous studies. Researchers investigating the goodness of confidence judgments have typically grouped forecasters' assessments into experimenter-defined categories, generally in equal widths of .10. In the present research, subjects created their own categories and later assigned confidence judgments to the categories, separating the tasks of discriminating categories (discrimination) and assigning numbers to categories (calibration). Further, the typical assessment procedure assumes that subjects are able to discriminate equally across the confidence scale. Since subjects in the present study defined their own assessment categories, they could locate those categories at any point on the scale. A final issue of interest was whether subjects were able to determine accurately the number of categories into which they could discriminate. Sixty subjects performed 1 of 2 tasks, general knowledge or forecasting, in both relatively easy and relatively hard conditions. Results showed a trade-off in performance: Calibration generally became worse as the number of categories increased, while discrimination generally improved. Overall accuracy was not affected by the number of categories used. Further, subjects partitioned categories more at the high end of the scale. Finally, measures showed that subjects were not accurate in their beliefs about their own discrimination ability.     

Confidence-judgment accuracy as a predictor of test performance

As part of the regular classroom testing procedure, undergraduate students in an introductory psychology course were asked to provide confidence judgments along with their answers to multiple-choice test items. The two objectives of the study were to determine the extent of students' confidence-judgment accuracy and the degree of relationship of this memory-monitoring ability to overall test performance. The results showed that even the students having the poorest test performance showed some confidence-judgment accuracy, and, more importantly, there was a strong positive relationship between confidence-judgment accuracy and test performance, r(43) = .49. Thus, students who know more also are better able to distinguish between known and unknown information.     

Metacognitive benefits of taking a test for children and young adolescents

The main purposes of this study were (a) to isolate monitoring of test performance from other forms of monitoring and (b) to determine the effect of taking a test on expectations about future performance. Children in grades 1-2, 4-5, and 7-8 were administered a vocabulary test. They either predicted their performance on tests like the one that was administered before taking the test, predicted after taking the test, or made postdictions about performance on the present test. There was unambiguous improvement in the accuracy of after-test predictions and postdictions compared to before-test predictions at grades 7-8 only. Although all age groups discriminated hard from easy items as they were doing them, such discrimination increased with age. In general, there were few sex differences, although whenever statistically significant sex differences in confidence were detected, boys tended to be more confident than girls. These results are consistent with claims that developmental changes in self-regulation could be tied to developmental changes in monitoring of performance and making predictions about future performance based on past performance.     

Effects of similarity and repetition on memory: registration without learning?

We investigated judgments of the frequency of test items (Y) that were highly similar to studied items (X) to test a prediction made by several memory models: that the judged frequency of Y should be proportional to the judged frequency of X. Whether stimuli were pictures or words, judged frequency of Y was bimodally distributed with 1 mode at zero, suggesting that frequency judgments involve a 2-stage process in which a zero judgment is made if there is a mismatch between retrieved information and the test item. Nonzero judgements, taken by themselves, were consistent with the prediction of proportionality. In 2 experiments, the percentage of zero judgments made to Y increased with repetition of X, but in 2 others the percentage did not change beyond frequency = 1. The percentage of "new" judgments in recognition memory followed this same pattern. Because the judged frequency of X increased even as X-Y discrimination showed no improvement, we characterize the result as "registration without learning."     

Availability of Information and the Aggregation of Confidence in Prior Decisions

Recent research on calibration has shown that judgments about aggregate performance are consistently lower in magnitude than confidence-judgments about single items (the “aggregation effect”). Three explanations of this effect have been proposed: Probabilistic Mental Models theory, the regression-to-the-mean hypothesis, and the dual-source hypothesis. In two studies, we tested predictions based on these explanations about the influence of availability of information on the aggregation effect. Study 1 showed that neither reducing the item set size for aggregate-item judgments nor delaying the single-item judgments eliminated the effect. Study 2 showed a persistent aggregation effect for different kinds of item lists and reminders. Further comparisons showed that discrimination (as distinguished from overconfidence) was (1) consistently better for single-item judgments than for aggregate-item judgments, and (2) improved when there is a delay between the choosing of an answer and the rating of confidence about that answer. The three proposed explanations of the aggregation effect are compared in light of these findings.     

The easy to hard effect and perceptual learning in flavor aversion conditioning

In experiments on the easy to hard effect, pretraining on an easy discrimination results in better performance on a harder version of the discrimination than pretraining on the hard discrimination itself. In addition, some theories posit that unreinforccd preexposure to the easy discrimination should be as effective as differentially reinforced easy pretraining in producing the easy to hard effect. Two experiments on flavor aversion conditioning in rats demonstrated the basic easy to hard effect. However, in neither experiment was easy preexposure more effective than hard preexposure in enhancing learning of the hard discrimination. Indeed, in Experiment 2, rats preexposed to an easy discrimination learned the hard discrimination significantly more slowly than those preexposed to the hard discrimination itself.     

SCALING THE EPISODIC FAMILIARITIES OF PICTURES AND WORDS

Abstract— Most current memory theories assume that judgments of past occurrence are based on a unidimensional familiarity signal In a test of this hypothesis, subjects studied mixed lists of pictures and words that occurred up to three times each They then were given two tests a forced-choice frequency discrimination test including all pairs of conditions (e g., picture seen twice vs word never seen) and a numerical frequency judgment test on individual items Forced-choice proportions for all pairings (picture-picture, word-word, and picture-word) were well fit by a one-dimensional scaling solution, suggesting a common basis of recognition and frequency judgments for both pictures and words Both forced-choice and numerical judgment data indicated that familiarities of pictures started lower than those of words but increased more rapidly with repetition Results are discussed in connection with the distinction between familiarity and recall, and the possible role of rescaling in the mirror effect.     

Confidence depends on level of aggregation

The credible intervals that people set around their point estimates are typically too narrow (cf. Lichtenstein, Fischhoff, & Phillips, 1982). That is, a set of many such intervals does not contain the actual values of the criterion variables as often as it should given the probability assigned to this event for each estimate. The typical interpretation of such data is that people are overconfident about the accuracy of their judgments. This paper presents data from two studies showing the typical levels of overconfidence for individual estimates of unknown quantities. However, data from the same subjects on a different measure of confidence for the same items, their own global assessment for the set of multiple estimates as a whole, showed significantly lower levels of confidence and overconfidence than their average individual assessment for items in the set. It is argued that the event and global assessments of judgment quality are fundamentally different and are affected by unique psychological processes. Finally, we discuss the implications of a difference between confidence in single and multiple estimates for confidence research and theory.     

Morality Information,Performance Information,and the Distinction Between Trust and Confidence1

A series of 3 studies explored the efficacy of a dual‐mode model of cooperation based on trust and confidence, the TCC model. The TCC model is founded on the distinction between morality‐relevant information and performance‐relevant information, with the former dominating—controlling the interpretation of—the latter in a variety of judgment contexts. According to the TCC model, trust is based on judgments of value similarity derived from morality‐relevant information, and confidence is based on judgments of past performance derived from performance‐relevant information. Results from all 3 studies supported the relations depicted in the TCC model, showing that judgments of trust dominated judgments of past performance and of confidence.     

Age differences in feeling-of-knowing and confidence judgements as a function of knowledge domain

For two semantic knowledge domains, general and computer-related, feeling-of-knowing (FOK) and confidence level (CL) ratings and their relative accuracy were assessed in young, middle-aged, and older adults, after test difficulty was equated across age groups. Global memory self-efficacy beliefs were also assessed for each domain. As expected, greater age was associated with poorer memory self-efficacy beliefs only in the computer domain. The oldest two groups were found to be more underconfident than young adults when rating their FOK but not their CL, for computer items but not for general items. Statistical control of age differences in memory self-efficacy beliefs in the relevant domain greatly reduced this age effect on computer-related FOK ratings. This finding suggests that absolute FOK judgments are more closely related to memory self-efficacy beliefs than are CL judgments. Gamma correlations between judgments and recognition performance revealed that all age groups were equally accurate in FOK and in CL judgments, in both domains.     

Retrieval of item frequency information in a continuous memory task

The accuracy and response latency of absolute frequency judgments were measured as a function of test lag (the number of intervening items between presentations of a test item) in a continuous memory task. Frequency was varied from one to three presentations in Experiments 1 and 2 and from one to five presentations in Experiment 3. The proportion of correct responses decreased as frequency increased, and correct mean response time tended to increase with frequency. Both accuracy and correct mean response time were found to be largely a function of the most recent test lag. The lag-latency functions were best described by piecewise linear functions, with the breakpoint occurring between lags 1 and 2. Continuous frequency estimation was also shown to improve with extended practice. The implications of the results are discussed with reference to trace strength, numerical-inference, and multiple-trace theories of frequency discrimination.     

Judgments relative to patterns: how temporal sequence patterns affect judgments and memory

Six experiments studied relative frequency judgment and recall of sequentially presented items drawn from 2 distinct categories (i.e., city and animal). The experiments show that judged frequencies of categories of sequentially encountered stimuli are affected by certain properties of the sequence configuration. We found (a) a first-run effect whereby people overestimated the frequency of a given category when that category was the first repeated category to occur in the sequence and (b) a dissociation between judgments and recall; respondents may judge 1 event more likely than the other and yet recall more instances of the latter. Specifically, the distribution of recalled items does not correspond to the frequency estimates for the event categories, indicating that participants do not make frequency judgments by sampling their memory for individual items as implied by other accounts such as the availability heuristic (Tversky & Kahneman, 1973) and the availability process model (Hastie & Park, 1986). We interpret these findings as reflecting the operation of a judgment heuristic sensitive to sequential patterns and offer an account for the relationship between memory and judged frequencies of sequentially encountered stimuli.     

Spacing one's study: evidence for a metacognitive control strategy

This article investigated individual control of spacing strategies during study. Three predictions were outlined: The spacing hypothesis suggests that people choose to space their study to improve long-term learning via the spacing effect. The massing hypothesis suggests that people choose to mass their study because of illusions of confidence during study. The metacognitive hypothesis suggests that people control their spacing schedules as a function of their metacognitive judgments of specific to-be-learned items. To test these hypotheses, the authors asked participants to study and make judgments of learning for cue-target pairs. Then, participants were given three choices; they could study the pair again immediately (massed), study the pair again after the entire list had been presented (spaced), or choose not to restudy (done). Results supported a metacognitively controlled spacing strategy-people spaced items that were judged to be relatively easy but massed items that were judged as relatively difficult.     

The detrimental effects of power on confidence, advice taking, and accuracy

Incorporating input from others can enhance decision quality, yet often people do not effectively utilize advice. We propose that greater power increases the propensity to discount advice, and that a key mechanism explaining this effect is elevated confidence in one’s judgment. We investigate the relationships across four studies: a field survey where working professionals rated their own power and confidence and were rated by coworkers on their level of advice taking; an advice taking task where power and confidence were self-reported; and two advice taking experiments where power was manipulated. Results consistently showed a negative relationship between power and advice taking, and evidence of mediation through confidence. The fourth study also revealed that higher power participants were less accurate in their final judgments. Power can thus exacerbate the tendency for people to overweight their own initial judgment, such that the most powerful decision makers can also be the least accurate.     

Metacognitive monitoring during criterion learning: When and why are judgments accurate?

Research on metacognitive judgment accuracy during retrieval practice has increased in recent years. However, prior work had not systematically evaluated item-level judgment accuracy and the underlying bases of judgment accuracy in a criterion-learning paradigm (in which items are practiced until correctly recalled during encoding). Understanding these relationships during criterion learning has important theoretical implications for self-regulated learning frameworks, and also has applied implications for student learning: If the factors that influence metacognitive judgments are not predictive of subsequent test performance, students may make poor decisions during self-regulated learning. In the present experiments, participants engaged in test–restudy practice until items were recalled correctly. Once a given item reached criterion, participants made an immediate or delayed judgment of learning (JOL) for the item. A final cued-recall test occurred 30 min later. We examined judgment accuracy (the relationship between JOLs and test performance) and the underlying bases of judgment accuracy by evaluating cue utilization (the relationship between cues and JOLs) and cue diagnosticity (the relationship between cues and test performance). Immediate JOLs were only modestly related to subsequent test performance, and further analyses revealed that the cues related to JOLs were only weakly predictive of test accuracy. However, delaying JOLs improved both the accuracy of the JOLs and the diagnosticity of the cues that influenced judgments.