Metacognitive monitoring during criterion learning: When and why are judgments accurate?

Research on metacognitive judgment accuracy during retrieval practice has increased in recent years. However, prior work had not systematically evaluated item-level judgment accuracy and the underlying bases of judgment accuracy in a criterion-learning paradigm (in which items are practiced until correctly recalled during encoding). Understanding these relationships during criterion learning has important theoretical implications for self-regulated learning frameworks, and also has applied implications for student learning: If the factors that influence metacognitive judgments are not predictive of subsequent test performance, students may make poor decisions during self-regulated learning. In the present experiments, participants engaged in test–restudy practice until items were recalled correctly. Once a given item reached criterion, participants made an immediate or delayed judgment of learning (JOL) for the item. A final cued-recall test occurred 30 min later. We examined judgment accuracy (the relationship between JOLs and test performance) and the underlying bases of judgment accuracy by evaluating cue utilization (the relationship between cues and JOLs) and cue diagnosticity (the relationship between cues and test performance). Immediate JOLs were only modestly related to subsequent test performance, and further analyses revealed that the cues related to JOLs were only weakly predictive of test accuracy. However, delaying JOLs improved both the accuracy of the JOLs and the diagnosticity of the cues that influenced judgments.     

Evidence that judgments of learning are causally related to study choice

Three experiments investigated whether study choice was directly related to judgments of learning (JOLs) by examining people’s choices in cases in which JOLs were dissociated from recall. In Experiment 1, items were given either three repetitions or one repetition on Trial 1. Items given three repetitions received one on Trial 2, and those given one repetition received three on Trial 2—equating performance at the end of Trial 2, but yielding different immediate Trial 2 JOLs. Study choice followed the “illusory” JOLs. A delayed JOL condition in Experiment 2 did not show this JOL bias and neither did study choice. Finally, using a paradigm (Koriat & Bjork, 2005) in which similar JOLs are given to forward and backward associative pairs, despite much worse performance on the backward pairs, study choice again followed the mistaken JOLs. We concluded that JOLs—what people believe they know—directly influence people’s study choices.     

People use the memory for past-test heuristic as an explicit cue for judgments of learning

When people estimate their memory for to-be-learned material over multiple study–test trials, they tend to base their judgments of learning (JOLs) on their test performance for those materials on the previous trial. Their use of this information—known as the memory for past-test (MPT) heuristic—is believed to be responsible for improvements in the relative accuracy (resolution) of people’s JOLs across learning trials. Although participants seem to use past-test information as a major basis for their JOLs, little is known about how learners translate this information into a judgment of learning. Toward this end, in two experiments, we examined whether participants factored past-test performance into their JOLs in either an explicit, theory-based way or an implicit way. To do so, we had one group of participants (learners) study paired associates, make JOLs, and take a test on two study–test trials. Other participants (observers) viewed learners’ protocols and made JOLs for the learners. Presumably, observers could only use theory-based information to make JOLs for the learners, which allowed us to estimate the contribution of explicit and implicit information to learners’ JOLs. Our analyses suggest that all participants factored simple past-test performance into their JOLs in an explicit, theory-based way but that this information made limited contributions to improvements in relative accuracy across trials. In contrast, learners also used other privileged, implicit information about their learning to inform their judgments (that observers had no access to) that allowed them to achieve further improvements in relative accuracy across trials.     

The effect of divided attention on global judgment of learning accuracy

This study examined the effect of divided attention (DA) on global judgment of learning (JOL) accuracy in a multitrial list learning paradigm. A word monitoring task was used to divide attention. Participants were assigned to an attention condition (DA at encoding, DA at judgment, DA at retrieval, or focused attention) and completed 4 learning trials, each comprising a study, judgment, and recall phase. Participants showed greater overconfidence in the DA at encoding (Trial 2) and DA at retrieval (Trials 1 and 2) conditions than in the focused attention condition. DA atjudgment did not affect JOL accuracy, and there was no effect of DA in Trials 3 and 4 on JOL accuracy across all attention conditions. Results indicate that participants consider conditions of encoding and retrieval but do not engage in recall when forming global JOLs. These findings suggest that people rely on extrinsic cues when making repeated, global metamemoryjudgments.     

Empirical analysis of the intrinsic-extrinsic distinction of judgments of learning (JOLs): effects of relatedness and serial position on JOLs

A. Koriat's (1997) cue-utilization framework provided a significant advance in understanding how people make judgments of learning (JOLs). A major distinction is made between intrinsic and extrinsic cues. JOLs are predicted to be sensitive to intrinsic cues (e.g., item relatedness) and less sensitive to extrinsic cues (e.g., serial position) because JOLs are comparative across items in a list. The authors evaluated predictions by having people make JOLs after studying either related (poker-flush) or unrelated (dog-spoon) items. Although some outcomes confirmed these predictions, others could not be readily explained by the framework. Namely, relatedness influenced JOLs even when manipulated between participants, primacy effects were evident on JOLs, and the order in which blocks of items were presented (either all related items first or all unrelated items first) influenced JOLs. The authors discuss the framework in relation to these and other outcomes.     

Multiple study trials and judgments of learning

We compared judgments of learning (JOLs) that were made either (a) after 1 study trial, (b) 2 study trials, or (c) in-between the 1st and 2nd study trials. In regard to the absolute accuracy of JOLs at predicting subsequent recall, we replicated previous findings of an underconfidence-with-practice effect for immediate JOLs and report for the first time a new finding of an underconfidence-with-practice effect for delayed JOLs (i.e., delayed JOLs after one trial overestimated the likelihood of subsequent recall, whereas delayed JOLs after two trials underestimated that likelihood). Also, although delayed JOLs always had a greater relative accuracy than did immediate JOLs, the relative accuracy of immediate and delayed JOLs was approximately the same after 1 versus 2 study trials. These results demonstrate that additional study trials affect the absolute accuracy of all JOLs but not the relative accuracy of any JOLs. Thus an increase in the number of study trials produced an increasing bias to be underconfident about the subsequent likelihood of recall but did not affect people's ordering of which items had been more (versus less) well-learned.     

加工流畅性和提取流畅性与学习不良儿童学习判断的关系

流畅性是个体做学习判断（Judgements of Learning, JOLs）时的重要线索,主要包括加工流畅性和提取流畅性。实验采用联想字对作为实验材料,把做JOLs的时间分为即刻JOLs (Immediate:I)（学习完一个项目立即判断）,短时延迟JOLs（Delay condition: D1）和长时延迟JOLs（D2）,考察不同JOLs时间条件下,流畅性线索对学习不良儿童和一般儿童的JOLs的不同预测模式。被试为普通小学五年级学习不良和一般儿童各20名,结果发现,一般儿童在I条件下主要利用加工流畅性,在D条件下主要利用提取流畅性做JOLs,而学习不良儿童无论在I还是D条件下均主要利用提取流畅性线索做JOLs。加工流畅性在I条件下效度较高,提取流畅性在D条件下效度较高     

How many dimensions underlie judgments of learning and recall? Evidence from state-trace methodology

The authors used state-trace methodology to investigate whether a single dimension (e.g., strength) is sufficient to account for recall and judgments of learning (JOLs) or whether multiple dimensions (e.g., intrinsic and extrinsic factors) are needed. The authors separately manipulated the independent variables of intrinsic and extrinsic cues, determining their state traces for recall and JOLs. In contrast to the supposition that intrinsic cues have similar effects on both recall and JOLs whereas extrinsic cues affect JOLs less strongly than recall (i.e., 2 dimensions underlying recall and JOLs), the authors found repeated support for the sufficiency of a single dimension for both recall and JOLs (not only immediate JOLs but also delayed JOLs) across a variety of intrinsic and extrinsic cues.     

The effects of categorical relatedness on judgements of learning (JOLs)

On-line monitoring during study can be influenced by the relatedness shared between the cue and target of a paired associate. We examined the effects on people's judgements of learning (JOLs) of a different kind of relatedness, which occurs in a list organised into sets of categorically related words and unrelated words. In two experiments, participants studied a list of words organised into a series of sets of four categorically related words or four unrelated words. In Experiment 1, JOLs were made immediately after each word had been studied, and JOL magnitude was greater for related than unrelated words. In Experiment 2, JOLs were delayed after study and, as expected, they were substantially greater for related sets of words. Serial position effects (an increase in JOL magnitude across the words of a related set) were evident with immediate JOLs but not with delayed JOLs. The relatedness effect was not present early in the list for immediate JOLs but was present throughout the list for delayed JOLs. We conclude by discussing some preliminary explanations for these new phenomena.     

The effects of categorical relatedness on judgements of learning (JOLs)

On-line monitoring during study can be influenced by the relatedness shared between the cue and target of a paired associate. We examined the effects on people's judgements of learning (JOLs) of a different kind of relatedness, which occurs in a list organised into sets of categorically related words and unrelated words. In two experiments, participants studied a list of words organised into a series of sets of four categorically related words or four unrelated words. In Experiment 1, JOLs were made immediately after each word had been studied, and JOL magnitude was greater for related than unrelated words. In Experiment 2, JOLs were delayed after study and, as expected, they were substantially greater for related sets of words. Serial position effects (an increase in JOL magnitude across the words of a related set) were evident with immediate JOLs but not with delayed JOLs. The relatedness effect was not present early in the list for immediate JOLs but was present throughout the list for delayed JOLs. We conclude by discussing some preliminary explanations for these new phenomena.     

The concreteness effect on judgments of learning: Evaluating the contributions of fluency and beliefs

Researchers have often determined how cues influence judgments of learning (JOLs; e.g., concrete words are assigned higher JOLs than are abstract words), and recently there has been an emphasis in understanding why cues influence JOLs (i.e., the mechanisms that underlie cue effects on JOLs). The analytic-processing (AP) theory posits that JOLs are constructed in accordance with participants’ beliefs of how a cue will influence memory. Even so, some evidence suggests that fluency is also important to cue effects on JOLs. In the present experiments, we investigated the contributions of participants’ beliefs and processing fluency to the concreteness effect on JOLs. To evaluate beliefs, participants estimated memory performance in a hypothetical experiment (Experiment 1), and studied concrete and abstract words and made a pre-study JOL for each (Experiments 2 and 3). Participants’ predictions demonstrated the belief that concrete words are more likely to be remembered than are abstract words, consistent with the AP theory. To evaluate fluency, response latencies were measured during lexical decision (Experiment 4), self-paced study (Experiment 5), and mental imagery (Experiment 7). Number of trials to acquisition was also evaluated (Experiment 6). Fluency did not differ between concrete and abstract words in Experiments 5 and 6, and it did not mediate the concreteness effect on JOLs in Experiments 4 and 7. Taken together, these results demonstrate that beliefs are a primary mechanism driving the concreteness effect on JOLs.     

The effect of identical word pairs on people's metamemory judgments: What are the contributions of processing fluency and beliefs about memory?

Judgments of learning (JOLs) are higher for identical pairs (dog–dog) than for related pairs (dog–cat). This identical effect may be mediated (a) by processing fluency (i.e., identical pairs are processed faster than related pairs) or (b) by a belief that identical pairs are better remembered or (c) by both factors. In the present work, we assessed the contribution of both factors. We evaluated whether a measure of processing fluency (i.e., self-paced study) mediated the relationship between pair type and JOLs (Experiment 1) and attempted to disrupt processing fluency using an AlTeRnAtInG presentation format (Experiment 2). We also evaluated whether judgments made in the absence of processing fluency demonstrated the identical effect (Experiment 3), and, finally, we had participants read a vignette about an experiment that included both pair types and estimate which pairs would be best remembered (Experiment 4). Evidence from all experiments converged on the conclusion that people's beliefs about how variables affect memory—and not differential fluency—best explain the identical effect, although we cannot entirely rule out the possibility that fluency plays a small role. The outcomes were consistent with the analytic-processing theory of JOLs—namely, when instructed to make JOLs, people adopt an analytic problem-solving approach that involves identifying variation across pairs that plausibly relate to memory and then use this variation to make JOLs.     

Simultaneous utilization of multiple cues in judgments of learning

There is much evidence that metacognitive judgments, such as people’s predictions of their future memory performance (judgments of learning, JOLs), are inferences based on cues and heuristics. However, relatively little is known about whether and when people integrate multiple cues in one metacognitive judgment or focus on a single cue without integrating further information. The current set of experiments systematically addressed whether and to what degree people integrate multiple extrinsic and intrinsic cues in JOLs. Experiment 1 varied two cues: number of study presentations (1 vs. 2) and font size (18 point vs. 48 point). Results revealed that people integrated both cues in their JOLs. Experiment 2 demonstrated that the two word characteristics concreteness (abstract vs. concrete) and emotionality (neutral vs. emotional) were integrated in JOLs. Experiment 3 showed that people integrated all four cues in their JOLs when manipulated simultaneously. Finally, Experiment 4 confirmed integration of three cues that varied on a continuum rather than in two easily distinguishable levels. These results demonstrate that people have a remarkable capacity to integrate multiple cues in metacognitive judgments. In addition, our findings render an explanation of cue effects on JOLs in terms of demand characteristics implausible.     

Immediate judgments of learning are insensitive to implicit interference effects at retrieval

We conducted three experiments to determine whether metamemory predictions at encoding, immediate judgments of learning (IJOLs) are sensitive to implicit interference effects that will occur at retrieval. Implicit interference was manipulated by varying the association set size of the cue (Experiments 1 and 2) or the target (Experiment 3). The typical finding is that memory is worse for large-set-size cues and targets, but only when the target is studied alone and later prompted with a related cue (extralist). When the pairs are studied together (intralist), recall is the same regardless of set size; set size effects are eliminated. Metamemory predictions at retrieval, such as delayed JOLs (DJOLs) and feeling-of-knowing (FOK) judgments accurately reflect implicit interference effects (e.g., Eakin & Hertzog, 2006. In all three experiments, we found that DJOLs and FOKs accurately predicted set size effects on retrieval but that IJOLs did not. The findings provide further evidence that metamemory predictions are inferred from information other than direct access to the state of the memory trace, as well as indicate that inferences are based on different sources depending on when in the memory process predictions are made.     

The sensitivity of judgment-of-learning resolution to past test performance,new learning,and forgetting

When people judge their learning of items across study–test trials, their accuracy in discriminating between learned and unlearned items improves on the second trial. We examined the source of this improvement by estimating the contribution of three factors—memory for past test performance (MPT), new learning, and forgetting—to accuracy on trial 2. In Experiment 1, during an initial trial, participants studied paired associates, made a judgment of learning (JOL) for each one, and were tested. During the second trial, we manipulated two variables: when the JOL was made (either immediately before or after studying an item) and whether participants were told the outcome of the initial recall attempt on trial 1. In Experiment 2, the same procedure was used with a 1-week retention interval between study and test on trial 2. In both experiments, JOL resolution was higher on trial 2 than on trial 1. Fine-grained analyses of JOL magnitude and decomposition of resolution supported several conclusions. First, MPT contributed the most to boosts in JOL magnitude and improvements in resolution across trials. Second, JOLs and subsequent resolution were sensitive to new learning and forgetting, but only when participants’ judgments were made after study. Thus, JOLs appear to integrate information from multiple factors, and these factors jointly contribute to JOL resolution.     

When people’s judgments of learning (JOLs) are extremely accurate at predicting subsequent recall: the “Displaced-JOL effect”

Judgments of learning (JOL) made after a delay more accurately predict subsequent recall than JOLs made immediately after learning. One explanation is that delayed JOLs involve retrieving information about the target item from secondary memory, whereas immediate JOLs involve retrieval from primary memory. One view of working memory claims that information in primary memory is displaced to secondary memory when attention is shifted to a secondary task. Thus, immediate JOLs might be as accurate as delayed JOLs if an intervening task displaces the target item from primary memory, requiring retrieval from secondary memory, prior to making the JOL. In four experiments, participants saw related word-pairs and made JOLs predicting later recall of the item. In Experiment 1, delayed JOLs were more accurate than JOLs made shortly after learning, regardless of whether a secondary task intervened between learning and JOL. In Experiments 2–4, the secondary task demands increased and JOLs made shortly after learning with an intervening task were just as accurate as delayed JOLs, and both were more accurate than immediate JOLs with no intervening task (Experiment 4). These results are consistent with a retrieval-based account of JOLs, and demonstrate that the “delayed-JOL effect” can be obtained without a long delay.     

Association, inhibition, and object permanence in dogs' (Canis familiaris) spatial search

The relative role of associative processes and the use of explicit cues about object location in search behavior in dogs (Canis familiaris) was assessed by using a spatial binary discrimination reversal paradigm in which reversal conditions featured: (1) a previously rewarded location and a novel location, (2) a previously nonrewarded location and a novel location, or (3) a previously rewarded location and a previously nonrewarded location. Rule mediated learning predicts a similar performance in these different reversal conditions whereas associative learning predicts the worst performance in Condition 3. Evidence for an associative control of search emerged when no explicit cues about food location were provided (Experiment 1) but also when dogs witnessed the hiding of food in the reversal trials (Experiment 2) and when they did so in both the prereversal and the reversal trials (Experiment 3). Nevertheless, dogs performed better in the prereversal phase of Experiment 3 indicating that their search could be informed by the knowledge of the food location. Experiment 4 confirmed the results of Experiments 1 and 2, under a different arrangement of search locations. We conclude that knowledge about object location guides search behavior in dogs but it cannot override associative processes.     

Testing an associative account of semantic satiation

How is the meaning of a word retrieved without interference from recently viewed words? The ROUSE theory of priming assumes a discounting process to reduce source confusion between subsequently presented words. As applied to semantic satiation, this theory predicted a loss of association between the lexical item and meaning. Four experiments tested this explanation in a speeded category-matching task. All experiments used lists of 20 trials that presented a cue word for 1 s followed by a target word. Randomly mixed across the list, 10 trials used cues drawn from the same category whereas the other 10 trials used cues from 10 other categories. In Experiments 1a and 1b, the cues were repeated category labels (FRUIT–APPLE) and responses gradually slowed for the repeated category. In Experiment 2, the cues were nonrepeated exemplars (PEAR–APPLE) and responses remained faster for the repeated category. In Experiment 3, the cues were repeated exemplars in a word matching task (APPLE–APPLE) and responses again remained faster for the repeated category.     

Multiple bases for young and older adults' judgments of learning in multitrial learning

Prior work has suggested that participants use a memory-for-past-tests (MPT) heuristic for judgments of learning (JOLs) in a multitrial learning scenario. That is, when learning the same material in multiple sessions, previous memory performance can be used as a basis for later memory predictions. We explored this issue by evaluating the impact of healthy aging on the use of MPT across trials. Young adults and healthy older adults learned pairs of words, made JOLs, and received a memory test in three study-test trials on the same material. Results indicated that both young and older adults relied on MPT as a basis for JOLs and changes in MPT across trials were nominal. Further, only the most-recent past test influenced JOLs, whereas earlier tests were unrelated to later judgments. JOLs were also influenced by prior-trial JOLs and were related to subsequent memory performance on the same trial. We suggest that these data support both indirect- and direct-memory mechanisms as the bases for the MPT heuristic. Further, in a multitrial learning scenario, in which the same information was being learned, young and older adults used the same bases for their JOLs.     

Influence of cue word perceptual information on metamemory accuracy in judgement of learning

Previous studies have suggested that perceptual information regarding to-be-remembered words in the study phase affects the accuracy of judgement of learning (JOL). However, few have investigated whether the perceptual information in the JOL phase influences JOL accuracy. This study examined the influence of cue word perceptual information in the JOL phase on immediate and delayed JOL accuracy through changes in cue word font size. In Experiment 1, large-cue word pairs had significantly higher mean JOL magnitude than small-cue word pairs in immediate JOLs and higher relative accuracy than small-cue pairs in delayed JOLs, but font size had no influence on recall performance. Experiment 2 increased the JOL time, and mean JOL magnitude did not reliably differ for large-cue compared with small-cue pairs in immediate JOLs. However, the influence on relative accuracy still existed in delayed JOLs. Experiment 3 increased the familiarity of small-cue words in the delayed JOL phase by adding a lexical decision task. The results indicated that cue word font size no longer affected relative accuracy in delayed JOLs. The three experiments in our study indicated that the perceptual information regarding cue words in the JOL phase affects immediate and delayed JOLs in different ways.