Impression formation of tests: retrospective judgments of performance are higher when easier questions come first

Four experiments are reported on the importance of retrospective judgments of performance (postdictions) on tests. Participants answered general knowledge questions and estimated how many questions they answered correctly. They gave higher postdictions when easy questions preceded difficult questions. This was true when time to answer each question was equalized and constrained, when participants were instructed not to write answers, and when questions were presented in a multiple-choice format. Results are consistent with the notion that first impressions predominate in overall perception of test difficulty.     

Using verification feedback to correct errors made on a multiple-choice test

A key educational challenge is how to correct students’ errors and misconceptions so that they do not persist. Simply labelling an answer as correct or incorrect on a short-answer test (verification feedback) does not improve performance on later tests; error correction requires receiving answer feedback. We explored the generality of this conclusion and whether the effectiveness of verification feedback depends on the type of test with which it is paired. We argue that, unlike for short-answer tests, learning whether one's multiple-choice selection is correct or incorrect should help participants narrow down the possible answers and identify specific lures as false. To test this proposition we asked participants to answer a series of general knowledge multiple-choice questions. They received no feedback, answer feedback, or verification feedback, and then took a short-answer test immediately and two days later. Verification feedback was just as effective as answer feedback for maintaining correct answers. Importantly, verification feedback allowed learners to correct more of their errors than did no feedback, although it was not as effective as answer feedback. Overall, verification feedback conveyed information to the learner, which has both practical and theoretical implications.     

Regulation of memory accuracy with multiple answers: the plurality option

We report two experiments that investigated the regulation of memory accuracy with a new regulatory mechanism: the plurality option. This mechanism is closely related to the grain-size option but involves control over the number of alternatives contained in an answer rather than the quantitative boundaries of a single answer. Participants were presented with a slideshow depicting a robbery (Experiment 1) or a murder (Experiment 2), and their memory was tested with five-alternative multiple-choice questions. For each question, participants were asked to generate two answers: a single answer consisting of one alternative and a plural answer consisting of the single answer and two other alternatives. Each answer was rated for confidence (Experiment 1) or for the likelihood of being correct (Experiment 2), and one of the answers was selected for reporting. Results showed that participants used the plurality option to regulate accuracy, selecting single answers when their accuracy and confidence were high, but opting for plural answers when they were low. Although accuracy was higher for selected plural than for selected single answers, the opposite pattern was evident for confidence or likelihood ratings. This dissociation between confidence and accuracy for selected answers was the result of marked overconfidence in single answers coupled with underconfidence in plural answers. We hypothesize that these results can be attributed to overly dichotomous metacognitive beliefs about personal knowledge states that cause subjective confidence to be extreme.     

Using verification feedback to correct errors made on a multiple-choice test

A key educational challenge is how to correct students' errors and misconceptions so that they do not persist. Simply labelling an answer as correct or incorrect on a short-answer test (verification feedback) does not improve performance on later tests; error correction requires receiving answer feedback. We explored the generality of this conclusion and whether the effectiveness of verification feedback depends on the type of test with which it is paired. We argue that, unlike for short-answer tests, learning whether one's multiple-choice selection is correct or incorrect should help participants narrow down the possible answers and identify specific lures as false. To test this proposition we asked participants to answer a series of general knowledge multiple-choice questions. They received no feedback, answer feedback, or verification feedback, and then took a short-answer test immediately and two days later. Verification feedback was just as effective as answer feedback for maintaining correct answers. Importantly, verification feedback allowed learners to correct more of their errors than did no feedback, although it was not as effective as answer feedback. Overall, verification feedback conveyed information to the learner, which has both practical and theoretical implications.     

The Effect of Type of Feedback in Multiple-Choice Testing on Long-Term Retention

We investigated how different types of feedback in multiple-choice testing influence long-term retention. Participants completed an initial multiple-choice test on general-knowledge questions. Then, they were randomly placed into one of the following four conditions: feedback displaying the original question and four alternative options including the correct answer (Feedback 1); feedback displaying the original question and the correct answer (Feedback 2); feedback displaying only the correct answer (Feedback 3); and no feedback (Control). On the final test two days later, participants in the feedback conditions showed improved retention—even participants who received Feedback 3 outperformed the control group, regardless of whether feedback followed correct or incorrect responses. There was no difference between the feedback conditions when feedback followed correct responses, but Feedback 1 was superior to Feedback 3 when feedback followed incorrect responses. The function of feedback in multiple-choice testing is to maintain correct responses and to correct erroneous responses.     

Using the regulation of accuracy to study performance when the correct answer is not known

We examined memory performance in multiple‐choice questions when correct answers were not always present. How do participants answer when they are aware that the correct alternative may not be present? To answer this question we allowed participants to decide on the number of alternatives in their final answer (the plurality option), and whether they wanted to report or withhold their answer (report option). We also studied the memory benefits when both the plurality and the report options were available. In two experiments participants watched a crime and then answered questions with five alternatives. Half of the questions were presented with the correct alternative and half were not. Participants selected one alternative and rated confidence, then selected three alternatives and again rated confidence, and finally indicated whether they preferred the answer with one or with three alternatives (plurality option). Lastly, they decided whether to report or withhold the answer (report option). Results showed that participants’ confidence in their selections was higher, that they chose more single answers, and that they preferred to report more often when the correct alternative was presented. We also attempted to classify a posteriori questions as either presented with or without the correct alternative from participants’ selection. Classification was better than chance, and encouraging, but the forensic application of the classification technique is still limited since there was a large percentage of responses that were incorrectly classified. Our results also showed that the memory benefits of both plurality and report options overlap.     

Memory for memory

Participants read short passages and 1 day later they answered questions via telephone about the passages (text facts) and about the experimental session (event facts). They were telephoned again 6 weeks later and answered the same questions about text and event facts. They also answered new questions about whether they remembered the answers they had given in the initial telephone interview (recall for prior memory performance). Although participants accurately remembered the majority of past memory successes, they were poor at remembering past memory failures. After being provided with the correct answer and tested again, the participants' performance improved somewhat, especially for memory failures. This suggests that some errors in recalling past forgetting might have been due to correctly remembering the answer previously given, but failing to realize that it had been wrong. These findings have implications for a variety of situations in which people are queried about past memory performance.     

Performance on the Swedish Scholastic Aptitude Test: Effects of Self-Assessment and Gender

Scholarly knowledge is traditionally believed to exist if a person answers correctly when tested. A test-taker that makes a lucky guess is thereby implicitly assumed to know as much as a person who both answers correctly and is sure of it. By incorporating sureness assessments, an additional dimension of knowledge can be obtained. In this study, 317 females and 233 males participated (predominantly White European; with mean age = 18.7 years). One group answered questions using a conventional multiple-choice answer sheet. Another group answered the same questions, but they were also instructed to assess their sureness. Significant differences were observed on the quantitative subtest; those who made self-assessments outscored those who did not, and especially individuals who rated themselves low on traits traditionally regarded as masculine (measured with Bem Sex Role Inventory) benefited from this process. Incorporating self-assessments provides extra information that makes it possible to differentiate between those who know the subject matter and those who are guessing, as well as a way to reduce the effect of the gender typing of the task on performance.     

Bias in conceptual priming

In recent years, Ratcliff, McKoon, and colleagues have argued that priming in perceptual implicit memory tests is the result of biases in information processing. Three experiments are presented that extend this framework to the conceptual implicit memory domain. Participants studied a list of words before receiving a set of general knowledge questions. For some questions, participants studied the correct answer; for others, they studied a similar but incorrect answer. Although study of a correct answer facilitated performance, study of the similar alternative hurt performance. Costs and benefits of previous study were observed in both production and forced-choice tasks. However, there was no benefit of previous study when participants studied both the correct answer and the similar but incorrect alternative. The pattern of results indicates that participants were biased to respond with previously studied words on the conceptual implicit memory test. This pattern is concordant with the biased information-processing approach to priming.     

Remembrance of remembrance past

Arnold and Lindsay (2002) found that individuals more often failed to remember they had previously recalled an item if that item had been cued in a qualitatively different way on two recall occasions: the "forgot-it-all-along" (FIA) effect. Experiment 1 was designed to determine if the FIA effect arises because participants incorrectly believe they have not been previously tested for an item, or because they incorrectly believe they have failed to recall the item when previously tested. Experiment 2 measured participants' confidence in their incorrect prior-recall judgements, and Experiment 3 tested participants' ability to "recover" their previous recollection when the prior-recall context was restored. Results indicated that participants usually believed they had not previously been cued for the items they failed to remember previously recalling; they were often confident in their incorrect judgements of prior non-remembering; and re-introducing the context of prior remembering sometimes enabled them to recapture their memories of previous recall.     

Remembrance of remembrance past

Arnold and Lindsay (2002) found that individuals more often failed to remember they had previously recalled an item if that item had been cued in a qualitatively different way on two recall occasions: the “forgot-it-all-along” (FIA) effect. Experiment 1 was designed to determine if the FIA effect arises because participants incorrectly believe they have not been previously tested for an item, or because they incorrectly believe they have failed to recall the item when previously tested. Experiment 2 measured participants' confidence in their incorrect prior-recall judgements, and Experiment 3 tested participants' ability to “recover” their previous recollection when the prior-recall context was restored. Results indicated that participants usually believed they had not previously been cued for the items they failed to remember previously recalling; they were often confident in their incorrect judgements of prior non-remembering; and re-introducing the context of prior remembering sometimes enabled them to recapture their memories of previous recall.     

Do distractors interfere with memory for study pairs in associative recognition?

In an associative recognition task, distractors generally consist of a rearrangement of the items composing the study pairs. This makes it possible that processing the distractors generates retroactive interference on memory for the study pairs. In Experiment 1, we explored this possibility in ayes/no recognition test concerning previously learned arbitrary associations between visual symbols and auditory syllables. Rearranged pairs had a deleterious impact on the accuracy and the speed of responses to related correct pairs. This effect did not vary as a function of the number of training blocks, and furthermore, in Experiment 2, the same effect was observed for overlearned small multiplication facts. These results suggest that exposure to potentially confounding information generates interference even if this information is known to be incorrect. Some implications are outlined, especially with regard to the widespread use of multiple-choice tests in knowledge evaluation.     

Phonological priming effects on word retrieval and tip-of-the-tongue experiences in young and older adults

In a repetition priming paradigm, young and older participants read aloud prime words that sometimes shared phonological components with a target word that answered a general knowledge question. In Experiment 1, prior processing of phonologically related words decreased tip-of-the-tongue states (TOTs) and increased correct responses to subsequent questions. In Experiment 2, the priming task occurred only when the participant could not answer the question. Processing phonologically related words increased correct recall, but only when the participant was in a TOT state. Phonological priming effects were age invariant, although older adults produced relatively more TOTs. Results support the transmission deficit model that the weak connections among phonological representations that cause TOTs are strengthened by production of phonologically related words. There was no evidence that phonologically related words block TOT targets.     

Relating familiarity-based recognition and the tip-of-the-tongue phenomenon: Detecting a word’s recency in the absence of access to the word

After viewing a list of single-word answers to general knowledge questions, participants received a test list containing general knowledge questions, some of whose answers were studied, and some of whose were not. Regardless of whether participants could provide the answer to a test question, they rated the likelihood that the answer had been studied. Across three experiments,participants consistently gave higher ratings to unanswerable questions whose answers were studied than to those whose answers were not studied. This discrimination ability persisted in the absence of reported tip-of-the-tongue (TOT) states and when no information about the answer could be articulated. Studying a question's answer did not increase the likelihood of a later TOT state for that question, yet participants gave higher recognition ratings when in a TOT state than when not in a TOT state. A possible theoretical mechanism for the present pattern is discussed, as are relevant theories of familiarity-based recognition and of the TOT phenomenon.     

Decision making under internal uncertainty: the case of multiple-choice tests with different scoring rules

This paper assesses framing effects on decision making with internal uncertainty, i.e., partial knowledge, by focusing on examinees' behavior in multiple-choice (MC) tests with different scoring rules. In two experiments participants answered a general-knowledge MC test that consisted of 34 solvable and 6 unsolvable items. Experiment 1 studied two scoring rules involving Positive (only gains) and Negative (only losses) scores. Although answering all items was the dominating strategy for both rules, the results revealed a greater tendency to answer under the Negative scoring rule. These results are in line with the predictions derived from Prospect Theory (PT) [Econometrica 47 (1979) 263]. The second experiment studied two scoring rules, which allowed respondents to exhibit partial knowledge. Under the Inclusion-scoring rule the respondents mark all answers that could be correct, and under the Exclusion-scoring rule they exclude all answers that might be incorrect. As predicted by PT, respondents took more risks under the Inclusion rule than under the Exclusion rule. The results illustrate that the basic process that underlies choice behavior under internal uncertainty and especially the effect of framing is similar to the process of choice under external uncertainty and can be described quite accurately by PT.     

Influence of Question Format and Text Availability on the Assessment of Expository Text Comprehension

This study examines how passage availability and reading comprehension question format (open-ended vs. multiple-choice) influence question answering. In two experiments, college undergraduates read an expository passage and answered open-ended and multiple-choice versions of text-based, local, and global bridging inference questions. Half the participants were allowed to refer to the passage when answering the questions and half were not. Participants' prior domain knowledge relating to the text contents was assessed using multiple-choice and open-ended questions. Correlation-based analyses in the two experiments indicated: (a) a decline in the relationship between prior domain knowledge and comprehension when the passage was available during question answering; and (b) a high correlation between multiple-choice and open-ended question answering performance when the passage was not available for reference. Overall the results indicate that the nature of the reading comprehension assessment is influenced by the specific task with which comprehension is assessed.     

Six-year-olds' difficulties handling intensional contexts

Previous research shows that children have difficulties handling intensional contexts even when they can pass a test of false belief (e.g. Cognition 67 (1998) 287; Cognition 25 (1987) 289). Some authors (Perner, J. (1991). Understanding the representational mind. Cambridge, MA: MIT Press; Cognition 25 (1987) 289) place these difficulties in the linguistic and not the mental representational domain. The experiments reported here examined whether 6-year-old children could answer questions in an intensional context that did not require the explicit verbal characterization of a belief. We replicated previous findings and found that children answered according to their own knowledge in an intensional context. This occurred even though they responded by choosing a picture to insert into a protagonist's thought bubble rather than report the belief verbally. Children could correctly answer questions about the knowledge state of the protagonist and pass a test of false belief. Further experiments ruled out methodological explanations. Experiment 2 showed that the difference in answering according to own knowledge between the false belief and intensional stories is not accounted for by procedural factors in the two types of test. Experiment 3 revealed that children did not answer according to their own knowledge by default. Experiment 4 suggested that answering according to own knowledge was not a result of pictorial salience. Results are discussed in relation to the simulation-theory debate.     

Regaining access to marginal knowledge in a classroom setting

Students learn large amounts of information, but not all of it is remembered after courses end – meaning that valuable class time is often spent reviewing background material. Crucially, laboratory research suggests different strategies will be effective when reactivating previously learned information (i.e. marginal knowledge), as opposed to learning new information. In two experiments, we evaluated whether these laboratory results translated to the classroom. Topics from prior courses were tested to document which information students could no longer retrieve. Half were assigned to a not-tested control and half to the intervention; for these topics, students answered multiple-choice questions (without feedback) that gave them the chance to recognize the information they had failed to retrieve. Weeks later, students completed a final assessment on all topics. Crucially, multiple-choice testing increased the retrieval of previously forgotten information, providing the first classroom demonstration of the reactivation of marginal knowledge.     

People's hypercorrection of high-confidence errors: did they know it all along?

This study investigated the "knew it all along" explanation of the hypercorrection effect. The hypercorrection effect refers to the finding that when people are given corrective feedback, errors that are committed with high confidence are easier to correct than low-confidence errors. Experiment 1 showed that people were more likely to claim that they knew it all along when they were given the answers to high-confidence errors as compared with low-confidence errors. Experiments 2 and 3 investigated whether people really did know the correct answers before being told or whether the claim in Experiment 1 was mere hindsight bias. Experiment 2 showed that (a) participants were more likely to choose the correct answer in a 2nd guess multiple-choice test when they had expressed an error with high rather than low confidence and (b) that they were more likely to generate the correct answers to high-confidence as compared with low-confidence errors after being told they were wrong and to try again. Experiment 3 showed that (c) people were more likely to produce the correct answer when given a 2-letter cue to high- rather than low-confidence errors and that (d) when feedback was scaffolded by presenting the target letters 1 by 1, people needed fewer such letter prompts to reach the correct answers when they had committed high- rather than low-confidence errors. These results converge on the conclusion that when people said that they knew it all along, they were right. This knowledge, no doubt, contributes to why they are able to correct those high-confidence errors so easily.     

Memory for questions and amount of processing

Two experiments were carried out on how questions are remembered. Subjects watched a videotape of a series of simple events and then answered 18 questions about these events. The questions were all of the same general syntactic form (e.g., “Did the pencil fall against the jug on A?”, where A refers to a particular location). They were designed to elicit three sorts of answer: “yes,” “no” because the event took place at another location, and “no” because the event did not take place at all. After the subjects had answered the questions, they were given an unexpected test of their ability to recall them. A difference in the memorability of the questions was predicted on the basis of a procedural theory of comprehension and a hypothesis about memory subjects should cease to process a question when they realize that it concerns an event that did not take place, and such questions should be harder to remember because they are processed to a lesser degree than the other sorts of question. Experiment 1 confirmed the predictions, but its results in part could be accounted for by assuming that subjects recalled the original events and used them as a cue to remembering the questions. Experiment 2 eliminated this explanation by showing that when subjects do not have to answer certain questions, their recall of them is very poor. However, the same differences in the memorability of the three sorts of question were obtained for both answered and unanswered questions.