Responding to Feedback after Multiple-choice Answers: The Influence of Response Confidence

A model is proposed that describes how people choose multiple-choice answers and deal with subsequent feedback. Central to the model is the construct of likelihood, a signal that: (1) indexes the match between task demands and what is known, (2) gives a basis for decisions, and (3) is estimated by response confidence ratings. In two experiments, subjects answered an item, rated confidence, and studied feedback (for 100 items), and then answered the items again. All predictions of the model were confirmed. Specifically, for right/wrong feedback messages, feedback study intervals were longer after correct than incorrect choices (for correct choices, these intervals decreased as confidence increased). Also, there was no relation between the probability of correcting errors and level of confidence. For feedback messages that consisted of an item with correct alternative marked, there was a positive relation between probability of correcting errors and level of confidence.     

Monitoring the source monitoring

The hypothesis that the retrieval of correct source memory cues, those leading to a correct source attribution, increases confidence, whereas the retrieval of incorrect source memory cues, those leading to a source misattribution, decreases confidence was tested. Four predictions were derived from this hypothesis: (1) confidence should be higher for correct than incorrect source attribution except; (2) when no source cues are retrieved; (3) only the source misattributions inferred from the retrieval of incorrect source cues will be rated with low confidence; and (4) the number of source cues retrieved, either correct or incorrect, will affect the confidence in the source attributions. To test these predictions, participants read two narratives from two witnesses to a bank robbery, a customer and a teller. Then, participants completed a source monitoring test with four alternatives, customer, teller, both, or neither, and rated their confidence in their source attribution. Results supported the first three predictions, but they also suggested that the number of correct source monitoring cues retrieved did not play a role in the monitoring of the accuracy of the source attributions. Attributions made from the recovery of incorrect source cues could be tagged as dubious or uncertain, thus leading to lowered confidence irrespective of the number of incorrect source cues or whether another correct source cue was also recovered. This research has potential applications for eyewitness memory because it shows that confidence can be an indicator of the accuracy of a source attribution.     

Confidence in masked orientation judgments is informed by both evidence and visibility

How do human observers determine their degree of belief that they are correct in a decision about a visual stimulus—that is, their confidence? According to prominent theories of confidence, the quality of stimulation should be positively related to confidence in correct decisions, and negatively to confidence in incorrect decisions. However, in a backward-masked orientation task with a varying stimulus onset asynchrony (SOA), we observed that confidence in incorrect decisions also increased with stimulus quality. Model fitting to our decision and confidence data revealed that the best explanation for the present data was the new weighted evidence-and-visibility model, according to which confidence is determined by evidence about the orientation as well as by the general visibility of the stimulus. Signal detection models, postdecisional accumulation models, two-channel models, and decision-time-based models were all unable to explain the pattern of confidence as a function of SOA and decision correctness. We suggest that the metacognitive system combines several cues related to the correctness of a decision about a visual stimulus in order to calculate decision confidence.     

How we make discoveries

In trying to make discoveries, we are trying to uncover knowledge of HIDDEN realities. It appears impossible to uncover knowledge of hidden realities. How can we evaluate results? (How can we find out whether they are true or even good approximation when we cannot compare them to the hidden realities?) But we are often able to do things which appear impossible; it depends on whether we have chanced onto, or discovered, or invented, the relevant OPERATING PRINCIPLES. It appeared impossible to fly to the moon in one lifetime. We discovered the principle of the rocket. If we can discover the operating principle for making discoveries, we know how we make discoveries. In this paper, I show step by step how we can discover this operating principle.     

Confidence–Accuracy Calibration with General Knowledge and Eyewitness Memory Cued Recall Questions

Summary: The confidence–accuracy relationship has primarily been studied through recognition tests and correlation analysis. However, cued recall is more ecological from a forensic perspective. Moreover, there may be more informative ways of analysing the confidence–accuracy relationship than correlations. In the present study, participants viewed a video of a bank robbery and were asked cued recall questions covering general knowledge and the video itself. Confidence ratings were collected, and correlations, calibration and discrimination measures were calculated. All measures indicated a strong confidence–accuracy relationship that was better for general knowledge than eyewitness memory questions. However, there were no differences in confidence ratings for correct answers, suggesting that the differences could be limited to the evaluation of incorrect answers. We concluded that confidence may be a good marker for accuracy with cued recall, but that further research using ecological tests and more informative data analysis techniques is needed. Copyright © 2011 John Wiley & Sons, Ltd.     

Using verification feedback to correct errors made on a multiple-choice test

A key educational challenge is how to correct students’ errors and misconceptions so that they do not persist. Simply labelling an answer as correct or incorrect on a short-answer test (verification feedback) does not improve performance on later tests; error correction requires receiving answer feedback. We explored the generality of this conclusion and whether the effectiveness of verification feedback depends on the type of test with which it is paired. We argue that, unlike for short-answer tests, learning whether one's multiple-choice selection is correct or incorrect should help participants narrow down the possible answers and identify specific lures as false. To test this proposition we asked participants to answer a series of general knowledge multiple-choice questions. They received no feedback, answer feedback, or verification feedback, and then took a short-answer test immediately and two days later. Verification feedback was just as effective as answer feedback for maintaining correct answers. Importantly, verification feedback allowed learners to correct more of their errors than did no feedback, although it was not as effective as answer feedback. Overall, verification feedback conveyed information to the learner, which has both practical and theoretical implications.     

Using verification feedback to correct errors made on a multiple-choice test

A key educational challenge is how to correct students' errors and misconceptions so that they do not persist. Simply labelling an answer as correct or incorrect on a short-answer test (verification feedback) does not improve performance on later tests; error correction requires receiving answer feedback. We explored the generality of this conclusion and whether the effectiveness of verification feedback depends on the type of test with which it is paired. We argue that, unlike for short-answer tests, learning whether one's multiple-choice selection is correct or incorrect should help participants narrow down the possible answers and identify specific lures as false. To test this proposition we asked participants to answer a series of general knowledge multiple-choice questions. They received no feedback, answer feedback, or verification feedback, and then took a short-answer test immediately and two days later. Verification feedback was just as effective as answer feedback for maintaining correct answers. Importantly, verification feedback allowed learners to correct more of their errors than did no feedback, although it was not as effective as answer feedback. Overall, verification feedback conveyed information to the learner, which has both practical and theoretical implications.     

Effects of reiteration,hindsight bias,and memory on realism in eyewitness confidence

This study investigated the effects of repetition, memory, feedback, and hindsight bias on the realism in confidence in answers to questions on a filmed kidnapping. In Experiment 1 the participants showed overconfidence in all conditions. In the Repeat condition (‘how confident are you now that your previous answers are correct’) overconfidence was reduced as a consequence of the decrease in confidence in both correct and incorrect answers compared with the Repeat condition when the participants received feedback on their answers and were asked to remember their initial confidence, the confidence level was higher for correct and lower for incorrect answers. In Experiment 2, recalled confidence (the Memory condition) increased compared with the original confidence both for correct and incorrect answers; the effect of this was increased overconfidence. The Hindsight condition showed a decrease in confidence in incorrect answers. The results suggest that a unique hindsight effect may be more clearly present for incorrect than for correct answers. Our study gives further evidence for the malleability of the realism in eyewitness confidence and we discuss both the theoretical and forensic implications of our findings. Copyright © 2000 John Wiley & Sons, Ltd.     

Initial Testing Does Not Necessarily Affect Eyewitness Recall Assessed by Specific Questioning

In eyewitness situations, open recall is followed by specific questioning about the witnessed event. We examined whether initial testing affects later recall of actions and specific details. After watching a video of a bank robbery, participants completed an initial testing phase that involved free recall, specific questions about the actions or details of the event or a control condition with no initial testing. In the final test, correct and incorrect answers, accuracy and response confidence for actions and details were analysed. Initial testing affected neither recall nor confidence. The participants were more accurate for actions than details. Response confidence was higher for correct than incorrect answers and higher for details than actions in correct answers. The results showed that specific questioning affects differently the recall of event actions and details and that remembering details increased confidence. Investigative interviewers can use this evidence when questioning information not reported in initial testing.Copyright © 2015 John Wiley & Sons, Ltd.     

Four applications of permutation methods to testing a single-mediator model

Four applications of permutation tests to the single-mediator model are described and evaluated in this study. Permutation tests work by rearranging data in many possible ways in order to estimate the sampling distribution for the test statistic. The four applications to mediation evaluated here are the permutation test of ab, the permutation joint significance test, and the noniterative and iterative permutation confidence intervals for ab. A Monte Carlo simulation study was used to compare these four tests with the four best available tests for mediation found in previous research: the joint significance test, the distribution of the product test, and the percentile and bias-corrected bootstrap tests. We compared the different methods on Type I error, power, and confidence interval coverage. The noniterative permutation confidence interval for ab was the best performer among the new methods. It successfully controlled Type I error, had power nearly as good as the most powerful existing methods, and had better coverage than any existing method. The iterative permutation confidence interval for ab had lower power than do some existing methods, but it performed better than any other method in terms of coverage. The permutation confidence interval methods are recommended when estimating a confidence interval is a primary concern. SPSS and SAS macros that estimate these confidence intervals are provided.     

Analysis of proctor marking accuracy in a computer-aided personalized system of instruction course

In a computer-aided version of Keller's personalized system of instruction (CAPSI), students within a course were assigned by a computer to be proctors for tests. Archived data from a CAPSI-taught behavior modification course were analyzed to assess proctor accuracy in marking answers as correct or incorrect. Overall accuracy was increased by having each test marked independently by two proctors, and was higher on incorrect answers when the degree of incorrectness was larger.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling.

The psychometric function relates an observer's performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper's principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one's choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling

The psychometric function relates an observer’s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper’s principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one’s choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

Constructing bootstrap confidence intervals for principal component loadings in the presence of missing data: a multiple-imputation approach

Earlier research has shown that bootstrap confidence intervals from principal component loadings give a good coverage of the population loadings. However, this only applies to complete data. When data are incomplete, missing data have to be handled before analysing the data. Multiple imputation may be used for this purpose. The question is how bootstrap confidence intervals for principal component loadings should be corrected for multiply imputed data. In this paper, several solutions are proposed. Simulations show that the proposed corrections for multiply imputed data give a good coverage of the population loadings in various situations.     

Intensity and number of alternatives in hue identification: Piéron’s law and choice reaction time

Two experiments investigated the effects of intensity and the number of response alternatives in hue identification. Three questions were addressed. First, what is the relationship between these effects? If intensity and number of alternatives affect different processes in series, these factors would have additive effects on reaction time (RT). Other models predict a multiplicative interaction. These models assume that when a stimulus is presented, evidence accrues over time for each response alternative, and that when the evidence for a particular alternative exceeds its criterion, the response is made. A natural auxiliary assumption is that intensity affects the rate of evidence accrual, while the number of alternatives affects the criterion. The second question addressed is, how does choice RT change as a function of intensity? Piéron used a power function to describe the effect of intensity on simple RT. This paper considers three candidates for intensity’s effect on choice RT: the power, logistic, and Michaelis functions. Each candidate function was tested in an additive model and a multiplicative model. The best account was given by the power function in an additive model. The last question is, does each process produce an output that can be classified as correct or incorrect? Schweickert (1985) proposed that factors that selectively influenced such processes would have additive effects on log percent correct. Instead, an interaction was found. Perhaps the assumption that the process affected by intensity produces a single output is wrong.     

Applying Principles to Cases and the Problem of Judgment

We sometimes decide what to do by applying moral principles to cases, but this is harder than it looks. Principles are more general than cases, and sometimes it is hard to tell whether and how a principle applies to a given case. Sometimes two conflicting principles seem to apply to the same case. To handle these problems, we use a kind of judgment to ascertain whether and how a principle applies to a given case, or which principle to follow when two principles seem to conflict. But what do we discern when we make such judgments—that is, what makes such judgments correct? The obvious answer is that they are made correct by whatever makes other moral judgments correct. However, that cannot be right, for a principle can be inconsistent with morality yet still apply in a particular way to a given case. If the principle is inconsistent with morality, then morality cannot be what we discern when we judge whether and how that principle applies to a given case. I offer an alternative account of what makes such judgments correct.     

基于概化理论的方差分量变异量估计

概化理论广泛应用于心理与教育测量实践中, 方差分量估计是进行概化理论分析的关键。方差分量估计受限于抽样, 需要对其变异量进行探讨。采用蒙特卡洛(Monte Carlo)数据模拟技术, 在正态分布下讨论不同方法对基于概化理论的方差分量变异量估计的影响。结果表明: Jackknife方法在方差分量变异量估计上不足取; 不采取Bootstrap方法的“分而治之”策略, 从总体上看, Traditional方法和有先验信息的MCMC方法在标准误及置信区间这两个变异量估计上优势明显。     

No pain no gain: The positive impact of punishment on the strategic regulation of accuracy

Previous studies have shown that punishing people through a large penalty for volunteering incorrect information typically leads them to withhold more information (metacognitive response bias), but it does not appear to influence their ability to distinguish between their own correct and incorrect answers (metacognitive accuracy discrimination). The goal of the current study was to demonstrate that punishing people for volunteering incorrect information—versus rewarding volunteering correct information—produces more effective metacognitive accuracy discrimination. All participants completed three different general-knowledge tests: a reward test (high points for correct volunteered answers), a baseline test (equal points/penalties for volunteered correct/incorrect answers) and a punishment test (high penalty for incorrect volunteered answers). Participants were significantly better at distinguishing between their own correct and incorrect answers on the punishment than reward test, which has implications for situations requiring effective accuracy monitoring.     

Fisher transformations for correlations corrected for selection and missing data

The validity of a test is often estimated in a nonrandom sample of selected individuals. To accurately estimate the relation between the predictor and the criterion we correct this correlation for range restriction. Unfortunately, this corrected correlation cannot be transformed using Fisher'sZ transformation, and asymptotic tests of hypotheses based on small or moderate samples are not accurate. We developed a Fisherr toZ transformation for the corrected correlation for each of two conditions: (a) the criterion data were missing due to selection on the predictor (the missing data were MAR); and (b) the criterion was missing at random, not due to selection (the missing data were MCAR). The twoZ transformations were evaluated in a computer simulation. The transformations were accurate, and tests of hypotheses and confidence intervals based on the transformations were superior to those that were not based on the transformations.     

Interpreting elliptical verb phrases

Elliptical verb phrases are anaphoric expressions whose correct interpretation depends on the exact form of the preceding text as well as its meaning. However, people are not very good at remembering surface details of what they read or hear, so how do they understand such expressions? One alternative to a linguistically based interpretation of elliptical verb phrases is to assign them meanings that are plausible, given general knowledge about the situation being described. In an experiment, subjects read passages in which context provided a plausible interpretation for an elliptical verb phrase that either was or was not at odds with the linguistically correct interpretation. There was a tendency for subjects to assign plausible, but incorrect, meanings to elliptical verb phrases. This tendency increased with the distance between the elliptical verb phrase and its antecedent. Incorrect interpretations were assigned slowly, and the speed of assignment increased with distance from the antecedent. These results suggest that people try to interpret elliptical verb phrases correctly, but that they are not always able to do so. Furthermore, they indicate that the intuition that elliptical verb phrases are easy to interpret is misleading. If text is to be comprehensible, speakers and writers must use such expressions with care.