The Overconfidence Phenomenon as a Consequence of Informal Experimenter-Guided Selection of Almanac Items

The paper argues for an ecological approach to realism of confidence in general knowledge. It is stressed that choice of answer to almanac items and confidence judgments derive from knowledge structures formed by adaptation to a natural environment. People are regarded as well-calibrated to their natural environments and the overconfidence phenomenon is seen as a consequence of the procedures involved in the creation of "traditional" general knowledge items, rather than as the result of a cognitive bias. An experiment is reported showing that when items are informally selected by human "selectors," instructed to select items that differentiate between more and less knowledgeable subjects, we observe poor calibration and the usual overconfidence phenomenon. However, when the item selection is "debiased" and the objects to be compared in the almanac items are selected randomly from a natural environment the overconfidence phenomenon disappears and we observe good calibration, and good resolution. These results support an ecological approach to realism of confidence and suggest that general procedures for debiasing overconfidence may be unwarranted.     

Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect

Two robust phenomena in research on confidence in one's general knowledge are the overconfidence phenomenon and the hard-easy effect. In this article, the authors propose that the hard-easy effect has been interpreted with insufficient attention to the scale-end effects, the linear dependency, and the regression effects in data and that the continued adherence to the idea of a "cognitive overconfidence bias" is mediated by selective attention to particular data sets. A quantitative review of studies with 2-alternative general knowledge items demonstrates that, contrary to widespread belief, there is (a) very little support for a cognitive-processing bias in these data; (b) a difference between representative and selected item samples that is not reducible to the difference in difficulty; and (c) near elimination of the hard-easy effect when there is control for scale-end effects and linear dependency.     

Internal Cue Theory: Calibration and Resolution of Confidence in General Knowledge

A theory is proposed in which beliefs in the form of internal cue validities mediate the processing of ecological cue validities in the assessment of confidence. The conditions necessary for perfect calibration are specified: (a) correspondence between ecological and internal validity, (b) perfect translation of internal validity into a confidence assessment, and (c) consistent utilization of cues. Process errors are then added to these conditions to investigate how calibration is affected by error variance of confidence assessments. To accomplish this, the calibration score (C) is decomposed into three additive parts: D² = bias, i.e., the squared difference between mean confidence and proportion correct; R² = resolution, i.e., the squared difference between the standard deviations of confidence and proportion correct; L = linearity, i.e., how closely the calibration curve follows a linear function. In the equation C = D² + R² + L, R² (resolution) reflects the subject′s ability to discriminate cue validities. Selection of items is a critical factor in studies of confidence. Informal selection with a tendency to avoid easy items results in overconfidence. Internal cue theory predicts both that overconfidence should disappear (in accordance with previous research) and that resolution should improve when item selection is made representative of the natural environment. Both predictions are confirmed by data from published studies on confidence in general knowledge. It is noteworthy that resolution is still poor and accounts for the major portion of miscalibration under representative item selection.     

“To‐be‐forgotten” statements become less true: Memory processes involved in selection and forgetting lead to truthfulness changes of ambiguous sentences

What happens when people try to forget something? What are the consequences of instructing people to intentionally forget a sentence? Recent studies employing the item‐method directed forgetting paradigm have shown that to‐be‐forgotten (TBF) items are, in a subsequent task, emotionally devaluated relative to to‐be‐remembered (TBR) items, an aftereffect of memory selection (Vivas, Marful, Panagiotidou & Bajo, 2016). As such, distractor devaluation by attentional selection generalizes to memory selection. In this study, we use the item‐method directed forgetting paradigm to test the effects of memory selection and inhibition on truth judgments of ambiguous sentences. We expected the relative standing of an item in the task (i.e., whether it was instructed to be remembered or forgotten) to affect the truthfulness value of that item, making TBF items less valid/truthful than TBR items. As predicted, ambiguous sentences associated with a “Forget” cue were subsequently judged as less true than sentences associated with a “Remember” cue, suggesting that instructions to intentionally forget a statement can produce changes in the validity/truthfulness of that statement. To our knowledge, this is the first study to show an influence of memory processes involved in selection and forgetting on the perceived truthfulness of sentences.     

Scale validation via quantifying item validity using the Dm index

While most validity indices are based on total test scores, this paper describes a method for quantifying the construct validity of items. The approach is based on the item selection technique originally described by Piazza in 1980. Unfortunately, Piazza's P2 index suffers from some substantial limitations. The Dm coefficient provides an alternative which can be used for item selection and provides a validity index for a set of items. The index is similar to that of traditional criterion-related validity indices. Criterion-related validity is used to demonstrate the accuracy of hypothesized relations of the measure with outcome variables of interest in research and practice. This method may be useful when the sample of items or persons is small, rendering more traditional approaches such as factor analysis or item response theory inappropriate. An example of how to use the technique is provided.     

Ecological and person‐oriented aspects of metacognitive processes in test‐taking

This study examines critical aspects of both the ecological and the person‐oriented accounts of observed biases in confidence judgements on tests of cognitive abilities. These biases reflect metacognitive processes involved in test‐taking. According to the ecological approach, poor realism of confidence judgements is due to the nature of the items included in general knowledge tests (test‐driven biases). The person‐oriented approach, however, argues that biases in confidence judgements may be due to a general self‐monitoring trait. The present study employed the ‘de‐biasing’ procedure proposed by Juslin ( 1994 ) for the selection of general knowledge test items, and used a newly developed geographical knowledge test suitable for the Australian population. Two other cognitive tests (Raven's Progressive Matrices and Line Length) were administered in order to determine whether there is a consistency in confidence ratings across diverse tasks. Statistical procedures traditional to both approaches‐calibration curves and factor analysis ‐ were employed. The results, with minor qualifications, support both perspectives. The study found a separate confidence factor, indicative of a self‐monitoring trait. Two other potential metacognitive factors (i.e. ‘expectation’ and ‘evaluation’, corresponding to self‐assessment/planning and self‐evaluation) could not be separated from accuracy and speed measures. Copyright © 2001 John Wiley & Sons, Ltd.     

Part-list cuing in amnesic patients: evidence for a retrieval deficit

Part-list cuing--the detrimental effect of the presentation of a subset of learned items on recall of the remaining items--was examined in amnesic patients and healthy control subjects. Subjects studied two types of categorized item lists: lists in which each category consisted of strong and moderate items and lists in which each category consisted of weak and moderate items. The subjects recalled a category's strong and weak items in either the presence of or absence of the moderate items serving as retrieval cues. In healthy subjects, part-list cuing impaired recall of the strong items but not of the weak items; in amnesics, part-list cuing impaired recall of both types of items. Part-list cuing is often attributed to a change in the retrieval process from a more effective one when cues are absent to a less effective one when they are present. On the basis of this view, our results indicate that part-list cuing causes a stronger retrieval inefficiency in amnesic patients than in healthy people.     

‘Grandpa,Do You Like Roller Coasters?’: Identifying Age‐Appropriate Personality Indicators

Personality development research heavily relies on the comparison of scale means across age. This approach implicitly assumes that the scales are strictly measurement invariant across age. We questioned this assumption by examining whether appropriate personality indicators change over the lifespan. Moreover, we identified which types of items (e.g. dispositions, behaviours, and interests) are particularly prone to age effects. We reanalyzed the German Revised NEO Personality Inventory normative sample (N = 11,724) and applied a genetic algorithm to select short scales that yield acceptable model fit and reliability across locally weighted samples ranging from 16 to 66 years of age. We then examined how the item selection changes across age points and item types. Emotion‐type items seemed to be interchangeable and generally applicable to people of all ages. Specific interests, attitudes, and social effect items—most prevalent within the domains of Extraversion, Agreeableness, and Openness—seemed to be more prone to measurement variations over age. A large proportion of items were systematically discarded by the item‐selection procedure, indicating that, independent of age, many items are problematic measures of the underlying traits. The implications for personality assessment and personality development research are discussed. © 2019 European Association of Personality Psychology     

Empirical analysis of the intrinsic-extrinsic distinction of judgments of learning (JOLs): effects of relatedness and serial position on JOLs

A. Koriat's (1997) cue-utilization framework provided a significant advance in understanding how people make judgments of learning (JOLs). A major distinction is made between intrinsic and extrinsic cues. JOLs are predicted to be sensitive to intrinsic cues (e.g., item relatedness) and less sensitive to extrinsic cues (e.g., serial position) because JOLs are comparative across items in a list. The authors evaluated predictions by having people make JOLs after studying either related (poker-flush) or unrelated (dog-spoon) items. Although some outcomes confirmed these predictions, others could not be readily explained by the framework. Namely, relatedness influenced JOLs even when manipulated between participants, primacy effects were evident on JOLs, and the order in which blocks of items were presented (either all related items first or all unrelated items first) influenced JOLs. The authors discuss the framework in relation to these and other outcomes.     

Reply to William R. Ferrell’s paper “A model for realism of confidence judgments: Implications for underconfidence in sensory discrimination”

Ferrell’s decision-variable partition model and our subjective distance model belong to the same family of Thurstonial models. The subjective distance model is limited to sensory discrimination with the method of constant stimuli and rooted in such notions as discriminal dispersion and sense distance. Ferrell’s model is intended to be wider in scope and to apply to both cognitive and sensory tasks. Both models need supplementary assumptions to predict calibration phenomena. The point of departure for us is the fact that the model predicts under-confidence under “guessing” and the empirical finding that people are about 100% correct when they report “absolutely certain.” Ferrell makes assumptions about cutoffs on the decision variable. The respondent is assumed to adjust or not adjust cutoffs according to “cues to difficulty.” We disagree with Ferrell’s claim that the hard-easy effect is explained by the respondent’s failure to adjust cutoffs sufficiently when there is a change in level of difficulty, and argue that this amounts to little more than a translation of the hard-easy effect into the lingua of Ferrell’s decision-variable partition model. Our argument is that the hard-easy effect is a consequence of the post hoc division of items according to solution probability. In addition, error variance may contribute to regression effects that enlarge the hard-easy effect. Finally, in contrast to Ferrell’s position, we regard inference (cognitive uncertainty) and discrimination (sensory uncertainty) as different psychological processes. An understanding of calibration in these two areas requires separate models.     

Ego-syntonicity in responses to items in the California Psychological Inventory

Responding to items on a personality questionnaire can evoke a variety of feelings, from discomfort to indifference to pleasure. Harrison Gough reported that when he wrote items for the California Psychological Inventory (CPI; Gough & Bradley, 1996), he tried to make the items as ego-syntonic as possible. Ego-syntonic items are those “which a respondent finds congenial, and on which giving an opinion is a rewarding act” (Gough & Bradley, 1996, p. 10). The present study asked 79 respondents to report how they felt after answering each CPI item. Average affect ratings were above neutral for a majority of items, indicating that Gough had some success in writing ego-syntonic items. Differences in item ego-syntonicity were attributable to other item characteristics. Respondents disliked responding to relatively odd and ambiguous items, items with linguistic negations, and items referring to negative feelings and situations. As predicted by Gough, respondents enjoyed responding to items on the communality scale, items with which most people agree. They also enjoyed items that referred to positive emotions and attitudes and to items indicating extraversion, conscientiousness, low neuroticism, and openness to experience. Highly ego-syntonic items were found to be more valid than less ego-syntonic items. Individuals who reported disliking many items were found to be socially anxious. The relation between reports of liking or disliking items, identity, and reputation are discussed, and further research on item response dynamics and validity is proposed.     

The disutility of the hard-easy effect in choice confidence

A common finding in confidence research is the hard-easy effect, in which judges exhibit greater overconfidence for more difficult sets of questions. Many explanations have been advanced for the hard-easy effect, including systematic cognitive mechanisms, experimenter bias, random error, and statistical artifact. In this article, I mathematically derive necessary and sufficient conditions for observing a hard-easy effect, and I relate these conditions to previous explanations for the effect. I conclude that all types of judges exhibit the hard-easy effect in almost all realistic situations. Thus, the effect’s presence cannot be used to distinguish between judges or to draw support for specific models of confidence elicitation.     

The accuracy of meta-metacognitive judgments: regulating the realism of confidence

Can people improve the realism of their confidence judgments about the correctness of their episodic memory reports by deselecting the least realistic judgments? An assumption of Koriat and Goldsmith's (Psychol Rev 103:490-517, 1996) model is that confidence judgments regulate the reporting of memory reports. We tested whether this assumption generalizes to the regulation of the realism (accuracy) of confidence judgments. In two experiments, 270 adults in separate conditions answered 50 recognition and recall questions about the contents of a just-seen video. After each answer, they made confidence judgments about the answer's correctness. In Experiment 1, the participants in the recognition conditions significantly increased their absolute bias when they excluded 15 questions. In Experiment 2, the participants in the recall condition significantly improved their calibration. The results indicate that recall, more than recognition, offers valid cues for participants to increase the realism of their report. However, the effects were small with only weak support for the conclusion that people have some ability to regulate the realism in their confidence judgments.     

认知诊断CAT中具有非统计约束选题方法的比较

项目曝光控制和内容约束关系到测验安全、测验的信度和效度, 是计算机化自适应测验(Computerized Adaptive Testing, CAT)中两类重要的非统计约束条件。本文在认知诊断CAT中针对内容约束和项目曝光控制要求, 运用5种方法选择测验项目。它们分别是：(1) Monte Carlo方法与项目合格方法相结合, 记为MC-IE; (2) Monte Carlo方法与最大优先指标方法相结合, 记为MC-MPI; (3) Monte Carlo方法与限制阈值方法相结合, 记为MC-RT; (4) Monte Carlo方法与限制进度指标方法相结合, 记为MC-RPG以及(5) Monte Carlo方法与最大后验概率方法相结合, 记为MC-PP。然后通过在线性、收敛、发散、无结构和独立五种属性结构下构建题库并运用重参化融融统和模型模拟被试反应比较它们的选题表现。研究发现, (1) 相同选题方法在不同属性结构下项目曝光率的分布类似, 测量精度按线性、收敛、发散、无结构和独立结构的顺序依次降低; (2) 相同属性结构下, 不同方法的测量精度高低依次为MC-PP、MC-IE、MC-RT、MC-MPI和MC-RPG方法; 项目曝光均匀性优劣依次为MC-RPG、MC-MPI、MC-RT、MC-IE和MC-PP方法。统一量纲值表明, MC-RPG方法的综合表现最好, MC-MPI方法的表现次之。     

CORRESPONDENCE BETWEEN ESTIMATES OF CONTENT AND CRITERION-RELATED VALIDITY VALUES

The correspondence between inferences made using two validation strategies–content and criterion-related–were examined in a specific personnel selection application. Empirical validity values and Law-she's (1975) content validity ratios (CVR) were obtained for items from three structured interview guides used in the selection of insurance agents. Ratings of each item by over 300 field managers were used to calculate the CVR values. Statistically significant, yet modest correlations were found between empirical item validities and content validities for an interview guide used to select applicants with prior insurance sales experience. No significant differences were found among these correlations by comparing job experts of different levels of managerial experience and experience in selection. Data for the interview guide used to select experienced applicants also indicated that a content validity approach can be useful in developing a selection instrument with an empirically valid composite rating. The hypotheses were not confirmed for interview guides used to select applicants with no prior insurance sales experience. The practical importance of these results are discussed, as are plans for future research.     

Sampling theory in item analysis

Since item values obtained by item analysis procedures are not always stable from one situation to another, it follows that selection of items for validity or difficulty is sometimes useless. An application of Chi Square to testing homogeneity of item values is made, in the case of theUL method, and illustrative data are presented. A method of applying sampling theory to Horst's maximizing function is outlined, as illustrative of author's observation that the results of item analysis by any of various methods may be similarly tested.     

眼动追踪在价值导向元记忆研究中的应用

价值导向元记忆关注人们在面对不同重要性信息时,通过元记忆监测和调节,有选择地优先加工高价值信息,以实现记忆效率最大化的目的。价值导向元记忆包括价值导向元记忆监测和控制,眼动追踪技术以其无干扰性、生态效度高等优势可以时时追踪这一监控过程。当前该领域研究中已采用的眼动指标集中在项目选择、学习时间分配、学习进程等方面。未来在项目选择、学习效率和策略比较等研究中可以探索眼动追踪技术的进一步应用。     

Feeling of knowing and restudy choices

Feeling-of-knowing judgments (FOK-Js) reflect people’s confidence that they would be able to recognize a currently unrecallable item. Although much research has been devoted to the factors determining the magnitude and accuracy of FOK-Js, much less work has addressed the issue of whether FOK-Js are related to any form of metacognitive control over memory processes. In the present study, we tested the hypothesis that FOK-Js are related to participants’ choices of which unrecallable items should be restudied. In three experiments, we showed that participants tend to choose for restudy items with high FOK-Js, both when they are explicitly asked to choose for restudy items that can be mastered in the restudy session (Exps. 1a and 2) and when such specific instructions are omitted (Exp. 1b). The study further demonstrated that increasing FOK-Js via priming cues affects restudy choices, even though it does not affect recall directly. Finally, Experiment 2 showed the strategy of restudying unrecalled items with high FOK-Js to be adaptive, because the efficacy of restudy is greater for these items than for items with low FOK-Js. Altogether, the present findings underscore an important role of FOK-Js for the metacognitive control of study operations.     

Subjective confidence in one's answers: the consensuality principle

In answering general-information questions, a within-person confidence-accuracy (C-A) correlation is typically observed, suggesting that people can monitor the correctness of their knowledge. However, because the correct answer is generally the consensual answer--the one endorsed by most participants--confidence judgment may actually monitor the consensuality of the answer rather than its correctness. Indeed, the C-A correlation was positive for items with a consensually correct answer but negative for items with a consensually wrong answer. Results suggest that the consensuality-confidence correlation may be mediated by 2 internal mnemonic cues that are correlated with consensuality: Consensual answers are reached faster and are selected more consistently by the same person on different occasions than nonconsensual answers. The results argue against a direct-access view of confidence judgments and suggest that such judgments will be accurate only as long as people's responses are by and large correct across the sampled items, thus stressing the criticality of a representative design.     

Examining the item response process to personality measures in high-stakes situations: Issues of measurement validity and predictive validity

We conducted two experimental studies with between-subjects and within-subjects designs to investigate the item response process for personality measures administered in high- versus low-stakes situations. Apart from assessing measurement validity of the item response process, we examined predictive validity; that is, whether or not different response models entail differential selection outcomes. We found that ideal point response models fit slightly better than dominance response models across high- versus low-stakes situations in both studies. Additionally, fitting ideal point models to the data led to fewer items displaying differential item functioning compared to fitting dominance models. We also identified several items that functioned as intermediate items in both the faking and honest conditions when ideal point models were fitted, suggesting that ideal point model is “theoretically” more suitable across these contexts for personality inventories. However, the use of different response models (dominance vs. ideal point) did not have any substantial impact on the validity of personality measures in high-stakes situations, or the effectiveness of selection decisions such as mean performance or percent of fakers selected. These findings are significant in that although prior research supports the importance and use of ideal point models for measuring personality, we find that in the case of personality faking, though ideal point models seem to have slightly better measurement validity, the use of dominance models may be adequate with no loss to predictive validity.