Response selection strategies and realism of confidence judgments

Two studies were conducted to examine how response selection strategy is related to confidence ratings and to performance on general knowledge questions. In both studies subjects were asked to answer 80 general knowledge questions and to rate their confidence in the correctness of the answer selected. A pilot study, in which subjects thought aloud while answering general knowledge questions, was carried out to identify different response selection strategies. In the first study, 40 subjects were asked to indicate which of four strategies (immediate recognition, inference, intuition, or guessing) they used for selecting an answer. In Study 2, think aloud reports from 20 subjects were coded into the same four strategies. The distribution of strategies differed between the studies, but there were very similar relations among strategy, confidence, and correctness of answer in the two studies. Response selection strategy was related to correctness of answer when confidence was partialed out. More specifically, immediate recognition was associated with higher proportion correct than with the other strategies. It was also found that ratings of how difficult the knowledge questions were to fellow students of the subjects were on a much more realistic level than the confidence ratings were. It is concluded that people could improve their confidence judgments by taking into account (a) how difficult a question is to other people, and (b) the response selection strategy used for answering the question.     

Influence of Question Format and Text Availability on the Assessment of Expository Text Comprehension

This study examines how passage availability and reading comprehension question format (open-ended vs. multiple-choice) influence question answering. In two experiments, college undergraduates read an expository passage and answered open-ended and multiple-choice versions of text-based, local, and global bridging inference questions. Half the participants were allowed to refer to the passage when answering the questions and half were not. Participants' prior domain knowledge relating to the text contents was assessed using multiple-choice and open-ended questions. Correlation-based analyses in the two experiments indicated: (a) a decline in the relationship between prior domain knowledge and comprehension when the passage was available during question answering; and (b) a high correlation between multiple-choice and open-ended question answering performance when the passage was not available for reference. Overall the results indicate that the nature of the reading comprehension assessment is influenced by the specific task with which comprehension is assessed.     

The Unconscious Feeling of Knowing: A Commentary on Koriat's Paper

In Koriat's paper “The Feeling of Knowing: Some Metatheoretical Implications for Consciousness and Control,” he asserts that the feeling of knowing straddles the implicit and explicit, and that these conscious feelings enter into a conscious control process that is necessary for controlled behavior. This assertion allows him to make many speculations on the nature of consciousness itself. We agree that feelings of knowing are produced through a monitoring of one's knowledge, and that this monitoring can affect the control of behavior such as whether or not to search memory for an answer. Further, we believe that monitoring of performance with a strategy can also affect cognition control and strategy selection; however, we also believe that frequently this monitoring and control occurs without conscious awareness. Feeling of knowing has received an inordinate amount of attention because it lies behind the highly recognizable tip-of-the-tongue phenomenon that represents one of the rare cases of conscious monitoring. There are other feelings of knowing which are much more common and are not accompanied by conscious awareness. These are evident in the early selection of a strategy for answering a problem. In our view, the research on feeling of knowing will not resolve the question of whether consciousness is merely epiphenomenal.     

Data-driven type checking in open domain question answering

Many open domain question answering systems answer questions by first harvesting a large number of candidate answers, and then picking the most promising one from the list. One criterion for this answer selection is type checking: deciding whether the candidate answer is of the semantic type expected by the question. We define a general strategy for building redundancy-based type checkers, built around the notions of comparison set and scoring method, where the former provide a set of potential answer types and the latter are meant to capture the relation between a candidate answer and an answer type. Our focus is on scoring methods. We discuss nine such methods, provide a detailed experimental comparison and analysis of these methods, and find that the best performing scoring method performs at the same level as knowledge-intensive methods, although our experiments do not reveal a clear-cut answer on the question whether any of the scoring methods we consider should be preferred over the others.     

Do multiplication and division strategies rely on executive and phonological working memory resources?

The role of executive and phonological working memory resources in simple arithmetic was investigated in two experiments. Participants had to solve simple multiplication problems (e.g., 4 x 8; Experiment 1) or simple division problems (e.g., 42 / 7; Experiment 2) under no-load, phonological-load, and executive-load conditions. The choice/no-choice method was used to investigate strategy execution and strategy selection independently. Results for strategy execution showed that executive working memory resources were involved in direct memory retrieval of both multiplication and division facts. Executive working memory resources were also involved in the use of nonretrieval strategies. Phonological working memory resources, on the other hand, tended to be involved in nonretrieval strategies only. Results for strategy selection showed no effects of working memory load. Finally, correlation analyses showed that both strategy execution and strategy selection correlated with individual-difference variables, such as gender, math anxiety, associative strength, calculator use, arithmetic skill, and math experience.     

Cognitive niches: an ecological model of strategy selection

How do people select among different strategies to accomplish a given task? Across disciplines, the strategy selection problem represents a major challenge. We propose a quantitative model that predicts how selection emerges through the interplay among strategies, cognitive capacities, and the environment. This interplay carves out for each strategy a cognitive niche, that is, a limited number of situations in which the strategy can be applied, simplifying strategy selection. To illustrate our proposal, we consider selection in the context of 2 theories: the simple heuristics framework and the ACT-R (adaptive control of thought-rational) architecture of cognition. From the heuristics framework, we adopt the thesis that people make decisions by selecting from a repertoire of simple decision strategies that exploit regularities in the environment and draw on cognitive capacities, such as memory and time perception. ACT-R provides a quantitative theory of how these capacities adapt to the environment. In 14 simulations and 10 experiments, we consider the choice between strategies that operate on the accessibility of memories and those that depend on elaborate knowledge about the world. Based on Internet statistics, our model quantitatively predicts people's familiarity with and knowledge of real-world objects, the distributional characteristics of the associated speed of memory retrieval, and the cognitive niches of classic decision strategies, including those of the fluency, recognition, integration, lexicographic, and sequential-sampling heuristics. In doing so, the model specifies when people will be able to apply different strategies and how accurate, fast, and effortless people's decisions will be.     

Answering yes-no questions about causes: Question acts and question categories

Three experiments examined the cognitive process of answering yes-no questions about causes. Singer’s VAIL model of question answering predicted that readers would take longer to correctly answer “no” than “don’t know” to such questions. In Experiment 1, the antecedent sentences used either the causal conjunction so orbecause. Experiment 2 compared so with an implicit causal link. In all conditions, the main prediction was strongly supported. However, when the questions referred to brief stories in Experiment 3, correct “no” and “don’t know” response latencies did not differ. It was concluded that (1) VAIL identifies the cognitive operations underlying the answering of causal questions; (2) answering yes-no questions about causes resembles answering yes-no questions about case-filling elements; (3) the yes-no versus wh- distinction is orthogonal to the type of relation asked about; and (4) studying question answering about sentences will contribute to the understanding of question answering about text.     

THE EFFECTS OF FORMING MULTI-PREDICTOR COMPOSITES ON GROUP DIFFERENCES AND ADVERSE IMPACT

A common assumption exists which asserts that the formation of composites of predictors represents a method for dealing with adverse impact. It is often expected that including predictors that demonstrate smaller group differences with others that demonstrate larger group differences will help to alleviate the amount of adverse impact observed at the composite level. The purpose of this paper is to answer the question “If two or more predictors are combined to form a composite, what will be the magnitude of group differences and, consequently, of adverse impact, of using that composite for selection?” In answering this question, a set of tables, figures, and formulas are presented that highlight variables influential in affecting how composites of predictors influence observed group differences. A number of conclusions are drawn that clarify the extent to which forming composites decreases group differences and subsequently adverse impact.     

Strategy and Misdirection in Forced Choice Memory Performance Testing in Deception Detection

We examined forced choice memory performance testing in deception detection from a theoretical perspective. Evidence suggests that participants form different strategies to defeat this test. We attempted to describe these strategies within the framework of Cognitive Hierarchy Theory, a theory that distinguishes strategies based on their degree of anticipation of opponents' strategies. Additionally, we explored whether the strategy selection process is malleable. Truth tellers and liars were subjected to a forced choice memory test about a mock crime. Additionally, half of the sample was subjected to a misdirection changing the appearance of the test to that of a polygraph examination. We found detection accuracies and strategies similar to previous experiments and our misdirection manipulation elicited new strategies and behaviour. Theoretical and practical applications are discussed.Copyright © 2017 John Wiley & Sons, Ltd.     

Sex differences in navigation strategy and efficiency

Research on human navigation has indicated that males and females differ in self-reported navigation strategy as well as objective measures of navigation efficiency. In two experiments, we investigated sex differences in navigation strategy and efficiency using an objective measure of strategy, the dual-solution paradigm (DSP; Marchette, Bakker, & Shelton, 2011). Although navigation by shortcuts and learned routes were the primary strategies used in both experiments, as in previous research on the DSP, individuals also utilized route reversals and sometimes found the goal location as a result of wandering. Importantly, sex differences were found in measures of both route selection and navigation efficiency. In particular, males were more likely to take shortcuts and reached their goal location faster than females, while females were more likely to follow learned routes and wander. Self-report measures of strategy were only weakly correlated with objective measures of strategy, casting doubt on their usefulness. This research indicates that the sex difference in navigation efficiency is large, and only partially related to an individual’s navigation strategy as measured by the dual-solution paradigm.     

Getting from neuron to checkmark: models and methods in cognitive survey research

Since the 1980s much work has been done in the field of Cognitive Survey Research. In an interdisciplinary endeavour, survey methodologists and cognitive psychologists (as well as social psychologists and linguists) have worked to unravel the cognitive processes underlying survey responses: to improve survey measurement, but also to obtain fundamental insight into the process of question answering, the nature of attitudes, the functioning of human memory, etc. Yet, despite the amount of work that has been done, less progress has been made than was deemed possible. This paper suggests ways in which to develop cognitive theories that may help to obtain an integrated understanding of question answering in surveys. One way to build better theories is to focus on model construction. Different types of cognitive models are discussed that can be used to model question answering. Second, a larger variety of methods from the cognitive toolbox can be used to further develop and test these models. An overview is given of the various tools available to researchers of cognitive aspects of survey research, with a special focus on newer methods from cognitive neuroscience. To connect different tools and methods into an integrative theory of question answering, the idea of hierarchical modelling is proposed. Copyright © 2007 John Wiley & Sons, Ltd.     

让自适应测验更知人善选——基于推荐系统的选题策略

基于推荐系统中协同过滤推荐的思想, 提出两种可以利用已有答题者数据的CAT选题策略：直接基于答题者推荐(DEBR)和间接基于答题者推荐(IEBR)。通过两个模拟研究, 在不同题库和不同长度的测验中, 比较了两种推荐选题策略与两种传统选题策略(FMI和BAS)在测量精度和对题目曝光率控制上的表现, 以及影响推荐选题策略表现的因素。结果发现：两种推荐选题策略对题目曝光率的控制优于两种传统选题策略, 测量精度不亚于BAS方法, 其中DEBR侧重选题精度, IEBR对题目曝光率控制最好。已有答题者数据的特点和质量是影响推荐选题策略表现的主要因素。     

Unpriming: the deactivation of thoughts through expression

Unpriming is a decrease in the influence of primed knowledge following a behavior expressing that knowledge. The authors investigated strategies for unpriming the knowledge of an answer that is activated when people are asked to consider a simple question. Experiment 1 found that prior correct answering eliminated the bias people normally show toward correct responding when asked to answer yes-no questions randomly. Experiment 2 revealed that prior answering intended to be random did not unprime knowledge on subsequent attempts to answer randomly. Experiment 3 found that exposure to the correct answer did not influence the knowledge bias but that exposure to the incorrect answer increased bias. Experiment 4 revealed that merely expressing the answer for oneself was sufficient to unprime knowledge. Experiment 5 found that each item of activated knowledge needs to be unprimed specifically, in that correctly answering 1 question does not reduce the knowledge bias in randomly answering another.     

A symposium: Why are the mentally retarded strategically deficient?

An overview is given for symposium papers originally presented at the Gatlinburg Conference on Research and Theory in Mental Retardation and Developmental Disabilities. Bray and Turner, in answering the symposium question “Why are the mentally retarded strategically deficient?” develop the position that situation-related variables and a person's comprehension of the “problem of remembering” leads to “production anomalies” rather than “strategy deficiencies.” Borkowski, Carr, and Pressley develop a case for deficiencies in metacognitive and motivational components as the reason for strategy deficiencies. Turnure attributes strategy deficiencies to impoverished learning of “situated meaning,” normally acquired as a result of informal instruction. The discussants, Belmont and Mitchell, provide an integration of these papers and discuss the influence of perceived task difficulty on strategy use.     

The face that launched a thousand ships: the mating-warring association in men

Questions about origins of human warfare continue to generate interesting theories with little empirical evidence. One of the proposed explanations is sexual selection theory. Within and supportive of this theoretical framework, the authors demonstrate a mating-warring association among young heterosexual men in four experiments. Male, but not female, participants exposed to attractive, as compared to unattractive, opposite-sex photographs were significantly more likely to endorse war-supporting statements on a questionnaire. The same mating effect was not found in answering trade conflict questions. Male participants primed by attractive faces or legs of young women were significantly faster in responding to images or words of war than those primed by unattractive faces or national flags. The same mating effect was not found in responding to farming concepts or general aggression expressions. Results underscore the link between mating and war, supporting the view that sexual selection provides an ultimate explanation for the origins of human warfare.     

CAT选题策略分类概述

选题是计算机化自适应测验（CAT）测试过程的关键环节,选题策略的目标是要达到较高的测量精度,同时也实现试题曝光率控制及其他测验目标的实现.本文根据选题策略的基本原理和衍生发展,将众多CAT选题策略分为五大选题策略系列：Fisher函数系列、K-LI函数系列、α分层系列、贝叶斯系列、b匹配系列;并根据测验目标（测验精度、试题曝光率控制、内容平衡、多条件约束）对这些选题策略进行了细分,并对CAT选题策略的选择思路进行归纳.     

Effects of task on the perceptual organization of sentences

Children and college students listened to sentences and responded to a question or a probe word after each sentence. To prevent Ss from using different perceptual strategies in performing the two tasks, question items and probe items were presented in mixed lists. In the probe task, the relationship between sentence structure and correct response latencies resembled the findings of an earlier study with block lists. The results from the question task were less similar to those obtained earlier, perhaps because, with mixed lists, Ss tended to organize all sentences in a manner especially suited to the probe task. A different type of organization might be used in the process of comprehending a sentence solely for the purpose of answering questions.     

SSL: a theory of how people learn to select strategies

The assumption that people possess a repertoire of strategies to solve the inference problems they face has been raised repeatedly. However, a computational model specifying how people select strategies from their repertoire is still lacking. The proposed strategy selection learning (SSL) theory predicts a strategy selection process on the basis of reinforcement learning. The theory assumes that individuals develop subjective expectations for the strategies they have and select strategies proportional to their expectations, which are then updated on the basis of subsequent experience. The learning assumption was supported in 4 experimental studies. Participants substantially improved their inferences through feedback. In all 4 studies, the best-performing strategy from the participants' repertoires most accurately predicted the inferences after sufficient learning opportunities. When testing SSL against 3 models representing extensions of SSL and against an exemplar model assuming a memory-based inference process, the authors found that SSL predicted the inferences most accurately.     

Decision making with the "adaptive toolbox": influence of environmental structure,intelligence, and working memory load

In 2 experiments with a total of 220 participants, the tendency to use simple heuristics such as the take the best heuristic in an adaptive manner was investigated. In a simulated stock market paradigm, the payoff structure of environments was varied, favoring either compensatory or noncompensatory decision strategies in terms of expected long-term payoff. In both experiments, the majority of participants were classified as using strategies that were adequate for the environment, supporting the notion of adaptive strategy selection. These strategy shifts were moderated by intelligence, as measured with common tests. Neither an additional learning phase (Experiment 1) nor working memory load or working memory capacity (Experiment 2) had additional effects on strategy selection.