Short-term gains, long-term pains: How cues about state aid learning in dynamic environments

Successful investors seeking returns, animals foraging for food, and pilots controlling aircraft all must take into account how their current decisions will impact their future standing. One challenge facing decision makers is that options that appear attractive in the short-term may not turn out best in the long run. In this paper, we explore human learning in a dynamic decision making task which places short- and long-term rewards in conflict. Our goal in these studies was to evaluate how people’s mental representation of a task affects their ability to discover an optimal decision strategy. We find that perceptual cues that readily align with the underlying state of the task environment help people overcome the impulsive appeal of short-term rewards. Our experimental manipulations, predictions, and analyses are motivated by current work in reinforcement learning which details how learners value delayed outcomes in sequential tasks and the importance that “state” identification plays in effective learning.     

Instance‐based learning in dynamic decision making

This paper presents a learning theory pertinent to dynamic decision making (DDM) called instancebased learning theory (IBLT). IBLT proposes five learning mechanisms in the context of a decision‐making process: instance‐based knowledge, recognition‐based retrieval, adaptive strategies, necessity‐based choice, and feedback updates. IBLT suggests in DDM people learn with the accumulation and refinement of instances, containing the decision‐making situation, action, and utility of decisions. As decision makers interact with a dynamic task, they recognize a situation according to its similarity to past instances, adapt their judgment strategies from heuristic‐based to instance‐based, and refine the accumulated knowledge according to feedback on the result of their actions. The IBLT's learning mechanisms have been implemented in an ACT‐R cognitive model. Through a series of experiments, this paper shows how the IBLT's learning mechanisms closely approximate the relative trend magnitude and performance of human data. Although the cognitive model is bounded within the context of a dynamic task, the IBLT is a general theory of decision making applicable to other dynamic environments.     

Of matchers and maximizers: How competition shapes choice under risk and uncertainty

In a world of limited resources, scarcity and rivalry are central challenges for decision makers—animals foraging for food, corporations seeking maximal profits, and athletes training to win, all strive against others competing for the same goals. In this article, we establish the role of competitive pressures for the facilitation of optimal decision making in simple sequential binary choice tasks. In two experiments, competition was introduced with a computerized opponent whose choice behavior reinforced one of two strategies: If the opponent probabilistically imitated participant choices, probability matching was optimal; if the opponent was indifferent, probability maximizing was optimal. We observed accurate asymptotic strategy use in both conditions irrespective of the provision of outcome probabilities, suggesting that participants were sensitive to the differences in opponent behavior. An analysis of reinforcement learning models established that computational conceptualizations of opponent behavior are critical to account for the observed divergence in strategy adoption. Our results provide a novel appraisal of probability matching and show how this individually ‘irrational’ choice phenomenon can be socially adaptive under competition.     

Feedback Delays: How Can Decision Makers Learn Not to Buy a New Car Every Time the Garage Is Empty?

Decision makers in dynamic environments (e.g., stock trading, inventory control, and firefighting) learn poorly in experiments where feedback about the outcomes of their actions is delayed. In searching for ways to mitigate these effects, this paper presents two computational models of learning with feedback delays and contrasts them against human decision-makers' performance. The no-memory model hypothesizes that decision makers always perceive feedback as immediate. The with-memory model hypothesizes that, over time, decision makers are able to develop internal representations of the task that help them to perform with delayed feedback. As borne out by human subjects, both models predict that a display of past history improves learning with delay and that increasing delay increasingly degrades performance. Even though the length of training in this task exceeds that used in many laboratory-based dynamic tasks, neither the two models nor the subjects are able to effectively learn without decision aids when faced with feedback delays. When given an amount of training that more closely approximates that provided in functioning dynamic environments, the with-memory model predicts that human decision makers may learn without decision aids over the long term if feedback delays are simple. These results raise several issues for continued theoretical investigation as well as potential suggestions for training and supporting decision makers in dynamic environments with feedback delays. Copyright 2000 Academic Press.     

Learning in noise: Dynamic decision-making in a variable environment

In engineering systems, noise is a curse, obscuring important signals and increasing the uncertainty associated with measurement. However, the negative effects of noise are not universal. In this paper, we examine how people learn sequential control strategies given different sources and amounts of feedback variability. In particular, we consider people’s behavior in a task where short- and long-term rewards are placed in conflict (i.e., the best option in the short-term is worst in the long-term). Consistent with a model based on reinforcement learning principles [Gureckis, T., & Love, B.C. Short term gains, long term pains: How cues about state aid learning in dynamic environments. Cognition (in press)], we find that learners differentially weight information predictive of the current task state. In particular, when cues that signal state are noisy, we find that participants’ ability to identify an optimal strategy is strongly impaired relative to equivalent amounts of noise that obscure the rewards/valuations of those states. In other situations, we find that noise and noise in reward signals may paradoxically improve performance by encouraging exploration. Our results demonstrate how experimentally-manipulated task variability can be used to test predictions about the mechanisms that learners engage in dynamic decision making tasks.     

Decision support for real-time, dynamic decision-making tasks

By definition, dynamic decision making dictates that multiple and interrelated decisions be made in a continuously changing environment. Such decision making is difficult and often taxes individuals’ cognitive resources. Here I investigated ways in which to support decision making in these environments. I evaluated three forms of decision support: outcome feedback, cognitive feedback, and feedforward that incorporated (to varying degrees) common features of learning theories associated with dynamic tasks. Participants in a laboratory experiment performed a real-time, dynamic decision-making task while receiving one of the different types of decision support. During the first 2 days, individuals received one type of decision support, but on the third day they performed the task without this support. Participants who received feedforward improved their performance considerably and continued to exhibit improved performance even after discontinuation of the decision support on the third day. Neither outcome feedback nor cognitive feedback resulted in improved performance. More research is necessary to conclusively identify the best forms of dynamic decision-making support and their durability when transferred to new tasks.     

Sensitivity to value-driven attention is predicted by how we learn from value

Reward learning is known to influence the automatic capture of attention. This study examined how the rate of learning, after high- or low-value reward outcomes, can influence future transfers into value-driven attentional capture. Participants performed an instrumental learning task that was directly followed by an attentional capture task. A hierarchical Bayesian reinforcement model was used to infer individual differences in learning from high or low reward. Results showed a strong relationship between high-reward learning rates (or the weight that is put on learning after a high reward) and the magnitude of attentional capture with high-reward colors. Individual differences in learning from high or low rewards were further related to performance differences when high- or low-value distractors were present. These findings provide novel insight into the development of value-driven attentional capture by showing how information updating after desired or undesired outcomes can influence future deployments of automatic attention.     

分布式群体在动态系统控制决策中的内隐学习

采用计算机网络模拟技术,考察分布式群体在动态系统控制决策过程中对系统规则的内隐学习效应及其迁移。结果表明,在决策者的显性规则知识量极少且没有增加的情况下,决策绩效随决策进程适应性提高。决策者所能报告的规则知识量与决策绩效的相关也不显著。同时,当决策任务目标转换时,决策的高绩效得以维持。这表明群体决策者能对动态系统规则进行内隐学习,且能正迁移到类似的决策情景中。研究进一步指出,群体决策行为的动态适应与内隐过程有关。     

Type of learning task impacts performance and strategy selection in decision making

In order to be adaptive, cognition requires knowledge about the statistical structure of the environment. We show that decision performance and the selection between cue-based and exemplar-based inference mechanisms can depend critically on how this knowledge is acquired. Two types of learning tasks are distinguished: learning by comparison, by which the decision maker learns which of two objects has a higher criterion value, and direct criterion learning, by which the decision maker learns an object's criterion value directly. In three experiments, participants were trained either with learning by comparison or with direct criterion learning and subsequently tested with paired-comparison, classification, and estimation tasks. Experiments 1 and 2 showed that although providing less information, learning by comparison led to better generalization (at test), both when generalizing to new objects and when the task format at test differed from the task format during training. Moreover, learning by comparison enabled participants to provide rather accurate continuous estimates. Computational modeling suggests that the advantage of learning by comparison is due to differences in strategy selection: whereas direct criterion learning fosters the reliance on exemplar processing, learning by comparison fosters cue-based mechanisms. The pattern in decision performance reversed when the task environment was changed from a linear (Experiments 1 and 2) to a nonlinear structure (Experiment 3), where direct criterion learning led to better decisions. Our results demonstrate the critical impact of learning conditions for the subsequent selection of decision strategies and highlight the key role of comparison processes in cognition.     

An Optimized Design of Choice Experiments: A New Approach for Studying Decision Behavior in Choice Task Experiments

In this paper, we present a new approach for the optimal experimental design problem of generating diagnostic choice tasks, where the respondent's decision strategy can be unambiguously deduced from the observed choice. In this new approach, we applied a genetic algorithm that creates a one‐to‐one correspondence between a set of predefined decision strategies and the alternatives of the choice task; it also manipulates the characteristics of the choice tasks. In addition, this new approach takes into account the measurement errors that can occur when the preferences of the decision makers are being measured. The proposed genetic algorithm is capable of generating diagnostic choice tasks even when the search space of possible choice tasks is very large. As proof‐of‐concept, we used this novel approach to generate respondent‐specific choice tasks with either low or high context‐based complexity that we operationalize by the similarity of alternatives and the conflict between alternatives. We find in an experiment that an increase in the similarity of the alternatives and an increase in the number of conflicts within the choice task lead to an increased use of non‐compensatory strategies and a decreased use of compensatory decision strategies. In contrast, the size of the choice tasks, measured by the number of attributes and alternatives, only weakly influences the strategy selection. Copyright © 2014 John Wiley & Sons, Ltd.     

Thinking about decisions: An integrative approach of person and task factors

Decisions vary. They may vary in both content and complexity. People also vary. An important way that people vary is how much they think. Some prior research investigating thinking and decision making largely conflicts with most traditional decision theories. For example, if considering an array of products to choose from, thinking more about the alternative's attributes should lead to a better decision. However, some research indicates that thinking more may also lead to focusing on irrelevant aspects of the decision and a less optimal outcome. We propose that this conflict in the literature exists because of a failure to consider the interaction between the individual and the decision task. To test this, we used separate methodologies that enhance or attenuate a person's thinking. In Study 1, we selected people who were especially high or low in need for cognition and had them complete a robust decision-making inventory, which included both complex and simple tasks. In Study 2, we manipulated participant's level of glucose, which acts as the brain's fuel to enhance or attenuate thinking ability. Both studies provide insight for understanding our central tenant that more thought leads to better decisions in complex tasks but does not influence simple decisions. These findings show how the individual's thinking can interact with the constructive elements of the task to shape decision choice.     

决策者角色及相关变量对风险偏好的影响

采用IAT技术测查被试的内隐自尊, 并引入收益和损失两种任务框架, 综合探讨了内隐自尊水平、任务得失框架对自我决策和为他人决策时风险偏好的影响。结果发现：(1)在收益框架下, 被试为他人决策时更冒险, 而在损失框架下, 为自我和他人决策的风险偏好无显著差异; (2)与为自我决策相比, 低内隐自尊者为他人决策时更冒险, 而高内隐自尊者为自我和他人决策的风险偏好无显著差异; (3)内隐自尊水平与任务框架的交互作用及内隐自尊水平、任务框架和决策者角色之间的三级交互作用均不显著。依据相关的理论和发现对这种结果进行了分析和讨论。     

Normative and Descriptive Models of Military Decisions to Deploy Precision Strike Capabilities

Precision strike capabilities represent a significant and highly controversial part of present day military operations. And yet, there is a surprising dearth of empirical research on military decision making in this domain. In this article, we therefore review different psychological perspectives on how these decisions can be made. Specifically, we compare the application of normative models of judgment and choice against the empirical research on human decision making, which suggests that people are more likely to employ heuristic strategies. We suggest that several features of decision tasks in the precision strike domain evoke the use of intuitive (heuristic) decision making whereas other features such as the sometimes unfamiliar (or novel) nature of the decision task requires analytic strategies to generate good solutions. Therefore, decisions about precision strike capabilities are best made with a mixture of intuitive and analytic thought, a mode of thinking known as quasirationality.     

Instance‐based Learning: A General Model of Repeated Binary Choice

A common practice in cognitive modeling is to develop new models specific to each particular task. We question this approach and draw on an existing theory, instance‐based learning theory (IBLT), to explain learning behavior in three different choice tasks. The same instance‐based learning model generalizes accurately to choices in a repeated binary choice task, in a probability learning task, and in a repeated binary choice task within a changing environment. We assert that, although the three tasks are different, the source of learning is equivalent and therefore, the cognitive process elicited should be captured by one single model. This evidence supports previous findings that instance‐based learning is a robust learning process that is triggered in a wide range of tasks from the simple repeated choice tasks to the most dynamic decision making tasks. Copyright © 2010 John Wiley & Sons, Ltd.     

专家和新手不确定决策认知过程的比较研究

本研究将风险投资决策作为不确定决策任务,采用专家-新手比较研究范型,通过出声思维法和访谈法,探讨了专家和新手不确定决策认知过程的差异。研究发现,这种差异体现在专家和新手所拥有的关于决策任务的认识和观念、决策信息选择、推理过程和决策策略等方面。     

Reinforcement,Rationality, and Intentions: How Robust Is Automatic Reinforcement Learning in Economic Decision Making?

Reinforcement learning is often observed in economic decision making and may lead to detrimental decisions. Because of its automaticity, it is difficult to avoid. In three experimental studies, we investigated whether this process could be controlled by goal intentions and implementation intentions. Participants' decisions were investigated in a probability‐updating task in which the normative rule to maximize expected payoff (Bayes' rule) conflicted with the reinforcement heuristic as a simple decision rule. Some participants were asked to set goal intentions designated to foster the optimization of rational decision making, while other participants were asked to furnish these goal intentions with implementation intentions. Results showed that controlling automatic processes of reinforcement learning is possible by means of goal intentions or implementation intentions that focus decision makers on the analysis of decision feedback. Importantly, such beneficial effects were not achieved by simply instructing participants to analyze the feedback, without defining a goal as the desired end state from a first‐person perspective. Regarding intentions supposed to shut down reinforcement processes by controlling negative affect, effects were more complex and depended on the specified goal‐directed behavior. The goal intention to suppress the disappointment elicited by negative feedback was not effective in controlling reinforcement processes. Furnishing this goal with an implementation intention even backfired and strengthened unwanted reinforcement processes. In contrast, asking participants to keep cool in response to negative decision outcomes through the use of goal intentions or implementation intentions increased decisions in line with Bayes' rule. Copyright © 2017 John Wiley & Sons, Ltd.     

愿望采择发展水平对幼儿情感决策的影响

选取148名4岁儿童, 运用实验法考察了幼儿的愿望采择发展水平对其情感决策的影响。结果表明: (1) 幼儿的愿望采择水平对其情感决策具有显著影响。情境中他人愿望信息充足时, 幼儿会采择他人愿望并以此为线索为他人决策, 但对线索的利用程度受其愿望采择水平的制约。(2) 在无关于他人愿望信息提示的条件下, 4岁儿童为他人与为自己的情感决策不存在显著差异。(3) 4岁儿童能够采择他人的单一愿望, 但其冲突愿望采择能力还不成熟。     

An effort based analysis of the paradoxical effects of incentives on decision‐aided performance

This study examines the effect of incentives on decision‐aided performance. In particular, the study provides further insight into whether, when, and how incentives affect task performance in the presence of decision aids by (1) replicating previous research showing the negative effects of incentives on performance; (2) investigating whether this effect generalizes to a more realistic scenario in which decision makers have access to additional contextual information not captured by the decision aid; and (3) applying an effort‐based framework to explain the link between incentives and performance. In contrast to the findings of prior research, our study shows that incentives do not necessarily decrease performance in the presence of decision aids. Rather, we demonstrate that the effect of incentives on decision‐aided performance depends on other contextual factors such as the absence or presence of additional contextual information. By further specifying the conditions under which incentives result in increases or decreases to decision‐aided task performance, our results have implications for both future research and the design of incentive systems in practice. Copyright © 2010 John Wiley & Sons, Ltd.     

One Night of Sleep Loss Impairs Innovative Thinking and Flexible Decision Making

Recent findings with clinically oriented neuropsychological tests suggest that one night without sleep causes particular impairment to tasks requiring flexible thinking and the updating of plans in the light of new information. This relatively little investigated field of sleep deprivation research has real-world implications for decision makers having lost a night's sleep. To explore this latter perspective further, we adapted a dynamic and realistic marketing decision making “game” embodying the need for these skills, and whereby such performance could be measured. As the task relied on the comprehension of a large amount of written information, a critical reasoning test was also administered to ascertain whether any failure at the marketing game might lie with information acquisition rather than with failures in decision making. Ten healthy highly motivated and trained participants underwent two counterbalanced 36 h trials, sleep vs no sleep. The critical reasoning task was unaffected by sleep loss, whereas performance at the game significantly deteri orated after 32–36 h of sleep loss, when sleep deprivation led to more rigid thinking, increased perseverative errors, and marked difficulty in appreciating an updated situation. At this point, and despite the sleep-deprived participants' best efforts to do well, their play collapsed, unlike that of the nonsleep-deprived participants.