Do humans produce the speed–accuracy trade-off that maximizes reward rate?

In this paper we investigate trade-offs between speed and accuracy that are produced by humans when confronted with a sequence of choices between two alternatives. We assume that the choice process is described by the drift diffusion model, in which the speed–accuracy trade-off is primarily controlled by the value of the decision threshold. We test the hypothesis that participants choose the decision threshold that maximizes reward rate, defined as an average number of rewards per unit of time. In particular, we test four predictions derived on the basis of this hypothesis in two behavioural experiments. The data from all participants of our experiments provide support only for some of the predictions, and on average the participants are slower and more accurate than predicted by reward rate maximization. However, when we limit our analysis to subgroups of 30–50% of participants who earned the highest overall rewards, all the predictions are satisfied by the data. This suggests that a substantial subset of participants do select decision thresholds that maximize reward rate. We also discuss possible reasons why the remaining participants select thresholds higher than optimal, including the possibility that participants optimize a combination of reward rate and accuracy or that they compensate for the influence of timing uncertainty, or both.     

Do not fear your opponent: suboptimal changes of a prevention strategy when facing stronger opponents

The time spent making a decision and its quality define a widely studied trade-off. Some models suggest that the time spent is set to optimize reward, as verified empirically in simple-decision making experiments. However, in a more complex perspective compromising components of regulation focus, ambitions, fear, risk and social variables, adjustment of the speed-accuracy trade-off may not be optimal. Specifically, regulatory focus theory shows that people can be set in a promotion mode, where focus is on seeking to approach a desired state (to win), or in a prevention mode, focusing to avoid undesired states (not to lose). In promotion, people are eager to take risks increasing speed and decreasing accuracy. In prevention, strategic vigilance increases, decreasing speed and improving accuracy. When time and accuracy have to be compromised, one can ask which of these 2 strategies optimizes reward, leading to optimal performance. This is investigated here in a unique experimental environment. Decision making is studied in rapid-chess (180 s per game), in which the goal of a player is to mate the opponent in a finite amount of time or, alternatively, time-out of the opponent with sufficient material to mate. In different games, players face strong and weak opponents. It was observed that (a) players adopt a more conservative strategy when facing strong opponents, with slower and more accurate moves, and (b) this strategy is suboptimal: Players increase their winning likelihood against strong opponents using the policy they adopt when confronting opponents with similar strength.     

Optimal classifier feedback improves cost-benefit but not base-rate decision criterion learning in perceptual categorization

Unequal payoffs engender separate reward- and accuracy-maximizing decision criteria; unequal base rates do not. When payoffs are unequal, observers place greater emphasis on accuracy than is optimal. This study compares objective classifier (the objectively correct response) with optimal classifier feedback (the optimal classifier's response) when payoffs or base rates are unequal. It provides a critical test of Maddox and Bohil's (1998) competition between reward and accuracy maximization (COBRA) hypothesis, comparing it with a competition between reward and probability matching (COBRM) and a competition between reward and equal response frequencies (COBRE) hypothesis. The COBRA prediction that optimal classifier feedback leads to better decision criterion leaning relative to objective classifier feedback when payoffs are unequal, but not when base rates are unequal, was supported. Model-based analyses suggested that the weight placed on accuracy was reduced for optimal classifier feedback relative to objective classifier feedback. In addition, delayed feedback affected learning of the reward-maximizing decision criterion.     

Toward a unified theory of decision criterion learning in perceptual categorization

Optimal decision criterion placement maximizes expected reward and requires sensitivity to the category base rates (prior probabilities) and payoffs (costs and benefits of incorrect and correct responding). When base rates are unequal, human decision criterion is nearly optimal, but when payoffs are unequal, suboptimal decision criterion placement is observed, even when the optimal decision criterion is identical in both cases. A series of studies are reviewed that examine the generality of this finding, and a unified theory of decision criterion learning is described (Maddox & Dodd, 2001). The theory assumes that two critical mechanisms operate in decision criterion learning. One mechanism involves competition between reward and accuracy maximization: The observer attempts to maximize reward, as instructed, but also places some importance on accuracy maximization. The second mechanism involves a flat-maxima hypothesis that assumes that the observer's estimate of the reward-maximizing decision criterion is determined from the steepness of the objective reward function that relates expected reward to decision criterion placement. Experiments used to develop and test the theory require each observer to complete a large number of trials and to participate in all conditions of the experiment. This provides maximal control over the reinforcement history of the observer and allows a focus on individual behavioral profiles. The theory is applied to decision criterion learning problems that examine category discriminability, payoff matrix multiplication and addition effects, the optimal classifier's independence assumption, and different types of trial-by-trial feedback. In every case the theory provides a good account of the data, and, most important, provides useful insights into the psychological processes involved in decision criterion learning.     

Children's Sensitivity to Cost and Reward in Decision Making across Distinct Domains of Probability,Effort, and Delay

Many behavioral paradigms used to study individuals' decision‐making tendencies do not capture the decision components that contribute to behavioral outcomes, such as differentiating decisions driven toward a reward from decisions driven away from a cost. This study tested a novel decision‐making task in a sample of 403 children (age 9 years) enrolled in an ongoing longitudinal study. The task consisted of three blocks representing distinct cost domains (delay, probability, and effort), wherein children were presented with a deck of cards, each of which consisted of a reward and a cost. Children elected whether to accept or skip the card at each trial. Reward–cost pairs were selected by using an adaptive algorithm to strategically sample the decision space in the fewest number of trials. Using person‐specific regression models, decision preferences were quantified for each cost domain with respect to general tolerance (intercept), as well as parameters estimating the effect of incremental increases in reward or cost on the probability of accepting a card. Results support the relative independence of decision‐making tendencies across cost domains, with moderate correlations observed between tolerance for delay and effort. Specific decision parameters showed unique associations with cognitive and behavioral measures including executive function, academic motivation, anxiety, and hyperactivity. Evidence indicates that sensitivity to reward is an important factor in incentivizing decisions to work harder or wait longer. Dissociating the relative contributions of reward and cost sensitivity in multiple domains may facilitate the identification of heterogeneity in suboptimal decision making. Copyright © 2017 John Wiley & Sons, Ltd.     

Acquisition of decision making criteria: reward rate ultimately beats accuracy

Speed-accuracy trade-offs strongly influence the rate of reward that can be earned in many decision-making tasks. Previous reports suggest that human participants often adopt suboptimal speed-accuracy trade-offs in single session, two-alternative forced-choice tasks. We investigated whether humans acquired optimal speed-accuracy trade-offs when extensively trained with multiple signal qualities. When performance was characterized in terms of decision time and accuracy, our participants eventually performed nearly optimally in the case of higher signal qualities. Rather than adopting decision criteria that were individually optimal for each signal quality, participants adopted a single threshold that was nearly optimal for most signal qualities. However, setting a single threshold for different coherence conditions resulted in only negligible decrements in the maximum possible reward rate. Finally, we tested two hypotheses regarding the possible sources of suboptimal performance: (1) favoring accuracy over reward rate and (2) misestimating the reward rate due to timing uncertainty. Our findings provide support for both hypotheses, but also for the hypothesis that participants can learn to approach optimality. We find specifically that an accuracy bias dominates early performance, but diminishes greatly with practice. The residual discrepancy between optimal and observed performance can be explained by an adaptive response to uncertainty in time estimation.     

Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates

Recent work has given rise to the view that reward-based decision making is governed by two key controllers: a habit system, which stores stimulus-response associations shaped by past reward, and a goal-oriented system that selects actions based on their anticipated outcomes. The current literature provides a rich body of computational theory addressing habit formation, centering on temporal-difference learning mechanisms. Less progress has been made toward formalizing the processes involved in goal-directed decision making. We draw on recent work in cognitive neuroscience, animal conditioning, cognitive and developmental psychology, and machine learning to outline a new theory of goal-directed decision making. Our basic proposal is that the brain, within an identifiable network of cortical and subcortical structures, implements a probabilistic generative model of reward, and that goal-directed decision making is effected through Bayesian inversion of this model. We present a set of simulations implementing the account, which address benchmark behavioral and neuroscientific findings, and give rise to a set of testable predictions. We also discuss the relationship between the proposed framework and other models of decision making, including recent models of perceptual choice, to which our theory bears a direct connection.     

Sensitivity to reward and risky driving,risky decision making,and risky health behaviour: A literature review

Young driver road safety has persisted as a global problem for decades, despite copious and diverse intervention. Recently the influence in reward sensitivity, which refers to the individual’s personal sensitivity to rewards, has received attention in health-related research, including more generally through decision making in risky circumstances, and in risky driving behaviour specifically. As such, a literature review and synthesis of the literature regarding reward sensitivity in relation to risky driving, risky decision making, and risky health behaviour, with a focus on literature in which adolescents and young adults feature, is timely. Thirty-one papers were identified, and the literature revealed that young drivers with greater reward sensitivity engage in more risky driving behaviours including speeding, crashes and traffic violations; and that individuals with greater reward sensitivity engage in more risky decision making and other risky health-related behaviours (such as drinking and drug use). Adolescents and young adults exhibit heightened sensitivity to rewards in the presence of peers, which has considerable implications for young driver road safety as research consistently demonstrates that carrying peer passengers places all vehicle occupants at greater risk of being involved in a road crash. Consideration of the influence of reward sensitivity in young driver road safety, and other adolescent/young adult health-related safety, appears to be a promising avenue of intervention, with gain-framed messages more likely to be accepted by young drivers with greater reward sensitivity. Future research in jurisdictions other than Australia and Europe will increase our understanding of the influence of reward sensitivity, and exploration of the differential impacts of reward-responsiveness and fun-seeking specifically are warranted.     

Eye blink rate predicts reward decisions in adolescents

The ventral striatum displays hyper‐responsiveness to reward in adolescents relative to other age groups, and animal research on the developmental trajectory of the dopaminergic system suggests that dopamine may underlie adolescent sensitivity to reward. However, practical limitations prevent the direct measurement of dopamine in healthy adolescents. Eye blink rate (EBR) shows promise as a proxy measure of striatal dopamine D2 receptor function. We investigated developmental differences in the relationship between EBR and reward‐seeking behavior on a risky decision‐making task. Increasing EBR was associated with greater reward maximization on the task for adolescent but not adult participants. Furthermore, adolescents demonstrated greater sensitivity to reward value than adults, as evinced by shifts in decision patterns based on increasing potential reward. These findings suggest that previously observed adolescent behavioral and neural hypersensitivity to reward may in fact be due to greater dopamine receptor activity, as represented by the relationship of blink rate and reward‐seeking behavior. They also demonstrate the feasibility and utility of using EBR as a proxy for dopamine in healthy youth in whom direct measurements of dopamine are prohibitively invasive.     

Optimal decision-making theories: linking neurobiology with behaviour

This article reviews recently proposed theories postulating that, during simple choices, the brain performs statistically optimal decision making. These theories are ecologically motivated by evolutionary pressures to optimize the speed and accuracy of decisions and to maximize the rate of receiving rewards for correct choices. This article suggests that the models of decision making that are proposed on different levels of abstraction can be linked by virtue of the same optimal computation. Also reviewed here are recent observations that many aspects of the circuit that involves the cortex and basal ganglia are the same as those that are required to perform statistically optimal choice. This review illustrates how optimal-decision theories elucidate current data and provide experimental predictions that concern both neurobiology and behaviour.     

Behavioral and neural predictors of upcoming decisions

Although it is widely known that brain regions such as the prefrontal cortex, the amygdala, and the ventral striatum play large roles in decision making, their precise contributions remain unclear. Here, we used functional magnetic resonance imaging and principles of reinforcement learning theory to investigate the relationship between current reinforcements and future decisions. In the experiment, subjects chose between high-risk (i.e., low probability of a large monetary reward) and low-risk (high probability of a small reward) decisions. For each subject, we estimated value functions that represented the degree to which reinforcements affected the value of decision options on the subsequent trial. Individual differences in value functions predicted not only trial-to-trial behavioral strategies, such as choosing high-risk decisions following high-risk rewards, but also the relationship between activity in prefrontal and subcortical regions during one trial and the decision made in the subsequent trial. These findings provide a novel link between behavior and neural activity by demonstrating that value functions are manifested both in adjustments in behavioral strategies and in the neural activity that accompanies those adjustments.     

Ostracism Reduces Reliance on Poor Advice from Others during Decision Making

Decision making is rarely context‐free, and often, both social information and non‐social information are weighed into one's decisions. Incorporating information into a decision can be influenced by previous experiences. Ostracism has extensive effects, including taxing cognitive resources and increasing social monitoring. In decision making situations, individuals are often faced with both objective and social information and must choose which information to include or filter out. How will ostracism affect the reliance on objective and social information during decision making? Participants (N = 245) in Experiment 1 were randomly assigned to be included or ostracized in a standardized, group task. They then performed a dynamic decision making task that involved the presentation of either non‐social (i.e. biased reward feedback) or social (i.e., poor advice from a previous participant) misleading information. In Experiment 2, participants (N = 105) completed either the ostracism non‐social condition or social misleading information condition with explicit instructions stating that the advice given was from an individual who did not partake in the group task. Ostracized individuals relied more on non‐social misleading information and performed worse than included individuals. However, ostracized individuals discounted misleading social information and outperformed included individuals. Results of Experiment 2 replicated the findings of Experiment 1. Across two experiments, ostracized individuals were more critical of advice from others, both individuals who may have ostracized them and unrelated individuals. In other words, compared with included individuals, ostracized individuals underweighted advice from another individual but overweighed non‐social information during decision making. We conclude that when deceptive objective information is present, ostracism results in disadvantageous decision making. Copyright © 2015 John Wiley & Sons, Ltd.     

Rhesus monkeys lack a consistent peak-end effect

In humans, the order of receiving sequential rewards can significantly influence the overall subjective utility of an outcome. For example, people subjectively rate receiving a large reward by itself significantly higher than receiving the same large reward followed by a smaller one (Do, Rupert, & Wolford, 2008). This result is called the peak-end effect. A comparative analysis of order effects can help determine the generality of such effects across primates, and we therefore examined the influence of reward-quality order on decision making in three rhesus macaque monkeys (Macaca mulatta). When given the choice between a high-low reward sequence and a low-high sequence, all three monkeys preferred receiving the high-value reward first. Follow-up experiments showed that for two of the three monkeys their choices depended specifically on reward-quality order and could not be accounted for by delay discounting. These results provide evidence for the influence of outcome order on decision making in rhesus monkeys. Unlike humans, who usually discount choices when a low-value reward comes last, rhesus monkeys show no such peak-end effect.     

Optimal correction for guessing in multiple-choice tests

Subjects’ decisions in multiple-choice tests are an interesting domain for the analysis of decision making under uncertainty. When the test is graded using a rule that penalizes wrong answers, each item can be viewed as a lottery where a rational examinee would choose whether to omit (sure reward) or answer (take the lottery) depending on risk aversion and level of knowledge. We formalize students as heterogeneous decision makers with different risk attitudes and levels of knowledge. Building on IRT, we compute the optimal penalty given students’ optimal behavior and the trade-off between bias and measurement error. Although MCQ examinations are frequently used, there is no consensus as to whether a penalty for wrong answers should be used or not. For example, examinations for medical licensing in some countries include MCQ sections with penalty while in others there is no penalty for wrong answers. We contribute to this discussion with a formal analysis of the effects of penalties; our simulations indicate that the optimal penalty is relatively high for perfectly rational students but also when they are not fully rational: even though penalty discriminates against risk averse students, this effect is small compared with the measurement error that it prevents.     

Judgment and decision making: Behavioral approaches

The area of judgment and decision making has given rise to the study of many interesting phenomena, including reasoning fallacies, which are also of interest to behavior analysts. Indeed, techniques and principles of behavior analysis may be applied to study these fallacies. This article reviews research from a behavioral perspective that suggests that humans are not the information-seekers we sometimes suppose ourselves to be. Nor do we utilize information effectively when it is presented. This is shown from the results of research utilizing matching to sample and other behavioral tools (monetary reward, feedback, instructional control) to study phenomena such as the conjunction fallacy, base-rate neglect, and probability matching. Research from a behavioral perspective can complement research from other perspectives in furthering our understanding of judgment and decision making.     

On the relation between base-rate and cost-benefit learning in simulated medical diagnosis.

Observers completed a series of simulated medical diagnosis tasks that differed in category discriminability and base-rate/cost-benefit ratio. Point, accuracy, and decision criterion estimates were closer to optimal (a) for category d' = 2.2 than for category d' = 1.0 or 3.2, (b) when base-rates as opposed to cost-benefits were manipulated, and (c) when the cost of an incorrect response resulted in no point loss (nonnegative cost) as opposed to a point loss (negative cost). These results support the "flat-maxima" and competition between reward and accuracy (COBRA) hypotheses. A hybrid model that instantiated simultaneously both hypotheses was applied to the data. The model parameters indicated that (a) the reward-maximizing decision criterion quickly approached the optimal criterion, (b) the importance placed on accuracy maximization early in learning was larger when the cost of an incorrect response was negative as opposed to nonnegative, and (c) by the end of training the importance placed on accuracy was equal for negative and nonnegative costs.     

On the generality of optimal versus objective classifier feedback effects on decision criterion learning in perceptual categorization

Biased category payoff matrices engender separate reward- and accuracy-maximizing decision criteria Although instructed to maximize reward, observers use suboptimal decision criteria that place greater emphasis on accuracy than is optimal. In this study, objective classifier feedback (the objectively correct response) was compared with optimal classifier feedback (the optimal classifier's response) at two levels of category discriminability when zero or negative costs accompanied incorrect responses for two payoff matrix multiplication factors. Performance was superior for optimal classifier feedback relative to objective classifier feedback for both zero- and negative-cost conditions, especially when category discriminability was low, but the magnitude of the optimal classifier advantage was approximately equal for zero- and negative-cost conditions. The optimal classifier feedback performance advantage did not interact with the payoff matrix multiplication factor. Model-based analyses suggested that the weight placed on accuracy was reduced for optimal classifier feedback relative to objective classifier feedback and for high category discriminability relative to low category discriminability. In addition, the weight placed on accuracy declined with training when feedback was based on the optimal classifier and remained relatively stable when feedback was based on the objective classifier. These results suggest that feedback based on the optimal classifier leads to superior decision criterion learning across a wide range of experimental conditions.     

Age-related differences in diffusion model boundary optimality with both trial-limited and time-limited tasks

In two-choice decision tasks, Starns and Ratcliff (Psychology and Aging 25: 377–390, 2010) showed that older adults are farther from the optimal speed–accuracy trade-off than young adults. They suggested that the age effect resulted from differences in task goals, with young participants focused on balancing speed and accuracy and older participants focused on minimizing errors. We compared speed–accuracy criteria with a standard procedure (blocks that had a fixed numbers of trials) to a condition in which blocks lasted a fixed amount of time and participants were instructed to get as many correct responses as possible within the time limit—a goal that explicitly required balancing speed and accuracy. Fits of the diffusion model showed that criteria differences persisted in the fixed-time condition, suggesting that age differences are not solely based on differences in task goals. Also, both groups produced more conservative criteria in difficult conditions when it would have been optimal to be more liberal.     

Integrative processing of perception and reward in an auditory localization paradigm

Under natural conditions, human beings routinely have to choose among multiple alternatives which are associated with specific outcomes of varying desirability. Typically, decisions are based upon the processing of perceptual input, which introduces additional noise to the system. Bayesian decision theory (BDT) allows us to formalize decision under risk and to predict statistically optimal choice behavior. In the present study, human observers performed a classification task characterized by an extensive amount of perceptual uncertainty (auditory localization). In addition, a spatial reward function was imposed on the task. We set up a BDT model with no free parameters to serve as a benchmark for statistically optimal choices, and tested it against a purely perceptual model and a hybrid, heuristic model. In addition, we tested these three models with free rather than fixed parameters for the perceptual uncertainty and the peak of the reward function. The log likelihoods of the models given the empirical data were determined by means of Monte Carlo simulations. Bayesian model comparison (BMC) revealed that the BDT model with two free parameters was the most plausible among the tested models. The fitted parameter values for the peak of the reward function were consistently smaller than the actual peak reward communicated to the participants. The results are discussed in the context of an internal underweighting of the reward function.     

Effort responses to suboptimal reward cues are related to striatal dopaminergic functioning

Reward cues have been found to increase the investment of effort in tasks even when cues are presented suboptimally (i.e. very briefly), making them hard to consciously detect. Such effort responses to suboptimal reward cues are assumed to rely mainly on the mesolimbic dopamine system, including the ventral striatum. To provide further support for this assumption, we performed two studies investigating whether these effort responses vary with individual differences in markers of striatal dopaminergic functioning. Study 1 investigated the relation between physical effort responses and resting state eye-blink rate. Study 2 examined cognitive effort responses in relation to individually averaged error-related negativity. In both studies effort responses correlated with the markers only for suboptimal, but not for optimal reward cues. These findings provide further support for the idea that effort responses to suboptimal reward cues are mainly linked to the mesolimbic dopamine system, while responses to optimal reward cues also depend on higher-level cortical functions.