Evidence for a magnitude effect in temporal discounting with pigeons

A magnitude effect in human intertemporal choice is well established-larger rewards or outcomes are discounted over time at a lower rate than are smaller rewards. However, many recent studies have failed to find a corresponding effect in nonhuman animals. Here we report a magnitude effect in temporal discounting for pigeons' choices involving a tradeoff between reward delay and amount. Pigeons chose between a small reward (1-s access to food) after a 2-s delay, and a large reward (4.5-s access to food) after a 28-s delay. Across conditions, the delays to the small and large rewards were increased or decreased, respectively. Temporal discounting functions obtained through a value-estimation procedure showed clear evidence of a magnitude effect: The value of the large reward decreased more slowly with increasing delay than the value of the small reward. We linked this result to a nonlinear relationship between choice and the delays associated with the small and large rewards. The nonlinearity was contrary to the generalized matching law but was predicted by the contextual choice model. Our results confirm the existence of a magnitude effect in nonhuman temporal discounting, showing that this adaptation is not unique to humans.     

Do humans produce the speed–accuracy trade-off that maximizes reward rate?

In this paper we investigate trade-offs between speed and accuracy that are produced by humans when confronted with a sequence of choices between two alternatives. We assume that the choice process is described by the drift diffusion model, in which the speed–accuracy trade-off is primarily controlled by the value of the decision threshold. We test the hypothesis that participants choose the decision threshold that maximizes reward rate, defined as an average number of rewards per unit of time. In particular, we test four predictions derived on the basis of this hypothesis in two behavioural experiments. The data from all participants of our experiments provide support only for some of the predictions, and on average the participants are slower and more accurate than predicted by reward rate maximization. However, when we limit our analysis to subgroups of 30–50% of participants who earned the highest overall rewards, all the predictions are satisfied by the data. This suggests that a substantial subset of participants do select decision thresholds that maximize reward rate. We also discuss possible reasons why the remaining participants select thresholds higher than optimal, including the possibility that participants optimize a combination of reward rate and accuracy or that they compensate for the influence of timing uncertainty, or both.     

Scaling Incentives: Determining Preference and Indifference in 8- to lO-Year-Old Girls

The present study adopted procedures similar to those used by Logan (1965) to determine choice behavior in children. The objective was to provide an improved methodology in delay-of-gratification studies, thus avoiding the problems associated with scaling different kinds of rewards, and to provide approximate functions relating a delay-of-token reward to choice behavior. Thus, 7 girls aged 8 to 10 years were exposed to a choice paradigm in which a larger reward (2 tokens) was pitted against a smaller reward (1 token); access to these rewards was delayed a certain period of time. The results of this experiment showed that as the delay of the larger reward increased, preference for the smaller reward increased in an orderly fashion. The finding that delay shifted preference from the larger to the smaller reward is discussed in relation to current theory. The results of this experiment also provide evidence for the utility of tokens in scaling incentives for choice studies.     

Opposing pupil responses to offered and anticipated reward values

Previous studies have shown that the pupils dilate more in anticipation of larger rewards. This finding raises the possibility of a more general association between reward amount and pupil size. We tested this idea by characterizing macaque pupil responses to offered rewards during evaluation and comparison in a binary choice task. To control attention, we made use of a design in which offers occurred in sequence. By looking at pupil responses after choice but before reward, we confirmed the previously observed positive association between pupil size and anticipated reward values. Surprisingly, however, we find that pupil size is negatively correlated with the value of offered gambles before choice, during both evaluation and comparison stages of the task. These results demonstrate a functional distinction between offered and anticipated rewards and present evidence against a narrow version of the simulation hypothesis; the idea that we represent offers by reactivating states associated with anticipating them. They also suggest that pupil size is correlated with relative, not absolute, values of offers, suggestive of an accept–reject model of comparison.     

Rewards and creative performance: a meta-analytic test of theoretically derived hypotheses

Although many scholars and practitioners are interested in understanding how to motivate individuals to be more creative, whether and how rewards affect creativity remain unclear. We argue that the conflicting evidence may be due to differences between studies in terms of reward conditions and the context in which rewards are offered. Specifically, we examine 5 potential moderators of the rewards-creative performance relationship: (a) the reward contingency, (b) the extent to which participants are provided information about their past or current creative performance, (c) the extent to which the reward and context offer choice or impose control, (d) the extent to which the context serves to enhance task engagement, and (e) the extent to which the performance tasks are complex. Using random-effects models, we meta-analyzed 60 experimental and nonexperimental studies (including 69 independent samples) that examined the rewards-creativity relationship with children or adults. Our results suggest that creativity-contingent rewards tend to increase creative performance-and are more positively related to creative performance when individuals are given more positive, contingent, and task-focused performance feedback and are provided more choice (and are less controlled). In contrast, performance-contingent or completion-contingent rewards tend to have a slight negative effect on creative performance.     

时间贴现的分段性

以延迟和相对延迟时间贴现的实验范式, 用选择法、匹配法确定价值主观相等点, 探讨时间贴现的分段性。三个实验和一个问卷调查发现, 时间贴现具有分段性。被试的时间贴现有三次显著变化, 表现为三个时段时间贴现心理状态的不同: 从现在到未来2周内规避损失、偏好风险、愿意短期等待, 从未来2周起直到未来10年采取非补偿性策略、需求与风险并重、愿意长期等待, 和从未来10年起直到未来50年规避风险、聊胜于无、不愿意等待。     

Failure to exploit learned spatial value information during visual search

ABSTRACT

In recent years there has been rapid proliferation of studies demonstrating how reward learning guides visual search. However, most of these studies have focused on feature-based reward, and there has been scant evidence supporting the learning of space-based reward. We raise the possibility that the visual search apparatus is impenetrable to spatial value contingencies, even when such contingencies are learned and represented online in a separate knowledge domain. In three experiments, we interleaved a visual choice task with a visual search task in which one display quadrant produced greater monetary rewards than the remaining quadrants. We found that participants consistently exploited this spatial value contingency during the choice task but not during the search task – even when these tasks were interleaved within the same trials and when rewards were contingent on response speed. These results suggest that the expression of spatial value information is task specific and that the visual search apparatus could be impenetrable to spatial reward information. Such findings are consistent with an evolutionary framework in which the search apparatus has little to gain from spatial value information in most real world situations.     

High temporal discounters overvalue immediate rewards rather than undervalue future rewards: An event-related brain potential study

Impulsivity is characterized in part by heightened sensitivity to immediate relative to future rewards. Although previous research has suggested that “high discounters” in intertemporal choice tasks tend to prefer immediate over future rewards because they devalue the latter, it remains possible that they instead overvalue immediate rewards. To investigate this question, we recorded the reward positivity, a component of the event-related brain potential (ERP) associated with reward processing, with participants engaged in a task in which they received both immediate and future rewards and nonrewards. The participants also completed a temporal discounting task without ERP recording. We found that immediate but not future rewards elicited the reward positivity. High discounters also produced larger reward positivities to immediate rewards than did low discounters, indicating that high discounters relatively overvalued immediate rewards. These findings suggest that high discounters may be more motivated than low discounters to work for monetary rewards, irrespective of the time of arrival of the incentives.     

Contributions of the nucleus accumbens and its subregions to different aspects of risk-based decision making

The nucleus accumbens (NAc) has been implicated in mediating different forms of decision making in humans and animals. In the present study, we observed that inactivation of the rat NAc, via infusion of GABA agonists, reduced preference for a large/risky option and increased response latencies on a probabilistic discounting task. Discrete inactivations of the NAc shell and core revealed further differences between these regions in mediating choice and response latencies, respectively. The effect on choice was attributable to reduced win–stay performance (i.e., choosing risky after a being rewarded for a risky choice on a preceding trial). Moreover, NAc inactivation altered choice only when the large/risky option had greater long-term value, in terms of the amount of food that could be obtained over multiple trials relative to the small/certain option. Inactivation of the NAc or the shell subregion also slightly reduced preference for larger rewards on a reward magnitude discrimination. Thus, the NAc seems to play a small role in biasing choice toward larger rewards, but its contribution to behavior is amplified when delivery of these rewards is uncertain, helping to direct response selection toward more favorable outcomes.     

Physical and cognitive effort discounting of hypothetical monetary rewards

Effort discounting refers to the decrease in the subjective value of a reward as the effort required to obtain the reward increases. The main aims of this study were to ascertain whether the amount of the reward affects the steepness of the effort discounting process for hypothetical monetary rewards, to identify whether this steepness depends on the type of effort that is required, and to determine whether the steepness of different types of effort covary at the individual level. Two types of effort were studied under hypothetical choice situations: physical effort and cognitive effort. Both physical and cognitive effort discounting were well described by the hyperbolic model. Large rewards were discounted less steeply than small rewards for both types of effort. This finding agrees with the results of prior studies which have found that larger rewards have greater motivational power. In addition, the steepness of physical effort discounting was positively correlated with the steepness of cognitive effort discounting, which suggests that the effort discounting process is a trait‐like characteristic within an individual.     

Observing response acquisition: preference for unpredictable appetitive rewards obtained under conditions predicted by DMOD

In five E-maze experiments, rats were given a choice between receiving reward and nonreward in a situation where stimuli were correlated with reward outcome (predictable situation) versus one where the stimuli were uncorrelated with reward outcome (unpredictable situation). Preference for the unpredictable situation occurred under the following conditions: (a) small (one 37-mg pellet), immediate rewards; (b) small, delayed (15 s) rewards, if the cues correlated with reward outcome were absent during the delay interval; (c) large (15 pellets), immediate rewards if a difficult discrimination was required; and (d) if the stimulus predicting nonreward was present at the choice point. Preference for the predictable situation was strongest if reinforcement was delayed and large or the stimulus predicting reward was present at the choice point. A weaker preference for the predictable situation occurred if reinforcement was immediate and large and a simple discrimination was required or if reinforcement was large and delayed and the cues that correlated with reward outcome were absent during the delay interval. The results support the predictions of DMOD (Daly modification of the Rescorla-Wagner model), a mathematical model of appetitive learning (Daly & Daly, 1982).     

Better late than never? The effect of feedback delay on ERP indices of reward processing

The feedback negativity (FN), an early neural response that differentiates rewards from losses, appears to be generated in part by reward circuits in the brain. A prominent model of the FN suggests that it reflects learning processes by which environmental feedback shapes behavior. Although there is evidence that human behavior is more strongly influenced by rewards that quickly follow actions, in nonlaboratory settings, optimal behaviors are not always followed by immediate rewards. However, it is not clear how the introduction of a delay between response selection and feedback impacts the FN. Thus, the present study used a simple forced choice gambling task to elicit the FN, in which feedback about rewards and losses was presented after either 1 or 6?s. Results suggest that, at short delays (1?s), participants clearly differentiated losses from rewards, as evidenced in the magnitude of the FN. At long delays (6?s), on the other hand, the difference between losses and rewards was negligible. Results are discussed in terms of eligibility traces and the reinforcement learning model of the FN.     

To be at the tail of the lions or the head of the foxes?

Agents seeking an opportunity for profit often have to compete with others who pursue the same opportunity. When having to choose between a number of opportunities differing in their value and if individuals differ in their chances of outperforming others, the choice can be cognitively and emotionally demanding. We explore choice between opportunities using stylized Lions–Foxes games. In such a game, each of three players, with different odds of beating others, has to choose one of two contests that offer different rewards. After game theoretically analyzing the games, which we have experimentally employed, we report four experiments that vary in choice elicitation (repeated play or strategy method), in players' matching (random strangers or partners) and in rewards. Regarding contest choices, we found the choice of the higher value (and seemingly more prestigious) contest to be positively related to winning odds, contrary to what four out of the five (mixed, partially mixed, or pure) equilibria predict. Participants started out rather optimistic, with a large majority choosing the higher value option, but with experience, they approached the only viable of two pure strategy equilibria. Still, mixing continued via reacting to past play and outcome, apparently balancing dissatisfaction from choosing either contest.     

An experimental analysis of impulsivity and impulse control in humans

In choosing between small, immediate and large, delayed reward, an organism behaves impulsively if it chooses the small reward and shows impulse control if it chooses the large reward. Work with nonhumans suggests that impulsivity and impulse control may be derived from gradients of delayed reinforcement. A model developed by Ainslie and by Rachlin suggests that preference for the rewards should be a function of when the choice is made: small reward with no delay may be preferred to large reward with delay X, but adding delay T to both alternatives should shift preference to the large reward. Three experiments investigated this preference reversal in humans, using termination of 90 dba white noise as the reinforcing event. Experiment 1 showed that under some instructional conditions 90-sec noise off with no delay was preferred over 120-sec noise off after a 60-sec delay, but that preference shifted to the large reward when a 15-sec delay (T) was added to both alternatives. Experiment 2 replicated this preference reversal under two conditions of large, delayed reward, and with three rather than two values of T. Experiment 3 confirmed this effect of T and showed that some humans committed themselves to the large reward when commitment could be made some time before presentation of the reward alternatives. These data support the Ainslie-Rachlin model and extend it to human choice behavior.     

Biases in preferences for sequences of outcomes in monkeys

Movies, vacations, and meals are all examples of events composed of a sequence of smaller events. How do we go from our evaluations of each scene in a movie to an evaluation of the sequence as a whole? In theory, we should simply average the values of the individual events. In practice, however, we are biased towards sequences where each element tends to be better than the previous, where the last value is large, and we overweight the best (or worst) part of the sequence. To study how general these biases are we examined monkeys’ preferences for sequences of rewards in a novel reward repeat task. Monkeys were first given a sequence of rewards and then chose between repeating the sequence or receiving a standard comparator sequence. We found that, like humans, monkeys overweight events that happen later in a sequence, so much so that adding a small reward to the end of a sequence can paradoxically reduce its value. Monkeys were also biased towards sequences with large peak values (the highest value in the sequence), but only following a working memory challenge, suggesting that this preference may be driven by memory limitations. These results demonstrate the cross-species nature of biases in preferences for sequences of outcomes. In addition, monkeys’ consistent preference for sequences in which large values occur later challenges the generality of discounting models of intertemporal choice in animals.     

Expectations,gains, and losses in the anterior cingulate cortex

The anterior cingulate cortex (ACC) participates in evaluating actions and outcomes. Little is known on how action-reward values are processed in ACC and if the context in which actions are performed influences this processing. In the present article, we report ACC unit activity of monkeys performing two tasks. The first task tested whether the encoding of reward values is context dependent—that is, dependent on the size of the other rewards that are available in the current block of trials. The second task tested whether unexpected events signaling a change in reward are represented. We show that the context created by a block design (i.e., the context of possible alternative rewards) influences the encoding of reward values, even if no decision or choice is required. ACC activity encodes the relative and not absolute expected reward values. Moreover, cingulate activity signals and evaluates when reward expectations are violated by unexpected stimuli, indicating reward gains or losses.     

PREFERENCE REVERSALS DUE TO MYOPIC DISCOUNTING OF DELAYED REWARD

Abstract— A basic stationarity axiom of economic theory assumes stable preference between two deferred goods separated by a fixed time. To test this assumption, we offered subjects choices between delayed rewards, while manipulating the delays to those rewards. Preferences typically reversed with changes in delay, as predicted by hyperbolic discounting models of impulsiveness. Of 36 subjects, 34 reversed preference from a larger, later reward to a smaller, earlier reward as the delays to both rewards decreased. We conclude that the stationarity axiom is not appropriate in models of human choice.     

A mechanism for reducing delay discounting by altering temporal attention

Rewards that are not immediately available are discounted compared to rewards that are immediately available. The more a person discounts a delayed reward, the more likely that person is to have a range of behavioral problems, including clinical disorders. This latter observation has motivated the search for interventions that reduce discounting. One surprisingly simple method to reduce discounting is an "explicit-zero" reframing that states default or null outcomes. Reframing a classical discounting choice as "something now but nothing later" versus "nothing now but more later" decreases discount rates. However, it is not clear how this "explicit-zero" framing intervention works. The present studies delineate and test two possible mechanisms to explain the phenomenon. One mechanism proposes that the explicit-zero framing creates the impression of an improving sequence, thereby enhancing the present value of the delayed reward. A second possible mechanism posits an increase in attention allocation to temporally distant reward representations. In four experiments, we distinguish between these two hypothesized mechanisms and conclude that the temporal attention hypothesis is superior for explaining our results. We propose a model of temporal attention whereby framing affects intertemporal preferences by modifying present bias.     

Undermining the Undermining Effect of Reward on Sustained Interest1

Sometimes, without recourse to controlling rewards, it is difficult to secure desirable behaviors. Yet, much work has demonstrated the damaging effect that such rewards can have on subsequent independent interest in the reward-induced behavior. Therefore, one who feels required to use controlling rewards to increase desirable action in another faces a dilemma: Failing to employ the rewards means foregoing the desired conduct in the immediate situation, hut applying them risks undermining the other's long-term interest in the conduct. We tested a technique designed to avoid this dilemma by (a) providing a controlling reward to obtain the desired action, but (b) then attributing the action to an abiding trait of the actor. Using this technique, we were able to reverse the damaging impact of a controlling reward on children's motivation to write well.