Erroneous choice and foregone gains in hoarding gray jays (Perisoreus canadensis)

Under the conventional assumption that natural selection favors choice behavior that maximizes some fitness-related currency, a forager making repeated binary choices should consistently choose the more valuable option. Under the alternative assumption that natural selection favors choice behavior that minimizes costly errors, erroneous choice is not only expected but is expected to be common when the cost of errors is low. This cost depends on the potential rate of return: the higher this rate, the smaller the cost of choosing the less valuable option. When this rate is very high, a forager may err frequently and yet forego no appreciable fitness gain. Errors should thus be more common when interruptions to foraging are shorter. Our experimental results supported this prediction: gray jays chose the less valuable option more frequently when subjected to shorter interruptions (experimentally imposed delays to access to food rewards). This tendency is consistent with the idea that an adaptive decision-making process may routinely produce errors, not because errors are in some way adaptive but because their fitness cost is minimal, particularly when delays are short. From a proximate perspective, this tendency to commit errors more frequently following shorter delays may be due to constraints on the jays’ information-processing capacity. In general, choice behavior should be viewed as the joint byproduct of adaptive decision making and cognitive constraints. Received: 8 June 2000 / Accepted after revision: 21 October 2000     

Choice between rewards differing in amount and delay: Toward a choice model of self control

A concurrent-chain procedure was used to study pigeons' choices between rewards differing in both amount and delay. The shorter delay terminated with a 2-second access to grain whereas the longer delay terminated with a 6-second access to grain. The ratio of the delays was constant within a given condition while their absolute values were varied. Over conditions, ratios of 6:1, 3:1, and 3:2 were studied. As the absolute values of the delays to reinforcement increased, preference for the longer-delayed but larger reward decreased under both the 6:1 and 3:1 ratios, but increased under the 3:2 ratio. These results are inconsistent with choice models predicting no change in preference when the ratios of delays and amounts are held constant. In addition, the change in preference under the 3:1 ratio is inconsistent with a simple multiplicative interaction of the trade off between reinforcer amount and delay, and suggests that delay is a more potent determinant of choice than is amount. These results have implications for models that view choice between small immediate rewards and large but delayed rewards as underlying the behavior commonly called self control.     

Magnitude effects for experienced rewards at short delays in the escalating interest task

A first-person shooter video game was adapted for the study of choice between smaller sooner and larger later rewards. Participants chose when to fire a weapon that increased in damage potential over a short interval. When the delay to maximum damage was shorter (5–8 s), people showed greater sensitivity to the consequences of their choices than when the delay was longer (17–20 s). Participants also evidenced a magnitude effect by waiting proportionally longer when the damage magnitudes were doubled for all rewards. The experiment replicated the standard magnitude effect with this new video game preparation over time scales similar to those typically used in nonhuman animal studies and without complications due to satiation or cost.     

Evidence for a magnitude effect in temporal discounting with pigeons

A magnitude effect in human intertemporal choice is well established-larger rewards or outcomes are discounted over time at a lower rate than are smaller rewards. However, many recent studies have failed to find a corresponding effect in nonhuman animals. Here we report a magnitude effect in temporal discounting for pigeons' choices involving a tradeoff between reward delay and amount. Pigeons chose between a small reward (1-s access to food) after a 2-s delay, and a large reward (4.5-s access to food) after a 28-s delay. Across conditions, the delays to the small and large rewards were increased or decreased, respectively. Temporal discounting functions obtained through a value-estimation procedure showed clear evidence of a magnitude effect: The value of the large reward decreased more slowly with increasing delay than the value of the small reward. We linked this result to a nonlinear relationship between choice and the delays associated with the small and large rewards. The nonlinearity was contrary to the generalized matching law but was predicted by the contextual choice model. Our results confirm the existence of a magnitude effect in nonhuman temporal discounting, showing that this adaptation is not unique to humans.     

Chimpanzees (Pan troglodytes) can wait,when they choose to: a study with the hybrid delay task

Self-control has been studied in nonhuman animals using a variety of tasks. The inter-temporal choice (ITC) task presents choices between smaller–sooner (SS) and larger–later (LL) options. Using food amounts as rewards, this presents two problems: (a) choices of the LL option could either reflect self-control or instead result from animals’ difficulty with pointing to smaller amounts of food; (b) there is no way to verify whether the subjects would not revert their choice for the LL option, if given the opportunity to do so during the ensuing delay. To address these problems, we have recently introduced a new protocol, the hybrid delay task, which combines an initial ITC with a subsequent accumulation phase in which selection of the SS option leads to its immediate delivery, but choice of the LL option then leads to one-by-one presentation of those items that continues only as long as the subject does not eat any of the accumulated items. The choice of the LL option therefore only reflects self-control when the number of items obtained from LL choices during the accumulation phase is higher than what could be received in the SS option. Previous research with capuchin monkeys demonstrated that their apparent self-control responses in the ITC task may have overestimated their general self-control abilities, given their poor performance in the hybrid delay task. Here, chimpanzees instead demonstrated that their choices for the LL option in the ITC phase of the hybrid delay task were confirmed by their ability to sustain long delays during accumulation of LL rewards.     

Pigeons' memory for number of events: Effects of intertrial interval and delay interval illumination

In Experiment 1, pigeons were trained at a 0-s baseline delay to discriminate sequences of light flashes (illumination of the feeder) that varied in number but not time (2f/4s and 8f/4s). During training, the intertrial interval was illuminated by the houselight for Group Light, but it was dark for Group Dark. Testing conducted with dark delay intervals produced a strong choose-small bias in both groups. All birds then received baseline training with a 5-s dark delay and were subsequently tested at shorter and longer dark delays. A choose-small bias was again observed at delays longer than the training delay, while a choose-large bias occurred at delays shorter than the training delay. Differentiating the ambient chamber illumination during the intertrial interval and the delay interval did not attenuate choose-small or choose-large effects. In Experiment 2, all birds received baseline training with a 5-s illuminated delay and were subsequently tested at shorter and longer illuminated delays. A choose-large bias was observed at delays longer than the training delay, while a choose-small bias occurred at delays shorter than the training delay. In Experiment 3, on intermittent test trials, when the duration of the second flash on small-sample trials was equal to the total flash duration on large-sample trials (i.e., 1600 ms), accuracy fell to approximately chance. These results suggest that pigeons discriminated the sequences of light flashes that varied in number but not in total duration of the sequence by relying on other temporal properties of the sequence rather than by using an event switch to count flashes.     

Determinants of pigeons' choices in token-based self-control procedures

Four pigeons were exposed to a token-based self-control procedure with stimulus lights serving as token reinforcers. Smaller-reinforcer choices produced one token immediately; larger-reinforcer choices produced three tokens following a delay. Each token could be exchanged for 2-s access to food during a signaled exchange period each trial. The main variables of interest were the exchange delays (delays from the choice to the exchange stimulus) and the food delays (also timed from the choice), which were varied separately and together across blocks of sessions. When exchange delays and food delays were shorter following smaller-reinforcer choices, strong preference for the smaller reinforcer was observed. When exchange delays and food delays were equal for both options, strong preference for the larger reinforcer was observed. When food delays were equal for both options but exchange delays were shorter for smaller-reinforcer choices, preference for the larger reinforcer generally was less extreme than under conditions in which both exchange and food delays were equal. When exchange delays were equal for both options but food delays were shorter for smaller-reinforcer choices, preference for the smaller reinforcer generally was less extreme than under conditions in which both exchange and food delays favored smaller-reinforcer choices. On the whole, the results were consistent with prior research on token-based self-control procedures in showing that choices are governed by reinforcer immediacy when exchange and food delays are unequal and by reinforcer amount when exchange and food delays are equal. Further, by decoupling the exchange delays from food delays, the results tentatively support a role for the exchange stimulus as a conditioned reinforcer.     

氟哌啶醇与丙咪嗪交互作用对成本效益决策的影响

本实验采用T迷宫延迟奖赏模型研究多巴胺D2受体拮抗剂氟哌啶醇和5-羟色胺重摄取抑制剂丙咪嗪的交互作用对成本效益决策的影响, 同时探讨了延迟时间对决策的影响。T迷宫两臂分别设置为低成本-低奖赏端和高成本-高奖赏端。实验结果发现：氟哌啶醇能够降低大鼠选择高成本-高奖赏端的次数, 丙咪嗪则能够增加大鼠选择高成本-高奖赏端的次数; 在同时注射这两种药物情况下, 丙咪嗪能够抑制由氟哌啶醇引起的对低成本-低奖赏端的选择倾向。另外, 实验发现, 随延迟时间的增加大鼠选择高成本-高奖赏端的次数相对减少。由此可见, 丙咪嗪能够反转由氟哌啶醇导致的对低成本-低奖赏端的选择倾向, 这可能是由于细胞间5-羟色胺含量的升高部分反转了由多巴胺系统受损导致的行为倾向; 延迟时间的改变可对决策倾向产生逆转, 因此成本的支出即延迟时间也是影响成本效益决策的重要因素。     

Choice and delay of reinforcement: Effects of terminal-link stimulus and response conditions

In two experiments, pigeons were exposed to concurrent-chains schedules in which a single initial-link variable-interval schedule led to access to terminal links composed of fixed-interval or fixed-delay schedules. In Experiment 1, an 8-s (or 16-s) delay to reinforcement was associated with the standard key, while reinforcer delay values associated with the experimental key were varied from 4 to 32 s. The results of Experiment 1 showed undermatching of response ratios to delay ratios with terminal-link fixed-delay schedules, whereas in some pigeons matching or overmatching was evident with the fixed-interval schedules. In Experiment 2, one pair of reinforcer delay values, either 8 versus 16 s or 16 versus 32 s, was used. In the first condition of Experiment 2, different delays were associated with different keylight stimuli (cued condition). In the second condition, different terminal-link delays were associated with the same stimulus, either a blackout (uncued-blackout condition) or a white key (uncued-white condition). To examine the role of responses emitted during delays, the keys were retracted during a delay (key-absent condition) in the third condition and responses were required by a fixed-interval schedule in the fourth condition. Experiment 2 demonstrated that the choice proportions for the shorter delay were more extreme in the cued condition than in the uncued-blackout condition, and that the response requirement imposed by the fixed-interval schedules did not affect choice of the shorter delay, nor did the key-absent and key-present conditions. These results indicate that the keylight-stimulus conditions affected preference for the shorter of two delays and that the findings obtained in Experiment 1 depended mainly on the keylight-stimulus conditions of the terminal links (i.e., the conditioned reinforcing value of the terminal-link stimuli).     

Effects of delay of prospective memory cues in an ongoing task on prospective memory task performance

To address the mixed results reported in previous studies, the present experiments examined forgetting in prospective memory (PM) by manipulating the delay between the PM instructions and cue presentation in event-based PM tasks. PM performance was measured for delays of 2–20 min in Experiment 1 and for delays of approximately 1–10 min in Experiment 2. Experiment 2 included both focal and nonfocal PM tasks, and speed on the ongoing task was measured to examine evidence for monitoring processes across the delays tested. The results suggest that nonfocal PM performance follows a nonlinear forgetting function (i.e., rapid decline for shorter delays and slower decline for longer delays) when tested over delays from 1 to 20 min. No effect of delay was seen for the focal task tested in Experiment 2 from 1 to 10 min. In Experiment 2, ongoing-task costs were also found for the first delay but not for longer delays, suggesting that monitoring was significantly reduced between 1 and 2.5 min of the ongoing-task trials.     

Timing, remembering, and discrimination

Four pigeons were first trained in a timing procedure. In one condition, each trial began with the presentation of an X on the center key, followed by a delay (short or long), after which two side keys were lit. If the delay was short, pecks to the red side key were reinforced. If the delay was long, pecks to the green side key were reinforced. In a second condition, the opposite contingencies applied following presentation of a square on the center key. Choice responses were then tested at 10 time intervals ranging from short to long (1 to 4 s and 4 to 7 s in different conditions). The two timing conditions were combined to create a remembering condition in which correct responding depended upon discrimination of both the sample stimulus (X or square) and the delay interval (short or long). Choices varied systematically across delay in timing conditions, but in remembering conditions, accurate choice at the training delays did not initially generalize to intermediate delays. However, with prolonged training in the remembering task, the response pattern began to resemble that of the timing conditions. Generalization gradients were asymmetrical, in accordance with Weber's Law, in that greater generalization occurred with longer delays than with shorter delays.     

Establishing preference for unreliable reinforcement in adults with dual diagnoses

We evaluated the choice responding of three adults dually diagnosed with developmental and psychiatric disabilities using concurrent schedules of reinforcement. Specifically, participants were given a choice between a response option resulting in reliable reinforcement and a response option resulting in unreliable reinforcement. Our primary purpose was to shift preference from reliable to unreliable reinforcement via the systematic presentation of stimuli during delay intervals. A second purpose was to evaluate the effectiveness of intervening stimuli in shifting preference at differing delay-to-reinforcement intervals. Preference for unreliable reinforcement was first examined in the absence of stimulus presentations during delays, at three different delay values. Next, we aimed to establish preference for unreliable reinforcement by presenting pictures of reinforcers during delays preceding unreliable reinforcement. Preference was again examined at three different delay values. In the absence of stimulus presentations during delays, participants were shown to prefer reliable reinforcement, particularly at the longer delay value. When stimuli were presented during the delays, two of the three participants preferred unreliable reinforcement, particularly the longer the delay value. These results suggest that the presentation of intervening stimuli during delays may help facilitate tolerance for unreliable reinforcement.     

Waiting for more: the performance of domestic dogs (Canis familiaris) on exchange tasks

Five domestic dogs (Canis familiaris) were tested in a cooperative exchange task with an experimenter, as previously tested in non-human primates. In the first task, the dogs exchanged to maximise payoffs when presented with food items of differing quality. All consistently exchanged lower-value for higher-value rewards, as determined by their individual food preference, and exchanges corresponded significantly with the spontaneous preferences of three dogs. Next, all subjects demonstrated an ability to perform two and three exchanges in succession, to gain both qualitative and quantitatively increased rewards (group mean = 72 and 92% successful triple exchanges, respectively). Finally, the ability to delay gratification over increasing intervals was tested; the dogs kept one food item to exchange later for a larger item. As previously reported in non-human primates, there was considerable individual variation in the tolerance of delays, between 10 s and 10 min for the largest rewards. For those who reached longer time lags (>40 s), the dogs gave up the chance to exchange earlier than expected by each subject’s general waiting capacity; the dogs anticipated delay duration and made decisions according to the relative reward values offered. Compared to primates, dogs tolerated relatively long delays for smaller value rewards, suggesting that the socio-ecological history of domestic dogs facilitates their performance on decision-making and delay of gratification tasks.     

Delay discounting by pathological gamblers

Discounting of delayed rewards by pathological gamblers was compared to discounting of delayed rewards by matched control nongambling participants. All participants completed a hypothetical choice task in which they made repeated choices between dollars 1,000 available after a delay and an equal or lesser amount of money available immediately. The delay to the large amount of money was varied from 1 week to 10 years across conditions. Indifference points between immediate money and delayed money were identified at each delay condition by varying the amount of immediate money across choice trials. For the majority of participants, indifference points decreased monotonically across delays. Overall, gamblers discounted the delayed rewards more steeply than did control participants.     

PREFERENCE REVERSALS DUE TO MYOPIC DISCOUNTING OF DELAYED REWARD

Abstract— A basic stationarity axiom of economic theory assumes stable preference between two deferred goods separated by a fixed time. To test this assumption, we offered subjects choices between delayed rewards, while manipulating the delays to those rewards. Preferences typically reversed with changes in delay, as predicted by hyperbolic discounting models of impulsiveness. Of 36 subjects, 34 reversed preference from a larger, later reward to a smaller, earlier reward as the delays to both rewards decreased. We conclude that the stationarity axiom is not appropriate in models of human choice.     

The Influence of Time Estimation and Time‐Saving Preferences on Learning to Make Temporally Dependent Decisions from Experience

In this paper, we merge research related to experiential learning, temporal perception, and the value of time and money by examining decisions where the timing of action (response) determines the outcome received. We predicted that time‐saving preferences and impatience would decrease maximization (i.e., taking action when it returned the largest reward), and that the constraints of temporal perception would compound their effects. Across three studies, participants undershot on average (i.e., responded earlier than the period of time during which a response would return the maximal reward) showed a preference for shorter‐delay options and often did not find the maximal reward. In addition, participants' reliance on temporal perception increased undershooting, increased preferences for shorter‐delay options, and reduced maximization. Nevertheless, participants who found the maximal reward continued to maximize at a high rate rather than opting for shorter delays and smaller rewards. Thus, while most participants appeared to have a preference for saving time, most behaved as reward maximizers rather than temporal discounters. Copyright © 2017 John Wiley & Sons, Ltd.     

Sensitivity to reinforcer duration in a self-control procedure

In a concurrent-chains procedure, pigeons' responses on left and right keys were followed by reinforcers of different durations at different delays following the choice responses. Three pairs of reinforcer delays were arranged in each session, and reinforcer durations were varied over conditions. In Experiment 1 reinforcer delays were unequal, and in Experiment 2 reinforcer delays were equal. In Experiment 1 preference reversal was demonstrated in that an immediate short reinforcer was chosen more frequently than a longer reinforcer delayed 6 s from the choice, whereas the longer reinforcer was chosen more frequently when delays to both reinforcers were lengthened. In both experiments, choice responding was more sensitive to variations in reinforcer duration at overall longer reinforcer delays than at overall shorter reinforcer delays, independently of whether fixed-interval or variable-interval schedules were arranged in the choice phase. We concluded that preference reversal results from a change in sensitivity of choice responding to ratios of reinforcer duration as the delays to both reinforcers are lengthened.     

Self-control and choice in humans: Effects of video game playing as a positive reinforcer

Participants chose between reinforcement schedules differing in delay of reinforcement (interval between a choice response and onset of a video game) and/or amount of reinforcement (duration of access to the game). Experiment 1 showed that immediate reinforcement was preferred to delayed reinforcement with amount of reinforcement held constant, and a large reinforcer was preferred to a small reinforcer when both were obtainable immediately. Imposing a delay before the large reinforcer produced a preference for the immediate, small reinforcer in 40% of participants. This suggested a limited degree of “impulsivity.” In Experiment 2, unequal delays were extended by equal intervals, the amounts being kept equal. Preference for the shorter delay decreased, an effect that presumably makes possible the “preference reversal” phenomenon in studies of self-control. Overall, the results demonstrate that video game playing can produce useful, systematic data when used as a positive reinforcer for choice behavior in humans.     

Transitional and steady-state choice behavior under an adjusting-delay schedule

Twelve rats made repeated choices on an adjusting-delay schedule between a smaller reinforcer (A) that was delivered immediately after a response and a larger reinforcer (B) that was delivered after a delay which increased or decreased by 20% depending on the subject's choices in successive blocks of trials. In two phases of the experiment (100 sessions and 40 sessions), reinforcer sizes were selected which enabled theoretical parameters expressing the rate of delay discounting and sensitivity to reinforcer size to be estimated from the ratio of the indifference delays obtained in the two phases. Indifference delays, calculated from adjusting delays in the last 10 sessions of each phase, were shorter when the sizes of A and B were 14 and 25 μl of a 0.6 M sucrose solution than when they were 25 and 100 μl of the same solution. The ratio of the indifference delays was significantly smaller than that predicted on the basis of an assumed linear relation between reinforcer size and instantaneous reinforcer value, consistent with a previous proposal that this relation may be hyperbolic in form. Estimates of the rate of delay discounting based on the ratio of the two indifference delays (mean, 0.08 s(-1)) were similar to values obtained previously using different intertemporal choice protocols. Estimates of the size-sensitivity parameter (mean 113 μl) were similar to estimates recently derived from performance on progressive-ratio schedules. In both phases of the experiment, adjusting delays in successive blocks of trials were analyzed using the Fourier transform. The power spectrum obtained from individual rats had a dominant frequency that corresponded to a period of oscillation of the adjusting delay between 30 and 100 trial blocks (mean, 78). Power in the dominant frequency band was highest in the early sessions of the first phase and declined with extended training. It is suggested that this experimental protocol may have utility in neurobehavioral studies of intertemporal choice.     

Relative sensitivity to reinforcer amount and delay in a self-control choice situation.

Rats were exposed to concurrent-chains schedules in which a single variable-interval schedule arranged entry into one of two terminal-link delay periods (fixed-interval schedules). The shorter delay ended with the delivery of a single food pellet; the longer day ended with a larger number of food pellets (two under some conditions and six under others). In Experiment 1, the terminal-link delays were selected so that under all conditions the ratio of delays would exactly equal the ratio of the number of pellets. But the absolute duration of the delays differed across conditions. In one condition, for example, rats chose between one pellet delayed 5 s and six pellets delayed 30 s; in another condition rats chose between one pellet delayed 10 s and six pellets delayed 60 s. The generalized matching law predicts indifference between the two alternatives, assuming that the sensitivity parameters for amount and delay of reinforcement are equal. The rats' choices were, in fact, close to indifference except when the choice was between one pellet delayed 5 s and six pellets delayed 30 s. That deviation from indifference suggests that the sensitivities to amount and delay differ from each other depending on the durations of the delays. In Experiment 2, rats chose between one pellet following a 5-s delay and six pellets following a delay that was systematically increased over sessions to find a point of indifference. Indifference was achieved when the delay to the six pellets was approximately 55 s. These results are consistent with the possibility that the relative sensitivities to amount and delay differ as a function of the delays.