DELAY‐AMOUNT TRADEOFFS IN CHOICES BY PIGEONS AND RATS: HYPERBOLIC VERSUS EXPONENTIAL DISCOUNTING

An adjusting‐delay procedure was used to study the choices of pigeons and rats when both delay and amount of reinforcement were varied. In different conditions, the choice alternatives included one versus two reinforcers, one versus three reinforcers, and three versus two reinforcers. The delay to one alternative (the standard alternative) was kept constant in a condition, and the delay to the other (the adjusting alternative) was increased or decreased many times a session so as to estimate an indifference point—a delay at which the two alternatives were chosen about equally often. Indifference functions were constructed by plotting the adjusting delay as a function of the standard delay for each pair of reinforcer amounts. The experiments were designed to test the prediction of a hyperbolic decay equation that the slopes of the indifference functions should increase as the ratio of the two reinforcer amounts increased. Consistent with the hyperbolic equation, the slopes of the indifference functions depended on the ratios of the two reinforcer amounts for both pigeons and rats. These results were not compatible with an exponential decay equation, which predicts slopes of 1 regardless of the reinforcer amounts. Combined with other data, these findings provide further evidence that delay discounting is well described by a hyperbolic equation for both species, but not by an exponential equation. Quantitative differences in the y‐intercepts of the indifference functions from the two species suggested that the rate at which reinforcer strength decreases with increasing delay may be four or five times slower for rats than for pigeons.     

DISCOUNTING OF DELAYED FOOD REWARDS IN PIGEONS AND RATS: IS THERE A MAGNITUDE EFFECT?

Temporal discounting refers to the decrease in the present, subjective value of a reward as the time to its receipt increases. Results from humans have shown that a hyperbola-like function describes the form of the discounting function when choices involve hypothetical monetary rewards. In addition, magnitude effects have been reported in which smaller reward amounts are discounted more steeply than larger amounts. The present research examines the cross-species generality of these findings using real rewards, namely food pellets, with both pigeons and rats. As with humans, an adjusting amount procedure was used to estimate the amount of immediate reward judged equal in value to a delayed reward. Different amounts of delayed food rewards (ranging from 5 to 32 pellets in pigeons and from 5 to 20 pellets in rats) were studied at delays varying from 1 s to 32 s. A simple hyperbola, similar to the hyperbola-like mathematical function that describes the discounting of hypothetical monetary rewards in humans, described the discounting of food rewards in both pigeons and rats. These results extend the generality of the mathematical model of discounting. Rates of discounting delayed food rewards were higher for pigeons than for rats. Unlike humans, however, neither pigeons nor rats showed a reliable magnitude effect: Rate of discounting did not vary systematically as a function of the amount of the delayed reward.     

Theories of probabilistic reinforcement.

In three experiments, pigeons chose between two alternatives that differed in the probability of reinforcement and the delay to reinforcement. A peck at a red key led to a delay of 5 s and then a possible reinforcer. A peck at a green key led to an adjusting delay and then a certain reinforcer. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In Experiments 1 and 2, the intertrial interval was varied across conditions, and these variations had no systematic effects on choice. In Experiment 3, the stimuli that followed a choice of the red key differed across conditions. In some conditions, a red houselight was presented for 5 s after each choice of the red key. In other conditions, the red houselight was present on reinforced trials but not on nonreinforced trials. Subjects exhibited greater preference for the red key in the latter case. The results were used to evaluate four different theories of probabilistic reinforcement. The results were most consistent with the view that the value or effectiveness of a probabilistic reinforcer is determined by the total time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. According to this view, probabilistic reinforcers are analogous to reinforcers that are delivered after variable delays.     

Delay discounting of qualitatively different reinforcers in rats

Humans discount larger delayed rewards less steeply than smaller rewards, whereas no such magnitude effect has been observed in rats (and pigeons). It remains possible that rats' discounting is sensitive to differences in the quality of the delayed reinforcer even though it is not sensitive to amount. To evaluate this possibility, Experiment 1 examined discounting of qualitatively different food reinforcers: highly preferred versus nonpreferred food pellets. Similarly, Experiment 2 examined discounting of highly preferred versus nonpreferred liquid reinforcers. In both experiments, an adjusting-amount procedure was used to determine the amount of immediate reinforcer that was judged to be of equal subjective value to the delayed reinforcer. The amount and quality of the delayed reinforcer were varied across conditions. Discounting was well described by a hyperbolic function, but no systematic effects of the quantity or the quality of the delayed reinforcer were observed.     

A Measure of Delay Discounting Within the Academic Domain

It is important to better understand the decision‐making processes involved in student procrastination, in order to develop interventions that reduce this common problem. Students may procrastinate because studying produces delayed reinforcers; however, no task measuring delay discounting of academic outcomes currently exists. In Experiment 1, we developed and piloted a measure of academic discounting modeled on titrating‐amount tasks successfully used in the discounting literature. Participants made hypothetical choices between working for money (the smaller, sooner reinforcer) and working on an assignment that was due at various times (the larger, later reinforcer). Participants showed systematic decreases in the subjective value of the assignment as a function of delay, and the hyperbolic and hyperboloid models described the shape of this decrease in value well. In general, larger delayed rewards are discounted less steeply than smaller delayed rewards (the magnitude effect). In Experiment 2, we observed the magnitude effect in academic discounting: Participants discounted a “not important” assignment more steeply than an “important” assignment. In the hyperboloid model, this change was captured by an increase in the s parameter. Results provide support for the validity of the academic discounting task. Copyright © 2018 John Wiley & Sons, Ltd.     

Effects of time between trials on rats' and pigeons' choices with probabilistic delayed reinforcers

Parallel experiments with rats and pigeons examined reasons for previous findings that in choices with probabilistic delayed reinforcers, rats' choices were affected by the time between trials whereas pigeons' choices were not. In both experiments, the animals chose between a standard alternative and an adjusting alternative. A choice of the standard alternative led to a short delay (1 s or 3 s), and then food might or might not be delivered. If food was not delivered, there was an "interlink interval," and then the animal was forced to continue to select the standard alternative until food was delivered. A choice of the adjusting alternative always led to food after a delay that was systematically increased and decreased over trials to estimate an indifference point--a delay at which the two alternatives were chosen about equally often. Under these conditions, the indifference points for both rats and pigeons increased as the interlink interval increased from 0 s to 20 s, indicating decreased preference for the probabilistic reinforcer with longer time between trials. The indifference points from both rats and pigeons were well described by the hyperbolic-decay model. In the last phase of each experiment, the animals were not forced to continue selecting the standard alternative if food was not delivered. Under these conditions, rats' choices were affected by the time between trials whereas pigeons' choices were not, replicating results of previous studies. The differences between the behavior of rats and pigeons appears to be the result of procedural details, not a fundamental difference in how these two species make choices with probabilistic delayed reinforcers.     

Effects of acute and chronic flunitrazepam on delay discounting in pigeons

Delay to delivery of a reinforcer can decrease responding for that reinforcer and increase responding for smaller reinforcers that are available concurrently and delivered without delay; acute administration of drugs can alter responding for large, delayed reinforcers, although the impact of chronic treatment on delay discounting is not well understood. In this experiment, the effects of repeated administration of the benzodiazepine flunitrazepam were studied in 6 pigeons responding on one key to receive food that was delivered immediately and on a second key to receive a larger amount of food that was delivered following delays which increased across a single session. Pigeons responded predominantly for the large reinforcer when there were no delays and when delays were short; however, as delays increased, responding for the large reinforcer decreased. Acutely, flunitrazepam (0.32, 1.0 and 3.2 mg/kg) dose-dependently increased responding for the large reinforcer, shifting the discounting curve rightward and upward. Repeated administration of flunitrazepam (0.32, 1.0 and 3.2 mg/kg, each for six sessions, separated by one session during which vehicle was administered) did not markedly alter its effects on responding for the large reinforcer, indicating that the development of tolerance to this effect of flunitrazepam is modest under these conditions.     

Pigeons play the percentages: computation of probability in a bird

The ability to compute probability, previously shown in nonverbal infants, apes, and monkeys, was examined in three experiments with pigeons. After responding to individually presented keys in an operant chamber that delivered reinforcement with varying probabilities, pigeons chose between these keys on probe trials. Pigeons strongly preferred a 75% reinforced key over a 25% reinforced key, even when the total number of reinforcers obtained on each key was equated. When both keys delivered 50% reinforcement, pigeons showed indifference between them, even though three times more reinforcers were obtained on one key than on the other. It is suggested that computation of probability may be common to many classes of animals and may be driven by the need to forage successfully for nutritional food items, mates, and areas with a low density of predators.     

A tribute to Howard Rachlin and his two-parameter discounting model: Reliable and flexible model fitting

Delay discounting reflects the rate at which a reward loses its subjective value as a function of delay to that reward. Many models have been proposed to measure delay discounting, and many comparisons have been made among these models. We highlight the two-parameter delay discounting model popularized by Howard Rachlin by demonstrating two key practical features of the Rachlin model. The first feature is flexibility; the Rachlin model fits empirical discounting data closely. Second, when compared with other available two-parameter discounting models, the Rachlin model has the advantage that unique best estimates for parameters are easy to obtain across a wide variety of potential discounting patterns. We focus this work on this second feature in the context of maximum likelihood, showing the relative ease with which the Rachlin model can be utilized compared with the extreme care that must be used with other models for discounting data, focusing on two illustrative cases that pass checks for data validity. Both of these features are demonstrated via a reanalysis of discounting data the authors have previously used for model selection purposes.     

Modeling the effect of reward amount on probability discounting

The present study with college students examined the effect of amount on the discounting of probabilistic monetary rewards. A hyperboloid function accurately described the discounting of hypothetical rewards ranging in amount from $20 to $10,000,000. The degree of discounting increased continuously with amount of probabilistic reward. This effect of amount was not due to changes in the rate parameter of the discounting function, but rather was due to increases in the exponent. These results stand in contrast to those observed with the discounting of delayed monetary rewards, in which the degree of discounting decreases with reward amount due to amount-dependent decreases in the rate parameter. Taken together, this pattern of results suggests that delay and probability discounting reflect different underlying mechanisms. That is, the fact that the exponent in the delay discounting function is independent of amount is consistent with a psychophysical scaling interpretation, whereas the finding that the exponent of the probability-discounting function is amount-dependent is inconsistent with such an interpretation. Instead, the present results are consistent with the idea that the probability-discounting function is itself the product of a value function and a weighting function. This idea was first suggested by Kahneman and Tversky (1979), although their prospect theory does not predict amount effects like those observed. The effect of amount on probability discounting was parsimoniously incorporated into our hyperboloid discounting function by assuming that the exponent was proportional to the amount raised to a power. The amount-dependent exponent of the probability-discounting function may be viewed as reflecting the effect of amount on the weighting of the probability with which the reward will be received.     

Choice with delayed and probabilistic reinforcers: effects of variability, time between trials, and conditioned reinforcers.

In a discrete-trials procedure with pigeons, a response on a green key led to a 4-s delay (during which green houselights were lit) and then a reinforcer might or might not be delivered. A response on a red key led to a delay of adjustable duration (during which red houselights were lit) and then a certain reinforcer. The delay was adjusted so as to estimate an indifference point--a duration for which the two alternatives were equally preferred. Once the green key was chosen, a subject had to continue to respond on the green key until a reinforcer was delivered. Each response on the green key, plus the 4-s delay that followed every response, was called one "link" of the green-key schedule. Subjects showed much greater preference for the green key when the number of links before reinforcement was variable (averaging four) than when it was fixed (always exactly four). These findings are consistent with the view that probabilistic reinforcers are analogous to reinforcers delivered after variable delays. When successive links were separated by 4-s or 8-s "interlink intervals" with white houselights, preference for the probabilistic alternative decreased somewhat for 2 subjects but was unaffected for the other 2 subjects. When the interlink intervals had the same green houselights that were present during the 4-s delays, preference for the green key decreased substantially for all subjects. These results provided mixed support for the view that preference for a probabilistic reinforcer is inversely related to the duration of conditioned reinforcers that precede the delivery of food.     

The Matching Law And Amount-dependent Exponential Discounting As Accounts Of Self-control Choice

Studies with humans have found evidence for amount-dependent temporal discounting, that is, that the sensitivity of choice to reinforcer delay varies inversely with reinforcer magnitude. To test whether similar results could be obtained with nonhumans, pigeons were trained on a two-component concurrent-chains procedure in which the durations of food reinforcement in the terminal links were equal within components but unequal between components. Terminal-link schedules were varied over four conditions to allow separate estimates of sensitivity to delay to be obtained for the large and small reinforcer-magnitude components. Although sensitivity to delay was greater in the small-magnitude component for all subjects, the effect was quite small. The difference in sensitivity was generally less than one standard error, and t tests on parameter differences failed to reach significance. Several models for temporal discounting, including an amount-dependent exponential function, were fitted to the data from the first four conditions. The resulting parameter estimates were used to make predictions for a self-control condition in which one terminal link arranged a smaller, less delayed reinforcer and the other arranged a larger, more delayed reinforcer. For all models, predictions were considerably more accurate when sensitivity to delay was the same regardless of reinforcer magnitude. The results support the independence of delay and magnitude as required by a version of the matching law, and provide strong evidence against amount-dependent exponential discounting as an account of self-control choice. A new two-parameter discounting function, consistent with the matching law, is proposed that has wide empirical generality for both human and nonhuman data.     

Introduction to the special issue on comparative cognition

Prior research has shown that nonhumans show an extreme preference for variable‐ over fixed‐delays to reinforcement. This well‐established preference for variability occurs because a reinforcer's strength or “value” decreases according to a curvilinear function as its delay increases. The purpose of the present experiments was to investigate whether this preference for variability occurs with human participants making hypothetical choices. In three experiments, participants recruited from Amazon Mechanical Turk made choices between variable and fixed monetary rewards. In a variable‐delay procedure, participants repeatedly chose between a reward delivered either immediately or after a delay (with equal probability) and a reward after a fixed delay (Experiments 1 and 2). In a double‐reward procedure, participants made choices between an alternative consisting of two rewards, one delivered immediately and one after a delay, and a second alternative consisting of a single reward delivered after a delay (Experiments 1 and 3). Finally, all participants completed a standard delay‐discounting task. Although we observed both curvilinear discounting and magnitude effects in the standard discounting task, we found no consistent evidence of a preference for variability—as predicted by two prominent models of curvilinear discounting (i.e., a simple hyperbola and a hyperboloid)—in our variable‐delay and double‐reward procedures. This failure to observe a preference for variability may be attributed to the hypothetical, rule‐governed nature of choices in the present study. In such contexts, participants may adopt relatively simple strategies for making more complex choices.     

Discounting by Probabilistic Waiting

In everyday life, many probabilistic situations may be characterized as probabilistic waiting. A gambler, for example, bets repeatedly at the racetrack, the casino, or the card table. The gambler may not win on the first try, but if a gamble is repeated enough times, a win is almost certain to occur eventually. If repeated gambles are structured as strings of losses ending in a win (probabilistic waiting) and the amount won is discounted by the delay caused by the series of losses, then strings with many losses will be discounted more than those with fewer losses, thereby causing subjective value of the series of gambles as a whole to increase. The current study used the opposite effect that amount has on the degree of delay and probability discounting as a marker to determine whether people evaluate situations involving probabilistic waiting as they evaluate situations involving delayed outcomes or as situations involving probabilistic outcomes. We find that the more likely a probabilistic waiting situation is to end in reward (e.g., a gamble is repeated indefinitely until reward is obtained), the more that situation conforms to delay discounting; the less likely a probabilistic waiting situation is to end in reward (e.g., a fixed, small number of gambles), the more that situation conforms to probability discounting. We argue that the former situation is applicable to pathological gambling, and that people with steep delay discount functions would therefore be more likely to have gambling problems. Copyright © 2015 John Wiley & Sons, Ltd.     

Subjective probability and delay.

Human subjects indicated their preference between a hypothetical $1,000 reward available with various probabilities or delays and a certain reward of variable amount available immediately. The function relating the amount of the certain-immediate reward subjectively equivalent to the delayed $1,000 reward had the same general shape (hyperbolic) as the function found by Mazur (1987) to describe pigeons' delay discounting. The function relating the certain-immediate amount of money subjectively equivalent to the probabilistic $1,000 reward was also hyperbolic, provided that the stated probability was transformed to odds against winning. In a second experiment, when human subjects chose between a delayed $1,000 reward and a probabilistic $1,000 reward, delay was proportional to the same odds-against transformation of the probability to which it was subjectively equivalent.     

Choice and multiple reinforcers

Pigeons chose between equivalent two-component mixed and multiple terminal-link schedules of reinforcement in the concurrent-chains procedure. The pigeons preferred the multiple schedule over the mixed when the components of the compound schedules were differentiated in terms of density of reinforcement, but the pigeons were indifferent when the components were differentiated in terms of number of reinforcers per cycle. Taken together, these results indicate that a local variable, the interval to the first reinforcer, but not a molar variable, the number of reinforcers, was sufficient to differentiate the components and thereby evoke preference.     

Soft commitment: self-control achieved by response persistence.

With reinforcement contingent on a single peck on either of two available keys (concurrent continuous reinforcement schedules) 4 pigeons, at 80% of free-feeding weights, preferred a smaller-sooner reinforcer (2.5 s of mixed grain preceded by a 0.5-s delay) to a larger-later reinforcer (4.5 s of mixed grain preceded by a 3.5-s delay). However, when the smaller-sooner and larger-later reinforcers were contingent on a concurrent fixed-ratio 31 schedule (the first 30 pecks distributed in any way on the two keys), all pigeons obtained the larger-later reinforcer much more often than they did when only a single peck was required. This "self-control" was achieved by beginning to peck the key leading to the larger-later reinforcer and persisting on that key until reinforcement occurred. We call this persistence "soft commitment" to distinguish it from strict commitment, in which self-control is achieved by preventing changeovers. Soft commitment also effectively achieved self-control when a brief (1-s) signal was inserted between the 30th and 31st response of the ratio and with concurrent fixed-interval 30-s schedules (rather than ratio schedules) of reinforcement. In a second experiment with the same subjects, the fixed ratio was interrupted by darkening both keys and lighting a third (center) key on which pecking was required for various fractions of the fixed-ratio count. The interruption significantly reduced self-control. When interruption was complete (30 responses on the center key followed by a single choice response), pigeons chose the smaller-sooner reinforcer as frequently as they did when only a single choice response was required.     

A second type of magnitude effect: Reinforcer magnitude differentiates delay discounting between substance users and controls

Basic research on delay discounting, examining preference for smaller–sooner or larger–later reinforcers, has demonstrated a variety of findings of considerable generality. One of these, the magnitude effect, is the observation that individuals tend to exhibit greater preference for the immediate with smaller magnitude reinforcers. Delay discounting has also proved to be a useful marker of addiction, as demonstrated by the highly replicated finding of greater discounting rates in substance users compared to controls. However, some research on delay discounting rates in substance users, particularly research examining discounting of small‐magnitude reinforcers, has not found significant differences compared to controls. Here, we hypothesize that the magnitude effect could produce ceiling effects at small magnitudes, thus obscuring differences in delay discounting between groups. We examined differences in discounting between high‐risk substance users and controls over a broad range of magnitudes of monetary amounts ($0.10, $1.00, $10.00, $100.00, and $1000.00) in 116 Amazon Mechanical Turk workers. We found no significant differences in discounting rates between users and controls at the smallest reinforcer magnitudes ($0.10 and $1.00) and further found that differences became more pronounced as magnitudes increased. These results provide an understanding of a second form of the magnitude effect: That is, differences in discounting between populations can become more evident as a function of reinforcer magnitude.     

Contiguity and conditioned reinforcement in probabilistic choice

In a baseline condition, pigeons chose between an alternative that always provided food following a 30-s delay (100% reinforcement) and an alternative that provided food half of the time and blackout half of the time following 30-s delays (50% reinforcement). The different outcomes were signaled by different-colored keylights. On average, each alternative was chosen approximately equally often, replicating the finding of suboptimal choice in probabilistic reinforcement procedures. The efficacy of the delay stimuli (keylights) as conditioned reinforcers was assessed in other conditions by interposing a 5-s gap (keylights darkened) between the choice response and one or more of the delay stimuli. The strength of conditioned reinforcement was measured by the decrease in choice of an alternative when the alternative contained a gap. Preference for the 50% alternative decreased in conditions in which the gap preceded either all delay stimuli, both delay stimuli for the 50% alternative, or the food stimulus for the 50% alternative, but preference was not consistently affected in conditions in which the gap preceded only the 100% delay stimulus or the blackout stimulus for the 50% alternative. These results support the notion that conditioned reinforcement underlies the finding of suboptimal preference in probabilistic reinforcement procedures, and that the signal for food on the 50% reinforcement alternative functions as a stronger conditioned reinforcer than the signal for food on the 100% reinforcement alternative. In addition, the results fail to provide evidence that the signal for blackout functions as a conditioned punisher.     

Effects of reinforcer probability, delay, and response requirements on the choices of rats and pigeons: possible species differences

In Experiment 1 with rats, a left lever press led to a 5-s delay and then a possible reinforcer. A right lever press led to an adjusting delay and then a certain reinforcer. This delay was adjusted over trials to estimate an indifference point, or a delay at which the two alternatives were chosen about equally often. Indifference points increased as the probability of reinforcement for the left lever decreased. In some conditions with a 20% chance of food, a light above the left lever was lit during the 5-s delay on all trials, but in other conditions, the light was only lit on those trials that ended with food. Unlike previous results with pigeons, the presence or absence of the delay light on no-food trials had no effect on the rats' indifference points. In other conditions, the rats showed less preference for the 20% alternative when the time between trials was longer. In Experiment 2 with rats, fixed-interval schedules were used instead of simple delays, and the presence or absence of the fixed-interval requirement on no-food trials had no effect on the indifference points. In Experiment 3 with rats and Experiment 4 with pigeons, the animals chose between a fixed-ratio 8 schedule that led to food on 33% of the trials and an adjusting-ratio schedule with food on 100% of the trials. Surprisingly, the rats showed less preference for the 33% alternative in conditions in which the ratio requirement was omitted on no-food trials. For the pigeons, the presence or absence of the ratio requirement on no-food trials had little effect. The results suggest that there may be differences between rats and pigeons in how they respond in choice situations involving delayed and probabilistic reinforcers.