Delay and amount of reward in a concurrent chain

Eight pigeons responded under a concurrent-chain schedule for rewards differing in both delay and amount, the larger reward being associated with the longer delay. Preference was examined as the absolute durations of the terminal-link delays were increased at four different delay ratios. Difficulties with other experiments of this type were controlled for by the use of (a) a single-tape initial link to equalize terminal-link entries, (b) a blackout following the more immediate reward to equalize terminal-link length, and (c) a photocell to measure reinforcer duration more accurately. Preference for the larger reward changed systematically as delays increased in all conditions, decreasing for the 6:1, 3:1, and 3:2 ratios, and increasing for the 1:1 ratio. These results were similar to, but significantly different from, those of previous investigations. The implications of these results for various models of concurrent-chain behavior are discussed.     

Preference for mixed versus constant delays of reinforcement: Effect of probability of the short, mixed delay

Preference for mixed versus constant delays of reinforcement was studied with a concurrent-chain procedure. Lever pressing by rats in concurrently available variable-interval 60-second initial links occasionally produced mutually exclusive terminal-link reinforcement delays. A constant delay of reinforcement (either 15 seconds or 30 seconds) composed one terminal link and mixed delays (.2 second and twice the value of the constant delay) were arranged in the other terminal link. The proportion of .2-second delays in the mixed-delay terminal link took on values of 0, .1, .25, .5, .75, .9, and 1.0 over experimental conditions. Based on relative rates of responding in the initial links, preference for the mixed delays was a negatively accelerated function of the proportion of short, mixed delays. Three of five rats preferred the mixed delays to the constant delays when the proportion of short, mixed delays was .1 or higher, and all five rats preferred the mixed delays when the proportion of short, mixed delays was .25 or higher. Neither Squires and Fantino's (1971) delay-reduction model of choice nor a model based on the harmonic mean reinforcement delay provided a close estimate of choice proportions over the range of short-delay proportions studied. The delay-reduction model underestimated choice for the mixed delays at low and intermediate proportions of short delays, and the harmonic-mean-delay model overestimated choice for the mixed delays at intermediate and high proportions of short delays.     

Differences in delay, not ratios, control choice in concurrent chains

In two experiments, pigeons were trained with concurrent-chains schedules, wherein responding to equal initial links measured preference between variable-interval terminal links. Absolute terminal-link duration was varied by keeping constant the difference between the terminal-link delays and forcing their ratio to change. Delay-reduction theory scales value relative to a common temporal context and requires that delay differences control choice. Thus, preference should remain invariant. Most competing accounts, including the matching law and a strong form of Weber's law, require that preference vary with the delay ratio. Experiment 1 employed standard concurrent chains, in which terminal-link position and color were confounded. Although average preference remained constant, individual preferences were highly variable and inconsistent, possibly due to carryover of position biases across conditions. In an attempt to reduce variability, Experiment 2 used a modified concurrent-chains procedure. Preference at different terminal-link durations was assessed simultaneously to prevent order effects, and terminal-link position was alternated randomly across trials to minimize the impact of position biases. In Experiment 2, both individual and mean preferences showed the constant-difference invariance. Overall, choice was controlled by terminal-link differences, not ratios.     

Limits to preference and the sensitivity of choice to rate and amount of food

Studies of choice holding food‐amount ratio constant while varying food‐rate ratio within sessions showed that local changes in preference depend on relative amount of food. The present study investigated whether sensitivity of choice to food‐rate ratio and sensitivity to food‐amount ratio are independent of one another when food‐rate ratios are varied across sessions and food‐amount ratios are varied within sessions. Food deliveries for rats’ presses on the left and right levers were scheduled according to three different food‐rate ratios of 1:1, 9:1, and 1:9; each food‐rate ratio lasted for 106 sessions and was arranged independently of seven food‐amount ratios (7:1, 6:2, 5:3, 4:4, 3:5, 2:6, and 1:7 food pellets) occurring within sessions in random sequence. Each amount ratio lasted for 10 food deliveries and was separated from another by a 60‐s blackout. Sensitivity to rate ratio was high (1.0) across food deliveries. Sensitivity to amount ratio was low when food rates were equal across alternatives, but was high when rate ratio and amount ratio opposed one another. When rate ratio and amount ratio went in the same direction, choice ratio reached an elevenfold limit which reduced sensitivity to approximately zero. We conclude that three factors affect sensitivity to amount: (1) the limit to preference, (2) the equal effect on preference of amounts greater than four pellets, and (3) the absence of differential effects of switches in amount in the equal‐rates (1:1) condition. Taken together, these findings indicate that rate and amount only sometimes combine independently as additive variables to determine preference when amount ratios vary frequently within sessions.     

Effects of signaled versus unsignaled delay of reinforcement on choice

Pigeons chose between 5-s and 15-s delay-of-reinforcement alternatives. The first key peck to satisfy the choice schedule began a delay timer, and food was delivered at the end of the interval. Key pecks during the delay interval were measured, but had no scheduled effect. In Experiment 1, signal conditions and choice schedules were varied across conditions. During unsignaled conditions, no stimulus change signaled the beginning of a delay interval. During differential and nondifferential signal conditions, offset of the choice stimuli and onset of a delay stimulus signaled the beginning of a delay interval. During differential signal conditions, different stimuli were correlated with the 5-s and 15-s delays, whereas the same stimulus appeared during both delay durations during nondifferential signal conditions. Pigeons showed similar, extreme levels of preference for the 5-s delay alternative during unsignaled and differentially signaled conditions. Preference levels were reliably lower with nondifferential signals. Experiment 2 assessed preference with two pairs of unsignaled delays in which the ratio of delays was held constant but the absolute duration was increased fourfold. No effect of absolute duration was found. The results highlight the importance of delayed primary reinforcement effects and challenge models of choice that focus solely on conditioned reinforcement.     

Evidence for a magnitude effect in temporal discounting with pigeons

A magnitude effect in human intertemporal choice is well established-larger rewards or outcomes are discounted over time at a lower rate than are smaller rewards. However, many recent studies have failed to find a corresponding effect in nonhuman animals. Here we report a magnitude effect in temporal discounting for pigeons' choices involving a tradeoff between reward delay and amount. Pigeons chose between a small reward (1-s access to food) after a 2-s delay, and a large reward (4.5-s access to food) after a 28-s delay. Across conditions, the delays to the small and large rewards were increased or decreased, respectively. Temporal discounting functions obtained through a value-estimation procedure showed clear evidence of a magnitude effect: The value of the large reward decreased more slowly with increasing delay than the value of the small reward. We linked this result to a nonlinear relationship between choice and the delays associated with the small and large rewards. The nonlinearity was contrary to the generalized matching law but was predicted by the contextual choice model. Our results confirm the existence of a magnitude effect in nonhuman temporal discounting, showing that this adaptation is not unique to humans.     

Exchange delays and impulsive choice in adult humans.

Choice responding by adult humans in a discrete-trial task was examined as a function of conditions that manipulated either the delay to point delivery or the delay between points and their exchange for money. In point-delay conditions, subjects chose between an "impulsive" alternative that provided a small amount of points immediately and a "self-control" alternative that provided a larger amount of points delayed by 15, 30, or 60 s. Points were exchanged for money immediately following the session. Subjects preferred the self-control alternative. In exchange-delay conditions, subjects chose between a small amount of points exchangeable for money immediately following the session and a larger amount of points exchangeable for money after 1 day, 3 weeks, or 6 weeks. A self-control preference observed for all subjects in the 1-day exchange-delay condition reversed to exclusive impulsive preference for 4 of the 6 subjects when choice conditions involved exchange delays of 3 or 6 weeks. These results show that human choice is sensitive to the manipulation of exchange delays and that impulsive preference can be obtained with exchange delays on the order of weeks.     

Self-control and choice in humans: Effects of video game playing as a positive reinforcer

Participants chose between reinforcement schedules differing in delay of reinforcement (interval between a choice response and onset of a video game) and/or amount of reinforcement (duration of access to the game). Experiment 1 showed that immediate reinforcement was preferred to delayed reinforcement with amount of reinforcement held constant, and a large reinforcer was preferred to a small reinforcer when both were obtainable immediately. Imposing a delay before the large reinforcer produced a preference for the immediate, small reinforcer in 40% of participants. This suggested a limited degree of “impulsivity.” In Experiment 2, unequal delays were extended by equal intervals, the amounts being kept equal. Preference for the shorter delay decreased, an effect that presumably makes possible the “preference reversal” phenomenon in studies of self-control. Overall, the results demonstrate that video game playing can produce useful, systematic data when used as a positive reinforcer for choice behavior in humans.     

Choice in a "self-control" paradigm: effects of a fading procedure

Pigeons chose between an immediate 2-second reinforcer (access to grain) and a 6-second reinforcer delayed 6 seconds. The four pigeons in the control group were exposed to this condition initially. The four experimental subjects first received a condition where both reinforcers were delayed 6 seconds. The small reinforcer delay was then gradually reduced to zero over more than 11,000 trials. Control subjects almost never chose the large delayed reinforcer. Experimental subjects chose the large delayed reinforcer significantly more often. Two experimental subjects showed preference for the large reinforcer even when the consequences for pecking the two keys were switched. The results indicate that fading procedures can lead to increased “self-control” in pigeons in a choice between a large delayed reinforcer and a small immediate reinforcer.     

PREFERENCE REVERSALS DUE TO MYOPIC DISCOUNTING OF DELAYED REWARD

Abstract— A basic stationarity axiom of economic theory assumes stable preference between two deferred goods separated by a fixed time. To test this assumption, we offered subjects choices between delayed rewards, while manipulating the delays to those rewards. Preferences typically reversed with changes in delay, as predicted by hyperbolic discounting models of impulsiveness. Of 36 subjects, 34 reversed preference from a larger, later reward to a smaller, earlier reward as the delays to both rewards decreased. We conclude that the stationarity axiom is not appropriate in models of human choice.     

Interruptions improve choice performance in gray jays: prolonged information processing versus minimization of costly errors

Under the assumption that selection favors minimization of costly errors, erroneous choice may be common when its fitness cost is low. According to an adaptive-choice model, this cost depends on the rate at which an animal encounters the choice: the higher this rate, the smaller the cost of choosing a less valuable option. Errors should thus be more common when interruptions to foraging are shorter. A previous experiment supported this prediction: gray jays, Perisoreus canadensis, were more error prone when subjected to shorter delays to access to food rewards. This pattern, though, is also predicted by an attentional-constraints model. Because the subjects were able to inspect the rewards during delays, their improved performance when subjected to longer delays could have been a byproduct of the experimentally prolonged opportunity for information processing. To evaluate this possibility, a follow-up experiment manipulated both delay to access and whether rewards could be inspected during delays. Depriving jays of the opportunity to inspect rewards (using opaque lids) induced only a small, nonsignificant increase in error rate. This effect was independent of length of delay and so the jays' improved performance when subjected to longer delays was not simply a byproduct of prolonged information processing. More definitively, even when the jays were prevented from inspecting rewards during delays, their performance improved when subjected to longer delays. The findings are thus consistent with the adaptive-choice model. Electronic Publication     

Delay discounting by pathological gamblers

Discounting of delayed rewards by pathological gamblers was compared to discounting of delayed rewards by matched control nongambling participants. All participants completed a hypothetical choice task in which they made repeated choices between dollars 1,000 available after a delay and an equal or lesser amount of money available immediately. The delay to the large amount of money was varied from 1 week to 10 years across conditions. Indifference points between immediate money and delayed money were identified at each delay condition by varying the amount of immediate money across choice trials. For the majority of participants, indifference points decreased monotonically across delays. Overall, gamblers discounted the delayed rewards more steeply than did control participants.     

Graded differential reinforcement: Response-dependent reinforcer amount

After key pecking had been autoshaped, six pigeons were exposed to a condition in which the duration of grain availability at the end of an 8-second trial depended on the number of responses emitted during the trial (0.25-second access to grain per response). This procedure, called correlated reinforcement, alternated across conditions with the automaintenance baseline in which the 8-second trial terminated with a constant 2.5-second access to grain. Two control procedures were run; in both, the reinforcer durations were yoked to those obtained in the last correlated session. In the yoked control no responses were required, but in the single-response yoked control at least one response was required to receive the yoked duration. The correlated condition maintained response rates above those produced by the two control conditions. These results may be accounted for by differential reinforcement.     

Differences, not ratios, control choice in an experimental analogue to foraging

In choice between outcomes with different delays to reinforcement, most theories require that choice be governed by the ratio of the delays, not by the difference between them, a requirement also consistent with Weber's law. Instead, delay-reduction theory and optimal-foraging theory stipulate, under conditions of the present experiments, that the difference between the delays, and not the ratio between them, controls choice. This prediction was assessed using a procedure, widely used in foraging experiments, in which pigeons chose between accepting and rejecting either of two delays when offered. Across conditions, the delays either differed by a constant amount, with the ratio between the delays varying, or differed by changing amounts, with the ratio between the delays constant. In each of six experiments, rate of acceptance of the longer delay depended only on the difference between the two delays and not on the ratio between them, supporting delay-reduction and foraging theory.     

A quantitative analysis of sensitivity to the conditioned reinforcing value of terminal-link stimuli in a concurrent-chains schedule.

Pigeons were exposed to a concurrent-chains schedule in which a single variable-interval 30-s schedule was used in the initial links and fixed-time schedules were used in the terminal links. Three types of keylight conditions were used in the terminal links. In the first condition, different delays were associated with different keylight stimuli (cued condition). In the second condition, different delays were associated with the same stimulus, either a blackout (uncued blackout condition) or a white key (uncued white condition). Paired values of terminal-link fixed-time schedules differed by a constant ratio of 3:1, while the absolute value of delays was varied from 3 s to 54 s. The results showed that choice proportions for the shorter of two delays increased when the absolute size of the delays was increased for all keylight conditions. Further, the choice proportions for the shorter delay increased from the uncued blackout condition, to the uncued white condition, to the cued condition. A modified version of Fantino's (1969) delay-reduction model (expressed as a function relating the response ratio to the delay-reduction ratio) can be applied to these data by showing that sensitivity to delay reduction increased from the uncued blackout condition, to the uncued white condition, to the cued condition. Thus, the present study demonstrated that a modified version of the delay-reduction model can be used to assess quantitative differences in the terminal-link keylight condition in terms of sensitivity to delay reduction (i.e., the conditioned reinforcing value of the terminal-link keylight stimuli).     

Relative sensitivity to reinforcer amount and delay in a self-control choice situation.

Rats were exposed to concurrent-chains schedules in which a single variable-interval schedule arranged entry into one of two terminal-link delay periods (fixed-interval schedules). The shorter delay ended with the delivery of a single food pellet; the longer day ended with a larger number of food pellets (two under some conditions and six under others). In Experiment 1, the terminal-link delays were selected so that under all conditions the ratio of delays would exactly equal the ratio of the number of pellets. But the absolute duration of the delays differed across conditions. In one condition, for example, rats chose between one pellet delayed 5 s and six pellets delayed 30 s; in another condition rats chose between one pellet delayed 10 s and six pellets delayed 60 s. The generalized matching law predicts indifference between the two alternatives, assuming that the sensitivity parameters for amount and delay of reinforcement are equal. The rats' choices were, in fact, close to indifference except when the choice was between one pellet delayed 5 s and six pellets delayed 30 s. That deviation from indifference suggests that the sensitivities to amount and delay differ from each other depending on the durations of the delays. In Experiment 2, rats chose between one pellet following a 5-s delay and six pellets following a delay that was systematically increased over sessions to find a point of indifference. Indifference was achieved when the delay to the six pellets was approximately 55 s. These results are consistent with the possibility that the relative sensitivities to amount and delay differ as a function of the delays.     

Token reinforcement, choice, and self-control in pigeons.

Pigeons were exposed to self-control procedures that involved illumination of light-emitting diodes (LEDs) as a form of token reinforcement. In a discrete-trials arrangement, subjects chose between one and three LEDs; each LED was exchangeable for 2-s access to food during distinct posttrial exchange periods. In Experiment 1, subjects generally preferred the immediate presentation of a single LED over the delayed presentation of three LEDs, but differences in the delay to the exchange period between the two options prevented a clear assessment of the relative influence of LED delay and exchange-period delay as determinants of choice. In Experiment 2, in which delays to the exchange period from either alternative were equal in most conditions, all subjects preferred the delayed three LEDs more often than in Experiment-1. In Experiment 3, subjects preferred the option that resulted in a greater amount of food more often if the choices also produced LEDs than if they did not. In Experiment 4, preference for the delayed three LEDs was obtained when delays to the exchange period were equal, but reversed in favor of an immediate single LED when the latter choice also resulted in quicker access to exchange periods. The overall pattern of results suggests that (a) delay to the exchange period is a more critical determinant of choice than is delay to token presentation; (b) tokens may function as conditioned reinforcers, although their discriminative properties may be responsible for the self-control that occurs under token reinforcer arrangements; and (c) previously reported differences in the self-control choices of humans and pigeons may have resulted at least in part from the procedural conventions of using token reinforcers with human subjects and food reinforcers with pigeon subjects.     

Determinants of pigeons' choices in token-based self-control procedures

Four pigeons were exposed to a token-based self-control procedure with stimulus lights serving as token reinforcers. Smaller-reinforcer choices produced one token immediately; larger-reinforcer choices produced three tokens following a delay. Each token could be exchanged for 2-s access to food during a signaled exchange period each trial. The main variables of interest were the exchange delays (delays from the choice to the exchange stimulus) and the food delays (also timed from the choice), which were varied separately and together across blocks of sessions. When exchange delays and food delays were shorter following smaller-reinforcer choices, strong preference for the smaller reinforcer was observed. When exchange delays and food delays were equal for both options, strong preference for the larger reinforcer was observed. When food delays were equal for both options but exchange delays were shorter for smaller-reinforcer choices, preference for the larger reinforcer generally was less extreme than under conditions in which both exchange and food delays were equal. When exchange delays were equal for both options but food delays were shorter for smaller-reinforcer choices, preference for the smaller reinforcer generally was less extreme than under conditions in which both exchange and food delays favored smaller-reinforcer choices. On the whole, the results were consistent with prior research on token-based self-control procedures in showing that choices are governed by reinforcer immediacy when exchange and food delays are unequal and by reinforcer amount when exchange and food delays are equal. Further, by decoupling the exchange delays from food delays, the results tentatively support a role for the exchange stimulus as a conditioned reinforcer.     

Rapid acquisition of choice and timing and the provenance of the terminal-link effect

Eight pigeons responded in a concurrent-chains procedure in which terminal-link schedules changed pseudorandomly across sessions. Pairs of terminal-link delays either summed to 15 s or to 45 s. Across sessions, the location of the shorter terminal link changed according to a pseudorandom binary sequence. On some terminal links, food was withheld to obtain start and stop times, measures of temporal control. Log initial-link response ratios stabilized within the first half of each session. Log response ratio was a monotonically-increasing but nonlinear function of programmed log terminal-link immediacy ratio. There was an effect of absolute terminal-link duration on log response ratio: For most subjects, preference for the relatively shorter terminal-link delay was stronger when absolute delays were long than when absolute delays were short. Polynomial regressions and model comparison showed that differences in degree of nonlinearity, not in sensitivity to log immediacy ratio, produced this effect. Temporal control of stop times was timescale invariant with scalar variability, but temporal control of start times was not consistent across subjects or terminal-link durations.     

Concurrent-schedule performance in transition: changeover delays and signaled reinforcer ratios

Six pigeons were trained in experimental sessions that arranged six or seven components with various concurrent-schedule reinforcer ratios associated with each. The order of the components was determined randomly without replacement. Components lasted until the pigeons had received 10 reinforcers, and were separated by 10-s blackout periods. The component reinforcer ratios arranged in most conditions were 27:1, 9:1, 3:1, 1:1, 1:3, 1:9 and 1:27; in others, there were only six components, three of 27:1 and three of 1:27. In some conditions, each reinforcement ratio was signaled by a different red-yellow flash frequency, with the frequency perfectly correlated with the reinforcer ratio. Additionally, a changeover delay was arranged in some conditions, and no changeover delay in others. When component reinforcer ratios were signaled, sensitivity to reinforcement values increased from around 0.40 before the first reinforcer in a component to around 0.80 before the 10th reinforcer. When reinforcer ratios were not signaled, sensitivities typically increased from zero to around 0.40. Sensitivity to reinforcement was around 0.20 lower in no-changeover-delay conditions than in changeover-delay conditions, but increased in the former after exposure to changeover delays. Local analyses showed that preference was extreme towards the reinforced alternative for the first 25 s after reinforcement in changeover-delay conditions regardless of whether components were signaled or not. In no-changeover-delay conditions, preference following reinforcers was either absent, or, following exposure to changeover delays, small. Reinforcers have both local and long-term effects on preference. The former, but not the latter, is strongly affected by the presence of a changeover delay. Stimulus control may be more closely associated with longer-term, more molar, reinforcer effects.