Choice in a "self-control" paradigm: effects of a fading procedure

Pigeons chose between an immediate 2-second reinforcer (access to grain) and a 6-second reinforcer delayed 6 seconds. The four pigeons in the control group were exposed to this condition initially. The four experimental subjects first received a condition where both reinforcers were delayed 6 seconds. The small reinforcer delay was then gradually reduced to zero over more than 11,000 trials. Control subjects almost never chose the large delayed reinforcer. Experimental subjects chose the large delayed reinforcer significantly more often. Two experimental subjects showed preference for the large reinforcer even when the consequences for pecking the two keys were switched. The results indicate that fading procedures can lead to increased “self-control” in pigeons in a choice between a large delayed reinforcer and a small immediate reinforcer.     

An experimental analysis of impulsivity and impulse control in humans

In choosing between small, immediate and large, delayed reward, an organism behaves impulsively if it chooses the small reward and shows impulse control if it chooses the large reward. Work with nonhumans suggests that impulsivity and impulse control may be derived from gradients of delayed reinforcement. A model developed by Ainslie and by Rachlin suggests that preference for the rewards should be a function of when the choice is made: small reward with no delay may be preferred to large reward with delay X, but adding delay T to both alternatives should shift preference to the large reward. Three experiments investigated this preference reversal in humans, using termination of 90 dba white noise as the reinforcing event. Experiment 1 showed that under some instructional conditions 90-sec noise off with no delay was preferred over 120-sec noise off after a 60-sec delay, but that preference shifted to the large reward when a 15-sec delay (T) was added to both alternatives. Experiment 2 replicated this preference reversal under two conditions of large, delayed reward, and with three rather than two values of T. Experiment 3 confirmed this effect of T and showed that some humans committed themselves to the large reward when commitment could be made some time before presentation of the reward alternatives. These data support the Ainslie-Rachlin model and extend it to human choice behavior.     

Probability and delay in commitment

In the first stage of a two-stage choice, human subjects chose between probabilistic access to a second choice (between a small high-probability reward and a large low-probability reward) and commitment to the large low-probability reward. When confronted with the second-stage choice, subjects strongly preferred the small high-probability reward. When the first-stage probability (of access to the second stage) was high, subjects strongly preferred the path leading to the choice in the second stage. But when the first-stage probability was low, subjects committed themselves to the large low-probability reward. These results parallel those obtained by Rachlin and Green (1972) with pigeons and constitute some evidence that probabilities may be interpreted as delays.     

An account of human “impulsivity” on self-control tasks

Three experiments examined the effects of economic constraint (time restriction—trials restriction) on adult humans' performance on concurrent chained schedules of reinforcement in which terminal links differed both in reward size and pre-reward delay levels. In the first, terminal links offered a long delay (20, 30, 40, 50 sec) followed by 2 points, or a short delay (10 sec) followed by 1 point. Initial links consisted of the same variable interval schedules for each alternative. Subjects showed a conditional sensitivity to increases in delay. Under a time constraint, there was an increasing preference for the smaller reward as the delay to the larger reward was increased, whereas under a trials constraint subjects were totally insensitive to such changes. In the second experiment, this finding was replicated using long and short periods of video game playing as large and small rewards. In the third experiment terminal link parameters remained constant (2 points after 15 sec, or 1 point immediately), but initial link parameters were manipulated (either VI 1-sec, 8-sec, or 20-sec) so that the higher relative rate of reward shifted from being associated with the small reward to being associated with the large reward. Again subjects chose differently under the two constraints. Subjects under the time constraint exhibited a preference for the small reward at all levels of initial link, but under the trials constraint subjects preferred the large reward. Subjects in the time constraint condition thus exhibited a maladaptive insensitivity to changes in the initial link duration. These findings are interpreted as supporting a miscalculation rather than a discounting interpretation of human performance on “self-control” tasks.     

Escape from freedom: Choosing not to choose in pigeons

Preference for the availability of food-reinforcement alternatives was investigated with Rachlin and Green's (1972) concurrent-chains self-control paradigm. The terminal link of one chain made available a choice between immediate access to food for T seconds and delayed access to food for 4 seconds. The terminal link of the other chain provided only delayed access to food. When T was reduced to .25 seconds, pigeons began to select the delayed-access key in both terminal links. Even so, the pigeons strongly preferred constraint over choice. This effect could not be accounted for by differences in the actual amount of food obtained in the terminal links, by avoidance of the immediate-reinforcement key when not presented as part of a choice, or by avoidance of a multi-key terminal link. Rather, constraint was preferred over freedom. Apparently, the preference for choice is determined by the particular type of choice offered.     

Observing response acquisition: preference for unpredictable appetitive rewards obtained under conditions predicted by DMOD

In five E-maze experiments, rats were given a choice between receiving reward and nonreward in a situation where stimuli were correlated with reward outcome (predictable situation) versus one where the stimuli were uncorrelated with reward outcome (unpredictable situation). Preference for the unpredictable situation occurred under the following conditions: (a) small (one 37-mg pellet), immediate rewards; (b) small, delayed (15 s) rewards, if the cues correlated with reward outcome were absent during the delay interval; (c) large (15 pellets), immediate rewards if a difficult discrimination was required; and (d) if the stimulus predicting nonreward was present at the choice point. Preference for the predictable situation was strongest if reinforcement was delayed and large or the stimulus predicting reward was present at the choice point. A weaker preference for the predictable situation occurred if reinforcement was immediate and large and a simple discrimination was required or if reinforcement was large and delayed and the cues that correlated with reward outcome were absent during the delay interval. The results support the predictions of DMOD (Daly modification of the Rescorla-Wagner model), a mathematical model of appetitive learning (Daly & Daly, 1982).     

Responding during reinforcement delay in a self-control paradigm.

Eight pigeons chose between a small, immediate reinforcer and a large, increasingly delayed reinforcer. Responding during the large-reinforcer delays was examined. During large-reinforcer delays, pecks on one key produced the small, immediate reinforcer; pecks on the other key had no effect. Thus, a pigeon could reverse its initial choice of the large, delayed reinforcer, or it could maintain its original choice. Pigeons that made a relatively high number of initial large-reinforcer choices tended to maintain these choices, and those pigeons that actually received a relatively high number of large reinforcers, tended to respond more frequently on the ineffective key during the delay periods. The findings suggest that some previous studies of self-control training in pigeons may have resulted in increased self-control partially due to a lack of opportunity for the pigeons to change their choices.     

Exchange delays and impulsive choice in adult humans.

Choice responding by adult humans in a discrete-trial task was examined as a function of conditions that manipulated either the delay to point delivery or the delay between points and their exchange for money. In point-delay conditions, subjects chose between an "impulsive" alternative that provided a small amount of points immediately and a "self-control" alternative that provided a larger amount of points delayed by 15, 30, or 60 s. Points were exchanged for money immediately following the session. Subjects preferred the self-control alternative. In exchange-delay conditions, subjects chose between a small amount of points exchangeable for money immediately following the session and a larger amount of points exchangeable for money after 1 day, 3 weeks, or 6 weeks. A self-control preference observed for all subjects in the 1-day exchange-delay condition reversed to exclusive impulsive preference for 4 of the 6 subjects when choice conditions involved exchange delays of 3 or 6 weeks. These results show that human choice is sensitive to the manipulation of exchange delays and that impulsive preference can be obtained with exchange delays on the order of weeks.     

Self-control in male and female rats

Eight male and 8 female Wistar rats were exposed to a discrete-trial procedure in which they chose between the presentation of a small (one pellet) or a large (three pellets) reinforcer. The delay to the small and large reinforcer was 6.0 s in the first condition of Experiment 1. Subjects consistently chose the large reinforcer. When the delay to the small reinforcer was decreased to 0.1 s in the next experimental condition, all subjects continued to choose the large 6.0-s delayed reinforcer. When the contingencies correlated with the two levers were reversed in the next experimental condition, the majority of subjects (5 males and 6 females) still chose the large delayed reinforcer over the small immediately presented reinforcer. The delay to the small reinforcer was maintained at 6.0 s, but the delay to the large reinforcer was varied among 9.0, 15.0, 24.0, and 36.0 s in Experiment 2, in which 4 males and 4 females participated. Most subjects consistently chose the large increasingly delayed reinforcer, although choice for the small 6.0-s delayed reinforcer developed in some females when the large reinforcer was delayed for 24.0 or 36.0 s. These choice patterns were not predicted from a literal application of a model that says choice should favor the alternative correlated with the higher (amount/delay) ratio.     

Choice between rewards differing in amount and delay: Toward a choice model of self control

A concurrent-chain procedure was used to study pigeons' choices between rewards differing in both amount and delay. The shorter delay terminated with a 2-second access to grain whereas the longer delay terminated with a 6-second access to grain. The ratio of the delays was constant within a given condition while their absolute values were varied. Over conditions, ratios of 6:1, 3:1, and 3:2 were studied. As the absolute values of the delays to reinforcement increased, preference for the longer-delayed but larger reward decreased under both the 6:1 and 3:1 ratios, but increased under the 3:2 ratio. These results are inconsistent with choice models predicting no change in preference when the ratios of delays and amounts are held constant. In addition, the change in preference under the 3:1 ratio is inconsistent with a simple multiplicative interaction of the trade off between reinforcer amount and delay, and suggests that delay is a more potent determinant of choice than is amount. These results have implications for models that view choice between small immediate rewards and large but delayed rewards as underlying the behavior commonly called self control.     

Do open space environments encourage children to seek immediate gratification?

This study evaluated the effects of 14 months' exposure to open space environments on preference for delayed reward. Three hundred eighty-nine pupils in open space and conventional classes in grades 4 and 5 were individually administered one of four tests for delay of gratification. These tests consisted of a choice between an immediate, smaller reward and a delayed, larger reward. Each of the four independent studies found that approximately the same percent of pupils from each educational environment selected the delayed alternative.     

The development of prudence in the face of varying future rewards

The choice to delay gratification, opting for a greater reward later instead of a smaller reward now, has been used as a measure of preschoolers' ability to make future-oriented decisions. The present studies investigated to what extent these choice tasks reflect children's ability to make choices in favour of their future self. In Experiment 1, preschoolers were presented with choices between a smaller immediate reward of one sticker and larger delayed rewards varying in amount from two to five stickers. Whereas 3-year-olds showed no increased tendency to choose the delayed option as quantity increased from two to five, 4-year-olds' future-oriented choices reflected the value of the delayed reward. In Experiment 2, preschoolers chose between immediate and delayed rewards that were equal in amount and varied between one and four stickers. Three-year-olds reliably chose the immediate reward and showed no differentiation among rewards, whereas 4-year-olds chose in a manner that suggested they were trying to satisfy both immediate and future desires. Together these results support the notion that future-oriented prudence reflects growth in concern for the future self.     

Summation of symbols by pigeons (Columba livia): the importance of number and mass of reward items

In each of 3 experiments, different sets of 4 pigeons (Columba livia) were trained to discriminate between 2 visual symbols that covered wells containing food items that varied in number, mass, or both. In Experiment 1, the symbols were associated with 0, 1, 3, 5, 7, or 9 pieces of grain reward. The pigeons learned to choose the symbol corresponding to the larger reward, and on summation tests, they chose the pair of symbols that summed to the larger total reward. When number of food pellets was varied but mass of reward was held constant in Experiment 2, preference for the larger number symbols failed to appear. When number was held constant and mass was varied in Experiment 3, the pigeons showed a clear preference for the larger mass symbols on single-symbol and summation tests. These findings show that pigeons summate the value of symbols and are more likely to represent symbols by mass of food reward than by number of food items.     

Negative reinforcement and choice in humans

Participants chose between reinforcement schedules differing in delay and/or duration of noise offset. In Experiment 1 it was found that (1) immediate reinforcement was preferred to delayed reinforcement when amounts (durations) of reinforcement were equal; (2) a relatively large reinforcer was preferred to a smaller one when both reinforcers were obtained immediately; and (3) preference for an immediate, small reinforcer versus a delayed, large reinforcer increased as the delay preceding the large reinforcer increased, a sign of “impulsivity”. In Experiment 2, the schedules differed in amount or delay and equal intervals were added either to the constant parameter or the varied parameter. A shift from virtually exclusive preference to indifference occurred in the latter case but not the former, a result supporting a model of self-control that assumes that the value of a schedule depends on the ratio of amount and delay, and that choice between schedules depends on the ratio of these values.     

Self-control and choice in humans: Effects of video game playing as a positive reinforcer

Participants chose between reinforcement schedules differing in delay of reinforcement (interval between a choice response and onset of a video game) and/or amount of reinforcement (duration of access to the game). Experiment 1 showed that immediate reinforcement was preferred to delayed reinforcement with amount of reinforcement held constant, and a large reinforcer was preferred to a small reinforcer when both were obtainable immediately. Imposing a delay before the large reinforcer produced a preference for the immediate, small reinforcer in 40% of participants. This suggested a limited degree of “impulsivity.” In Experiment 2, unequal delays were extended by equal intervals, the amounts being kept equal. Preference for the shorter delay decreased, an effect that presumably makes possible the “preference reversal” phenomenon in studies of self-control. Overall, the results demonstrate that video game playing can produce useful, systematic data when used as a positive reinforcer for choice behavior in humans.     

Effects of intertrial reinforcers on self-control choice.

In three experiments, pigeons chose between a small amount of food delivered after a short delay and a larger amount delivered after a longer delay. A discrete-trial adjusting-delay procedure was used to estimate indifference points--pairs of delay-amount combinations that were chosen about equally often. In Experiment 1, when additional reinforcers were available during intertrial intervals on a variable-interval schedule, preference for the smaller, more immediate reinforcer increased. Experiment 2 found that this shift in preference occurred partly because the variable-interval schedule started sooner after the smaller, more immediate reinforcer, but there was still a small shift in preference when the durations and temporal locations of the variable-interval schedules were identical for both alternatives. Experiment 3 found greater increases in preference for the smaller, more immediate reinforcer with a variable-interval 15-s schedule than with a variable-interval 90-s schedule. The results were generally consistent with a model that states that the impact of any event that follows a choice response declines according to a hyperbolic function with increasing time since the moment of choice.     

Procrastination by pigeons: preference for larger, more delayed work requirements.

In three experiments, pigeons chose between alternatives that required the completion of a small ratio schedule early in the trial or a larger ratio schedule later in the trial. Completion of the ratio requirement did not lead to an immediate reinforcer, but simply allowed the events of the trial to continue. In Experiment 1, the ratio requirements interrupted periods in which food was delivered on a variable-time schedule. In Experiments 2 and 3, each ratio requirement was preceded and followed by a delay, and only one reinforcer was delivered, at the end of each trial. Two of the experiments used an adjusting-ratio procedure in which the ratio requirement was increased and decreased over trials so as to estimate an indifference point--a ratio size at which the two alternatives were chosen about equally often. These experiments found clear evidence for "procrastination"--the choice of a larger but more delayed response requirement. In some cases, subjects chose the more delayed ratio schedule even when it was larger than the more immediate alternative by a factor of four or more. The results suggest that as the delay to the start of a ratio requirement is increased, it has progressively less effect on choice behavior, in much the same way that delaying a positive reinforcer reduces it effect on choice.     

Token reinforcement, choice, and self-control in pigeons.

Pigeons were exposed to self-control procedures that involved illumination of light-emitting diodes (LEDs) as a form of token reinforcement. In a discrete-trials arrangement, subjects chose between one and three LEDs; each LED was exchangeable for 2-s access to food during distinct posttrial exchange periods. In Experiment 1, subjects generally preferred the immediate presentation of a single LED over the delayed presentation of three LEDs, but differences in the delay to the exchange period between the two options prevented a clear assessment of the relative influence of LED delay and exchange-period delay as determinants of choice. In Experiment 2, in which delays to the exchange period from either alternative were equal in most conditions, all subjects preferred the delayed three LEDs more often than in Experiment-1. In Experiment 3, subjects preferred the option that resulted in a greater amount of food more often if the choices also produced LEDs than if they did not. In Experiment 4, preference for the delayed three LEDs was obtained when delays to the exchange period were equal, but reversed in favor of an immediate single LED when the latter choice also resulted in quicker access to exchange periods. The overall pattern of results suggests that (a) delay to the exchange period is a more critical determinant of choice than is delay to token presentation; (b) tokens may function as conditioned reinforcers, although their discriminative properties may be responsible for the self-control that occurs under token reinforcer arrangements; and (c) previously reported differences in the self-control choices of humans and pigeons may have resulted at least in part from the procedural conventions of using token reinforcers with human subjects and food reinforcers with pigeon subjects.     

Pigeons are attracted to a perceived gain without an actual gain

Reference dependence refers to the reduced value of a reward that is less than expected, or the added value of a reward that is greater than expected. There is evidence that when pigeons are offered an alternative that has 1 pellet versus an alternative that has 2 pellets, but one of the two pellets offered will be removed, the pigeons prefer the originally presented 1 pellet (loss aversion). In the present research, we tested for the opposite effect (gain attraction). In Experiments 1 and 2, pigeons could choose between 2 pellets, each one on a distinctive background. If they chose the optimal alternative, they received a second pellet. In Experiment 2, the second pellet obtained was the one not initially chosen (a task sometimes referred to as the ephemeral reward task). Pigeons learned to choose optimally in both experiments. In Experiment 3, we tested the pigeons for reference dependence. Pigeons were given an alternative that offered them one pellet or two pellets, if they chose the one-pellet alternative, they received an additional pellet, and if they chose the two-pellet alternative, they received the two pellets. In keeping with the reference dependence hypothesis, the pigeons preferred the 1-pellet alternative that gave them an extra pellet. These effects are related to similar findings with humans, including the endowment effect.

     

Inhibitory stimulus control and the magnitude of delayed reinforcement

The key pecking of pigeons was reinforced according to a variable-interval 1-min schedule during each of two successively presented stimuli. When the key was illuminated by a black line on a white background, reinforcement was delayed for 10 sec. When the key was illuminated by a plain white light, reinforcement was not delayed. For half of the subjects, the delayed reinforcer was 4.0-sec access to mixed grain, and for the remaining subjects it was 1.5-sec access. The immediate reinforcer was 1.5-sec access for all subjects. All subjects responded at a lower rate during the presentation of the black line; no between-group difference in terms of terminal response rate during the presentation of the line was found. However, subjects that received 4.0 sec of delayed reinforcement responded at a lower terminal rate during presentation of the plain white light than subjects that received 1.5 sec of delayed reinforcement. A subsequent generalization test along the line-orientation dimension produced flatter U-shaped gradients for subjects that received 4.0-sec of delayed reinforcement.