Token reinforcement, choice, and self-control in pigeons.

Pigeons were exposed to self-control procedures that involved illumination of light-emitting diodes (LEDs) as a form of token reinforcement. In a discrete-trials arrangement, subjects chose between one and three LEDs; each LED was exchangeable for 2-s access to food during distinct posttrial exchange periods. In Experiment 1, subjects generally preferred the immediate presentation of a single LED over the delayed presentation of three LEDs, but differences in the delay to the exchange period between the two options prevented a clear assessment of the relative influence of LED delay and exchange-period delay as determinants of choice. In Experiment 2, in which delays to the exchange period from either alternative were equal in most conditions, all subjects preferred the delayed three LEDs more often than in Experiment-1. In Experiment 3, subjects preferred the option that resulted in a greater amount of food more often if the choices also produced LEDs than if they did not. In Experiment 4, preference for the delayed three LEDs was obtained when delays to the exchange period were equal, but reversed in favor of an immediate single LED when the latter choice also resulted in quicker access to exchange periods. The overall pattern of results suggests that (a) delay to the exchange period is a more critical determinant of choice than is delay to token presentation; (b) tokens may function as conditioned reinforcers, although their discriminative properties may be responsible for the self-control that occurs under token reinforcer arrangements; and (c) previously reported differences in the self-control choices of humans and pigeons may have resulted at least in part from the procedural conventions of using token reinforcers with human subjects and food reinforcers with pigeon subjects.     

A Measure of Delay Discounting Within the Academic Domain

It is important to better understand the decision‐making processes involved in student procrastination, in order to develop interventions that reduce this common problem. Students may procrastinate because studying produces delayed reinforcers; however, no task measuring delay discounting of academic outcomes currently exists. In Experiment 1, we developed and piloted a measure of academic discounting modeled on titrating‐amount tasks successfully used in the discounting literature. Participants made hypothetical choices between working for money (the smaller, sooner reinforcer) and working on an assignment that was due at various times (the larger, later reinforcer). Participants showed systematic decreases in the subjective value of the assignment as a function of delay, and the hyperbolic and hyperboloid models described the shape of this decrease in value well. In general, larger delayed rewards are discounted less steeply than smaller delayed rewards (the magnitude effect). In Experiment 2, we observed the magnitude effect in academic discounting: Participants discounted a “not important” assignment more steeply than an “important” assignment. In the hyperboloid model, this change was captured by an increase in the s parameter. Results provide support for the validity of the academic discounting task. Copyright © 2018 John Wiley & Sons, Ltd.     

Determination of discount functions in rats with an adjusting-amount procedure.

An adjusting-amount procedure was used to measure discounting of reinforcer value by delay. Eight rats chose between a varying amount of immediate water and a fixed amount of water given after a delay. The amount of immediate water was systematically adjusted as a function of the rats' previous choices. This procedure was used to determine the indifference point at which each rat chose the immediate amount and the delayed amount with equal frequency. The amount of immediate water at this indifference point was used to estimate the value of the delayed amount of water. In Experiment 1, the effects of daily changes in the delay to the fixed reinforcer (100 microliters of water delivered after 0, 2, 4, 8, or 16 s) were tested. Under these conditions, the rats reached indifference points within the first 30 trials of each 60-trial session. In Experiment 2, the effects of water deprivation level on discounting of value by delay were assessed. Altering water deprivation level affected the speed of responding but did not affect delay discounting. In Experiment 3, the effects of varying the magnitude of the delayed water (100, 150, and 200 microliters) were tested. There was some tendency for the discounting function to be steeper for larger than for smaller reinforcers, although this difference did not reach statistical significance. In all three experiments, the obtained discount functions were well described by a hyperbolic function. These experiments demonstrate that the adjusting-amount procedure provides a useful tool for measuring the discounting of reinforcer value by delay.     

Self-control and impulsiveness in children and adults: Effects of food preferences

Experiment 1 used 6 preschool boys and Experiment 2 used 6 adult women to explore the effects of food preference on humans' choice in self-control paradigms. The boys showed a higher proportion of responses for more delayed, larger reinforcers (a measure of self-control) when those choices resulted in receipt of the most preferred food compared to when those choices resulted in the least preferred food. Further, the boys chose the less delayed, smaller reinforcers significantly more often when only those choices, as opposed to both choices, resulted in the most preferred food. Conversely, they chose the more delayed, larger reinforcers significantly more often when only those choices, as opposed to both choices, resulted in the most preferred food. Finally, the women demonstrated significantly less sensitivity to reinforcer amount relative to sensitivity to reinforcer delay (another measure of self-control) when they had a higher preference for the juice received as the less delayed, smaller reinforcer than for the juice received as the more delayed, larger reinforcer. Together, the results show that subjects' food preferences can influence self-control for food reinforcers.     

Self-control in adult humans: variation in positive reinforcer amount and delay.

In five experiments, choice responding of female human adults was examined, as a function of variations in reinforcer amount and reinforcer delay. Experiment 1 used a discrete-trials procedure, and Experiments 2, 3, 4, and 5 used a concurrent variable-interval variable-interval schedule. Reinforcer amount and reinforcer delay were varied both separately and together. In contrast to results previously reported with pigeons, the subjects in the present experiments usually chose the larger reinforcers even when those reinforcers were delayed. Together, the results from all the experiments suggest that the subjects followed a maximization strategy in choosing reinforcers. Such behavior makes it easy to observe self-control and difficult to observe impulsiveness in traditional laboratory experiments that use adult human subjects.     

SAVING THE BEST FOR LAST? A CROSS-SPECIES ANALYSIS OF CHOICES BETWEEN REINFORCER SEQUENCES

Two experiments were conducted to compare choices between sequences of reinforcers in pigeon (Experiment 1) and human (Experiment 2) subjects, using functionally analogous procedures. The subjects made pairwise choices among 3 sequence types, all of which provided the same overall reinforcerment rate, but differed in their temporal patterning. Token reinforcement schedules were used in both experiments and the type of exchange schedule varied across blocks of sessions. Some conditions permitted immediate exchange of tokens for consumable reinforcers (food for pigeons, video access for humans); in other conditions, tokens accumulated and were exchanged for consumable reinforcers only at the end of the sequence. Choice patterns in the immediate-exchange conditions were generally similar across species, with both pigeons and humans preferring sequences with the shortest delay to the initial reinforcer in the series. The results are broadly consistent with models of temporal discounting expanded to include the impact of sequences of delayed reinforcers acting in parallel from the time of the choice. Preferences were less consistent with discounting models in the delayed exchange conditions. Questionnaire data gathered at the end of the experiment were consistent with prior results of questionnaire studies, but showed no straightforward relation to the observed choice patterns, urging caution in the extrapolation of results from one decision-making domain to the other.     

Sensitivity to reinforcer duration in a self-control procedure

In a concurrent-chains procedure, pigeons' responses on left and right keys were followed by reinforcers of different durations at different delays following the choice responses. Three pairs of reinforcer delays were arranged in each session, and reinforcer durations were varied over conditions. In Experiment 1 reinforcer delays were unequal, and in Experiment 2 reinforcer delays were equal. In Experiment 1 preference reversal was demonstrated in that an immediate short reinforcer was chosen more frequently than a longer reinforcer delayed 6 s from the choice, whereas the longer reinforcer was chosen more frequently when delays to both reinforcers were lengthened. In both experiments, choice responding was more sensitive to variations in reinforcer duration at overall longer reinforcer delays than at overall shorter reinforcer delays, independently of whether fixed-interval or variable-interval schedules were arranged in the choice phase. We concluded that preference reversal results from a change in sensitivity of choice responding to ratios of reinforcer duration as the delays to both reinforcers are lengthened.     

EVALUATION OF A BRIEF STIMULUS PREFERENCE ASSESSMENT

We evaluated the utility of a brief (5-min) stimulus preference assessment for individuals with developmental disabilities. Participants had noncontingent (free) access to an array of stimuli and could interact with any of the stimuli at any time. Stimuli were never withdrawn or withheld from the participants during a 5-min session. In Experiment 1, the brief preference assessment was conducted for 10 participants to identify differentially preferred stimuli, and reinforcer assessments were conducted to test the reinforcing efficacy of those stimuli identified as highly preferred. In Experiment 2, a comparison was conducted between the brief preference assessment and a commonly used paired-stimulus preference assessment. Collectively, results demonstrated that the brief preference assessment identified stimuli that functioned as reinforcers for a simple operant response, identified preferred stimuli that were differentially effective as reinforcers compared to nonpreferred stimuli, was associated with fewer problem behaviors, and required less time to complete than a commonly used paired-stimulus preference assessment.     

ASSESSMENT OF STIMULUS PREFERENCE AND REINFORCER VALUE WITH PROFOUNDLY RETARDED INDIVIDUALS

We evaluated a procedure for identifying potential reinforcers with profoundly retarded individuals. In Experiment 1, six persons were repeatedly exposed to 16 stimuli, and approach behaviors to each stimulus were used to identify preferred and nonpreferred stimuli. In Experiment 2, we examined the reinforcing properties of preferred and nonpreferred stimuli by delivering them contingently on the occurrence of arbitrarily selected responses. Results revealed that the preferred stimulus conditions typically produced higher rates of responding than did either the baseline or the nonpreferred stimulus conditions, suggesting that the procedure can be used to assess reinforcer value for individuals with limited behavioral repertoires.     

IMPULSIVITY IN STUDENTS WITH SERIOUS EMOTIONAL DISTURBANCE: THE INTERACTIVE EFFECTS OF REINFORCER RATE,DELAY, AND QUALITY

We conducted two studies extending basic matching research on self-control and impulsivity to the investigation of choices of students diagnosed as seriously emotionally disturbed. In Study 1 we examined the interaction between unequal rates of reinforcement and equal versus unequal delays to reinforcer access on performance of concurrently available sets of math problems. The results of a reversal design showed that when delays to reinforcer access were the same for both response alternatives, the time allocated to each was approximately proportional to obtained reinforcement. When the delays to reinforcer access differed between the response alternatives, there was a bias toward the response alternative and schedule with the lower delays, suggesting impulsivity (i.e., immediate reinforcer access overrode the effects of rate of reinforcement). In Study 2 we examined the interactive effects of reinforcer rate, quality, and delay. Conditions involving delayed access to the high-quality reinforcers on the rich schedule (with immediate access to low-quality reinforcers earned on the lean schedule) were alternated with immediate access to low-quality reinforcers on the rich schedule (with delayed access to high-quality reinforcers on the lean schedule) using a reversal design. With 1 student, reinforcer quality overrode the effects of both reinforcer rate and delay to reinforcer access. The other student tended to respond exclusively to the alternative associated with immediate access to reinforcers. The studies demonstrate a methodology based on matching theory for determining influential dimensions of reinforcers governing individuals' choices.     

Negative reinforcement and choice in humans

Participants chose between reinforcement schedules differing in delay and/or duration of noise offset. In Experiment 1 it was found that (1) immediate reinforcement was preferred to delayed reinforcement when amounts (durations) of reinforcement were equal; (2) a relatively large reinforcer was preferred to a smaller one when both reinforcers were obtained immediately; and (3) preference for an immediate, small reinforcer versus a delayed, large reinforcer increased as the delay preceding the large reinforcer increased, a sign of “impulsivity”. In Experiment 2, the schedules differed in amount or delay and equal intervals were added either to the constant parameter or the varied parameter. A shift from virtually exclusive preference to indifference occurred in the latter case but not the former, a result supporting a model of self-control that assumes that the value of a schedule depends on the ratio of amount and delay, and that choice between schedules depends on the ratio of these values.     

Stimulus Equivalence: Effects Of A Default-response Option On Emergence Of Untrained Stimulus Relations

Human subjects were exposed to a concurrent-chains schedule in which reinforcer amounts, delays, or both were varied in the terminal links, and consummatory responses were required to receive points that were later exchangeable for money. Two independent variable-interval 30-s schedules were in effect during the initial links, and delay periods were defined by fixed-time schedules. In Experiment 1, subjects were exposed to three different pairs of reinforcer amounts and delays, and sensitivity to reinforcer amount and delay was determined based on the generalized matching law. The relative responding (choice) of most subjects was more sensitive to reinforcer amount than to reinforcer delay. In Experiment 2, subjects chose between immediate smaller reinforcers and delayed larger reinforcers in five conditions with and without timeout periods that followed a shorter delay, in which reinforcer amounts and delays were combined to make different predictions based on local reinforcement density (i.e., points per delay) or overall reinforcement density (i.e., points per total time). In most conditions, subjects' choices were qualitatively in accord with the predictions from the overall reinforcement density calculated by the ratio of reinforcer amount and total time. Therefore, the overall reinforcement density appears to influence the preference of humans in the present self-control choice situation.     

Zebrafish choice behavior is sensitive to reinforcer rate,immediacy, and magnitude ratios

Behavioral flexibility has, in part, been defined by choice behavior changing as a function of changes in reinforcer payoffs. We examined whether the generalized matching law quantitatively described changes in choice behavior in zebrafish when relative reinforcer rates, delays/immediacy, and magnitudes changed between two alternatives across conditions. Choice was sensitive to each of the three reinforcer properties. Sensitivity estimates to changes in relative reinforcer rates were greater when 2 variable-interval schedules were arranged independently between alternatives (Experiment 1a) than when a single schedule pseudorandomly arranged reinforcers between alternatives (Experiment 1b). Sensitivity estimates for changes in relative reinforcer immediacy (Experiment 2) and magnitude (Experiment 3) were similar but lower than estimates for reinforcer rates. These differences in sensitivity estimates are consistent with studies examining other species, suggesting flexibility in zebrafish choice behavior in the face of changes in payoff as described by the generalized matching law.     

DELAY‐AMOUNT TRADEOFFS IN CHOICES BY PIGEONS AND RATS: HYPERBOLIC VERSUS EXPONENTIAL DISCOUNTING

An adjusting‐delay procedure was used to study the choices of pigeons and rats when both delay and amount of reinforcement were varied. In different conditions, the choice alternatives included one versus two reinforcers, one versus three reinforcers, and three versus two reinforcers. The delay to one alternative (the standard alternative) was kept constant in a condition, and the delay to the other (the adjusting alternative) was increased or decreased many times a session so as to estimate an indifference point—a delay at which the two alternatives were chosen about equally often. Indifference functions were constructed by plotting the adjusting delay as a function of the standard delay for each pair of reinforcer amounts. The experiments were designed to test the prediction of a hyperbolic decay equation that the slopes of the indifference functions should increase as the ratio of the two reinforcer amounts increased. Consistent with the hyperbolic equation, the slopes of the indifference functions depended on the ratios of the two reinforcer amounts for both pigeons and rats. These results were not compatible with an exponential decay equation, which predicts slopes of 1 regardless of the reinforcer amounts. Combined with other data, these findings provide further evidence that delay discounting is well described by a hyperbolic equation for both species, but not by an exponential equation. Quantitative differences in the y‐intercepts of the indifference functions from the two species suggested that the rate at which reinforcer strength decreases with increasing delay may be four or five times slower for rats than for pigeons.     

Effects of acute and chronic flunitrazepam on delay discounting in pigeons

Delay to delivery of a reinforcer can decrease responding for that reinforcer and increase responding for smaller reinforcers that are available concurrently and delivered without delay; acute administration of drugs can alter responding for large, delayed reinforcers, although the impact of chronic treatment on delay discounting is not well understood. In this experiment, the effects of repeated administration of the benzodiazepine flunitrazepam were studied in 6 pigeons responding on one key to receive food that was delivered immediately and on a second key to receive a larger amount of food that was delivered following delays which increased across a single session. Pigeons responded predominantly for the large reinforcer when there were no delays and when delays were short; however, as delays increased, responding for the large reinforcer decreased. Acutely, flunitrazepam (0.32, 1.0 and 3.2 mg/kg) dose-dependently increased responding for the large reinforcer, shifting the discounting curve rightward and upward. Repeated administration of flunitrazepam (0.32, 1.0 and 3.2 mg/kg, each for six sessions, separated by one session during which vehicle was administered) did not markedly alter its effects on responding for the large reinforcer, indicating that the development of tolerance to this effect of flunitrazepam is modest under these conditions.     

Choosing schedules of signaled appetitive events over schedules of unsignaled ones

Two experiments were completed allowing albino rats to choose between signaled and unsignaled reward conditions. These experiments examined the effects on preference of (1) response dependent versus response-independent reward and, (2) food pellets versus chocolate milk as the reward. All subjects preferred the signaled condition over the unsignaled condition, whether exposed to response-dependent, or to response-independent delivery of rewards. Preference was controlled most effectively by presenting both the signal itself and the correlated stimulus identifying the signaled condition. The signal presented alone (Extinction 3) controlled preference more effectively than did the stimulus correlated with the signaled condition (Extinction 2). The second experiment showed that the quality of the reinforcer (pellets and chocolate milk) did not affect preference for signaled reward since all subjects preferred the signaled condition at levels comparable to those observed in Experiment 1, with food pellets. These results, along with others, argue against species differences, response-dependency, and reinforcer quality as variables affecting the direction of preference.     

Effects of intertrial reinforcers on self-control choice.

In three experiments, pigeons chose between a small amount of food delivered after a short delay and a larger amount delivered after a longer delay. A discrete-trial adjusting-delay procedure was used to estimate indifference points--pairs of delay-amount combinations that were chosen about equally often. In Experiment 1, when additional reinforcers were available during intertrial intervals on a variable-interval schedule, preference for the smaller, more immediate reinforcer increased. Experiment 2 found that this shift in preference occurred partly because the variable-interval schedule started sooner after the smaller, more immediate reinforcer, but there was still a small shift in preference when the durations and temporal locations of the variable-interval schedules were identical for both alternatives. Experiment 3 found greater increases in preference for the smaller, more immediate reinforcer with a variable-interval 15-s schedule than with a variable-interval 90-s schedule. The results were generally consistent with a model that states that the impact of any event that follows a choice response declines according to a hyperbolic function with increasing time since the moment of choice.     

Effects of pre-trial response requirements on self-control choices by rats and pigeons

Parallel experiments with rats and pigeons examined whether the size of a pre-trial ratio requirement would affect choices in a self-control situation. In different conditions, either 1 response or 40 responses were required before each trial. In the first half of each experiment, an adjusting-ratio schedule was used, in which subjects could choose a fixed-ratio schedule leading to a small reinforcer, or an adjusting-ratio schedule leading to a larger reinforcer. The size of the adjusting ratio requirement was increased and decreased over trials based on the subject's responses, in order to estimate an indifference point-a ratio at which the two alternatives were chosen about equally often. The second half of each experiment used an adjusting-delay procedure-fixed and adjusting delays to the small and large reinforcers were used instead of ratio requirements. In some conditions, particularly with the reinforcer delays, the rats had consistently longer adjusting delays with the larger pre-trial ratios, reflecting a greater tendency to choose the larger, delayed reinforcer when more responding was required to reach the choice point. No consistent effects of the pre-trial ratio were found for the pigeons in any of the conditions. These results may indicate that rats are more sensitive to the long-term reinforcement rates of the two alternatives, or they may result from a shallower temporal discounting rate for rats than for pigeons, a difference that has been observed in previous studies.     

Self-Control and Impulsiveness in Adult Human Females: Effects of Food Preferences

Two experiments explored the effects of food preferences on humans’ choice in a self-control paradigm. Reinforcers consisted of programmed periods of access time to drinking juice during the experimental session. Experiment 1 used 14 adult women and Experiment 2 used 9 adult women. In Experiment 1, the women demonstrated significantly less sensitivity to reinforcer amount relative to sensitivity to reinforcer delay (a measure of self-control) when they reported a higher preference for the juice received as the less delayed, smaller reinforcer than for the juice received as the more delayed, larger reinforcer. Conversely in Experiment 2, the women demonstrated significantly more sensitivity to reinforcer amount relative to sensitivity to reinforcer delay when they reported a higher preference for the juice received as the more delayed, larger reinforcer than for the juice received as the less delayed, smaller reinforcer. Together, the results show that participants’ food preferences can influence self-control for food reinforcers.     

ON THE RELATION BETWEEN REINFORCER EFFICACY AND PREFERENCE

Most research on stimulus preference and reinforcer assessment involves a preference assessment that is followed by a reinforcer assessment. Typically, the most and least preferred stimuli are tested as reinforcers. In the current study, we first quantified the reinforcing efficacies of six food items and then assessed relative preference for each item. Relative preference ranking and reinforcer efficacies showed almost perfect concordance for 1 participant and partial concordance for the other. Discordance tended to occur with the weakest reinforcers.