Choice and reinforcement delay

Previous studies of choice between two delayed reinforcers have indicated that the relative immediacy of the reinforcer is a major determinant of the relative frequency of responding. Parallel studies of choice between two interresponse times have found exceptions to this generality. The present study looked at the choice by pigeons between two delays, one of which was always four times longer than the other, but whose absolute durations were varied across conditions. The results indicated that choice is not uniquely determined by the relative immediacy of reinforcement, but that absolute delays are also involved. Models for concurrent chained schedules appear to be more applicable to the present data than the matching relation; however, these too failed to predict choice for long delays.     

Choice and delay of reinforcement

Pigeons were trained to peck either of two response keys for food reinforcement on equated aperiodic schedules. The distribution of responding at the two keys was studied as reinforcement was delayed for various durations. The relative frequency of responding at each key was shown to match the relative immediacy of reinforcement, immediacy defined as the reciprocal of the delay of reinforcement.     

Human self-control and the density of reinforcement

Choice responding in adult humans on a discrete-trial button-pressing task was examined as a function of amount, delay, and overall density (points per unit time) of reinforcement. Reinforcement consisted of points that were exchangeable for money. In T 0 conditions, an impulsive response produced 4 points immediately and a self-control response produced 10 points after a delay of 15 s. In T 15 conditions, a constant delay of 15 s was added to both prereinforcer delays. Postreinforcer delays, which consisted of 15 s added to the end of each impulsive trial, equated trial durations regardless of choice, and was manipulated in both T 0 and T 15 conditions. In all conditions, choice was predicted directly from the relative reinforcement densities of the alternatives. Self-control was observed in all conditions except T 0 without postreinforcer delays, where the impulsive choices produced the higher reinforcement density. These results support previous studies showing that choice is a direct function of the relative reinforcement densities when conditioned (point) reinforcers are used. In contrast, where responding produces intrinsic (immediately consumable) reinforcers, immediacy of reinforcement appears to account for preference when density does not.     

Rapid acquisition of preference in concurrent chains: effects of d-amphetamine on sensitivity to reinforcement delay

The purpose of this study was to examine effects of d-amphetamine on choice controlled by reinforcement delay. Eight pigeons responded under a concurrent-chains procedure in which one terminal-link schedule was always fixed-interval 8 s, and the other terminal-link schedule changed from session to session between fixed-interval 4 s and fixed-interval 16 s according to a 31-step pseudorandom binary sequence. After sufficient exposure to these contingencies (at least once through the pseudorandom binary sequence), the pigeons acquired a preference for the shorter reinforcement delay within each session. Estimates of the sensitivity to reinforcement immediacy were similar to those obtained in previous studies. For all pigeons, at least one dose of d-amphetamine attenuated preference and, hence, decreased estimates of sensitivity to reinforcement immediacy; in most cases, this effect occurred without a change in overall response rates. In many cases, the reduced sensitivity to reinforcement delay produced by d-amphetamine resulted primarily from a decrease in the asymptotic level of preference achieved within the session; in some cases, d-amphetamine produced complete indifference. These findings suggest that a reduction in the sensitivity to reinforcement delay may be an important behavioral mechanism of the effects of psychomotor stimulants.     

Responding on concurrent-chains schedules in open and closed economies.

Pigeons' key pecks were reinforced according to concurrent-chains schedules of reinforcement. The programmed average time from the onset of the initial links to a terminal link entry was held constant across conditions while the value of variable-interval schedules in the terminal links was varied. Performance was assessed under two economic conditions: (a) an open economy in which session duration was limited to 1 hr and subjects were maintained at 80% of their free-feeding body weights with postsession food when necessary; and (b) a closed economy in which sessions were 23.5 hr long and no deprivation regimen was in effect. In all cases, the relative rate of responding in the initial links matched the reduction in overall delay to primary reinforcement correlated with entry into one terminal link relative to the reduction in delay correlated with entry into the other terminal link. Although the sum of responses made in the initial links and terminal links was found to increase, then decrease, as the rate of food presentation decreased in the closed economy, there was no consistent effect of overall rate of food presentation on total responding in the open economy. The choice data suggest that relative delay reduction predicts choice accurately, regardless of economic context.     

Assessment of impulsivity and the development of self-control in students with attention deficit hyperactivity disorder.

We examined a combined approach of manipulating reinforcer dimensions and delay fading to promote the development of self‐control with 3 students diagnosed with attention deficit hyperactivity disorder. First, we administered a brief computer‐based assessment to determine the relative influence of reinforcer rate (R), reinforcer quality (Q), reinforcer immediacy (I), and effort (E) on the students' choices between concurrently presented math problems. During each session, one of these dimensions was placed in direct competition with another dimension (e.g., RvI involving math problem alternatives associated with high‐rate delayed reinforcement vs. low‐rate immediate reinforcement), with all possible pairs of dimensions presented across the six assessment conditions (RvQ, RvI, RvE, QvI, QvE, IvE). The assessment revealed that the choices of all 3 students were most influenced by immediacy of reinforcement, reflecting impulsivity. We then implemented a self‐control training procedure in which reinforcer immediacy competed with another influential dimension (RvI or QvI), and the delay associated with the higher rate or quality reinforcer alternative was progressively increased. The students allocated the majority of their time to the math problem alternatives yielding more frequent (high‐rate) or preferred (high‐quality) reinforcement despite delays of up to 24 hr. Subsequent readministration of portions of the assessment showed that self‐control transferred across untrained dimensions of reinforcement.     

BASIC AND APPLIED RESEARCH ON CHOICE RESPONDING

Choice responding refers to the manner in which individuals allocate their time or responding among available response options. In this article, we first review basic investigations that have identified and examined variables that influence choice responding, such as response effort and reinforcement rate, immediacy, and quality. We then describe recent bridge and applied studies that illustrate how the results of basic research on choice responding can help to account for human behavior in natural environments and improve clinical assessments and interventions.     

Maximizing present value: A model to explain why moderate response rates obtain on variable-interval schedules

In Phases 1 and 3, two Japanese monkeys responded on a multiple variable-ratio 80 variable-interval X schedule, where the value of X was adjusted to ensure equal between-schedule reinforcement rates. Components strictly alternated following the delivery of a food pellet, and each session ended following 50 components. Phase 2 differed from the others only in that the 50 pellets previously earned during the session were delivered together at session's end. Variable-ratio response rates did not decrease across phases, but variable-interval response rates decreased substantially during the Phase 2 procedure. This rate decrease was attributed to the food-at-session's-end manipulation removing the greater immediacy of reinforcement provided by short interresponse times relative to long interresponse times. Without this time preference for short interresponse times, the variable-interval interresponse-time reinforcement feedback function largely controlled response emission, dictating a response-rate reduction. This result was explained in terms of the economic notion of “maximizing present value.”     

Dry pants: A rapid method of toilet training children

Toilet training sometimes requires considerable time. An intensive learning procedure was devised for shortening this training time and tested with 34 children who were experiencing toilet training problems. The procedure had the following major characteristics: (1) a distraction-free environment, (2) an increased frequency of urination by increased fluid intake, (3) continuous practice and reinforcement of the necessary dressing skills, (4) continuous practice and reinforcement in approaching the toilet, (5) detailed and continuing instruction for each act required in toileting, (6) gradual elimination of the need for reminders to toilet, (7) immediate detection of accidents, (8) a period of required practice in toilet-approach after accidents as well as (9) negative reinforcement for the accident, (10) immediate detection of correct toileting, (11) immediacy of reinforcement for correct toiletings, (12) a multiple reinforcement system including imagined social benefits as well as actual praise, hugging and sweets, (13) continuing reinforcement for having dry pants, (14) learning by imitation, (15) gradual reduction of the need for immediate reinforcement and (16) post-training attention to cleanliness. All 34 children were trained and in an average of 4 hr; children over 26 months old required an average of 2 hr of training. After training, accidents decreased to a near-zero level and remained near zero during 4 months of follow-up. The results suggest that virtually all healthy children who have reached 20 months of age can be toilet trained and within a few hours.     

Choice and rate of reinforcement

Pigeons' responses in the presence of two concurrently available (initial-link) stimuli produced one of two different (terminal-link) stimuli. The rate of reinforcement in the presence of one terminal-link stimulus was three times that of the other. Three different pairs of identical but independent variable-interval schedules controlled entry into the terminal links. When the intermediate pair was in effect, the pigeons distributed their (choice) responses in the presence of the concurrently available stimuli of the initial links in the same proportion as reinforcements were distributed in the mutually exclusive terminal links. This finding was consistent with those of earlier studies. When either the pair of larger or smaller variable-interval schedules was in effect, however, proportions of choice responses did not match proportions of reinforcements. In addition, matching was not obtained when entry into the terminal links was controlled by unequal variable-interval schedules. A formulation consistent with extant data states that choice behavior is dependent upon the amount of reduction in the expected time to primary reinforcement, as signified by entry into one terminal link, relative to the amount of reduction in expected time to reinforcement signified by entry into the other terminal link.     

Independence of terminal-link entry rate and immediacy in concurrent chains

In Phase 1, 4 pigeons were trained on a three-component multiple concurrent-chains procedure in which components differed only in terms of relative terminal-link entry rate. The terminal links were variable-interval schedules and were varied across four conditions to produce immediacy ratios of 4:1, 1:4, 2:1, and 1:2. Relative terminal-link entry rate and relative immediacy had additive and independent effects on initial-link response allocation, and the data were well-described by a generalized-matching model. Regression analyses showed that allowing sensitivity to immediacy to vary across components produced only trivial increases in variance accounted for. Phase 2 used a three-component concurrent-schedules procedure in which the schedules were the same as the initial links of Phase 1. Across two conditions, the relative reinforcer magnitude was varied. Sensitivity to relative reinforcer rate was independent of relative magnitude, confirming results of prior studies. Sensitivity to relative reinforcer rate in Phase 2 did not vary systematically across subjects compared to sensitivity to relative entry rate in Phase 1, and regression analyses confirmed again that only small increases in variance accounted for were obtained when sensitivities were estimated independently compared with a single estimate for both phases. Overall, the data suggest that conditioned and primary reinforcers have functionally equivalent effects on choice and support the independence of relative terminal-link entry rate and immediacy as determiners of response allocation. These results are consistent with current models for concurrent chains, including Grace's (1994) contextual choice model and Mazur's (2001) hyperbolic value-added model.     

Effects of reinforcement magnitude on pigeons' preference for different fixed-ratio schedules of reinforcement

In concurrent, two-member chains, the completion of one or the other of two initial percentage fixed-interval 90-sec links produced a terminal link in which the completion of a fixed ratio produced food reinforcement. The fixed ratios and the duration of reinforcement in the terminal links were varied. Relative response rate in initial links was proportional to the relative reinforcement duration per ratio response (reinforcement duration divided by fixed ratio) in terminal links. The rate of responding in the terminal fixed-ratio links was insensitive to both ratio size and reinforcement duration and therefore did not vary sufficiently to distinguish between responses per reinforcement and immediacy of reinforcement as controlling variables in terminal links.     

Zebrafish choice behavior is sensitive to reinforcer rate,immediacy, and magnitude ratios

Behavioral flexibility has, in part, been defined by choice behavior changing as a function of changes in reinforcer payoffs. We examined whether the generalized matching law quantitatively described changes in choice behavior in zebrafish when relative reinforcer rates, delays/immediacy, and magnitudes changed between two alternatives across conditions. Choice was sensitive to each of the three reinforcer properties. Sensitivity estimates to changes in relative reinforcer rates were greater when 2 variable-interval schedules were arranged independently between alternatives (Experiment 1a) than when a single schedule pseudorandomly arranged reinforcers between alternatives (Experiment 1b). Sensitivity estimates for changes in relative reinforcer immediacy (Experiment 2) and magnitude (Experiment 3) were similar but lower than estimates for reinforcer rates. These differences in sensitivity estimates are consistent with studies examining other species, suggesting flexibility in zebrafish choice behavior in the face of changes in payoff as described by the generalized matching law.     

Temporal Patterning in Choice among Delayed Outcomes

Undergraduates chose repeatedly between two equal monetary outcomes. One outcome was more immediate, but the other resulted in less delay on future trials and a lower average delay per outcome overall. Minimizing average delay per outcome maximized overall reinforcement rate (money earned). The difference in immediacy between the two alternatives on a given trial was easily perceptible but the (opposed) difference in overall reinforcement rate was gradual and difficult to perceive. Subjects frequently chose the more immediate outcome and thus failed to maximize overall reinforcement rate. However, when choice-outcome (CO) units were grouped in triplets (as opposed to being presented singly at fixed intervals) subjects chose the more immediate outcome less often and thereby increased overall reinforcement rate. Findings are discussed in terms of impulsiveness and risk-aversion in laboratory studies and in everyday life.     

How to teach a pigeon to maximize overall reinforcement rate

In two experiments deviations from matching earned higher overall reinforcement rates than did matching. In Experiment 1 response proportions were calculated over a 360-response moving average, updated with each response. Response proportions that differed from the nominal reinforcement proportions, by a criterion that was gradually increased, were eligible for reinforcement. Response proportions that did not differ from matching were not eligible for reinforcement. When the deviation requirement was relatively small, the contingency proved to be effective. However, there was a limit as to how far response proportions could be pushed from matching. Consequently, when the deviation requirement was large, overall reinforcement rate decreased and pecking was eventually extinguished. In Experiment 2 a discriminative stimulus was added to the procedure. The houselight was correlated with the relationship between response proportions and the nominal (programmed) reinforcement proportions. When the difference between response and reinforcement proportions met the deviation requirement, the light was white and responses were eligible for reinforcement. When the difference between response and reinforcement proportions failed to exceed the deviation requirement, the light was blue and responses were not eligible for reinforcement. With the addition of the light, it proved to be possible to shape deviations from matching without any apparent limit. Thus, in Experiment 2 overall reinforcement rate predicted choice proportions and relative reinforcement rate did not. In contrast, in previous experiments on the relationship between matching and overall reinforcement maximization, relative reinforcement rate was usually the better predictor of responding. The results show that whether overall or relative reinforcement rate better predicts choice proportions may in part be determined by stimulus conditions.     

Resistance to change produced by access to fixed-delay versus variable-delay terminal links

Pigeons' responding was reinforced on a multiple schedule consisting of two two-link chain schedules presented in regular alternation. Responding in initial links (always variable-interval 60-s) produced a key-color change and access to a terminal link. The terminal link for one chain provided food after a fixed delay (fixed-interval or fixed-time); the terminal link for the other provided food after a variable delay (variable-interval or variable-time). The average duration of the terminal-link schedules was varied across conditions, but in every condition the arithmetic mean of the variable-delay terminal-link schedule was equal to the duration of the fixed delay. Response rates were higher in the initial links of the chains with the variable-delay terminal links. Response-decreasing operations (satiation, extinction) were used after performances reached asymptote. Response rates maintained by access to variable-delay terminal links tended to be more resistant to change than were rates maintained by access to fixed-delay terminal links. These results are consistent with the preference for variable- over fixed-interval terminal links observed with concurrent-chains schedules, suggesting (1) that immediacy of reinforcement influences the conditioned reinforcing potency of access to a terminal link and (2) that choice in concurrent chains and resistance of responding to change may be manifestations of the same effect of reinforcement.     

Matching, maximizing, and the behavioral unit: concurrent reinforcement of response sequences.

Pigeons pecked two keys in a probability matching situation in which four two-peck sequences were intermittently reinforced: left-left, left-right, right-left and right-right. In Phase 1, relative reinforcement rate was varied with respect to the first response of a sequence: reinforcers were differentially assigned for left-left and left-right sequences as opposed to right-left and right-right sequences. The second response of reinforced sequences occurred equally on the left and right keys across conditions. In Phase II, relative reinforcement rate was varied for sequences that involve an alternation as opposed to those that did not. The relative outputs of the different sequences matched the relative reinforcement rates for the different sequences in both phases. Relative response rates for key pecks did not always match relative reinforcement rates. The intertrial interval separating responses was varied in both phases; increases in the intertrial interval affected the relative frequency of different sequences. The results demonstrate that response sequences acted as functional units influencing choice and thus support a structural account of choice. At the same time, the matching of relative sequence proportion and relative reinforcement rate supports a matching account.     

Variation, repetition, and choice

Experiment 1 investigated the controlling properties of variability contingencies on choice between repeated and variable responding. Pigeons were exposed to concurrent-chains schedules with two alternatives. In the REPEAT alternative, reinforcers in the terminal link depended on a single sequence of four responses. In the VARY alternative, a response sequence in the terminal link was reinforced only if it differed from the n previous sequences (lag criterion). The REPEAT contingency generated low, constant levels of sequence variation whereas the VARY contingency produced levels of sequence variation that increased with the lag criterion. Preference for the REPEAT alternative tended to increase directly with the degree of variation required for reinforcement. Experiment 2 examined the potential confounding effects in Experiment 1 of immediacy of reinforcement by yoking the interreinforcer intervals in the REPEAT alternative to those in the VARY alternative. Again, preference for REPEAT was a function of the lag criterion. Choice between varying and repeating behavior is discussed with respect to obtained behavioral variability, probability of reinforcement, delay of reinforcement, and switching within a sequence.     

Matching and conditioned reinforcement rate

Attempts to examine the effects of variations in relative conditioned reinforcement rate on choice have been confounded by changes in rates of primary reinforcement or changes in the value of the conditioned reinforcer. To avoid these problems, this experiment used concurrent observing responses to examine sensitivity of choice to relative conditioned reinforcement rate. In the absence of observing responses, unsignaled periods of food delivery on a variable-interval 90-s schedule alternated with extinction on a center key (i.e., a mixed schedule was in effect). Two concurrently available observing responses produced 15-s access to a stimulus differentially associated with the schedule of food delivery (S+). The relative rate of S+ deliveries arranged by independent variable-interval schedules for the two observing responses varied across conditions. The relation between the ratio of observing responses and the ratio of S+ deliveries was well described by the generalized matching law, despite the absence of changes in the rate of food delivery. In addition, the value of the S+ deliveries likely remained constant across conditions because the ratio of S+ to mixed schedule food deliveries remained constant. Assuming that S+ deliveries serve as conditioned reinforcers, these findings are consistent with the functional similarity between primary and conditioned reinforcers suggested by general choice theories based on the concatenated matching law (e.g., contextual choice and hyperbolic value-added models). These findings are inconsistent with delay reduction theory, which has no terms for the effects of rate of conditioned reinforcement in the absence of changes in rate of primary reinforcement.