The effects of unsignalled delayed reinforcement

Pigeons' pecks were reinforced according to a variable-interval schedule. A delay-of-reinforcement procedure was then added to the schedule, or a yoked-control procedure was arranged where the reinforcers occurred independently of responding according to the same variable-interval schedule. During the delay-of-reinforcement procedure, the first peck after a reinforcer was scheduled began a delay timer and the reinforcer was delivered at the end of the interval. No stimulus change signalled the delay interval and responses could occur during it, so that the obtained delays were often shorter than those scheduled. Responding under this procedure was highly variable but, in general, behavior was substantially reduced even with the shortest delay used, 3 sec. In addition, the rates maintained by delayed reinforcement were only slightly greater than those maintained by the yoked-control procedure, suggesting that adventitious pairings of response and reinforcer were responsible for some of the maintenance of behavior that did occur. The results challenge recent conceptions of reinforcement as involving response-reinforcer correlations and re-emphasize the role of temporal proximity between response and reinforcer.     

Signal functions in delayed reinforcement

Three experiments were conducted with pigeons to examine the role of the signal in delay-of-reinforcement procedures. In the first, a blackout accompanying a period of nonreinforcement increased key-peck response rates maintained by immediate reinforcement. The effects of dissociating the blackout from the delay interval were examined in the second experiment. In three conditions, blackouts and unsignaled delays were negatively correlated or occurred randomly with respect to one another. A signaled delay and an unsignaled delay that omitted the blackouts were studied in two other conditions. All delay-of-reinforcement conditions generally produced response rates lower than those produced by immediate reinforcement. Signaled delays maintained higher response rates than did any of the various unsignaled-delay conditions, with or without dissociated blackouts. The effects of these latter conditions did not differ systematically from one another. The final experiment showed that response rates varied as a function of the frequency with which a blackout accompanied delay intervals. By eliminating a number of methodological difficulties present in previous delay-of-reinforcement experiments, these results suggest the importance of the signal in maintaining responding during delay-of-reinforcement procedures and, conversely, the importance of the delay interval in decreasing responding.     

The role of the response-reinforcer relation in delay-of-reinforcement effects

The role of the response-reinforcer relation in maintaining operant behavior under conditions of delayed reinforcement was investigated by using a two-operandum (i.e., two-key) procedure with pigeons. Responding on one key was reinforced under a tandem variable-interval differential-reinforcement-of-other-behavior (tandem VI DRO) schedule. The schedule defined a resetting unsignaled delay-of-reinforcement procedure in that a response was required when the interfood interval of the VI schedule lapsed, but further responding during the DRO component on either key reset the time interval. This ensured a fixed delay duration between any response and reinforcement. Responding on another key, physically identical to the first one except for spatial location, otherwise was without consequence. The location of the key correlated with the delay-of-reinforcement procedure varied between sessions according to a semirandom sequence. Differences in response rates between the two keys were greater, with proportionally higher rates on the key correlated with the delay-of-reinforcement procedure, the longer the delay-of-reinforcement procedure remained correlated with the same key. Differences in responding on the two keys also increased within individual sessions. These results suggest that the response-reinforcer relation is the primary determinant of responding when responding is acquired and maintained with delayed reinforcement.     

Effects of signaled versus unsignaled delay of reinforcement on choice

Pigeons chose between 5-s and 15-s delay-of-reinforcement alternatives. The first key peck to satisfy the choice schedule began a delay timer, and food was delivered at the end of the interval. Key pecks during the delay interval were measured, but had no scheduled effect. In Experiment 1, signal conditions and choice schedules were varied across conditions. During unsignaled conditions, no stimulus change signaled the beginning of a delay interval. During differential and nondifferential signal conditions, offset of the choice stimuli and onset of a delay stimulus signaled the beginning of a delay interval. During differential signal conditions, different stimuli were correlated with the 5-s and 15-s delays, whereas the same stimulus appeared during both delay durations during nondifferential signal conditions. Pigeons showed similar, extreme levels of preference for the 5-s delay alternative during unsignaled and differentially signaled conditions. Preference levels were reliably lower with nondifferential signals. Experiment 2 assessed preference with two pairs of unsignaled delays in which the ratio of delays was held constant but the absolute duration was increased fourfold. No effect of absolute duration was found. The results highlight the importance of delayed primary reinforcement effects and challenge models of choice that focus solely on conditioned reinforcement.     

Unsignalled delay of reinforcement in variable-interval schedules

Three pigeons responded on several tandem variable-interval fixed-time schedules in which the value of the fixed-time component was varied to assess the effects of different unsignalled delays of reinforcement. Actual (obtained) delays between the last key peck in an interval and reinforcement were consistently shorter than the nominal (programmed) delay. When nominal delays were relatively short, response rates were higher during the delay condition than during the corresponding nondelay condition. At longer nominal delay intervals, response rates decreased monotonically with increasing delays. The results were consistent with those obtained from delay-of-reinforcement procedures that impose either a stimulus change (signal) or a no-response requirement during the delay interval.     

Changes in functional response units with briefly delayed reinforcement

In two experiments, key-peck responding of pigeons was compared under variable-interval schedules that arranged immediate reinforcement and ones that arranged unsignaled delays of reinforcement. Responses during the nominal unsignaled delay periods had no effect on the reinforcer presentations. In Experiment 1, the unsignaled delays were studied using variable-interval schedules as baselines. Relative to the immediate reinforcement condition, 0.5-s unsignaled delays decreased the duration of the reinforced interresponse times and increased the overall frequency of short (<0.5-s) interresponse times. Longer, 5.0-s unsignaled delays increased the duration of the reinforced interresponse times and decreased the overall frequency of the short interresponse times. In Experiment 2, similar effects to those of Experiment 1 were obtained when the 0.5-s unsignaled delays were imposed upon a baseline schedule that explicitly arranged reinforcement of short interresponse times and therefore already generated a large number of short interresponse times. The results support earlier suggestions that the unsignaled 0.5-s delays change the functional response unit from a single key peck to a multiple key-peck unit. These findings are discussed in terms of the mechanisms by which contingencies control response structure in the absence of specific structural requirements.     

Responding of pigeons under variable-interval schedules of unsignaled, briefly signaled, and completely signaled delays to reinforcement

In Experiment 1, three pigeons' key pecking was maintained under a variable-interval 60-s schedule of food reinforcement. A 1-s unsignaled nonresetting delay to reinforcement was then added. Rates decreased and stabilized at values below those observed under immediate-reinforcement conditions. A brief stimulus change (key lit red for 0.5 s) was then arranged to follow immediately the peck that began the delay. Response rates quickly returned to baseline levels. Subsequently, rates near baseline levels were maintained with briefly signaled delays of 3 and 9 s. When a 27-s briefly signaled delay was instituted, response rates decreased to low levels. In Experiment 2, four pigeons' responding was first maintained under a multiple variable-interval 60-s (green key) variable-interval 60-s (red key) schedule. Response rates in both components fell to low levels when a 3-s unsignaled delay was added. In the first component delays were then briefly signaled in the same manner as Experiment 1, and in the second component they were signaled with a change in key color that remained until food was delivered. Response rates increased to near baseline levels in both components, and remained near baseline when the delays in both components were lengthened to 9 s. When delays were lengthened to 27 s, response rates fell to low levels in the briefly signaled delay component for three of four pigeons while remaining at or near baseline in the completely signaled delay component. In Experiment 3, low response rates under a 9-s unsignaled delay to reinforcement (tandem variable-interval 60 s fixed-time 9 s) increased when the delay was briefly signaled. The role of the brief stimulus as conditioned reinforcement may be a function of its temporal relation to food, and thus may be related to the eliciting function of the stimulus.     

Mechanisms underlying the effects of unsignaled delayed reinforcement on key pecking of pigeons under variable-interval schedules.

Three experiments were conducted to test an interpretation of the response-rate-reducing effects of unsignaled nonresetting delays to reinforcement in pigeons. According to this interpretation, rates of key pecking decrease under these conditions because key pecks alternate with hopper-observing behavior. In Experiment 1, 4 pigeons pecked a food key that raised the hopper provided that pecks on a different variable-interval-schedule key met the requirements of a variable-interval 60-s schedule. The stimuli associated with the availability of the hopper (i.e., houselight and keylight off, food key illuminated, feedback following food-key pecks) were gradually removed across phases while the dependent relation between hopper availability and variable-interval-schedule key pecks was maintained. Rates of pecking the variable-interval-schedule key decreased to low levels and rates of food-key pecks increased when variable-interval-schedule key pecks did not produce hopper-correlated stimuli. In Experiment 2, pigeons initially pecked a single key under a variable-interval 60-s schedule. Then the dependent relation between hopper presentation and key pecks was eliminated by arranging a variable-time 60-s schedule. When rates of pecking had decreased to low levels, conditions were changed so that pecks during the final 5 s of each interval changed the keylight color from green to amber. When pecking produced these hopper-correlated stimuli, pecking occurred at high rates, despite the absence of a peck-food dependency. When peck-produced changes in keylight color were uncorrelated with food, rates of pecking fell to low levels. In Experiment 3, details (obtained delays, interresponse-time distributions, eating times) of the transition from high to low response rates produced by the introduction of a 3-s unsignaled delay were tracked from session to session in 3 pigeons that had been initially trained to peck under a conventional variable-interval 60-s schedule. Decreases in response rates soon after the transition to delayed reinforcement were accompanied by decreases in eating times and alterations in interresponse-time distributions. As response rates decreased and became stable, eating times increased and their variability decreased. These findings support an interpretation of the effects of delayed reinforcement that emphasizes the importance of hopper-observing behavior.     

Responding of pigeons under variable-interval schedules of signaled-delayed reinforcement: effects of delay-signal duration.

Two experiments with pigeons examined the relation of the duration of a signal for delay ("delay signal") to rates of key pecking. The first employed a multiple schedule comprised of two components with equal variable-interval 60-s schedules of 27-s delayed food reinforcement. In one component, a short (0.5-s) delay signal, presented immediately following the key peck that began the delay, was increased in duration across phases; in the second component the delay signal initially was equal to the length of the programmed delay (27 s) and was decreased across phases. Response rates prior to delays were an increasing function of delay-signal duration. As the delay signal was decreased in duration, response rates were generally higher than those obtained under identical delay-signal durations as the signal was increased in duration. In Experiment 2 a single variable-interval 60-s schedule of 27-s delayed reinforcement was used. Delay-signal durations were again increased gradually across phases. As in Experiment 1, response rates increased as the delay-signal duration was increased. Following the phase during which the signal lasted the entire delay, shorter delay-signal-duration conditions were introduced abruptly, rather than gradually as in Experiment 1, to determine whether the gradual shortening of the delay signal accounted for the differences observed in response rates under identical delay-signal conditions in Experiment 1. Response rates obtained during the second exposures to the conditions with shorter signals were higher than those observed under identical conditions as the signal duration was increased, as in Experiment 1. In both experiments, rates and patterns of responding during delays varied greatly across subjects and were not systematically related to delay-signal durations. The effects of the delay signal may be related to the signal's role as a discriminative stimulus for adventitiously reinforced intradelay behavior, or the delay signal may have served as a conditioned reinforcer by virtue of the temporal relation between it and presentation of food.     

Body weight and response acquisition with delayed reinforcement.

The relation between body weight and responding established with unsignaled delayed reinforcement was investigated. In three experiments, naive rats were deprived to either 70%, 80%, or 90% of ad libitum weight and were then exposed to tandem variable-interval 15-s differential-reinforcement-of-other-behavior 30-s schedules. The tandem schedule defined a resetting unsignaled delay-of-reinforcement procedure. In the first experiment, speed of magazine training, acquisition of lever pressing, and final rate of lever pressing were related to body weight. In the next experiment, lever pressing was established and maintained in rats that were magazine trained at 70% of ad libitum weight but that were then exposed to the delay procedure at 90% of ad libitum weight. Responding did not change consistently either across or within subjects in subsequent conditions in which body weight was manipulated. In the final experiment, lever pressing was established and maintained with delayed reinforcement in the absence of magazine training for each of 2 rats at 70% and for 1 of 2 rats at 90% of ad libitum weight. The results further illuminate the conditions under which responding can be established in the absence of training and when such responses are reinforced only following an unsignaled delay period.     

Fixed and variable ratios and delays: further tests of an equivalence rule

A discrete-trial procedure was used to measure pigeons' choices between fixed and variable ratio schedules and between fixed and variable delays before reinforcement. A peck at a green key produced a reinforcement schedule that was constant within a condition but varied across conditions. A peck at a red key produced a ratio schedule (or, in other conditions, a simple delay) whose size was increased or decreased many times a session, depending on the subject's previous choices. The purpose of these adjustments was to estimate an indifference point--a ratio size (or delay duration) at which the subject chose each key about equally often. The results were used to test a simple "equivalence rule" for choices between fixed and variable schedules (Mazur, 1984). This rule, which was applied without using free parameters, predicted the major trends in the obtained indifference points from both ratio and delay conditions. However, some small but consistent deviations from the predictions were apparent. Better predictions were generated with a more complex equation, which included parameters reflecting the subjects' sensitivities to delay of reinforcement and to events of different probabilities. It was concluded that a successful equivalence rule must include parameters that can be adjusted to describe the effects of delay and probability in a given experimental setting. Once these parameters are estimated, however, choices involving both fixed and variable delays and fixed and variable ratios can be accurately predicted with the same equation.     

DISCRIMINATION ACQUISITION IN CHILDREN WITH DEVELOPMENTAL DISABILITIES UNDER IMMEDIATE AND DELAYED REINFORCEMENT

We evaluated the discrimination acquisition of individuals with developmental disabilities under immediate and delayed reinforcement. In Experiment 1, discrimination between two alternatives was examined when reinforcement was immediate or delayed by 20 s, 30 s, or 40 s. In Experiment 2, discrimination between 2 alternatives was compared across an immediate reinforcement condition and a delayed reinforcement condition in which subjects could respond during the delay. In Experiment 3, discrimination among 4 alternatives was compared across immediate and delayed reinforcement. In Experiment 4, discrimination between 2 alternatives was examined when reinforcement was immediate and 0‐s or 30‐s intertrial intervals (ITI) were programmed. For most subjects, discrimination acquisition occurred under immediate reinforcement. However, for some subjects, introducing delays slowed or prevented discrimination acquisition under some conditions. Results from Experiment 4 suggest that longer ITIs cannot account for the lack of discrimination under delayed reinforcement.     

Disruption of responding maintained by conditioned reinforcement: alterations in response-conditioned-reinforcer relations

An observing procedure was used to investigate the effects of alterations in response-conditioned-reinforcer relations on observing. Pigeons responded to produce schedule-correlated stimuli paired with the availability of food or extinction. The contingency between observing responses and conditioned reinforcement was altered in three experiments. In Experiment 1, after a contingency was established in baseline between the observing response and conditioned reinforcement, it was removed and the schedule-correlated stimuli were presented independently of responding according to a variable-time schedule. The variable-time schedule was constructed such that the rate of stimulus presentations was yoked from baseline. The removal of the observing contingency reliably reduced rates of observing. In Experiment 2, resetting delays to conditioned reinforcement were imposed between observing responses and the schedule-correlated stimuli they produced. Delay values of 0, 0.5, 1, 5, and 10 s were examined. Rates of observing varied inversely as a function of delay value. In Experiment 3, signaled and unsignaled resetting delays between observing responses and schedule-correlated stimuli were compared. Baseline rates of observing were decreased less by signaled delays than by unsignaled delays. Disruptions in response-conditioned-reinforcer relations produce similar behavioral effects to those found with primary reinforcement.     

Repeated,within‐session resurgence

Resurgence is a reliable, transient effect that only occasionally is replicated more than once within a single experiment or subject. In the present experiments, within‐session resurgence was generated repeatedly by dividing individual sessions into three phases (Training, Alternative‐Reinforcement, and Resurgence‐Test). In Experiments 1 and 2, resurgence reliably occurred in most of the 22‐30 daily sessions when responding was reinforced on, respectively, fixed‐ and variable‐interval schedules. Resurgence magnitude and duration did decrease across replications for some subjects, but not for others. To examine the utility of the procedure in studying the effects of an independent variable on resurgence, in Experiment 3 the effects of rich and lean baseline and alternative reinforcement rates on resurgence were compared. The target response was eliminated more rapidly, resurgence occurred more often, and usually was greater following rich alternative reinforcement rates. Resurgence was of greater magnitude when the baseline reinforcement rate was relatively lean compared to the alternative reinforcement rate. These experiments provide a reliable method for generating resurgence within individual sessions, instead of across multiple‐session conditions, that can be repeated over many successive sessions.     

Rapid acquisition of preference in concurrent chains: effects of d-amphetamine on sensitivity to reinforcement delay

The purpose of this study was to examine effects of d-amphetamine on choice controlled by reinforcement delay. Eight pigeons responded under a concurrent-chains procedure in which one terminal-link schedule was always fixed-interval 8 s, and the other terminal-link schedule changed from session to session between fixed-interval 4 s and fixed-interval 16 s according to a 31-step pseudorandom binary sequence. After sufficient exposure to these contingencies (at least once through the pseudorandom binary sequence), the pigeons acquired a preference for the shorter reinforcement delay within each session. Estimates of the sensitivity to reinforcement immediacy were similar to those obtained in previous studies. For all pigeons, at least one dose of d-amphetamine attenuated preference and, hence, decreased estimates of sensitivity to reinforcement immediacy; in most cases, this effect occurred without a change in overall response rates. In many cases, the reduced sensitivity to reinforcement delay produced by d-amphetamine resulted primarily from a decrease in the asymptotic level of preference achieved within the session; in some cases, d-amphetamine produced complete indifference. These findings suggest that a reduction in the sensitivity to reinforcement delay may be an important behavioral mechanism of the effects of psychomotor stimulants.     

Choice between single and multiple delayed reinforcers.

Pigeons chose between alternatives that differed in the number of reinforcers and in the delay to each reinforcer. A peck on a red key produced the same consequences on every trial within a condition, but between conditions the number of reinforcers varied from one to three and the reinforcer delays varied between 5 s and 30 s. A peck on a green key produced a delay of adjustable duration and then a single reinforcer. The green-key delay was increased or decreased many times per session, depending on a subject's previous choices, which permitted estimation of an indifference point, or a delay at which a subject chose each alternative about equally often. The indifference points decreased systematically with more red-key reinforcers and with shorter red-key delays. The results did not support the suggestion of Moore (1979) that multiple delayed reinforcers have no effect on preference unless they are closely grouped. The results were well described in quantitative detail by a simple model stating that each of a series of reinforcers increases preference, but that a reinforcer's effect is inversely related to its delay. The success of this model, which considers only delay of reinforcement, suggested that the overall rate of reinforcement for each alternative had no effect on choice between those alternatives.     

Unsignaled delay of reinforcement, relative time, and resistance to change

Two experiments with pigeons examined the effects of unsignaled, nonresetting delays of reinforcement on responding maintained by different reinforcement rates. In Experiment 1, 3-s unsignaled delays were introduced into each component of a multiple variable-interval (VI) 15-s VI 90-s VI 540-s schedule. When considered as a proportion of the preceding immediate reinforcement baseline, responding was decreased similarly for the three multiple-schedule components in both the first six and last six sessions of exposure to the delay. In addition, the relation between response rates and reinforcement rates was altered such that both parameters of the single-response version of the matching law (i.e., k and Re) were decreased. Experiment 2 examined the effects of unsignaled delays ranging from 0.5 s to 8.0 s on responding maintained by a multiple VI 20-s VI 120-s schedule of reinforcement. Response rates in both components increased with brief unsignaled delays and decreased with longer delays. As in Experiment 1, response rates as a proportion of baseline were affected similarly for the two components in both the first six and last six sessions of exposure to the delay. Unlike delays imposed between two stimulus events, the effects of delays between responses and reinforcers do not appear to be attenuated when the average time between reinforcers is longer. In addition, the disruptions produced by unsignaled delays appear to be inconsistent with the general finding that responding maintained by higher rates of reinforcement is less resistant to change.     

Varying reinforcement magnitude on interval schedules

In Experiment 1, rats were trained on either a random-interval or a variable-interval 60-sec schedule of reinforcement, and reinforcement magnitude was varied across conditions between one and four pellets. Although the two schedules maintained different patterns of behaviour, patterns and rates of responding were not systematically affected by the variation in reinforcement magnitude. In Experiment 2, a regulated probability interval schedule that generated similar rates of reinforcement to those of the schedules of Experiment 1 was used, with the pattern of behaviour generated resembling that typical of a random-interval schedule. Changing reinforcement magnitude again produced few systematic changes in behaviour. In Experiment 3, a variable-ratio schedule was used within a procedure that otherwise resembled that of Experiments 1 and 2. Increasing the reinforcement magnitude now decreased the rates of responding, and examination of the patterns of responding showed that this came about because rates of responding were higher early in the interreinforcer interval in the one-pellet condition. These experiments demonstrate the insensitivity of behaviour under interval schedules to changes in reinforcement magnitude and suggest the operation of mechanisms different from those engaged by ratio schedules and discretetrial learning procedures.     

A comparison of delays and ratio requirements in self-control choice.

In a discrete-trial procedure, pigeons could choose between 2-s and 6-s access to grain by making a single key peck. In Phase 1, the pigeons obtained both reinforcers by responding on fixed-ratio schedules. In Phase 2, they received both reinforcers after simple delays, arranged by fixed-time schedules, during which no responses were required. In Phase 3, the 2-s reinforcer was available through a fixed-time schedule and the 6-s reinforcer was available through a fixed-ratio schedule. In all conditions, the size of the delay or ratio leading to the 6-s reinforcer was systematically increased or decreased several times each session, permitting estimation of an "indifference point," the schedule size at which a subject chose each alternative equally often. By varying the size of the schedule for the 2-s reinforcer across conditions, several such indifference points were obtained from both fixed-time conditions and fixed-ratio conditions. The resulting "indifference curves" from fixed-time conditions and from fixed-ratio conditions were similar in shape, and they suggested that a hyperbolic equation describes the relation between ratio size and reinforcement value as well as the relation between reinforcer delay and its reinforcement value. The results from Phase 3 showed that subjects chose fixed-time schedules over fixed-ratio schedules that generated the same average times between a choice response and reinforcement.     

Relative sensitivity to reinforcer amount and delay in a self-control choice situation.

Rats were exposed to concurrent-chains schedules in which a single variable-interval schedule arranged entry into one of two terminal-link delay periods (fixed-interval schedules). The shorter delay ended with the delivery of a single food pellet; the longer day ended with a larger number of food pellets (two under some conditions and six under others). In Experiment 1, the terminal-link delays were selected so that under all conditions the ratio of delays would exactly equal the ratio of the number of pellets. But the absolute duration of the delays differed across conditions. In one condition, for example, rats chose between one pellet delayed 5 s and six pellets delayed 30 s; in another condition rats chose between one pellet delayed 10 s and six pellets delayed 60 s. The generalized matching law predicts indifference between the two alternatives, assuming that the sensitivity parameters for amount and delay of reinforcement are equal. The rats' choices were, in fact, close to indifference except when the choice was between one pellet delayed 5 s and six pellets delayed 30 s. That deviation from indifference suggests that the sensitivities to amount and delay differ from each other depending on the durations of the delays. In Experiment 2, rats chose between one pellet following a 5-s delay and six pellets following a delay that was systematically increased over sessions to find a point of indifference. Indifference was achieved when the delay to the six pellets was approximately 55 s. These results are consistent with the possibility that the relative sensitivities to amount and delay differ as a function of the delays.