Changes in functional response units with briefly delayed reinforcement

In two experiments, key-peck responding of pigeons was compared under variable-interval schedules that arranged immediate reinforcement and ones that arranged unsignaled delays of reinforcement. Responses during the nominal unsignaled delay periods had no effect on the reinforcer presentations. In Experiment 1, the unsignaled delays were studied using variable-interval schedules as baselines. Relative to the immediate reinforcement condition, 0.5-s unsignaled delays decreased the duration of the reinforced interresponse times and increased the overall frequency of short (<0.5-s) interresponse times. Longer, 5.0-s unsignaled delays increased the duration of the reinforced interresponse times and decreased the overall frequency of the short interresponse times. In Experiment 2, similar effects to those of Experiment 1 were obtained when the 0.5-s unsignaled delays were imposed upon a baseline schedule that explicitly arranged reinforcement of short interresponse times and therefore already generated a large number of short interresponse times. The results support earlier suggestions that the unsignaled 0.5-s delays change the functional response unit from a single key peck to a multiple key-peck unit. These findings are discussed in terms of the mechanisms by which contingencies control response structure in the absence of specific structural requirements.     

The role of the response-reinforcer relation in delay-of-reinforcement effects

The role of the response-reinforcer relation in maintaining operant behavior under conditions of delayed reinforcement was investigated by using a two-operandum (i.e., two-key) procedure with pigeons. Responding on one key was reinforced under a tandem variable-interval differential-reinforcement-of-other-behavior (tandem VI DRO) schedule. The schedule defined a resetting unsignaled delay-of-reinforcement procedure in that a response was required when the interfood interval of the VI schedule lapsed, but further responding during the DRO component on either key reset the time interval. This ensured a fixed delay duration between any response and reinforcement. Responding on another key, physically identical to the first one except for spatial location, otherwise was without consequence. The location of the key correlated with the delay-of-reinforcement procedure varied between sessions according to a semirandom sequence. Differences in response rates between the two keys were greater, with proportionally higher rates on the key correlated with the delay-of-reinforcement procedure, the longer the delay-of-reinforcement procedure remained correlated with the same key. Differences in responding on the two keys also increased within individual sessions. These results suggest that the response-reinforcer relation is the primary determinant of responding when responding is acquired and maintained with delayed reinforcement.     

Unsignaled delay of reinforcement, relative time, and resistance to change

Two experiments with pigeons examined the effects of unsignaled, nonresetting delays of reinforcement on responding maintained by different reinforcement rates. In Experiment 1, 3-s unsignaled delays were introduced into each component of a multiple variable-interval (VI) 15-s VI 90-s VI 540-s schedule. When considered as a proportion of the preceding immediate reinforcement baseline, responding was decreased similarly for the three multiple-schedule components in both the first six and last six sessions of exposure to the delay. In addition, the relation between response rates and reinforcement rates was altered such that both parameters of the single-response version of the matching law (i.e., k and Re) were decreased. Experiment 2 examined the effects of unsignaled delays ranging from 0.5 s to 8.0 s on responding maintained by a multiple VI 20-s VI 120-s schedule of reinforcement. Response rates in both components increased with brief unsignaled delays and decreased with longer delays. As in Experiment 1, response rates as a proportion of baseline were affected similarly for the two components in both the first six and last six sessions of exposure to the delay. Unlike delays imposed between two stimulus events, the effects of delays between responses and reinforcers do not appear to be attenuated when the average time between reinforcers is longer. In addition, the disruptions produced by unsignaled delays appear to be inconsistent with the general finding that responding maintained by higher rates of reinforcement is less resistant to change.     

Discrimination of response-reinforcer and response-stimulus contingencies in pigeons

For three pigeons (Experiment 1), the presentation of a red response key ended with a food presentation either following two responses separated by at least 10 seconds (a DRL contingency) or following a 10-second response-free period (a DRO contingency). For three other birds (Experiment 2), a brief stimulus presentation terminated the DRL and DRO contingencies. A white side key was presented next and ended with response-dependent food following one contingency and a timeout following the other. Since the contingency on the red key was unsignaled, differential responding on the white side key could indicate that the two response-reinforcer relations had been discriminated. In Experiment 1, the red-key duration and number of responses influenced white-key responding following the contingency that predicted the timeout. A response-initiated DRO was instated, and the influence of red-key duration and response number on white-key responding was diminished. In both experiments, the 10-second time criterion in both contingencies was varied from 0.34 second to 10 seconds. Even at short time intervals the DRO and DRL contingencies were readily discriminated. Pigeons tended to class the two contingencies according to a rule that did not involve simply stimulus duration, numbers of responses, or even the time between a response and its consequence.     

Responding of pigeons under variable-interval schedules of unsignaled, briefly signaled, and completely signaled delays to reinforcement

In Experiment 1, three pigeons' key pecking was maintained under a variable-interval 60-s schedule of food reinforcement. A 1-s unsignaled nonresetting delay to reinforcement was then added. Rates decreased and stabilized at values below those observed under immediate-reinforcement conditions. A brief stimulus change (key lit red for 0.5 s) was then arranged to follow immediately the peck that began the delay. Response rates quickly returned to baseline levels. Subsequently, rates near baseline levels were maintained with briefly signaled delays of 3 and 9 s. When a 27-s briefly signaled delay was instituted, response rates decreased to low levels. In Experiment 2, four pigeons' responding was first maintained under a multiple variable-interval 60-s (green key) variable-interval 60-s (red key) schedule. Response rates in both components fell to low levels when a 3-s unsignaled delay was added. In the first component delays were then briefly signaled in the same manner as Experiment 1, and in the second component they were signaled with a change in key color that remained until food was delivered. Response rates increased to near baseline levels in both components, and remained near baseline when the delays in both components were lengthened to 9 s. When delays were lengthened to 27 s, response rates fell to low levels in the briefly signaled delay component for three of four pigeons while remaining at or near baseline in the completely signaled delay component. In Experiment 3, low response rates under a 9-s unsignaled delay to reinforcement (tandem variable-interval 60 s fixed-time 9 s) increased when the delay was briefly signaled. The role of the brief stimulus as conditioned reinforcement may be a function of its temporal relation to food, and thus may be related to the eliciting function of the stimulus.     

Pavlovian contingencies and resistance to change in a multiple schedule

According to theoretical accounts of behavioral momentum, the Pavlovian stimulus—reinforcer contingency determines resistance to change. To assess this prediction, 8 pigeons were exposed to an unsignaled delay-of-reinforcement schedule (a tandem variable-interval fixed-time schedule), a signaled delay-of-reinforcement schedule (a chain variable-interval fixed-time schedule), and an immediate, zero-delay schedule of reinforcement in a three-component multiple schedule. The unsignaled delay and signaled delay schedules employed equal fixed-time delays, with the only difference being a stimulus change in the signaled delay schedule. Overall rates of reinforcement were equated for the three schedules. The Pavlovian contingency was identical for the unsignaled and immediate schedules, and response—reinforcer contiguity was degraded for the unsignaled schedule. Results from two disruption procedures (prefeeding subjects prior to experimental sessions and adding a variable-time schedule to timeout periods separating baseline components) demonstrated that response—reinforcer contiguity does play a role in determining resistance to change. The results from the extinction manipulation were not as clear. Responding in the unsignaled delay component was consistently less resistant to change than was responding in both the immediate and presignaled segments of the signaled delay components, contrary to the view that Pavlovian contingencies determine resistance to change. Probe tests further supported the resistance-to-change results, indicating consistency between resistance to change and preference, both of which are putative measures of response strength.     

Resistance to change of responding maintained by unsignaled delays to reinforcement: a response-bout analysis

Previous experiments have shown that unsignaled delayed reinforcement decreases response rates and resistance to change. However, the effects of different delays to reinforcement on underlying response structure have not been investigated in conjunction with tests of resistance to change. In the present experiment, pigeons responded on a three-component multiple variable-interval schedule for food presented immediately, following brief (0.5 s), or following long (3 s) unsignaled delays of reinforcement. Baseline response rates were lowest in the component with the longest delay; they were about equal with immediate and briefly delayed reinforcers. Resistance to disruption by presession feeding, response-independent food during the intercomponent interval, and extinction was slightly but consistently lower as delays increased. Because log survivor functions of interresponse times (IRTs) deviated from simple modes of bout initiations and within-bout responding, an IRT-cutoff method was used to examine underlying response structure. These analyses suggested that baseline rates of initiating bouts of responding decreased as scheduled delays increased, and within-bout response rates tended to be lower in the component with immediate reinforcers. The number of responses per bout was not reliably affected by reinforcer delay, but tended to be highest with brief delays when total response rates were higher in that component. Consistent with previous findings, resistance to change of overall response rate was highly correlated with resistance to change of bout-initiation rates but not with within-bout responding. These results suggest that unsignaled delays to reinforcement affect resistance to change through changes in the probability of initiating a response bout rather than through changes in the underlying response structure.     

Behavioral control by the response–reinforcer correlation

Using a discrete‐trials procedure, two experiments examined the effects of response–reinforcer correlations on responding while controlling molecular variables that operated at the moment of reinforcer delivery (e.g., response–reinforcer temporal contiguity, interresponse times preceding reinforcement). Each trial consisted of three successive components: Response, Timeout, and Reinforcement, with the duration of each component held constant. The correlation between the number of responses in the Response component and reinforcer deliveries in the Reinforcement component was varied. In the Positive‐correlation condition, a larger number of responses in the Response component programmed a higher reinforcement rate (Experiment 1) or a shorter time to reinforcement (Experiment 2) in the Reinforcement component. Although programmed in this way, the actual reinforcer delivery was dependent on, and occurred immediately after, a response in the Reinforcement component. In the Zero‐correlation condition, the programmed rates of reinforcement (Experiment 1) or the times to reinforcement (Experiment 2) in the Reinforcement component of each trial were yoked to those in the preceding Positive‐correlation condition. Responding in the Response component was higher in the Positive‐ than in the Zero‐correlation condition, without systematic changes in molecular variables. The results suggest that the response–reinforcer correlation can be a controlling variable of behavior.     

Reinforcement Delay Fading During Differential Reinforcement of Communication: The Effects of Signals on Response Maintenance

Signals during delays to reinforcement may lessen reductions in responding that typically occur when there is a delay between a response and its reinforcer. Sparse applied research has been devoted to understanding the conditions under which responding may be maintained when delays to reinforcement are introduced. We evaluated the extent to which providing signals during delay fading affected responding in the context of differential reinforcement of communication responses. Three individuals were exposed to gradually increasing signaled and unsignaled reinforcement delays in multiple‐schedule and/or withdrawal designs. Results for 2 of 3 participants suggested that (a) the presence of signals facilitated response maintenance under delayed reinforcement and (b) coordinated basic and applied research may advance both conceptual understanding and clinical outcomes of delayed reinforcement.     

The schedule dependency of the signaled reinforcement effect

In Experiment 1, rats leverpressed for food reinforcement on either a variable ratio (VR) 30 schedule or a variable interval (VI) 15-s schedule. One group in each condition received a signal filling a 500-ms delay of reinforcement. This treatment enhanced rates on the VR schedule, and attenuated rates on the VI schedule, relative to the rate seen in an unsignaled control condition. In Experiment 2 there was no delay of reinforcement and the signal and food were presented simultaneously. Attenuated rates of responding were observed on VI schedules with a range of mean interval values (15 to 300 s). Experiment 3 used a range of VR schedules (10 to 150) with simultaneous presentations of signal and food. A signal-induced enhancement of response rate was found at all VR values. In Experiment 4, a signal elevated response rates on a tandem VI VR schedule, but depressed rates on a tandem VR VI schedule, compared to control conditions receiving unsignaled delayed reinforcement. These results are taken to show that the effect of a signal accompanying reinforcement depends upon the nature of the behavior that is reinforced during exposure to a given schedule.     

Responding of pigeons under variable-interval schedules of signaled-delayed reinforcement: effects of delay-signal duration.

Two experiments with pigeons examined the relation of the duration of a signal for delay ("delay signal") to rates of key pecking. The first employed a multiple schedule comprised of two components with equal variable-interval 60-s schedules of 27-s delayed food reinforcement. In one component, a short (0.5-s) delay signal, presented immediately following the key peck that began the delay, was increased in duration across phases; in the second component the delay signal initially was equal to the length of the programmed delay (27 s) and was decreased across phases. Response rates prior to delays were an increasing function of delay-signal duration. As the delay signal was decreased in duration, response rates were generally higher than those obtained under identical delay-signal durations as the signal was increased in duration. In Experiment 2 a single variable-interval 60-s schedule of 27-s delayed reinforcement was used. Delay-signal durations were again increased gradually across phases. As in Experiment 1, response rates increased as the delay-signal duration was increased. Following the phase during which the signal lasted the entire delay, shorter delay-signal-duration conditions were introduced abruptly, rather than gradually as in Experiment 1, to determine whether the gradual shortening of the delay signal accounted for the differences observed in response rates under identical delay-signal conditions in Experiment 1. Response rates obtained during the second exposures to the conditions with shorter signals were higher than those observed under identical conditions as the signal duration was increased, as in Experiment 1. In both experiments, rates and patterns of responding during delays varied greatly across subjects and were not systematically related to delay-signal durations. The effects of the delay signal may be related to the signal's role as a discriminative stimulus for adventitiously reinforced intradelay behavior, or the delay signal may have served as a conditioned reinforcer by virtue of the temporal relation between it and presentation of food.     

Effects of unsignaled delay of reinforcement on preference and resistance to change

In Phase 1, pigeons were trained on a concurrent chain in which a 3-s unsignaled delay of reinforcement was imposed on responding in a terminal link in some conditions. Preference for that terminal link was always reduced in comparison with conditions in which there was no delay, substantially so for 3 of the 4 pigeons. In Phase 2, pigeons responded in a two-component multiple schedule. The scheduled rates of reinforcement were equal, but a 3-s unsignaled delay was imposed in one component. Resistance of responding to prefeeding and extinction was reduced in the delay component for the same 3 subjects for which the data had shown strong effects of delay on preference. Systematic observation revealed differences in response topography. In the delay component, subjects oriented more closely to the key and responses were less forceful compared with the no-delay component. Our results give further evidence that preference and resistance to change covary within subjects. However, they challenge the premise that the critical determiners of preference (i.e., terminal-link value) and resistance to change (behavioral mass) may be quantified purely in terms of stimulus—reinforcer relations.     

Briefly delayed reinforcement: An interresponse time analysis

Key-peck responding of pigeons was compared under VI or DRL schedules arranging immediate reinforcement and briefly (.5 sec) delayed reinforcement. Delays were either signaled by a blackout in the chamber, unsignaled, or unsignaled with an additional requirement that responding not occur during the .5 sec interval immediately preceding reinforcement (response delay). Relative to the immediate reinforcement condition, response rates increased during the unsignaled delay, decreased during the signaled delay, and were inconsistent during the response delay condition. An analysis of interresponse times (IRTs) under the different conditions revealed a substantial increase in the frequency of short (0 to .5 sec) IRTs during the unsignaled condition and generally during the response delay conditions compared to that during the immediate reinforcement baseline. Signaled delays decreased the frequency of short (0 to .5 sec) IRTs relative to the immediate reinforcement condition. The results suggest that brief unsignaled delays and, in many instances, response delays increase the frequency of short IRTs by eliminating constraints on responding.     

Context extinction following conditioning with delayed reward enhances subsequent instrumental responding

In Experiment 1, delayed reward generated low response rates relative to immediate reward delivered with the same frequency. Lister rats exposed to delayed reward subsequently responded at a higher rate in extinction if they received nonreinforced exposure to the conditioning context after instrumental training and prior to test, compared with animals that received home cage exposure. In Experiment 2, a signaled delay of reinforcement resulted in higher rates than an unsignaled delay. Nonreinforced exposure to the conditioning context elevated response rate for subjects in the unsignaled condition relative to a home cage group, but had no effect on response rates for subjects that had received the signaled delay. In Experiment 3, following an unsignaled reinforcement delay, groups receiving either no event or signaled food in the context responded faster in extinction than groups receiving no context exposure or unsignaled food.     

Choice, changing over, and reinforcement delays

In three experiments, pigeons were used to examine the independent effects of two normally confounded delays to reinforcement associated with changing between concurrently available variable-interval schedules of reinforcement. In Experiments 1 and 2, combinations of changeover-delay durations and fixed-interval travel requirements were arranged in a changeover-key procedure. The delay from a changeover-produced stimulus change to a reinforcer was varied while the delay between the last response on one alternative and a reinforcer on the other (the total obtained delay) was held constant. Changeover rates decreased as a negative power function of the total obtained delay. The delay between a changeover-produced stimulus change had a small and inconsistent effect on changeover rates. In Experiment 3, changeover delays and fixed-interval travel requirements were arranged independently. Changeover rates decreased as a negative power function of the total obtained delay despite variations in the delay from a change in stimulus conditions to a reinforcer. Periods of high-rate responding following a changeover, however, were higher near the end of the delay from a change in stimulus conditions to a reinforcer. The results of these experiments suggest that the effects of changeover delays and travel requirements primarily result from changes in the delay between a response at one alternative and a reinforcer at the other, but the pattern of responding immediately after a changeover depends on the delay from a changeover-produced change in stimulus conditions to a reinforcer.     

Choice behavior and the accessibility of the reinforcer

In Experiment 1, matching of relative response rates to relative rates of reinforcement was obtained in concurrent variable-interval schedules when the absolute values of the two concurrent variable-interval schedules varied from 6 sec and 12 sec to 600 sec and 1200 sec. Increases in the duration of the changeover delay, however, produced decreases in the relative response rates and, consequently, some deviation from matching. In Experiment 2, matching of relative response rates to the relative duration of the reinforcer failed to occur when the equal variable-interval schedules arranging access to the two different reinforcer durations (1.5 and 6 sec) were varied in size from concurrent variable-interval 10-sec schedules to concurrent variable-interval 600-sec schedules.     

Disruption of responding maintained by conditioned reinforcement: alterations in response-conditioned-reinforcer relations

An observing procedure was used to investigate the effects of alterations in response-conditioned-reinforcer relations on observing. Pigeons responded to produce schedule-correlated stimuli paired with the availability of food or extinction. The contingency between observing responses and conditioned reinforcement was altered in three experiments. In Experiment 1, after a contingency was established in baseline between the observing response and conditioned reinforcement, it was removed and the schedule-correlated stimuli were presented independently of responding according to a variable-time schedule. The variable-time schedule was constructed such that the rate of stimulus presentations was yoked from baseline. The removal of the observing contingency reliably reduced rates of observing. In Experiment 2, resetting delays to conditioned reinforcement were imposed between observing responses and the schedule-correlated stimuli they produced. Delay values of 0, 0.5, 1, 5, and 10 s were examined. Rates of observing varied inversely as a function of delay value. In Experiment 3, signaled and unsignaled resetting delays between observing responses and schedule-correlated stimuli were compared. Baseline rates of observing were decreased less by signaled delays than by unsignaled delays. Disruptions in response-conditioned-reinforcer relations produce similar behavioral effects to those found with primary reinforcement.     

PREFERENCE FOR A STIMULUS THAT FOLLOWS A RELATIVELY AVERSIVE EVENT: CONTRAST OR DELAY REDUCTION?

Several types of contrast effects have been identified including incentive contrast, anticipatory contrast, and behavioral contrast. Clement, Feltus, Kaiser, and Zentall (2000) proposed a type of contrast that appears to be different from these others and called it within-trial contrast. In this form of contrast the relative value of a reinforcer depends on the events that occur immediately prior to the reinforcer. Reinforcers that follow relatively aversive events are preferred over those that follow less aversive events. In many cases the delay reduction hypothesis proposed by Fantino (1969) also can account for such effects. The current experiments provide a direct test of the delay reduction and contrast hypotheses by manipulating the schedule of reinforcement while holding trial duration constant. In Experiment 1, preference for fixed-interval (FI) versus differential-reinforcement-of-other-behavior (DRO) schedules of reinforcement was assessed. Some pigeons preferred one schedule over the other while others demonstrated a position (side) preference. Thus, no systematic preference was found. In Experiment 2, a simultaneous color discrimination followed the FI or DRO schedule, and following training, preference was assessed by presenting the two positive stimuli simultaneously. Consistent with the contrast hypothesis, pigeons showed a significant preference for the positive stimulus that in training had followed their less preferred schedule.     

Within-subject testing of the signaled-reinforcement effect on operant responding as measured by response rate and resistance to change

Response rates under random-interval schedules are lower when a brief (500 ms) signal accompanies reinforcement than when there is no signal. The present study examined this signaled-reinforcement effect and its relation to resistance to change. In Experiment 1, rats responded on a multiple random-interval 60-s random-interval 60-s schedule, with signaled reinforcement in only one component. Response resistance to alternative reinforcement, prefeeding, and extinction was compared between these components. Lower response rates, and greater resistance to change, occurred in the component with the reinforcement signal. In Experiment 2, response rates and resistance to change were compared after training on a multiple random-interval 60-s random-interval 60-s schedule in which reinforcer delivery was unsignaled in one component and a response-produced uncorrelated stimulus was presented in the other component. Higher response rates and greater resistance to change occurred with the uncorrelated stimulus. These results highlight the significance of considering the effects of an uncorrelated signal when used as a control condition, and challenge accounts of resistance to change that depend solely on reinforcer rate.     

Within-session delay-of-reinforcement gradients

Within-session delay-of-reinforcement gradients were generated with pigeons by progressively increasing delays to reinforcement within each session. In Experiment 1, the effects of imposing progressive delays on variable-interval and fixed-interval schedules were investigated while controlling for simultaneous decreases in reinforcer rate across the session via a within-subject yoked-control procedure. Rate of key pecking decreased as a negatively decelerated function of delay of reinforcement within a session. These rate decreases were greater than those during a yoked-interval session in which the rate of immediate reinforcement decreased at the same rate as it did under the progressive-delay procedure. In Experiment 2, delay-of-reinforcement gradients were shallower when the progressive delay intervals were signaled by a blackout than when they were unsignaled. The delay gradients obtained in each experiment were similar to those generated under conditions in which different delays of reinforcement are imposed across blocks of sessions. The present procedure offers a technique for rapidly generating delay-of-reinforcement gradients that might serve as baselines for assessing the effects of other behavioral and pharmacological variables.