Responding of pigeons under variable-interval schedules of unsignaled, briefly signaled, and completely signaled delays to reinforcement

In Experiment 1, three pigeons' key pecking was maintained under a variable-interval 60-s schedule of food reinforcement. A 1-s unsignaled nonresetting delay to reinforcement was then added. Rates decreased and stabilized at values below those observed under immediate-reinforcement conditions. A brief stimulus change (key lit red for 0.5 s) was then arranged to follow immediately the peck that began the delay. Response rates quickly returned to baseline levels. Subsequently, rates near baseline levels were maintained with briefly signaled delays of 3 and 9 s. When a 27-s briefly signaled delay was instituted, response rates decreased to low levels. In Experiment 2, four pigeons' responding was first maintained under a multiple variable-interval 60-s (green key) variable-interval 60-s (red key) schedule. Response rates in both components fell to low levels when a 3-s unsignaled delay was added. In the first component delays were then briefly signaled in the same manner as Experiment 1, and in the second component they were signaled with a change in key color that remained until food was delivered. Response rates increased to near baseline levels in both components, and remained near baseline when the delays in both components were lengthened to 9 s. When delays were lengthened to 27 s, response rates fell to low levels in the briefly signaled delay component for three of four pigeons while remaining at or near baseline in the completely signaled delay component. In Experiment 3, low response rates under a 9-s unsignaled delay to reinforcement (tandem variable-interval 60 s fixed-time 9 s) increased when the delay was briefly signaled. The role of the brief stimulus as conditioned reinforcement may be a function of its temporal relation to food, and thus may be related to the eliciting function of the stimulus.     

Key pecking of pigeons under variable-interval schedules of briefly signaled delayed reinforcement: effects of variable-interval value.

Key pecking of 4 pigeons was maintained under a multiple variable-interval 20-s variable-interval 120-s schedule of food reinforcement. When rates of key pecking were stable, a 5-s unsignaled, nonresetting delay to reinforcement separated the first peck after an interval elapsed from reinforcement in both components. Rates of pecking decreased substantially in both components. When rates were stable, the situation was changed such that the peck that began the 5-s delay also changed the color of the keylight for 0.5 s (i.e., the delay was briefly signaled). Rates increased to near-immediate reinforcement levels. In subsequent conditions, delays of 10 and 20 s, still briefly signaled, were tested. Although rates of key pecking during the component with the variable-interval 120-s schedule did not change appreciably across conditions, rates during the variable-interval 20-s component decreased greatly in 1 pigeon at the 10-s delay and decreased in all pigeons at the 20-s delay. In a control condition, the variable-interval 20-s schedule with 20-s delays was changed to a variable-interval 35-s schedule with 5-s delays, thus equating nominal rates of reinforcement. Rates of pecking increased to baseline levels. Rates of pecking, then, depended on the value of the briefly signaled delay relative to the programmed interfood times, rather than on the absolute delay value. These results are discussed in terms of similar findings in the literature on conditioned reinforcement, delayed matching to sample, and classical conditioning.     

Changes in functional response units with briefly delayed reinforcement

In two experiments, key-peck responding of pigeons was compared under variable-interval schedules that arranged immediate reinforcement and ones that arranged unsignaled delays of reinforcement. Responses during the nominal unsignaled delay periods had no effect on the reinforcer presentations. In Experiment 1, the unsignaled delays were studied using variable-interval schedules as baselines. Relative to the immediate reinforcement condition, 0.5-s unsignaled delays decreased the duration of the reinforced interresponse times and increased the overall frequency of short (<0.5-s) interresponse times. Longer, 5.0-s unsignaled delays increased the duration of the reinforced interresponse times and decreased the overall frequency of the short interresponse times. In Experiment 2, similar effects to those of Experiment 1 were obtained when the 0.5-s unsignaled delays were imposed upon a baseline schedule that explicitly arranged reinforcement of short interresponse times and therefore already generated a large number of short interresponse times. The results support earlier suggestions that the unsignaled 0.5-s delays change the functional response unit from a single key peck to a multiple key-peck unit. These findings are discussed in terms of the mechanisms by which contingencies control response structure in the absence of specific structural requirements.     

Disruption of responding maintained by conditioned reinforcement: alterations in response-conditioned-reinforcer relations

An observing procedure was used to investigate the effects of alterations in response-conditioned-reinforcer relations on observing. Pigeons responded to produce schedule-correlated stimuli paired with the availability of food or extinction. The contingency between observing responses and conditioned reinforcement was altered in three experiments. In Experiment 1, after a contingency was established in baseline between the observing response and conditioned reinforcement, it was removed and the schedule-correlated stimuli were presented independently of responding according to a variable-time schedule. The variable-time schedule was constructed such that the rate of stimulus presentations was yoked from baseline. The removal of the observing contingency reliably reduced rates of observing. In Experiment 2, resetting delays to conditioned reinforcement were imposed between observing responses and the schedule-correlated stimuli they produced. Delay values of 0, 0.5, 1, 5, and 10 s were examined. Rates of observing varied inversely as a function of delay value. In Experiment 3, signaled and unsignaled resetting delays between observing responses and schedule-correlated stimuli were compared. Baseline rates of observing were decreased less by signaled delays than by unsignaled delays. Disruptions in response-conditioned-reinforcer relations produce similar behavioral effects to those found with primary reinforcement.     

Resistance to change of responding maintained by unsignaled delays to reinforcement: a response-bout analysis

Previous experiments have shown that unsignaled delayed reinforcement decreases response rates and resistance to change. However, the effects of different delays to reinforcement on underlying response structure have not been investigated in conjunction with tests of resistance to change. In the present experiment, pigeons responded on a three-component multiple variable-interval schedule for food presented immediately, following brief (0.5 s), or following long (3 s) unsignaled delays of reinforcement. Baseline response rates were lowest in the component with the longest delay; they were about equal with immediate and briefly delayed reinforcers. Resistance to disruption by presession feeding, response-independent food during the intercomponent interval, and extinction was slightly but consistently lower as delays increased. Because log survivor functions of interresponse times (IRTs) deviated from simple modes of bout initiations and within-bout responding, an IRT-cutoff method was used to examine underlying response structure. These analyses suggested that baseline rates of initiating bouts of responding decreased as scheduled delays increased, and within-bout response rates tended to be lower in the component with immediate reinforcers. The number of responses per bout was not reliably affected by reinforcer delay, but tended to be highest with brief delays when total response rates were higher in that component. Consistent with previous findings, resistance to change of overall response rate was highly correlated with resistance to change of bout-initiation rates but not with within-bout responding. These results suggest that unsignaled delays to reinforcement affect resistance to change through changes in the probability of initiating a response bout rather than through changes in the underlying response structure.     

Effects of delayed conditioned reinforcement in chain schedules.

The contingency between responding and stimulus change on a chain variable-interval 33-s, variable-interval 33-s, variable-interval 33-s schedule was weakened by interposing 3-s delays between either the first and second or the second and third links. No stimulus change signaled the delay interval and responses could occur during it, so the obtained delays were often shorter than the scheduled delay. When the delay occurred after the initial link, initial-link response rates decreased by an average of 77% with no systematic change in response rates in the second or third links. Response rates in the second link decreased an average of 59% when the delay followed that link, again with little effect on response rates in the first or third links. Because the effect of delaying stimulus change was comparable to the effect of delaying primary reinforcement in a simple variable-interval schedule, and the effect of the unsignaled delay was specific to the link in which the delay occurred, the results provide strong evidence for the concept of conditioned reinforcement.     

The schedule dependency of the signaled reinforcement effect

In Experiment 1, rats leverpressed for food reinforcement on either a variable ratio (VR) 30 schedule or a variable interval (VI) 15-s schedule. One group in each condition received a signal filling a 500-ms delay of reinforcement. This treatment enhanced rates on the VR schedule, and attenuated rates on the VI schedule, relative to the rate seen in an unsignaled control condition. In Experiment 2 there was no delay of reinforcement and the signal and food were presented simultaneously. Attenuated rates of responding were observed on VI schedules with a range of mean interval values (15 to 300 s). Experiment 3 used a range of VR schedules (10 to 150) with simultaneous presentations of signal and food. A signal-induced enhancement of response rate was found at all VR values. In Experiment 4, a signal elevated response rates on a tandem VI VR schedule, but depressed rates on a tandem VR VI schedule, compared to control conditions receiving unsignaled delayed reinforcement. These results are taken to show that the effect of a signal accompanying reinforcement depends upon the nature of the behavior that is reinforced during exposure to a given schedule.     

Pavlovian contingencies and resistance to change in a multiple schedule

According to theoretical accounts of behavioral momentum, the Pavlovian stimulus—reinforcer contingency determines resistance to change. To assess this prediction, 8 pigeons were exposed to an unsignaled delay-of-reinforcement schedule (a tandem variable-interval fixed-time schedule), a signaled delay-of-reinforcement schedule (a chain variable-interval fixed-time schedule), and an immediate, zero-delay schedule of reinforcement in a three-component multiple schedule. The unsignaled delay and signaled delay schedules employed equal fixed-time delays, with the only difference being a stimulus change in the signaled delay schedule. Overall rates of reinforcement were equated for the three schedules. The Pavlovian contingency was identical for the unsignaled and immediate schedules, and response—reinforcer contiguity was degraded for the unsignaled schedule. Results from two disruption procedures (prefeeding subjects prior to experimental sessions and adding a variable-time schedule to timeout periods separating baseline components) demonstrated that response—reinforcer contiguity does play a role in determining resistance to change. The results from the extinction manipulation were not as clear. Responding in the unsignaled delay component was consistently less resistant to change than was responding in both the immediate and presignaled segments of the signaled delay components, contrary to the view that Pavlovian contingencies determine resistance to change. Probe tests further supported the resistance-to-change results, indicating consistency between resistance to change and preference, both of which are putative measures of response strength.     

The role of the response-reinforcer relation in delay-of-reinforcement effects

The role of the response-reinforcer relation in maintaining operant behavior under conditions of delayed reinforcement was investigated by using a two-operandum (i.e., two-key) procedure with pigeons. Responding on one key was reinforced under a tandem variable-interval differential-reinforcement-of-other-behavior (tandem VI DRO) schedule. The schedule defined a resetting unsignaled delay-of-reinforcement procedure in that a response was required when the interfood interval of the VI schedule lapsed, but further responding during the DRO component on either key reset the time interval. This ensured a fixed delay duration between any response and reinforcement. Responding on another key, physically identical to the first one except for spatial location, otherwise was without consequence. The location of the key correlated with the delay-of-reinforcement procedure varied between sessions according to a semirandom sequence. Differences in response rates between the two keys were greater, with proportionally higher rates on the key correlated with the delay-of-reinforcement procedure, the longer the delay-of-reinforcement procedure remained correlated with the same key. Differences in responding on the two keys also increased within individual sessions. These results suggest that the response-reinforcer relation is the primary determinant of responding when responding is acquired and maintained with delayed reinforcement.     

Mechanisms underlying the effects of unsignaled delayed reinforcement on key pecking of pigeons under variable-interval schedules.

Three experiments were conducted to test an interpretation of the response-rate-reducing effects of unsignaled nonresetting delays to reinforcement in pigeons. According to this interpretation, rates of key pecking decrease under these conditions because key pecks alternate with hopper-observing behavior. In Experiment 1, 4 pigeons pecked a food key that raised the hopper provided that pecks on a different variable-interval-schedule key met the requirements of a variable-interval 60-s schedule. The stimuli associated with the availability of the hopper (i.e., houselight and keylight off, food key illuminated, feedback following food-key pecks) were gradually removed across phases while the dependent relation between hopper availability and variable-interval-schedule key pecks was maintained. Rates of pecking the variable-interval-schedule key decreased to low levels and rates of food-key pecks increased when variable-interval-schedule key pecks did not produce hopper-correlated stimuli. In Experiment 2, pigeons initially pecked a single key under a variable-interval 60-s schedule. Then the dependent relation between hopper presentation and key pecks was eliminated by arranging a variable-time 60-s schedule. When rates of pecking had decreased to low levels, conditions were changed so that pecks during the final 5 s of each interval changed the keylight color from green to amber. When pecking produced these hopper-correlated stimuli, pecking occurred at high rates, despite the absence of a peck-food dependency. When peck-produced changes in keylight color were uncorrelated with food, rates of pecking fell to low levels. In Experiment 3, details (obtained delays, interresponse-time distributions, eating times) of the transition from high to low response rates produced by the introduction of a 3-s unsignaled delay were tracked from session to session in 3 pigeons that had been initially trained to peck under a conventional variable-interval 60-s schedule. Decreases in response rates soon after the transition to delayed reinforcement were accompanied by decreases in eating times and alterations in interresponse-time distributions. As response rates decreased and became stable, eating times increased and their variability decreased. These findings support an interpretation of the effects of delayed reinforcement that emphasizes the importance of hopper-observing behavior.     

Responding of pigeons under variable-interval schedules of signaled-delayed reinforcement: effects of delay-signal duration.

Two experiments with pigeons examined the relation of the duration of a signal for delay ("delay signal") to rates of key pecking. The first employed a multiple schedule comprised of two components with equal variable-interval 60-s schedules of 27-s delayed food reinforcement. In one component, a short (0.5-s) delay signal, presented immediately following the key peck that began the delay, was increased in duration across phases; in the second component the delay signal initially was equal to the length of the programmed delay (27 s) and was decreased across phases. Response rates prior to delays were an increasing function of delay-signal duration. As the delay signal was decreased in duration, response rates were generally higher than those obtained under identical delay-signal durations as the signal was increased in duration. In Experiment 2 a single variable-interval 60-s schedule of 27-s delayed reinforcement was used. Delay-signal durations were again increased gradually across phases. As in Experiment 1, response rates increased as the delay-signal duration was increased. Following the phase during which the signal lasted the entire delay, shorter delay-signal-duration conditions were introduced abruptly, rather than gradually as in Experiment 1, to determine whether the gradual shortening of the delay signal accounted for the differences observed in response rates under identical delay-signal conditions in Experiment 1. Response rates obtained during the second exposures to the conditions with shorter signals were higher than those observed under identical conditions as the signal duration was increased, as in Experiment 1. In both experiments, rates and patterns of responding during delays varied greatly across subjects and were not systematically related to delay-signal durations. The effects of the delay signal may be related to the signal's role as a discriminative stimulus for adventitiously reinforced intradelay behavior, or the delay signal may have served as a conditioned reinforcer by virtue of the temporal relation between it and presentation of food.     

Briefly delayed reinforcement: An interresponse time analysis

Key-peck responding of pigeons was compared under VI or DRL schedules arranging immediate reinforcement and briefly (.5 sec) delayed reinforcement. Delays were either signaled by a blackout in the chamber, unsignaled, or unsignaled with an additional requirement that responding not occur during the .5 sec interval immediately preceding reinforcement (response delay). Relative to the immediate reinforcement condition, response rates increased during the unsignaled delay, decreased during the signaled delay, and were inconsistent during the response delay condition. An analysis of interresponse times (IRTs) under the different conditions revealed a substantial increase in the frequency of short (0 to .5 sec) IRTs during the unsignaled condition and generally during the response delay conditions compared to that during the immediate reinforcement baseline. Signaled delays decreased the frequency of short (0 to .5 sec) IRTs relative to the immediate reinforcement condition. The results suggest that brief unsignaled delays and, in many instances, response delays increase the frequency of short IRTs by eliminating constraints on responding.     

Effects of signaled versus unsignaled delay of reinforcement on choice

Pigeons chose between 5-s and 15-s delay-of-reinforcement alternatives. The first key peck to satisfy the choice schedule began a delay timer, and food was delivered at the end of the interval. Key pecks during the delay interval were measured, but had no scheduled effect. In Experiment 1, signal conditions and choice schedules were varied across conditions. During unsignaled conditions, no stimulus change signaled the beginning of a delay interval. During differential and nondifferential signal conditions, offset of the choice stimuli and onset of a delay stimulus signaled the beginning of a delay interval. During differential signal conditions, different stimuli were correlated with the 5-s and 15-s delays, whereas the same stimulus appeared during both delay durations during nondifferential signal conditions. Pigeons showed similar, extreme levels of preference for the 5-s delay alternative during unsignaled and differentially signaled conditions. Preference levels were reliably lower with nondifferential signals. Experiment 2 assessed preference with two pairs of unsignaled delays in which the ratio of delays was held constant but the absolute duration was increased fourfold. No effect of absolute duration was found. The results highlight the importance of delayed primary reinforcement effects and challenge models of choice that focus solely on conditioned reinforcement.     

Variable-interval reinforcement schedule value influences responding following REM sleep deprivation

The effects of rapid-eye movement sleep deprivation (REMSD) in rats were studied in relation to variable-interval (VI) reinforcement schedule value. Initially, lever pressing was maintained on a VI 30-s schedule of food pellet delivery. After a baseline was established, rats were repeatedly exposed to 96 hr of REMSD and control conditions of an equivalent duration. Responding decreased following REMSD but not after exposure to control conditions. Lever pressing was then maintained on a VI 15-s schedule of food pellet delivery and exposure to the REMSD and control conditions was repeated. Under this condition following repeated REMSD exposures, rates of lever pressing became similar to baseline responding. A VI 30-s schedule of food pellet delivery was then reinstated and REMSD and control conditions were repeated. Lever pressing following exposure to the REMSD condition decreased for 3 of 4 rats. Results suggest that VI schedule value influences the effects of REMSD on responding.     

Effects of cocaine on briefly signaled versus completely signaled delays to reinforcement.

Key pecking by 4 pigeons was maintained by a multiple schedule consisting of two variable-interval 60-s schedules wherein each food presentation followed a nonresetting 27-s delay that was either briefly signaled at its outset or completely signaled. Brief-signal duration was adjusted so that response rates maintained by the briefly and completely signaled delays of reinforcement were similar. In general, acute administration of small to intermediate doses (0.3 to 3.0 mg/kg) of cocaine produced either small increases in response rates in both components or no change, and larger doses (5.6 to 13.0 mg/kg) decreased response rates. Chronic (i.e., daily) cocaine administration (10.0 mg/kg) resulted in tolerance to the rate-decreasing effects in both components. Cocaine's effects were generally similar whether delays were completely or briefly signaled. Discontinuation of cocaine administration and subsequent removal of the delay signals also had similar effects in both components of the multiple schedule. Taken together, these results are consistent with the view that the two types of delay signals were equally effective in maintaining responding during the variable-interval schedules.     

A comparison of signaled and unsignaled delay of reinforcement

Pigeons were trained on either a variable-interval 60-second schedule, or on a schedule that differentially reinforced responses that were spaced at least 20 seconds apart. The birds were then exposed to several durations of reinforcement delay, with comparisons between signaled and unsignaled delays. Although unsignaled delays of 5 and 10 seconds produced large decreases in response rate, signaled delays of up to 10 seconds produced only moderate decreases in response rates. In addition, some subjects responded more rapidly with a .5 or 1.0 second duration of unsignaled delay than with immediate reinforcement. These response rate changes occurred regardless of whether the rate of reinforcement concomitantly decreased or increased.     

Effects of reinforcement rate and delay on the acquisition of lever pressing by rats.

The acquisition of lever pressing by naive rats, in the absence of shaping, was studied as a function of different rates and unsignaled delays of reinforcement. Groups of 3 rats were each exposed to tandem schedules that differed in either the first or the second component. First-component schedules were either continuous reinforcement or random-interval 15, 30, 60 or 120 s; second-component schedules were fixed-time 0, 1, 3, 6, 12, or 24 s. Rate of responding was low under continuous immediate reinforcement and higher under random-interval 15 s. Random interval 30-s and 60-s schedules produced lower rates that were similar to each other. Random-interval 120 s controlled the lowest rate in the immediate-reinforcement condition. Adding a constant 12-s delay to each of the first-component schedule parameters controlled lower response rates that did not vary systematically with reinforcement rate. The continuous and random-interval 60-s schedules of immediate reinforcement controlled higher global and first-component response rates than did the same schedules combined with longer delays, and first-component rates showed some graded effects of delay duration. In addition, the same schedules controlled higher second-component response rates in combination with a 1-s delay than in combination with longer delays. These results were related to those from previous studies on acquisition with delayed reinforcement as well as to those from similar reinforcement procedures used during steady-state responding.     

A comparison of delays and ratio requirements in self-control choice.

In a discrete-trial procedure, pigeons could choose between 2-s and 6-s access to grain by making a single key peck. In Phase 1, the pigeons obtained both reinforcers by responding on fixed-ratio schedules. In Phase 2, they received both reinforcers after simple delays, arranged by fixed-time schedules, during which no responses were required. In Phase 3, the 2-s reinforcer was available through a fixed-time schedule and the 6-s reinforcer was available through a fixed-ratio schedule. In all conditions, the size of the delay or ratio leading to the 6-s reinforcer was systematically increased or decreased several times each session, permitting estimation of an "indifference point," the schedule size at which a subject chose each alternative equally often. By varying the size of the schedule for the 2-s reinforcer across conditions, several such indifference points were obtained from both fixed-time conditions and fixed-ratio conditions. The resulting "indifference curves" from fixed-time conditions and from fixed-ratio conditions were similar in shape, and they suggested that a hyperbolic equation describes the relation between ratio size and reinforcement value as well as the relation between reinforcer delay and its reinforcement value. The results from Phase 3 showed that subjects chose fixed-time schedules over fixed-ratio schedules that generated the same average times between a choice response and reinforcement.     

Sensitivity to relative reinforcer rate in concurrent schedules: independence from relative and absolute reinforcer duration

Twelve pigeons responded on two keys under concurrent variable-interval (VI) schedules. Over several series of conditions, relative and absolute magnitudes of reinforcement were varied. Within each series, relative rate of reinforcement was varied and sensitivity of behavior ratios to reinforcer-rate ratios was assessed. When responding at both alternatives was maintained by equal-sized small reinforcers, sensitivity to variation in reinforcer-rate ratios was the same as when large reinforcers were used. This result was observed when the overall rate of reinforcement was constant over conditions, and also in another series of concurrent schedules in which one schedule was kept constant at VI ached 120 s. Similarly, reinforcer magnitude did not affect the rate at which response allocation approached asymptote within a condition. When reinforcer magnitudes differred between the two responses and reinforcer-rate ratios were varied, sensitivity of behavior allocation was unaffected although response bias favored the schedule that arranged the larger reinforcers. Analysis of absolute response rates ratio sensitivity to reinforcement occurrred on the two keys showed that this invariance of response despite changes in reinforcement interaction that were observed in absolute response rates on the constant VI 120-s schedule. Response rate on the constant VI 120-s schedule was inversely related to reinforcer rate on the varied key and the strength of this relation depended on the relative magnitude of reinforcers arranged on varied key. Independence of sensitivity to reinforcer-rate ratios from relative and absolute reinforcer magnitude is consistent with the relativity and independence assumtions of the matching law.     

A parametric variation of delayed reinforcement in infants.

This study is an exploration of the parameters of delayed reinforcement with 6 infants (2 to 6 months old) in two experiments using single-subject repeated-reversal designs. In Experiment 1, unsignaled 3-s delayed reinforcement was used to increase infant vocalization rate when compared to a differential-reinforcement-of-other-than-vocalization condition and a yoked, no-contingency comparison condition. In Experiment 2, unsignaled 5-s delayed reinforcement was used to increase infant vocalization rate when compared to an alternating-treatments comparison condition. The alternating-treatments comparison consisted of 3-min components of differential reinforcement of other behavior and 3-min components of a nontreatment baseline. Successful conditioning was obtained in both experiments. These results contrast with those of previous infancy researchers who did not obtained conditioning with delays of 3 s and who attributed their findings to the limitations of the infant's memory capacity. We present an alternative conceptual framework and methodology for the analysis of delayed reinforcement in infants.