Briefly delayed reinforcement: An interresponse time analysis

Key-peck responding of pigeons was compared under VI or DRL schedules arranging immediate reinforcement and briefly (.5 sec) delayed reinforcement. Delays were either signaled by a blackout in the chamber, unsignaled, or unsignaled with an additional requirement that responding not occur during the .5 sec interval immediately preceding reinforcement (response delay). Relative to the immediate reinforcement condition, response rates increased during the unsignaled delay, decreased during the signaled delay, and were inconsistent during the response delay condition. An analysis of interresponse times (IRTs) under the different conditions revealed a substantial increase in the frequency of short (0 to .5 sec) IRTs during the unsignaled condition and generally during the response delay conditions compared to that during the immediate reinforcement baseline. Signaled delays decreased the frequency of short (0 to .5 sec) IRTs relative to the immediate reinforcement condition. The results suggest that brief unsignaled delays and, in many instances, response delays increase the frequency of short IRTs by eliminating constraints on responding.     

Response-rate differences in variable-interval and variable-ratio schedules: An old problem revisited

In Experiment 1, a variable-ratio 10 schedule became, successively, a variable-interval schedule with only the minimum interreinforcement intervals yoked to the variable ratio, or a variable-interval schedule with both interreinforcement intervals and reinforced interresponse times yoked to the variable ratio. Response rates in the variable-interval schedule with both interreinforcement interval and reinforced interresponse time yoking fell between the higher rates maintained by the variable-ratio schedule and the lower rates maintained by the variable-interval schedule with only interreinforcement interval yoking. In Experiment 2, a tandem variable-interval 15-s variable-ratio 5 schedule became a yoked tandem variable-ratio 5 variable-interval x-s schedule, and a tandem variable-interval 30-s variable-ratio 10 schedule became a yoked tandem variable-ratio 10 variable-interval x-s schedule. In the yoked tandem schedules, the minimum interreinforcement intervals in the variable-interval components were those that equated overall interreinforcement times in the two phases. Response rates did not decline in the yoked schedules even when the reinforced interresponse times became longer. Experiment 1 suggests that both reinforced interresponse times and response rate–reinforcement rate correlations determine response-rate differences in variable-ratio 10 and yoked variable-interval schedules in rats. Experiment 2 suggests a minimal role for the reinforced interresponse time in determining response rates on tandem variable-interval 30-s variable-ratio 10 and yoked tandem variable-ratio 10 variable-interval x-s schedules in rats.     

Unsignaled delay of reinforcement, relative time, and resistance to change

Two experiments with pigeons examined the effects of unsignaled, nonresetting delays of reinforcement on responding maintained by different reinforcement rates. In Experiment 1, 3-s unsignaled delays were introduced into each component of a multiple variable-interval (VI) 15-s VI 90-s VI 540-s schedule. When considered as a proportion of the preceding immediate reinforcement baseline, responding was decreased similarly for the three multiple-schedule components in both the first six and last six sessions of exposure to the delay. In addition, the relation between response rates and reinforcement rates was altered such that both parameters of the single-response version of the matching law (i.e., k and Re) were decreased. Experiment 2 examined the effects of unsignaled delays ranging from 0.5 s to 8.0 s on responding maintained by a multiple VI 20-s VI 120-s schedule of reinforcement. Response rates in both components increased with brief unsignaled delays and decreased with longer delays. As in Experiment 1, response rates as a proportion of baseline were affected similarly for the two components in both the first six and last six sessions of exposure to the delay. Unlike delays imposed between two stimulus events, the effects of delays between responses and reinforcers do not appear to be attenuated when the average time between reinforcers is longer. In addition, the disruptions produced by unsignaled delays appear to be inconsistent with the general finding that responding maintained by higher rates of reinforcement is less resistant to change.     

Responding of pigeons under variable-interval schedules of unsignaled, briefly signaled, and completely signaled delays to reinforcement

In Experiment 1, three pigeons' key pecking was maintained under a variable-interval 60-s schedule of food reinforcement. A 1-s unsignaled nonresetting delay to reinforcement was then added. Rates decreased and stabilized at values below those observed under immediate-reinforcement conditions. A brief stimulus change (key lit red for 0.5 s) was then arranged to follow immediately the peck that began the delay. Response rates quickly returned to baseline levels. Subsequently, rates near baseline levels were maintained with briefly signaled delays of 3 and 9 s. When a 27-s briefly signaled delay was instituted, response rates decreased to low levels. In Experiment 2, four pigeons' responding was first maintained under a multiple variable-interval 60-s (green key) variable-interval 60-s (red key) schedule. Response rates in both components fell to low levels when a 3-s unsignaled delay was added. In the first component delays were then briefly signaled in the same manner as Experiment 1, and in the second component they were signaled with a change in key color that remained until food was delivered. Response rates increased to near baseline levels in both components, and remained near baseline when the delays in both components were lengthened to 9 s. When delays were lengthened to 27 s, response rates fell to low levels in the briefly signaled delay component for three of four pigeons while remaining at or near baseline in the completely signaled delay component. In Experiment 3, low response rates under a 9-s unsignaled delay to reinforcement (tandem variable-interval 60 s fixed-time 9 s) increased when the delay was briefly signaled. The role of the brief stimulus as conditioned reinforcement may be a function of its temporal relation to food, and thus may be related to the eliciting function of the stimulus.     

A comparison of signaled and unsignaled delay of reinforcement

Pigeons were trained on either a variable-interval 60-second schedule, or on a schedule that differentially reinforced responses that were spaced at least 20 seconds apart. The birds were then exposed to several durations of reinforcement delay, with comparisons between signaled and unsignaled delays. Although unsignaled delays of 5 and 10 seconds produced large decreases in response rate, signaled delays of up to 10 seconds produced only moderate decreases in response rates. In addition, some subjects responded more rapidly with a .5 or 1.0 second duration of unsignaled delay than with immediate reinforcement. These response rate changes occurred regardless of whether the rate of reinforcement concomitantly decreased or increased.     

Molar And Molecular Control In Variable-interval And Variable-ratio Schedules

Response rates are typically higher under variable-ratio than under variable-interval schedules of reinforcement, perhaps because of differences in the dependence of reinforcement rate on response rate or because of differences in the reinforcement of long interresponse times. A variable-interval-with-added-linear-feedback schedule is a variable-interval schedule that provides a response rate/reinforcement rate correlation by permitting the minimum interfood interval to decrease with rapid responding. Four rats were exposed to variable-ratio 15, 30, and 60 food reinforcement schedules, variable-interval 15-, 30-, and 60-s food reinforcement schedules, and two versions of variable-interval-with-added-linear-feedback 15-, 30-, and 60-s food reinforcement schedules. Response rates on the variable-interval-with-added-linear-feedback schedule were similar to those on the variable-interval schedule; all three schedules led to lower response rates than those on the variable-ratio schedules, especially when the schedule values were 30. Also, reinforced interresponse times on the variable-interval-with-added-linear-feedback schedule were similar to those on variable interval and much longer than those produced by variable ratio. The results were interpreted as supporting the hypothesis that response rates on variable-interval schedules in rats are lower than those on comparable variable-ratio schedules, primarily because the former schedules reinforce long interresponse times.     

Resistance to change of responding maintained by unsignaled delays to reinforcement: a response-bout analysis

Previous experiments have shown that unsignaled delayed reinforcement decreases response rates and resistance to change. However, the effects of different delays to reinforcement on underlying response structure have not been investigated in conjunction with tests of resistance to change. In the present experiment, pigeons responded on a three-component multiple variable-interval schedule for food presented immediately, following brief (0.5 s), or following long (3 s) unsignaled delays of reinforcement. Baseline response rates were lowest in the component with the longest delay; they were about equal with immediate and briefly delayed reinforcers. Resistance to disruption by presession feeding, response-independent food during the intercomponent interval, and extinction was slightly but consistently lower as delays increased. Because log survivor functions of interresponse times (IRTs) deviated from simple modes of bout initiations and within-bout responding, an IRT-cutoff method was used to examine underlying response structure. These analyses suggested that baseline rates of initiating bouts of responding decreased as scheduled delays increased, and within-bout response rates tended to be lower in the component with immediate reinforcers. The number of responses per bout was not reliably affected by reinforcer delay, but tended to be highest with brief delays when total response rates were higher in that component. Consistent with previous findings, resistance to change of overall response rate was highly correlated with resistance to change of bout-initiation rates but not with within-bout responding. These results suggest that unsignaled delays to reinforcement affect resistance to change through changes in the probability of initiating a response bout rather than through changes in the underlying response structure.     

Responding of pigeons under variable-interval schedules of signaled-delayed reinforcement: effects of delay-signal duration.

Two experiments with pigeons examined the relation of the duration of a signal for delay ("delay signal") to rates of key pecking. The first employed a multiple schedule comprised of two components with equal variable-interval 60-s schedules of 27-s delayed food reinforcement. In one component, a short (0.5-s) delay signal, presented immediately following the key peck that began the delay, was increased in duration across phases; in the second component the delay signal initially was equal to the length of the programmed delay (27 s) and was decreased across phases. Response rates prior to delays were an increasing function of delay-signal duration. As the delay signal was decreased in duration, response rates were generally higher than those obtained under identical delay-signal durations as the signal was increased in duration. In Experiment 2 a single variable-interval 60-s schedule of 27-s delayed reinforcement was used. Delay-signal durations were again increased gradually across phases. As in Experiment 1, response rates increased as the delay-signal duration was increased. Following the phase during which the signal lasted the entire delay, shorter delay-signal-duration conditions were introduced abruptly, rather than gradually as in Experiment 1, to determine whether the gradual shortening of the delay signal accounted for the differences observed in response rates under identical delay-signal conditions in Experiment 1. Response rates obtained during the second exposures to the conditions with shorter signals were higher than those observed under identical conditions as the signal duration was increased, as in Experiment 1. In both experiments, rates and patterns of responding during delays varied greatly across subjects and were not systematically related to delay-signal durations. The effects of the delay signal may be related to the signal's role as a discriminative stimulus for adventitiously reinforced intradelay behavior, or the delay signal may have served as a conditioned reinforcer by virtue of the temporal relation between it and presentation of food.     

Local patterns of responding maintained by concurrent and multiple schedules

Local patterns of responding were studied when pigeons pecked for food in concurrent variable-interval schedules (Experiment I) and in multiple variable-interval schedules (Experiment II). In Experiment I, similarities in the distribution of interresponse times on the two keys provided further evidence that responding on concurrent schedules is determined more by allocation of time than by changes in local pattern of responding. Relative responding in local intervals since a preceding reinforcement showed consistent deviations from matching between relative responding and relative reinforcement in various postreinforcement intervals. Response rates in local intervals since a preceding changeover showed that rate of responding is not the same on both keys in all postchangeover intervals. The relative amount of time consumed by interchangeover times of a given duration approximately matched relative frequency of reinforced interchangeover times of that duration. However, computer simulation showed that this matching was probably a necessary artifact of concurrent schedules. In Experiment II, when component durations were 180 sec, the relationship between distribution of interresponse times and rate of reinforcement in the component showed that responding was determined by local pattern of responding in the components. Since responding on concurrent schedules appears to be determined by time allocation, this result would establish a behavioral difference between multiple and concurrent schedules. However, when component durations were 5 sec, local pattern of responding in a component (defined by interresponse times) was less important in determining responding than was amount of time spent responding in a component (defined by latencies). In fact, with 5-sec component durations, the relative amount of time spent responding in a component approximately matched relative frequency of reinforcement in the component. Thus, as component durations in multiple schedules decrease, multiple schedules become more like concurrent schedules, in the sense that responding is affected by allocation of time rather than by local pattern of responding.     

Synthetic variable-interval schedules of reinforcement

Three pigeons pecked for food on a synthetic variable-interval schedule of reinforcement that had two independent parts: a variable-interval schedule that arranged a distribution of interreinforcement intervals, and a device that randomly assigned each reinforcement to one of 10 classes of interresponse times. The frequencies of reinforcement for the 10 classes of interresponse times were systematically varied, while the overall frequency of reinforcement was held within a comparatively narrow range. The 10 classes extended either from 0.1 to 0.6 sec in 0.05-sec intervals, or from 1.0 to 6.0 sec in 0.5-sec intervals. In the former case, some control by reinforcement was obtained, but it was weak and no simple relationships were discernible. In the latter case, the relative frequency of an interresponse time was a generally increasing function of its relative frequency of reinforcement, and two simple controlling relationships were found. First, the function relating interresponse times per opportunity to reinforcements per opportunity was, over a restricted range, approximately linear with a slope of unity. Second, when all 10 classes of interresponse times were reinforced equally often, the relative frequency of an interresponse time approximately equalled the relative reciprocal of its length.     

Effects of signaled versus unsignaled delay of reinforcement on choice

Pigeons chose between 5-s and 15-s delay-of-reinforcement alternatives. The first key peck to satisfy the choice schedule began a delay timer, and food was delivered at the end of the interval. Key pecks during the delay interval were measured, but had no scheduled effect. In Experiment 1, signal conditions and choice schedules were varied across conditions. During unsignaled conditions, no stimulus change signaled the beginning of a delay interval. During differential and nondifferential signal conditions, offset of the choice stimuli and onset of a delay stimulus signaled the beginning of a delay interval. During differential signal conditions, different stimuli were correlated with the 5-s and 15-s delays, whereas the same stimulus appeared during both delay durations during nondifferential signal conditions. Pigeons showed similar, extreme levels of preference for the 5-s delay alternative during unsignaled and differentially signaled conditions. Preference levels were reliably lower with nondifferential signals. Experiment 2 assessed preference with two pairs of unsignaled delays in which the ratio of delays was held constant but the absolute duration was increased fourfold. No effect of absolute duration was found. The results highlight the importance of delayed primary reinforcement effects and challenge models of choice that focus solely on conditioned reinforcement.     

Key pecking of pigeons under variable-interval schedules of briefly signaled delayed reinforcement: effects of variable-interval value.

Key pecking of 4 pigeons was maintained under a multiple variable-interval 20-s variable-interval 120-s schedule of food reinforcement. When rates of key pecking were stable, a 5-s unsignaled, nonresetting delay to reinforcement separated the first peck after an interval elapsed from reinforcement in both components. Rates of pecking decreased substantially in both components. When rates were stable, the situation was changed such that the peck that began the 5-s delay also changed the color of the keylight for 0.5 s (i.e., the delay was briefly signaled). Rates increased to near-immediate reinforcement levels. In subsequent conditions, delays of 10 and 20 s, still briefly signaled, were tested. Although rates of key pecking during the component with the variable-interval 120-s schedule did not change appreciably across conditions, rates during the variable-interval 20-s component decreased greatly in 1 pigeon at the 10-s delay and decreased in all pigeons at the 20-s delay. In a control condition, the variable-interval 20-s schedule with 20-s delays was changed to a variable-interval 35-s schedule with 5-s delays, thus equating nominal rates of reinforcement. Rates of pecking increased to baseline levels. Rates of pecking, then, depended on the value of the briefly signaled delay relative to the programmed interfood times, rather than on the absolute delay value. These results are discussed in terms of similar findings in the literature on conditioned reinforcement, delayed matching to sample, and classical conditioning.     

Effects of reinforcement rate and delay on the acquisition of lever pressing by rats.

The acquisition of lever pressing by naive rats, in the absence of shaping, was studied as a function of different rates and unsignaled delays of reinforcement. Groups of 3 rats were each exposed to tandem schedules that differed in either the first or the second component. First-component schedules were either continuous reinforcement or random-interval 15, 30, 60 or 120 s; second-component schedules were fixed-time 0, 1, 3, 6, 12, or 24 s. Rate of responding was low under continuous immediate reinforcement and higher under random-interval 15 s. Random interval 30-s and 60-s schedules produced lower rates that were similar to each other. Random-interval 120 s controlled the lowest rate in the immediate-reinforcement condition. Adding a constant 12-s delay to each of the first-component schedule parameters controlled lower response rates that did not vary systematically with reinforcement rate. The continuous and random-interval 60-s schedules of immediate reinforcement controlled higher global and first-component response rates than did the same schedules combined with longer delays, and first-component rates showed some graded effects of delay duration. In addition, the same schedules controlled higher second-component response rates in combination with a 1-s delay than in combination with longer delays. These results were related to those from previous studies on acquisition with delayed reinforcement as well as to those from similar reinforcement procedures used during steady-state responding.     

The concurrent reinforcement of two interresponse times: the relative frequency of an interresponse time equals its relative harmonic length

The relative lengths of two concurrently reinforced interresponse times were varied in an experiment in which three pigeons obtained food by pecking on a single key. Visual discriminative stimuli accompanied the two time intervals in which reinforcements were scheduled according to a one-minute variable-interval. The steady-state relative frequency of an interresponse time approximately equalled the complement of its relative length, that is, its relative harmonic length. Thus, lengths of interresponse times and delays of reinforcement have the same effect on the relative frequencies of interresponse times and choices in one-key and two-key concurrent variable-interval schedules, respectively. A second experiment generalized further the functional equivalence between the effects of these one-key and two-key concurrent schedules by revealing that the usual matching-to-relative-immediacy in two-key concurrent schedules is undisturbed if reinforcement depends upon the occurrence of a response at the end of the delay interval, as it does in the one-key schedules. The results of both experiments are consistent with a quantitative theory of concurrent operant behavior.     

Two-key concurrent paced variable-interval paced variable-interval schedules of reinforcement

Nine pigeons were used in two experiments in which a response was reinforced if a variable-interval schedule had assigned a reinforcement and if the response terminated an interresponse time within a certain interval, or class, of interresponse times. One such class was scheduled on one key, and a second class was scheduled on a second key. The procedure was, therefore, a two-key concurrent paced variable-interval paced variable-interval schedule. In Exp. I, the lengths of the two reinforced interresponse times were varied. The relative frequency of responding on a key approximately equalled the relative reciprocal of the length of the interresponse time reinforced on that key. In Exp. II, the relative frequency and relative magnitude of reinforcement were varied. The relative frequency of responding on the key for which the shorter interresponse time was reinforced was a monotonically increasing, negatively accelerated function of the relative frequency of reinforcement on that key. The relative frequency of responding depended on the relative magnitude of reinforcement in approximately the same way as it depended on the relative frequency of reinforcement. The relative frequency of responding on the key for which the shorter interresponse time was reinforced depended on the lengths of the two reinforced interresponse times and on the relative frequency and relative magnitude of reinforcement in the same way as the relative frequency of the shorter interresponse time depended on these variables in previous one-key concurrent schedules of reinforcement for two interresponse times.     

Within-session delay-of-reinforcement gradients

Within-session delay-of-reinforcement gradients were generated with pigeons by progressively increasing delays to reinforcement within each session. In Experiment 1, the effects of imposing progressive delays on variable-interval and fixed-interval schedules were investigated while controlling for simultaneous decreases in reinforcer rate across the session via a within-subject yoked-control procedure. Rate of key pecking decreased as a negatively decelerated function of delay of reinforcement within a session. These rate decreases were greater than those during a yoked-interval session in which the rate of immediate reinforcement decreased at the same rate as it did under the progressive-delay procedure. In Experiment 2, delay-of-reinforcement gradients were shallower when the progressive delay intervals were signaled by a blackout than when they were unsignaled. The delay gradients obtained in each experiment were similar to those generated under conditions in which different delays of reinforcement are imposed across blocks of sessions. The present procedure offers a technique for rapidly generating delay-of-reinforcement gradients that might serve as baselines for assessing the effects of other behavioral and pharmacological variables.     

Sensitivity to reinforcer duration in a self-control procedure

In a concurrent-chains procedure, pigeons' responses on left and right keys were followed by reinforcers of different durations at different delays following the choice responses. Three pairs of reinforcer delays were arranged in each session, and reinforcer durations were varied over conditions. In Experiment 1 reinforcer delays were unequal, and in Experiment 2 reinforcer delays were equal. In Experiment 1 preference reversal was demonstrated in that an immediate short reinforcer was chosen more frequently than a longer reinforcer delayed 6 s from the choice, whereas the longer reinforcer was chosen more frequently when delays to both reinforcers were lengthened. In both experiments, choice responding was more sensitive to variations in reinforcer duration at overall longer reinforcer delays than at overall shorter reinforcer delays, independently of whether fixed-interval or variable-interval schedules were arranged in the choice phase. We concluded that preference reversal results from a change in sensitivity of choice responding to ratios of reinforcer duration as the delays to both reinforcers are lengthened.     

Mechanisms underlying the effects of unsignaled delayed reinforcement on key pecking of pigeons under variable-interval schedules.

Three experiments were conducted to test an interpretation of the response-rate-reducing effects of unsignaled nonresetting delays to reinforcement in pigeons. According to this interpretation, rates of key pecking decrease under these conditions because key pecks alternate with hopper-observing behavior. In Experiment 1, 4 pigeons pecked a food key that raised the hopper provided that pecks on a different variable-interval-schedule key met the requirements of a variable-interval 60-s schedule. The stimuli associated with the availability of the hopper (i.e., houselight and keylight off, food key illuminated, feedback following food-key pecks) were gradually removed across phases while the dependent relation between hopper availability and variable-interval-schedule key pecks was maintained. Rates of pecking the variable-interval-schedule key decreased to low levels and rates of food-key pecks increased when variable-interval-schedule key pecks did not produce hopper-correlated stimuli. In Experiment 2, pigeons initially pecked a single key under a variable-interval 60-s schedule. Then the dependent relation between hopper presentation and key pecks was eliminated by arranging a variable-time 60-s schedule. When rates of pecking had decreased to low levels, conditions were changed so that pecks during the final 5 s of each interval changed the keylight color from green to amber. When pecking produced these hopper-correlated stimuli, pecking occurred at high rates, despite the absence of a peck-food dependency. When peck-produced changes in keylight color were uncorrelated with food, rates of pecking fell to low levels. In Experiment 3, details (obtained delays, interresponse-time distributions, eating times) of the transition from high to low response rates produced by the introduction of a 3-s unsignaled delay were tracked from session to session in 3 pigeons that had been initially trained to peck under a conventional variable-interval 60-s schedule. Decreases in response rates soon after the transition to delayed reinforcement were accompanied by decreases in eating times and alterations in interresponse-time distributions. As response rates decreased and became stable, eating times increased and their variability decreased. These findings support an interpretation of the effects of delayed reinforcement that emphasizes the importance of hopper-observing behavior.     

Molecular contingencies in schedules of intermittent punishment

In two experiments, key pecking of pigeons was maintained by a variable-interval 180-s schedule of food presentation. Conjointly, a second schedule delivered response-dependent electric shock. In the first experiment, shocks were presented according to either a variable-interval or a nondifferential interval-percentile schedule. The variable-interval shock schedule differentially delivered shocks following long interresponse times. Although the nondifferential shock schedules delivered shocks less differentially with respect to interresponse times, the two shock schedules equally reduced the relative frequency of long interresponse times. The second experiment differentially shocked long or short interresponse times in different conditions, with resulting decreases in the relative frequency of the targeted interresponse times. These experiments highlight the importance of selecting the appropriate level of analysis for the interaction of behavior and environment. Orderly relations present at one level of analysis (e.g., interresponse times) may not be revealed at other levels of analysis (e.g., overall response rate).