Behavioral momentum and resistance to extinction across repeated extinction tests

The present experiments assessed whether resistance to extinction of pigeons' key pecking decreased across repeated extinction tests. An additional impetus for this research was to determine how the quantitative framework provided by behavioral momentum theory might be used to describe any such changes across tests. Pigeons pecked keys in two‐component multiple schedules (one component associated with a higher reinforcer rate and the other with a lower rate) in which baseline and extinction conditions alternated. In Experiment 1, baseline and extinction conditions alternated every session, and, in Experiment 2, these conditions lasted for 10 and 7 sessions, respectively. Resistance to extinction decreased across successive extinction conditions in both experiments. Fits of the behavioral‐momentum based model of extinction to the data returned uncertain results in Experiment 1 but implicated both generalization decrement and response–reinforcer contingency termination as the possible mechanisms responsible for behavior change in Experiment 2. Thus, these data suggest that experimental manipulations that affect discrimination of changes in reinforcement contingencies may influence resistance to extinction by modulating the disruptive impacts of removing reinforcers from the experimental context and of suspending response–reinforcer contingencies.     

Alternative reinforcement effects on fixed-interval performance

Pigeons' key pecks were reinforced with food on a fixed-interval schedule. Food also was available at variable time periods either independently of responding or for not key pecking (a differential-reinforcement-of-other-behavior schedule). The latter condition arranged reinforcement following the first pause of t seconds after it became available according to a variable-time schedule. This schedule allowed separation of the effects of pause requirements ≤ five-seconds and reinforcement frequency. The time spent pausing increased as the duration of the pause required for reinforcement increased from 0 to 30 seconds and as the frequency of reinforcement for pausing increased from 0 to 2 reinforcers per minute. Key pecking was more evenly distributed within each fixed interval with shorter required pauses and with more frequent reinforcement for pausing. The results complement those obtained with other concurrent schedules in which the same operant response was reinforced in both components.     

On the effects of component durations and component reinforcement rates in multiple schedules

Four experiments, each using the same six pigeons, investigated the effects of varying component durations and component reinforcement rates in multiple variable-interval schedules. Experiment 1 used unequal component durations in which one component was five times the duration of the other, and the shorter component was varied over conditions from 120 seconds to 5 seconds. The schedules were varied over five values for each pair of component durations. Sensitivity to reinforcement rate changes was the same at all component durations. In Experiment 2, both component durations were 5 seconds, and the schedules were again varied using both one and two response keys. Sensitivity to reinforcement was not different from the values found in Experiment 1. In Experiment 3, various manipulations, including body-weight changes, reinforcer duration changes, blackouts, hopper lights correlated with keylights, and overall reinforcement rate changes were carried out. No reliable increase in reinforcement sensitivity resulted from any manipulation. Finally, in Experiment 4, reinforcement rates in the two components were kept constant and unequal, and the component durations were varied. Shorter components produced significantly increased response rates normally in the higher reinforcement rate component, but schedule reversals at short component durations eliminated the response rate increases. The effects of component duration on multiple schedule performance cannot be interpreted as changing sensitivity to reinforcement nor to changing bias.     

Effects of concurrent response-independent reinforcement on fixed-interval schedule performance

In three experiments, behavior maintained by fixed-interval schedules changed when response-independent reinforcement was delivered concurrently according to fixed- or variable-time schedules. In Experiment I, a pattern of positively accelerated responding during fixed interval was changed to a linear pattern when response-independent reinforcement occurred under a variable-time schedule. Overall response rates (total responses/total time) decreased as the frequency of response-independent reinforcement increased. Experiment II showed that the response-rate changes in the first experiment were controlled by the response-reinforcer relation, but the changes in patterns of responding were similar whether concurrently available reinforcement at varying times was response-dependent or response-independent. In the final experiment, the addition of response-independent reinforcement at fixed times to a fixed-interval schedule resulted in changes in both local and overall response rates and in the occurrence of positively accelerated responding between reinforcements. These results suggest that the temporal distribution of reinforcers determines response patterns and that both the response-reinforcement dependency and the schedule of reinforcement determine overall response rates during concurrently scheduled response-dependent and response-independent reinforcement.     

Frequency versus magnitude of reinforcement: New data with a different procedure

Two pigeons, with previous exposure to concurrent schedules, were submitted to 29 sessions of 8 hours each with concurrent variable-interval variable-interval schedules in which reinforcement parameters changed from session to session. In the first nine sessions reinforcement durations were equal in both schedules while reinforcement frequencies varied; in Sessions 10 through 18, both frequency and duration of reinforcement were varied; in Sessions 19 through 29, only reinforcement duration was varied. Results with this different procedure confirm previous findings that behavior is more sensitive to changes in reinforcement frequency than to reinforcement magnitude.     

Conditioned reinforcement versus time to reinforcement in chain schedules.

Pigeons were trained on three-component chain schedules in which the initial component was either a fixed-interval or variable-interval schedule. The middle and terminal components were varied among fixed-interval fixed-interval, variable-interval variable-interval, and an interdependent variable-interval variable-interval schedule in which the sum of the durations of the two variable-interval components was always equal to the sum of the fixed-interval fixed-interval components. At issue was whether the response rate in the initial component was controlled by its time to primary reinforcement or by the temporal parameters of the stimulus correlated with the middle terminal link. The fixed-interval initial-link schedule maintained much lower response rates than the variable-interval initial-link schedule regardless of the schedules in the middle and terminal links. Nevertheless, the intervening schedules played some role: With fixed-interval schedules in the initial links, response rates were consistently highest with independent variable-interval schedules in the middle and terminal links and intermediate with the interdependent variable-interval schedules; these initial-link differences were predicted by the response rates in the middle link of the chain. With variable-interval schedules in the initial links, response rates were lowest with the fixed-interval fixed-interval schedules following the initial link and were not systematically different for the two types of variable-interval variable-interval schedules. The results suggest that time to reinforcement itself accounts for little if any variance in initial-link responding.     

Early extinction effects following intermittent reinforcement: Little evidence of extinction bursts

The occurrence of extinction bursts—transient increases in response rate in excess of those observed in baseline during the period immediately following discontinuation of reinforcement of a response—was examined. In Experiment 1, key pecking of pigeons was reinforced according to a multiple schedule in which a variable-ratio schedule alternated with an interval schedule in which the reinforcers were yoked to the preceding variable-ratio component. In Experiments 2 and 3, rats were screened such that the lever-press response rates of different rats maintained by variable-interval schedules were either relatively high or relatively low. Following these baseline conditions, in Experiments 1 and 2 responding was extinguished by eliminating the food reinforcer and in Experiment 3 by removing the response–reinforcer dependency. Responses immediately following extinction implementation were examined. Response increases relative to baseline during the first 20 min of a 324.75-min extinction session (Experiment 1) or during the first 30-min extinction session (Experiments 2 and 3) were rare and unsystematic. The results (a) reinforce earlier meta-analyses concluding that extinction bursts may be a less ubiquitous early effect of extinction than has been suggested and (b) invite further experimentation to establish their generality as a function of preceding reinforcement conditions.     

Dependency, temporal contiguity, and response-independent reinforcement

A comparison was made of the effects of variable-interval, variable-time, and tandem variable-interval fixed-time schedules on key-peck responding of pigeons. The variable-interval component of the tandem schedule retained the response-reinforcement dependency; the fixed-time component allowed the temporal proximity between responding and reinforcement to vary, constrained only by the duration of the fixed-time interval. Response rates were highest during the variable-interval and lowest during the variable-time schedule. Intermediate response rates occurred during the tandem schedule. The results of a yoked control condition showed that the effects of the tandem schedule were not due simply to changes in reinforcement distribution or frequency. The results suggest that substantial reductions in responding occur when reinforcement is response-dependent but not necessarily contiguous with the response required to produce reinforcement.     

Effects of unsignaled delay of reinforcement on preference and resistance to change

In Phase 1, pigeons were trained on a concurrent chain in which a 3-s unsignaled delay of reinforcement was imposed on responding in a terminal link in some conditions. Preference for that terminal link was always reduced in comparison with conditions in which there was no delay, substantially so for 3 of the 4 pigeons. In Phase 2, pigeons responded in a two-component multiple schedule. The scheduled rates of reinforcement were equal, but a 3-s unsignaled delay was imposed in one component. Resistance of responding to prefeeding and extinction was reduced in the delay component for the same 3 subjects for which the data had shown strong effects of delay on preference. Systematic observation revealed differences in response topography. In the delay component, subjects oriented more closely to the key and responses were less forceful compared with the no-delay component. Our results give further evidence that preference and resistance to change covary within subjects. However, they challenge the premise that the critical determiners of preference (i.e., terminal-link value) and resistance to change (behavioral mass) may be quantified purely in terms of stimulus—reinforcer relations.     

Key pecking during extinction after intermittent or continuous reinforcement as a function of the number of reinforcers delivered during training.

Key pecking by 7 pigeons was established and maintained on a multiple variable-ratio variable-ratio (VR) schedule of food presentation. The schedule in one of the components was then changed to fixed-ratio (FR) 1 for a predetermined number of reinforcers. Both components were then changed to extinction (i.e., multiple extinction, extinction). This sequence was repeated a different number of times for each pigeon to determine the relation between the number of reinforcers delivered during each component of the multiple VR FR 1 schedule and the number of responses during extinction. For most pigeons, there were fewer responses during extinction in the presence of a stimulus recently correlated with FR 1, regardless of the number of reinforcers received. The ratio of the total responses in extinction in the former VR component to the total responses in the former FR 1 component increased as the number of reinforcers delivered during each component of the multiple schedule increased. Within-subject replications of the partial-reinforcement extinction effect generally occurred, and there were no overall reductions in the number of responses in extinction with repeated exposures to extinction.     

Neurochemical changes correlated with behavior maintained under fixed-interval and fixed-ratio schedules of reinforcement.

Key pecking of 4 pigeons was maintained under a multiple 3-min fixed-interval, 30-response fixed-ratio schedule of food presentation. Only one schedule was in effect during an experimental session, and each was correlated with a different keylight stimulus and location (left vs. right). The different schedule components alternated across days or weeks. Cerebrospinal fluid was collected from chronically implanted intracerebroventricular cannulae following sessions with the different schedules, as well as following sessions in which reinforcement was withheld (extinction), when response-independent food was delivered, and when the experimental chamber was dark and there were no scheduled events. Metabolites of the neurotransmitters serotonin, norepinephrine, and dopamine were assayed in cerebrospinal fluid using high-performance liquid chromatography with electrochemical detection. Compared to the fixed-ratio condition, responding maintained under the fixed-interval schedule resulted in consistently higher levels of the serotonin metabolite 5-hydroxyindoleacetic acid and of the dopamine metabolite homovanillic acid in all pigeons. Levels of 3-methoxy-4-hydroxyphenylethylene glycol, a metabolite of norepinephrine, and dihydroxyphenylacetic acid, another dopamine metabolite, were also higher in 3 of the 4 pigeons following exposure to the fixed-interval schedules when compared to levels of these metabolites after exposure to the fixed-ratio schedule. Extinction of fixed-ratio responding resulted in large increases in 5-hydroxyindoleacetic acid compared to levels of this metabolite under the fixed-ratio schedule, whereas this serotonin metabolite decreased during extinction of responding under the fixed-interval schedule. Control procedures suggested that the neurochemical changes were not related to the rate of responding but were a function of the specific experimental conditions. Distinctive neurochemical changes that accompany schedule-controlled responding show the sensitivity of the neurochemical environment to behavioral contingencies and demonstrate further the profound impact that such contingencies have on biobehavioral processes.     

Unsignalled delay of reinforcement in variable-interval schedules

Three pigeons responded on several tandem variable-interval fixed-time schedules in which the value of the fixed-time component was varied to assess the effects of different unsignalled delays of reinforcement. Actual (obtained) delays between the last key peck in an interval and reinforcement were consistently shorter than the nominal (programmed) delay. When nominal delays were relatively short, response rates were higher during the delay condition than during the corresponding nondelay condition. At longer nominal delay intervals, response rates decreased monotonically with increasing delays. The results were consistent with those obtained from delay-of-reinforcement procedures that impose either a stimulus change (signal) or a no-response requirement during the delay interval.     

Persistence during and resurgence following noncontingent reinforcement implemented with and without extinction

Noncontingent reinforcement (NCR) is typically implemented with extinction (EXT) for destructive behavior reinforced by social consequences and without EXT for destructive behavior reinforced by sensory consequences. Behavioral momentum theory (BMT) predicts that responding will be more persistent, and treatment relapse in the form of response resurgence more likely, when NCR is implemented without EXT due to the greater overall rate of reinforcement associated with this intervention. We used an analogue arrangement to test these predictions of BMT by comparing NCR implemented with and without EXT. For two of three participants, we observed more immediate reductions in responding during NCR without EXT. However, for all participants, NCR without EXT produced greater resurgence than NCR with EXT when we discontinued all reinforcers during an EXT Only phase, although there was variability in response patterns across and within participants. Implications for treatment of destructive behavior using NCR are discussed.     

Resistance to reinforcement change in multiple and concurrent schedules assessed in transition and at steady state

Behavioral momentum theory relates resistance to change of responding in a multiple-schedule component to the total reinforcement obtained in that component, regardless of how the reinforcers are produced. Four pigeons responded in a series of multiple-schedule conditions in which a variable-interval 40-s schedule arranged reinforcers for pecking in one component and a variable-interval 360-s schedule arranged them in the other. In addition, responses on a second key were reinforced according to variable-interval schedules that were equal in the two components. In different parts of the experiment, responding was disrupted by changing the rate of reinforcement on the second key or by delivering response-independent food during a blackout separating the two components. Consistent with momentum theory, responding on the first key in Part 1 changed more in the component with the lower reinforcement total when it was disrupted by changes in the rate of reinforcement on the second key. However, responding on the second key changed more in the component with the higher reinforcement total. In Parts 2 and 3, responding was disrupted with free food presented during intercomponent blackouts, with extinction (Part 2) or variable-interval 80-s reinforcement (Part 3) arranged on the second key. Here, resistance to change was greater for the component with greater overall reinforcement. Failures of momentum theory to predict short-term differences in resistance to change occurred with disruptors that caused greater change between steady states for the richer component. Consistency of effects across disruptors may yet be found if short-term effects of disruptors are assessed relative to the extent of change observed after prolonged exposure.     

Choice and conditioned reinforcement.

A potential weakness of one formulation of delay-reduction theory is its failure to include a term for rate of conditioned reinforcement, that is, the rate at which the terminal-link stimuli occur in concurrent-chains schedules. The present studies assessed whether or not rate of conditioned reinforcement has an independent effect upon choice. Pigeons responded on either modified concurrent-chains schedules or on comparable concurrent-tandem schedules. The initial link was shortened on only one of two concurrent-chains schedules and on only one of two corresponding concurrent-tandem schedules. This manipulation increased rate of conditioned reinforcement sharply in the chain but not in the tandem schedule. According to a formulation of delay-reduction theory, when the outcomes chosen (the terminal links) are equal, as in Experiment 1, choice should depend only on rate of primary reinforcement; thus, choice should be equivalent for the tandem and chain schedules despite a large difference in rate of conditioned reinforcement. When the outcomes chosen are unequal, however, as in Experiment 2, choice should depend upon both rate of primary reinforcement and relative signaled delay reduction; thus, larger preferences should occur in the chain than in the tandem schedules. These predictions were confirmed, suggesting that increasing the rate of conditioned reinforcement on concurrent-chains schedules may have no independent effect on choice.     

Interresponse-time shaping by variable-interval-like interresponse-time reinforcement contingencies

The interresponse-time reinforcement contingencies and distributions of interreinforcement intervals characteristic of certain variable-interval schedules were mimicked by reinforcing each key peck with a probability equal to the duration of the interresponse time it terminated, divided by the scheduled mean interreinforcement interval. The interresponse-time reinforcement contingency was then eliminated by basing the probability of reinforcement on the fifth interresponse time preceding the key peck. Even though distributions of interreinforcement intervals were unaffected by this manipulation, response rates consistently increased. A second experiment replicated this effect and showed it to combine additively with that of mean reinforcement rate. These results provide strong support for the contention that current analyses of variable-interval response rates that ignore the inherent interresponse-time reinforcement contingency may be seriously in error.     

The effects of delayed reinforcement on free-operant responding

In previous studies of delayed reinforcement, response rate has been found to vary inversely with the response-reinforcer interval. However, in all of these studies the independent variable, response-reinforcer time, was confounded with the number of reinforcers presented in a fixed period of time (reinforcer frequency). In the present study, the frequency of available reinforcers was held constant, while temporal separation between response and reinforcer was independently manipulated. A repeating time cycle, T, was divided into two alternating time periods, t^D and t^Δ. The first response in t^D was reinforced at the end of the prevailing T cycle and extinction prevailed in t^Δ. Two placements for t^D were defined, an early t^D placement in which t^D precedes t^Δ and a late t^D placement in which t^D follows t^Δ. The duration of the early and late t^D was systematically decreased from 30 seconds (i.e., t^D = T) to 0.1 second. Manipulation of t^D placement and duration controlled the temporal separation between response and reinforcement, but it did not affect the frequency of programmed reinforcers, which was 1/T. The results show that early and late t^D placements of equal duration have similar overall effects upon response rate, reinforcer frequency, responses per reinforcer, and obtained response-reinforcer temporal separation. A stepwise regression analysis using log response rate as the dependent variable showed that the obtained delay was a significant first-step variable for six of eight subjects, with obtained reinforcer frequency significant for the remaining two subjects.     

Independence of response force and reinforcement rate on concurrent variable-interval schedule performance

Five pigeons were trained over 43 experimental conditions on a variety of concurrent variable-interval schedules on which the forces required on the response keys were varied. The results were well described by the generalized matching law with log reinforcement ratios and log force ratios exerting independent (noninteractive) effects on preference. A further analysis using the Akaike criterion, an information-theoretic measure of the efficiency of a model, showed that overall reinforcement rate and overall force requirement did not affect preference. Unlike reinforcement rate changes, force requirement increases did not change the response rate on the alternate key, and an extension of Herrnstein's absolute response rate function for force variation on a single variable-interval schedule is suggested.     

Effects of delayed conditioned reinforcement in chain schedules.

The contingency between responding and stimulus change on a chain variable-interval 33-s, variable-interval 33-s, variable-interval 33-s schedule was weakened by interposing 3-s delays between either the first and second or the second and third links. No stimulus change signaled the delay interval and responses could occur during it, so the obtained delays were often shorter than the scheduled delay. When the delay occurred after the initial link, initial-link response rates decreased by an average of 77% with no systematic change in response rates in the second or third links. Response rates in the second link decreased an average of 59% when the delay followed that link, again with little effect on response rates in the first or third links. Because the effect of delaying stimulus change was comparable to the effect of delaying primary reinforcement in a simple variable-interval schedule, and the effect of the unsignaled delay was specific to the link in which the delay occurred, the results provide strong evidence for the concept of conditioned reinforcement.