Defining reinforcess — A problem in communication for consultation in behavior modification

This paper reviews the respondent (Hull-Spence) and operant (Skinnerian) conditioning definitions of reinforcers and reinforcement and demonstrates the need to keep the systems separate when consulting about behavior modification. The two systems are shown to lead to different modification procedures.One important distinction between the systems is whether a reinforcer is simply associated with a response (respondent) or whether it must follow the response (operant). A second important distinction is the definition of negative reinforcement. In respondent conditioning, negative reinforcement entails presenting an aversive stimulus in association with the response and results in a decrease in response rate. In operant conditioning, negative reinforcement entails the removal of an aversive stimulus following a correct response, which results in an increase in response rate.     

Suppression of operant behavior and schedule-induced licking in rats

The first experiment studied the effects of punishment on rats' lever pressing maintained by a fixed-interval schedule of food reinforcement and on the associated schedule-induced licking. When licking was followed by shock, licking was suppressed but lever pressing was largely unaffected. When lever pressing was followed by shock, lever pressing was suppressed but licking was unaffected. In both cases, the punished behavior recovered its previous unpunished level when the shocks were discontinued. In a second experiment, the rats' lever pressing was maintained by a variable-interval schedule of food reinforcement under which polydipsic licking also developed. Both lever pressing and licking were partially suppressed during a stimulus correlated with occasional unavoidable electric shocks. With a higher shock intensity, both behaviors were suppressed further. Both lever pressing and licking recovered their previous levels when shocks were discontinued. These results show that schedule-induced licking, which has been described as adjunctive behavior, can be suppressed by procedures that suppress reinforced lever pressing, an operant behavior.     

Context Conditioning and Free operant Acquisition under Delayed Reinforcement

Three experiments examined the effect of context conditioning on the acquisition of freeoperant lever pressing by hungry rats when the presentation of the food reinforcer was delayed for 32 sec. The first study replicated the preexposure effect reported by Dickinson, Watt, and Griffiths (1992): Exposure to the contextual cues with the lever withdrawn prior to each instrumental training session enhanced acquisition, an effect that was attenuated by the presentation of non-contingent reinforcement during the preexposure periods. Signalling the non-contingent reinforcers during the preexposure periods with a brief auditory stimulus enhanced acquisition in a second study, suggesting that the non-contingent reinforcement interferes with acquisition through context conditioning. The final study confirmed this conclusion using a within-subject procedure in which pressing different levers was reinforced in two contexts, one of which was also associated with non-contingent reinforcers.     

Overshadowing of a context aversion by a novel incentive in operant conditioning

Two experiments examined the processes underlying the suppression of instrumental behaviours by lithium in rats, as reported by Meachum (1988 and this issue). Experiment 1 examined whether presenting a novel sucrose solution prior to lithium chloride administration would overshadow aversion learning to either the stimuli of the operant chamber or to familiar food pellets. After lever pressing had been established, and in the absence of responding, animals received free deliveries of a novel sucrose solution, familiar food pellets, or both, or they were exposed to only the cues of the operant chamber, prior to lithium injections. Lever pressing for food pellets was then assessed. It was found that the animals receiving the novel sucrose, either alone or with the familiar food pellets, pressed more for pellets than either the group receiving only food pellets or the group exposed to only the context. In addition, there was no appreciable difference in the response rates between the context-only group and the group that received the familiar food pellets. These outcomes were interpreted in terms of the novel sucrose overshadowing aversion learning to the context. Experiment 2 investigated whether in fact aversive contextual conditioning could be obtained using the present parameters. This was accomplished by directly manipulating the contexts. In this experiment animals were trained to lever press in two distinctive contexts. Subsequently, one context was paired with the novel sucrose, and the second was experienced in the absence of reinforcement prior to toxicosis. During a subsequent non-reinforced test it was found that responding in the context paired with the novel sucrose was considerably higher than responding in the context that was experienced alone. These findings stand in contrast to the taste-mediated contextual potentiation observed when a consumatory response is used to assess aversive contextual conditioning.     

Conditional acceleration and external disinhibition of operant lever pressing by prereward, neutral, and reinforcing stimuli

Rats responding under a differential-reinforcement-of-low-rate schedule increased their rates of lever pressing during a 20-second click/flash stimulus that preceded the delivery of a response-independent food pellet. The increase could not be attributed to suppression of collateral behavior that has been said to mediate temporally-spaced responding. We propose that the prereward stimulus functioned as an external disinhibitor of lever pressing that had been inhibited by the constraints of the operant schedule. Support is derived from the observed disinhibitory effects of a 10-second unpaired click/flash stimulus and of unsignaled, response-independent pellets that were presented while the animals were responding under the same schedule.     

Contingency effects with maintained instrumental reinforcement

In three experiments we investigated the effect on the performance of thirsty rats of varying the instrumental contingency between lever pressing and the delivery of a saccharin reinforcer. In Experiment 1, the subjects performed more slowly in a non-contingent condition, in which the momentary probability of reinforcement was unaffected by whether or not the animals pressed, than in a contingent condition in which the reinforcer was never presented except following a lever press. This was true of performance under both random ratio and interval schedules in which the function determining the probability of reinforcement following a lever press remained the same across the contingent and non-contingent conditions. Experiment 2 demonstrated that instrumental performance was less affected when the contingency was degraded by the introduction of free reinforcers if these reinforcers were signalled. In Experiment 3, lever pressing was reinstated to some degree after non-contingent training by giving non-reinforced exposure to the operant chamber in the absence of the lever. These results suggest that free reinforcers depress instrumental behaviour through a performance mechanism engaged by their ability to support conditioning of the contextual cues.     

Temporal control in a complex environment: An analysis of schedule-related behavior

Three experiments conducted in an automated ten-compartment chamber recorded collateral activities of rats reinforced for lever pressing on differential-reinforcement-of-low-rate schedules. In Experiment 1, the rate of lever pressing increased when stimulus support for collateral activities was removed, thus confirming earlier findings. However, there were no temporal or sequential patterns of collateral activities that predicted operant responding. In Experiment 2, the rate of lever pressing increased only if (a) access to all stimulus support for collateral activities was simultaneously prevented, and (b) the rat was forced to remain in the presence of the lever and food tray. The availability of any of the stimuli related to collateral activity was sufficient to keep lever-pressing rates from increasing. Experiment 3 examined collateral activities under a signaled differential-reinforcement-of-low-rate schedule. Preventing access to stimuli supporting collateral activities had little effect on stable lever pressing when the signal was maintained. When the signal was removed, collateral activities continued, but lever-pressing rates increased in three of the four rats and rates of food presentation declined in all rats. Hypotheses that collateral activities have (a) a timekeeping or discriminative function, or (b) directly inhibit operant responding were not supported. The results suggest that collateral activities may facilitate operant responding by simply removing the subject from the presence of reinforcement-related stimuli.     

Behavioral momentum theory fails to account for the effects of reinforcement rate on resurgence

The behavioral‐momentum model of resurgence predicts reinforcer rates within a resurgence preparation should have three effects on target behavior. First, higher reinforcer rates in baseline (Phase 1) produce more persistent target behavior during extinction plus alternative reinforcement. Second, higher rate alternative reinforcement during Phase 2 generates greater disruption of target responding during extinction. Finally, higher rates of either reinforcement source should produce greater responding when alternative reinforcement is suspended in Phase 3. Recent empirical reports have produced mixed results in terms of these predictions. Thus, the present experiment further examined reinforcer‐rate effects on persistence and resurgence. Rats pressed target levers for high‐rate or low‐rate variable‐interval food during Phase 1. In Phase 2, target‐lever pressing was extinguished, an alternative nose‐poke became available, and nose‐poking produced either high‐rate variable‐interval, low‐rate variable‐interval, or no (an extinction control) alternative reinforcement. Alternative reinforcement was suspended in Phase 3. For groups that received no alternative reinforcement, target‐lever pressing was less persistent following high‐rate than low‐rate Phase‐1 reinforcement. Target behavior was more persistent with low‐rate alternative reinforcement than with high‐rate alternative reinforcement or extinction alone. Finally, no differences in Phase‐3 responding were observed for groups that received either high‐rate or low‐rate alternative reinforcement, and resurgence occurred only following high‐rate alternative reinforcement. These findings are inconsistent with the momentum‐based model of resurgence. We conclude this model mischaracterizes the effects of reinforcer rates on persistence and resurgence of operant behavior.     

Compounding of discriminative stimuli correlated with chained and multiple schedules

In studies of stimulus compounding (1) the stimuli are presented randomly, (2) primary reinforcement is correlated with each stimulus, (3) a specific response is emitted during each stimulus, and (4) the response is necessary to produce the reinforcer. The present experiments assessed the importance of these procedures by (1) presenting light and tone stimuli in fixed order, (2) removing reinforcement (food) during one stimulus, (3) preventing the response (lever pressing) from being emitted, and (4) eliminating the contingency between lever pressing and food. These variables were presented in various combinations within the context of chained and multiple schedules. When the stimuli were combined in the schedule component correlated with each stimulus, the frequency of lever pressing increased in most instances (additive summation). This suggests that the effect of combining stimuli was not closely tied to the specific procedures used in previous experiments. However, presenting the stimuli in a fixed order did have an effect: the level of responding to the compound was generally greatest when the stimuli were combined in the component correlated with the higher frequency of lever pressing to the single stimulus. Additive summation failed to occur consistently when response-independent food was correlated with each stimulus, and when both lever pressing and food were eliminated during one stimulus.     

Response–reinforcer contiguity versus response-rate–reinforcer-rate covariance in rats' lever pressing: Support for a multiscale view

The multiscale molar view sees behavior as a flow, like a river, extended in time. Matching theory expresses the way activities compete for time. Relative time taken by any activity depends on relative induction. The present experiment tested matching theory applied to concurrent contingent and noncontingent food. As adjunctive activities that compete with operant activity, we recorded hopper head entries and presses on a lever near the food hopper that had no programmed consequences. Eight naïve rats were first exposed to a variable-time 60 s schedule, which across conditions was gradually transformed into a variable-interval 60 s schedule by increasing the proportion of food that was delivered contingent on pressing a lever far from the hopper. Another group of 4 rats that had been trained to press a lever near a food hopper were introduced in the second condition, in which one food delivery was contingent on far-lever pressing. We found induction following a power function to describe pressing on the far lever (operant activity). Matching theory combined with power-function induction also accounted for adjunctive activity. Results with single contingent food deliveries provided little support for the molecular view that behavior consists of discrete responses “strengthened” by immediately following reinforcers.     

Comparing Locomotion With Lever-press Travel In An Operant Simulation Of Foraging

An operant model of foraging was studied. Rats searched for food by pressing on the left lever, the patch, which provided one, two, or eight reinforcers before extinction (i.e., zero reinforcers). Obtaining each reinforcer lowered the probability of receiving another reinforcer, simulating patch depletion. Rats traveled to another patch by pressing the right lever, which restored reinforcer availability to the left lever. Travel requirement changed by varying the probability of reset for presses on the right lever; in one condition, additional locomotion was required. That is, rats ran 260 cm from the left to the right lever, made one response on the right lever, and ran back to a fresh patch on the left lever. Another condition added three hurdles to the 260-cm path. The lever-pressing and simple locomotion conditions generated equivalent travel times. Adding the hurdles produced longer times in patches than did the lever-pressing and simple locomotion requirements. The results contradict some models of optimal foraging but are in keeping with McNair's (1982) optimal giving-up time model and add to the growing body of evidence that different environments may produce different foraging strategies.     

Overshadowing of a stimulus-reinforcer association by an instrumental response

In two experiments, rats were first exposed to pairings of a clicker and food; they were subsequently, in order to measure the effectiveness of the clicker as a conditioned reinforcer, given the opportunity to press a lever which turned the clicker on. For one group of animals the food originally delivered in the presence of the clicker had been contingent on their performance of an instrumental response (running in a running wheel); for a second the contingency between clicker and food had been purely classical. Although the actual correlation between clicker and food was identical for the two groups, the clicker was a less effective conditioned reinforcer for the first group than for the second. In a third experiment, all animals were initially required to run to obtain food in the presence of the clicker, but one group received additional trials on which food was delivered contingent on running in the absence of the clicker. This group showed less tendency to lever press for the clicker than a second group that had received free food on trials when the clicker was not presented. The results of all three experiments suggest that conditioning to the clicker could be overshadowed if the occurrence of food was more reliably predicted by the execution of an instrumental running response; they thus support the view that instrumental conditioning depends on the establishment of an association between response and reinforcer similar to the association between stimulus and reinforcer underlying classical conditioning.     

The control of appetitive instrumental responding does not depend on classical conditioning to the discriminative stimulus

In Experiment I, rats were exposed to a classical relationship between a clicker-light compound and response-independent food. Conditioning to the light was blocked if the clicker had previously served as a classical signal for food, but not if it had been established as a discriminative stimulus for food-reinforced lever pressing. In Experiment II, a tone-light compound served as a discriminative stimulus for lever pressing. Control by the light was blocked if the tone was independently trained as a discriminative stimulus, but not if it was trained as a classical signal for response-independent food. These results suggest that discriminative stimuli do not come to control appetitive instrumental responding by virtue of their implicit classical relationship to the instrumental reinforcer.     

The role of the instrumental contingency in the motivational control of performance

In four experiments we investigated an irrelevant incentive effect based upon a transition from hunger to thirst. Hungry rats were trained to lever press either for sucrose solution or for food pellets before performance was tested in extinction while they were thirsty. Reinforcer-specific motivational control was found in the first experiment in that the animals pressed the lever more on tests following training with the sucrose solution rather than with food pellets. Moreover, this effect was seen only when testing was conducted following water, but not following food deprivation. The outcome of the remaining experiments suggests that this motivational control is not mediated by the instrumental contingency between lever pressing and the sucrose reinforcer during training. In these studies lever pressing and chain pulling were reinforced concurrently, one with sucrose and the other with food pellets, in order to equate the noninstrumental functions of the incentives. Following this training, lever pressing in extinction under thirst was unaffected by the type of incentive used as its reinforcer during training.     

Post-Conditioning Devaluation of an Instrumental Reinforcer has no Effect on Extinction Performance

The extent to which a representation of the reinforcer controls an instrumental response can be assessed by studying the effect of post-conditioning changes in the reinforcer value. In the first experiment rats were trained to press a lever for sucrose pellets on a variable-interval (VI) schedule. The sucrose was subsequently devalued by pairing with Lithium Chloride (LiCl). This had no effect on lever pressing in extinction, although it profoundly reduced reacquisition responding and consumption. In Experiment II rats were trained to shuttle between the two distinctive chambers of a choice-box, in which lever pressing was reinforced in one chamber by sucrose and in the other chamber by food pellets programmed on independent VI schedules. A LiCl-induced taste-aversion was conditioned to the sucrose, and although this markedly affected reacquisition, extinction responding in the sucrose chamber and chamber preference were unaffected. These results indicate that instrumental performance can be independent of the current value of the reinforcer, and are discussed with reference to stimulus-response theory and second-order Pavlovian conditioning.     

Rate changes after unscheduled omission and presentation of reinforcement

Changes in response rate similar to frustration effects were studied in a two-lever situation. Responding on one lever on a fixed-interval schedule produced access to water for 5 sec and an exteroceptive stimulus. In the presence of this stimulus, responding on another lever on a fixed-interval schedule produced access to water for 5 sec and terminated the stimulus. Occasional omission of a previously scheduled reinforcer after responding on the first lever resulted consistently in increases in rate on the second lever during the immediately succeeding interval. In another procedure, occasional presentation of a previously unscheduled reinforcer after responding on the first lever resulted consistently in decreases in rate on the second lever during the immediately succeeding interval. Changes occurred after the first omissions or presentations and were about the same in magnitude as the procedure continued over several sessions. Typically, an increase or decrease in rate was maintained throughout an entire 100-sec interval. Changes in rate on the second lever of approximately the same magnitude also occurred when rate on the first lever was near-zero under a schedule that differentially reinforced behavior other than lever pressing.     

Stimulus- and response-reinforcer contingencies in autoshaping, operant, classical, and omission training procedures in rats

Separate groups of rats received 500 trials of lever-press training under autoshaping (food delivery followed 10-second lever presentations, or occurred immediately following a response); operant conditioning (responding was necessary for food delivery); and classical conditioning (food followed lever presentations regardless of responding). Each group then received 500 trials on an omission procedure in which food was omitted on trials with a response. Another group received 1000 trials on the omission procedure, and a fifth group, random control, received 1000 uncorrelated presentations of lever and food. The autoshaping, operant, and classical groups reached high response levels by the end of initial training. Acquisition was fastest in the autoshaping group. Responding remained consistently low in the control group. The omission group responded at a level between the control group and the other three groups. During omission training, responding in these three groups declined to the omission-group level. During omission training, the rats continued contacting the lever frequently after lever pressing had declined. Response maintenance under omission training seems not to require topographic similarity between the response and reinforcer-elicited consummatory behaviors.     

Effects of reinforcement schedule on facilitation of operant extinction by chlordiazepoxide

Learning and memory are central topics in behavioral neuroscience, and inbred mice strains are widely investigated. However, operant conditioning techniques are not as extensively used in this field as they should be, given the effectiveness of the methodology of the experimental analysis of behavior. In the present study, male C57B1/6 mice, widely used as background for transgenic studies, were trained to lever press on discrete-trial fixed-ratio 5 or fixed-interval (11 s or 31 s) schedules of food reinforcement and then exposed to 15 extinction sessions following vehicle or chlordiazepoxide injections (15 mg/kg i.p., administered either prior to all extinction sessions, or prior to the final 10 extinction sessions). Extinction of operant behavior was facilitated by drug administration following training on either schedule, but this facilitation only occurred once a number of extinction sessions had taken place. The extinction process proceeded more rapidly following fixed-interval training. Resistance to extinction was equally high following training with either schedule type, and was reduced by drug administration in both cases. These phenomena were evident in individual cumulative records and in analyses of group data. Results are interpreted in terms of phenomena of operant extinction identified in Skinner's (1938) Behavior of Organisms, and by behavioral momentum theory. These procedures could be used to extend the contribution of operant conditioning to contemporary behavioral neuroscience.     

Towards mouse models of perseveration: a heritable component in extinction of operant behavior in fourteen standard and recombinant inbred mouse lines

Extinction of instrumental responses is an essential skill for adaptive behavior such as foraging. So far, only few studies have focused on extinction following appetitive conditioning in mice. We studied extinction of appetitive operant lever-press behavior in six standard inbred mouse strains (A/J, C3H/HeJ, C57BL/6J, DBA/2J, BALB/cByJ and NOD/Ltj) and eight recombinant inbred mouse lines. From the response rates at the end of operant and extinction training we computed an extinction index, with higher values indicating better capability to omit behavioral responding in absence of reward. This index varied highly across the mouse lines tested, and the variability was partially due to a significant heritable component of 12.6%.To further characterize the relationship between operant learning and extinction, we calculated the slope of the time course of extinction across sessions. While many strains showed a considerable capacity to omit responding when lever pressing was no longer rewarded, we found a few lines showing an abnormally high perseveration in lever press behavior, showing no decay in response scores over extinction sessions.No correlation was found between operant and extinction response scores, suggesting that appetitive operant learning and extinction learning are dissociable, a finding in line with previous studies indicating that these forms of learning are dependent on different brain areas. These data shed light on the heritable basis of extinction learning and may help develop animal models of addictive habits and other perseverative disorders, such as compulsive food seeking and eating.