Post-Conditioning Devaluation of an Instrumental Reinforcer has no Effect on Extinction Performance

The extent to which a representation of the reinforcer controls an instrumental response can be assessed by studying the effect of post-conditioning changes in the reinforcer value. In the first experiment rats were trained to press a lever for sucrose pellets on a variable-interval (VI) schedule. The sucrose was subsequently devalued by pairing with Lithium Chloride (LiCl). This had no effect on lever pressing in extinction, although it profoundly reduced reacquisition responding and consumption. In Experiment II rats were trained to shuttle between the two distinctive chambers of a choice-box, in which lever pressing was reinforced in one chamber by sucrose and in the other chamber by food pellets programmed on independent VI schedules. A LiCl-induced taste-aversion was conditioned to the sucrose, and although this markedly affected reacquisition, extinction responding in the sucrose chamber and chamber preference were unaffected. These results indicate that instrumental performance can be independent of the current value of the reinforcer, and are discussed with reference to stimulus-response theory and second-order Pavlovian conditioning.     

The control of appetitive instrumental responding does not depend on classical conditioning to the discriminative stimulus

In Experiment I, rats were exposed to a classical relationship between a clicker-light compound and response-independent food. Conditioning to the light was blocked if the clicker had previously served as a classical signal for food, but not if it had been established as a discriminative stimulus for food-reinforced lever pressing. In Experiment II, a tone-light compound served as a discriminative stimulus for lever pressing. Control by the light was blocked if the tone was independently trained as a discriminative stimulus, but not if it was trained as a classical signal for response-independent food. These results suggest that discriminative stimuli do not come to control appetitive instrumental responding by virtue of their implicit classical relationship to the instrumental reinforcer.     

Contingency effects with maintained instrumental reinforcement

In three experiments we investigated the effect on the performance of thirsty rats of varying the instrumental contingency between lever pressing and the delivery of a saccharin reinforcer. In Experiment 1, the subjects performed more slowly in a non-contingent condition, in which the momentary probability of reinforcement was unaffected by whether or not the animals pressed, than in a contingent condition in which the reinforcer was never presented except following a lever press. This was true of performance under both random ratio and interval schedules in which the function determining the probability of reinforcement following a lever press remained the same across the contingent and non-contingent conditions. Experiment 2 demonstrated that instrumental performance was less affected when the contingency was degraded by the introduction of free reinforcers if these reinforcers were signalled. In Experiment 3, lever pressing was reinstated to some degree after non-contingent training by giving non-reinforced exposure to the operant chamber in the absence of the lever. These results suggest that free reinforcers depress instrumental behaviour through a performance mechanism engaged by their ability to support conditioning of the contextual cues.     

Instrumental responding following reinforcer devaluation

In two experiments, hungry rats were given instrumental lever-press training for an appetitive reinforcer and, in addition, were exposed to another type of food which was not contingent on lever pressing. In the first experiment, exposure to each type of food was on separate days, whereas in the second experiment rats were exposed to each type of food in strict alternation within each session. Subsequently, a food aversion was conditioned to the reinforcer for the experimental group and to the non-contingent food for the control group. In both experiments, animals with an aversion to the reinforcer responded less in an extinction test than animals with an aversion to the non-contingent food. Subsequent reacquisition tests confirmed that the aversion to the non-contingent food in the control group was of comparable strength with that to the reinforcer in the experimental group. The results were discussed in terms of whether the reinforcer is encoded in the associative structure set up by exposure to an instrumental contingency.     

Alcohol seeking by rats: Action or habit?

In two experiments, we examined the relative susceptibility to outcome devaluation of lever pressing by rats for either a 10% ethanol solution or food pellets. The rats were trained to press different levers for these two reinforcers using a sucrose-substitution procedure. An aversion was then conditioned from either the ethanol solution or the food pellets by pairing consumption with illness induced by lithium chloride. When instrumental performance was subsequently tested in extinction, the rats pressed less on the pellet lever if the pellets, rather than the ethanol, had been devalued by aversion conditioning. By contrast, performance on the ethanol lever was unaffected by whether the ethanol or pellets were devalued. Moreover, noncontingent presentations of the devalued reinforcer had no impact on test performance. The differential resistance to outcome devaluation suggests that, in contrast to food seeking, alcohol seeking is a stimulus-response habit rather than a goal-directed action mediated by a representation of the action-outcome contingency.     

Stimulus- and response-reinforcer contingencies in autoshaping, operant, classical, and omission training procedures in rats

Separate groups of rats received 500 trials of lever-press training under autoshaping (food delivery followed 10-second lever presentations, or occurred immediately following a response); operant conditioning (responding was necessary for food delivery); and classical conditioning (food followed lever presentations regardless of responding). Each group then received 500 trials on an omission procedure in which food was omitted on trials with a response. Another group received 1000 trials on the omission procedure, and a fifth group, random control, received 1000 uncorrelated presentations of lever and food. The autoshaping, operant, and classical groups reached high response levels by the end of initial training. Acquisition was fastest in the autoshaping group. Responding remained consistently low in the control group. The omission group responded at a level between the control group and the other three groups. During omission training, responding in these three groups declined to the omission-group level. During omission training, the rats continued contacting the lever frequently after lever pressing had declined. Response maintenance under omission training seems not to require topographic similarity between the response and reinforcer-elicited consummatory behaviors.     

The role of response-contingent incentives in lithium chloride-mediated suppression of an operant response

Three experiments investigated what role a novel incentive plays in the development of operant response suppression mediated by lithium chloride. In all experiments animals were trained to press two levers under concurrent schedules of reinforcement. In Experiment 1 responding on one lever delivered a familiar incentive (food pellets), whereas responding on an alternative lever delivered a novel incentive (sucrose solution) prior to lithium chloride injections. If lithium was administered immediately after the instrumental session, the action associated with the novel, but not with the familiar, incentive was suppressed. By comparison, in a control group for which responding on both levers led to the familiar incentive, both actions were suppressed. Experiment 2 examined whether the novelty, rather than the sensory properties, of the incentive is crucial for observing performance suppression. It was found that animals familiarized with the “target” incentive were insensitive to aversion conditioning by lithium, in that there was no difference in response rates between the action that delivered the familiar incentive from that which earned the “target”. In contrast, if animals were unfamiliar with the “target” incentive at the time of aversion conditioning, they suppressed responding on the lever that was associated with the novel incentive but did not suppress responding on the lever associated with the familiar incentive. Experiment 3 investigated the mechanism underlying instrumental performance suppression. After the completion of concurrent lever press training, novel sucrose was introduced in conjunction with the pellets for responding on one lever; responding on the other lever continued to deliver only familiar pellets. Lithium injections were then administered either immediately following the sessions or several hours after the sessions. It was found that the rate of responding on the lever associated with the contingent delivery of sucrose was suppressed below that of the pellet-alone action. By comparison, if lithium injections were administered several hours following the session, an elevation in responding on the sucrose-plus-pellet lever was observed. The outcomes of all three experiments demonstrate not only that the novelty of an incentive is important in obtaining performance suppression, but also that a novel incentive can punish instrumental responding if it has been associated with toxicosis.     

Response duration is sensitive to both immediate and delayed reinforcement

We investigated the duration of lever pressing by rats when the delivery of appetitive reinforcers was contingent upon response duration. In the first experiment, response durations increased when duration requirements were imposed, and they decreased when duration requirements were removed. This effect occurred whether reinforcers were immediate or delayed by 8 s. In order to maintain the integrity of the delay intervals, reinforcer delivery was dependent upon both lever depression and release. In a second experiment, lever depression only and a response duration of at least 4 s were required for reinforcer delivery. Compared to immediate reinforcement conditions, delayed reinforcers increased both variability and the length of the maximum response durations. In a third experiment, immediate reinforcers were delivered contingent upon lever depression and release under a variety of duration requirements. Median lever‐press durations tracked the contingencies rapidly. Across all three experiments, rats emitted numerous response durations that were too short to satisfy the reinforcer requirements, and bimodal distributions similar to those produced by differential reinforcement of low rate schedules were evident for most rats. In many aspects, response duration responds to reinforcement parameters in a fashion similar to rate of discrete responding, but an examination of this continuous dimension of behavior may provide additional information about environment–behavior relationships.     

Autoshaping the pigeon's gape response: acquisition and topography as a function of reinforcer type and magnitude.

The pigeon's key-pecking response is experimentally dissociable into transport (head movement) and gape (jaw movement) components. During conditioning of the key-pecking response, both components come under the control of the conditioned stimulus. To study the acquisition of gape conditioned responses and to clarify the contribution of unconditioned stimulus (reinforcer) variables to the form of the response, gape and key-contact responses were recorded during an autoshaping procedure and reinforcer properties were systematically varied. One group of 8 pigeons was food deprived and subgroups of 2 birds each were exposed to four different pellet sizes as reinforcers, each reinforcer signaled by a keylight conditioned stimulus. A second group was water deprived and received water reinforcers paired with the conditioned stimulus. Water- or food-deprived control groups received appropriate water or food reinforcers that were randomly delivered with respect to the keylight stimulus. Acquisition of the conditioned gape response frequently preceded key-contact responses, and gape conditioned responses were generally elicited at higher rates than were key contacts. The form of the conditioned gape was similar to, but not identical with, the form of the unconditioned gape. The gape component is a critical topographical feature of the conditioned key peck, a sensitive measure of conditioning during autoshaping, and an important source of the observed similarities in the form of conditioned and consummatory responses.     

Transfer of persistence to the acquisition of a new behaviour

Two experiments tested whether the degree of effort required for the reinforcement of one behaviour would affect the acquisition of a second behaviour. In the first experiment, rats were placed in a conditioning chamber and: (a) were required to press a lever for food pellets on a fixed ratio schedule, (b) received free presentation of the pellets, or (c) did not receive pellets. Next, all rats were rewarded for a new behaviour, round trips across the length of a runway. As predicted, the fixed-ratio group had the greatest shuttle rate. In the second experiment, two groups were required to press a lever, and the number of presses per pellet was varied. For two other groups not required to press the lever, the amount of food presented per approach to the feeder was varied. The greater required number of lever presses and the lesser number of pellets per approach to the feeder produced the higher subsequent shuttle rates. Two alternative explanations were compared: the degree of accustomed effort per reinforcer becomes a learned component of behaviour, or high effort increases the habituation of frustration-produced disruptive responses.     

Response–reinforcer contiguity versus response-rate–reinforcer-rate covariance in rats' lever pressing: Support for a multiscale view

The multiscale molar view sees behavior as a flow, like a river, extended in time. Matching theory expresses the way activities compete for time. Relative time taken by any activity depends on relative induction. The present experiment tested matching theory applied to concurrent contingent and noncontingent food. As adjunctive activities that compete with operant activity, we recorded hopper head entries and presses on a lever near the food hopper that had no programmed consequences. Eight naïve rats were first exposed to a variable-time 60 s schedule, which across conditions was gradually transformed into a variable-interval 60 s schedule by increasing the proportion of food that was delivered contingent on pressing a lever far from the hopper. Another group of 4 rats that had been trained to press a lever near a food hopper were introduced in the second condition, in which one food delivery was contingent on far-lever pressing. We found induction following a power function to describe pressing on the far lever (operant activity). Matching theory combined with power-function induction also accounted for adjunctive activity. Results with single contingent food deliveries provided little support for the molecular view that behavior consists of discrete responses “strengthened” by immediately following reinforcers.     

Motivational control of instrumental performance: The role of prior experience of the reinforcer

In four experiments we investigated the conditions that are necessary for instrumental performance to adjust appropriately to a change in drive state. Rats were trained to press a lever and pull a chain concurrently, with one action being reinforced by sucrose solution and the other by food pellets. In Experiment 1 for animals that were hungry throughout training the rate of lever pressing in an extinction test under thirst was unaffected by the type of reinforcer produced by this action during training, even though the sucrose solution would maintain a higher rate than the food pellets during training under thirst. In contrast, animals that experienced the instrumental contingencies arranged by the concurrent schedule while thirsty at some point during training pressed the lever more during the extinction test under thirst when this action had been reinforced with the sucrose solution rather than the food pellets. The remaining three experiments demonstrated that for this incentive effect to occur it is sufficient that the sucrose solution be delivered non-contingently under thirst at some stage of training. Thus, it would appear that performance mediated by an instrumental contingency adjusts appropriately to reinforcer revaluation brought about by a drive shift only if the animals have had prior experience with the incentive under the test drive state. This observation was interpreted in terms of Tolman's “cathexis” theory of motivation.     

Effects of reinforcer delays on choice as a function of income level

Three rats earned their daily food ration by responding during individual trials either on a lever that delivered one food pellet immediately or on a second lever that delivered three pellets after a delay that was continuously adjusted to ensure substantial responding to both alternatives. Choice of the delayed reinforcer increased when the number of trials per session was reduced. This result suggests that models seeking closure on choice effects must include a parameter reflecting how preference changes with sessionwide income. Moreover, models positing that reinforcer probability and immediacy (1/delay) function equivalently in choice are called into question by the finding that probability and immediacy produce opposing effects when income level is changed.     

Effects of reinforcer probability, delay, and response requirements on the choices of rats and pigeons: possible species differences

In Experiment 1 with rats, a left lever press led to a 5-s delay and then a possible reinforcer. A right lever press led to an adjusting delay and then a certain reinforcer. This delay was adjusted over trials to estimate an indifference point, or a delay at which the two alternatives were chosen about equally often. Indifference points increased as the probability of reinforcement for the left lever decreased. In some conditions with a 20% chance of food, a light above the left lever was lit during the 5-s delay on all trials, but in other conditions, the light was only lit on those trials that ended with food. Unlike previous results with pigeons, the presence or absence of the delay light on no-food trials had no effect on the rats' indifference points. In other conditions, the rats showed less preference for the 20% alternative when the time between trials was longer. In Experiment 2 with rats, fixed-interval schedules were used instead of simple delays, and the presence or absence of the fixed-interval requirement on no-food trials had no effect on the indifference points. In Experiment 3 with rats and Experiment 4 with pigeons, the animals chose between a fixed-ratio 8 schedule that led to food on 33% of the trials and an adjusting-ratio schedule with food on 100% of the trials. Surprisingly, the rats showed less preference for the 33% alternative in conditions in which the ratio requirement was omitted on no-food trials. For the pigeons, the presence or absence of the ratio requirement on no-food trials had little effect. The results suggest that there may be differences between rats and pigeons in how they respond in choice situations involving delayed and probabilistic reinforcers.     

Irrelevant incentive learning during instrumental conditioning: The role of the drive-reinforcer and response-reinforcer relationships

Four experiments investigated the processes by which a motivationally-induced change in the value of the training reinforcer affects instrumental performance. Initially, thirsty rats were trained to lever press for either a sodium or non-sodium solution. In Experiment I sodium-trained rats responded faster in extinction following the induction of a sodium appetite, but not following either food or water deprivation. Thus, enhanced extinction performance depends upon the relevance of the training reinforcer to the test drive state. The remaining experiments examined the role of the instrumental contingency. Animals received response-contingent presentations of one solution alternated either within (Experiments II and III) or between sessions (Experiment IV) with non-contingent presentations of another solution. Neither procedure yielded convincing evidence that contingent sodium presentations generated more responding in extinction under a sodium appetite than did non-contingent sodium presentations. On the basis of these results, we argue that the instrumental contingency itself does not play a major role in this irrelevant incentive effect.     

Discrimination between reinforced action patterns in the rat

Laboratory rats were rewarded for face-washing, rearing, or scratching by being given the opportunity to press retractable levers for food reward. Yoked control animals received the same number of lever presentations and food rewards, but did not have to face-wash, rear, or scratch to obtain the levers. The experimental animals showed increases in number of bouts of reinforced target behavior above control levels, and the total amount of time spent face-washing increased when a 1.5-sec criterion for reinforced bout length was introduced. The activities in this experiment were made to serve a discriminative as well as an instrumental function, since the cue to tell the rat which lever to press for reward when the levers were presented was the activity that the rat had engaged in to obtain lever presentation. In two separate experiments high levels of discrimination between behaviors were obtained. Discrimination was worse following scratching than after other actions, and scratching also showed relatively poor instrumental conditioning. The relationship between Pavlovian and instrumental conditioning processes in this situation is discussed.     

Choice with delayed and probabilistic reinforcers: effects of prereinforcer and postreinforcer stimuli.

In Experiment 1, pigeons' pecks on a green key led to a 5-s delay with green houselights, and then food was delivered on 20% (or, in other conditions, 50%) of the trials. Pecks on a red key led to an adjusting delay with red houselights, and then food was delivered on every trial. The adjusting delay was used to estimate indifference points: delays at which the two alternatives were chosen about equally often. Varying the presence or absence of green houselights during the delays that preceded possible food deliveries had large effects on choice. In contrast, varying the presence of the green or red houselights in the intertrial intervals had no effects on choice. In Experiment 2, pecks on the green key led to delays of either 5 s or 30 s with green houselights, and then food was delivered on 20% of the trials. Varying the duration of the green houselights on nonreinforced trials had no effect on choice. The results suggest that the green houselights served as a conditioned reinforcer at some times but not at others, depending on whether or not there was a possibility that a primary reinforcer might be delivered. Given this interpretation of what constitutes a conditioned reinforcer, most of the results were consistent with the view that the strength of a conditioned reinforcer is inversely related to its duration.     

Consequences of response-contingent change in unconditioned stimulus intensity upon the rabbit (Oryctolagus cuniculus) nictitating membrane response.

In a between-groups design, an instrumental contingency was superimposed on classical conditioning of the rabbit nictitating membrane response. Subjects were exposed to a tonal conditioned stimulus (CS) and, if no conditioned response (CR) occurred, a 5.0-ma. paraorbital electric shock unconditioned stimulus (US). USs of 5.0, 3.3, 1.7, and 0.0 ma., respectively, were contingent on occurrence of a CR in the CS-US interval for subjects in the 4 groups of the experiment. The group exposed to contingent US omission differed from the other 3 groups in percentage CR, onset latency, amplitude, and 2 indices of CR form, and those 3 groups generally did not differ significantly among themselves on these dependent variables. Results were interpreted to be contrary to "law of effect" formulations of classical conditioning.     

The role of the instrumental contingency in the motivational control of performance

In four experiments we investigated an irrelevant incentive effect based upon a transition from hunger to thirst. Hungry rats were trained to lever press either for sucrose solution or for food pellets before performance was tested in extinction while they were thirsty. Reinforcer-specific motivational control was found in the first experiment in that the animals pressed the lever more on tests following training with the sucrose solution rather than with food pellets. Moreover, this effect was seen only when testing was conducted following water, but not following food deprivation. The outcome of the remaining experiments suggests that this motivational control is not mediated by the instrumental contingency between lever pressing and the sucrose reinforcer during training. In these studies lever pressing and chain pulling were reinforced concurrently, one with sucrose and the other with food pellets, in order to equate the noninstrumental functions of the incentives. Following this training, lever pressing in extinction under thirst was unaffected by the type of incentive used as its reinforcer during training.     

Variations in the sensitivity of instrumental responding to reinforcer devaluation

In five experiments hungry rats were trained to make a lever press response for a sucrose reinforcer. That sucrose was subsequently devalued by conditioning a food-aversion to it, and the ability of the rats to integrate knowledge about the instrumental contingency with that gained from aversion training was assessed in an extinction test. Experiment I showed successful integration following limited but not extended instrumental training. Experiment II suggested that the crucial factor was the spacing of training; successful integration was seen after massed but not distributed training. The third experiment implicated distributed experience with the reinforcer, rather than distributed response practice, in failures of integration. Experiment IV showed that if the distribution of food-aversion learning was dissimilar to that of instrumental training then a failure of integration could result; this finding was able to account for the distribution of training effects seen in previous studies, but not the effect of extended training. Experiment V replicated the extended training effect seen in Experiment I, and provided evidence that this may reflect the degree of exposure to the reinforcer rather than the extent of response practice.