Signalling and incentive processes in instrumental reinforcer devaluation

We have previously reported that conditioning an aversion to the reinforcer using an isotonic lithium chloride (LiCl) solution following instrumental training reduces performance in a subsequent extinction test only if animals are re-exposed to the reinforcer prior to the test. Rescorla (1992), in contrast, reported an immediate devaluation effect using a hypertonic LiCl solution that did not depend upon re-exposure. In two experiments we examined the effect of using a hypertonic LiCl solution to condition the aversion to the reinforcer on subsequent instrumental performance in extinction, with and without re-exposure. In Experiment 1 thirsty rats were trained to press a lever for a sucrose solution before being injected with 0.6 M LiCl either immediately or after a delay. Half of the immediate and delay groups were then re-exposed to the sucrose in the absence of the lever, with the remainder being exposed to water. Contrary to the previously reported effects of isotonic LiCl, a hypertonic solution induced a reinforcer devaluation effect in all the immediately poisoned animals, which did not depend upon re-exposure to the reinforcer. In Experiment 2 the possibility that this devaluation effect was induced by the discomfort associated with the hypertonicity of the solution was assessed by replicating Experiment 1 but, in addition, using two immediately poisoned groups given the LiCl injection under anaesthesia. In the absence of anaesthesia, the devaluation effect observed without re-exposure to the reinforcer in Experiment 1 was replicated. When the injection was given under anaesthesia, however, a reinforcer devaluation effect was observed only in animals that were re-exposed to the reinforcer prior to the extinction test. These results were interpreted as evidence that a reinforcer devaluation effect induced by pairing the reinforcer with illness depends upon a process of incentive learning, whereas a devaluation effect mediated by learning a signalling relationship between the reinforcer and somatic discomfort does not.     

Incentive learning following reinforcer devaluation is not conditional upon the motivational state during re-exposure

Three experiments analysed the effect of re-exposure to the reinforcer following aversion conditioning on instrumental performance. In the first experiment, groups of hungry and thirsty rats were trained to press a lever for sucrose, which was then followed by a single injection of lithium chloride (LiCl). On the following day, half the animals in each motivational condition received re-exposure to the sucrose solution; the remaining animals were not re-exposed. In a subsequent extinction test animals that had received re-exposure to the sucrose pressed less than animals that were not re-exposed. Moreover, the effect of re-exposure to the sucrose solution was similar following training under hunger and thirst. In the remaining studies, animals were trained to lever-press for sucrose while either hungry or thirsty. They were then injected with LiCl and re-exposed to the sucrose while either hungry or thirsty, i.e. in the same or different motivational state employed during training, or they were not re-exposed. Lever pressing was then tested in extinction in the training motivational state. As in the first experiment, re-exposure to the reinforcer after aversion conditioning enhanced the magnitude of the reinforcer devaluation effect. More importantly, re-exposure to the sucrose produced a comparable effect on instrumental performance, whether re-exposure was given under the same or different motivational state to that employed during training. These results suggest that the instrumental reinforcer devaluation effect depends upon a process of incentive learning, but that this process is not conditional upon the current motivational state of the animal.     

Irrelevant incentive learning during instrumental conditioning: The role of the drive-reinforcer and response-reinforcer relationships

Four experiments investigated the processes by which a motivationally-induced change in the value of the training reinforcer affects instrumental performance. Initially, thirsty rats were trained to lever press for either a sodium or non-sodium solution. In Experiment I sodium-trained rats responded faster in extinction following the induction of a sodium appetite, but not following either food or water deprivation. Thus, enhanced extinction performance depends upon the relevance of the training reinforcer to the test drive state. The remaining experiments examined the role of the instrumental contingency. Animals received response-contingent presentations of one solution alternated either within (Experiments II and III) or between sessions (Experiment IV) with non-contingent presentations of another solution. Neither procedure yielded convincing evidence that contingent sodium presentations generated more responding in extinction under a sodium appetite than did non-contingent sodium presentations. On the basis of these results, we argue that the instrumental contingency itself does not play a major role in this irrelevant incentive effect.     

The effect of the instrumental training contingency on susceptibility to reinforcer devaluation

Two experiments investigated performance of instrumental lever pressing by rats following post-conditioning devaluation of the sucrose reinforcer produced by establishing an aversion to it. In Experiment I rats responded less in an extinction test after being averted from the sucrose following training on a ratio schedule, but not following an equivalent amount of training on an interval schedule. This was true even though the devalued sucrose would not act as an effective reinforcer on either the ratio or interval schedule. Experiment II provided a further investigation of the insensitivity of interval responding to reinforcer devaluation by comparing test performance under simple extinction with responding when the devalued reinforcer was presented on either a response-contingent or non-contingent schedule during the test. Once again simple extinction performance was unaffected by prior reinforcer devaluation. Furthermore, neither non-contingent nor contingent presentations of the devalued reinforcer significantly depressed responding below the level seen in the extinction condition. Ratio, but not interval performance appears to be controlled by knowledge about the instrumental contingency that encodes specific properties of the training reinforcer.     

Motivational control of instrumental performance following a shift from thirst to hunger

Thirsty rats were trained to press a lever for either a sucrose solution or saline before performance was tested in extinction while the animals were either hungry alone or experiencing both hunger and a sodium appetite. Reinforcer-specific motivational control was observed in that the animals trained with the sucrose solution pressed more than those trained with the saline when they were tested hungry, but not when they were tested under combined hunger and sodium appetite. In order to assess the role of a Pavlovian incentive process in this effect, thirsty animals received non-contingent pairings of one stimulus with the sucrose solution and another with saline in the second experiment. In an extinction test the sucrose stimulus augmented lever pressing relative to the saline stimulus when the animals were hungry, but not when they were thirsty. In the subsequent experiments the contribution of the Pavlovian process was equated by giving concurrent training with both incentives. Lever pressing and chain pulling were reinforced concurrently, one with the sucrose solution and the other with saline, while the animals were thirsty. Once again, the animals pressed more in extinction if this action had been trained with the sucrose solution rather than the saline, but only if they were hungry rather than thirsty. Thus, instrumental performance across a thirst-to-hunger shift can also be controlled by an instrumental incentive process. The direct engagement of the instrumental process by this motivational shift contrasts to the absence of such control following a hunger-to-thirst transition (Dickinson & Dawson, 1987a), a fact attributed to the asymmetrical motivational interactions produced by water and food deprivation.     

Incentive learning and the motivational control of instrumental performance

Hungry rats were trained to press a lever and pull a chain concurrently, with one action being reinforced with a sucrose solution and the other with food pellets. In addition, in the first two experiments all animals experienced non-contingent presentations of the two incentives in the absence of the operant manipulanda while either thirsty or hungry and either before (Experiment 1A) or after (Experiment 1B) the instrumental training. When lever pressing was assessed subsequently in extinction under thirst, the animals pressed at a relatively high rate only if (1) this action had been reinforced with the sucrose solution rather than the food pellets during training and (2) they had received the non-contingent presentations of the sucrose solution and food pellets on days on which they were thirsty rather than hungry. A third experiment demonstrated that non-contingent exposure to the sucrose solution alone, but not to water under thirst was sufficient to bring about this type of motivational control of instrumental performance.     

Motivational control of instrumental performance: The role of prior experience of the reinforcer

In four experiments we investigated the conditions that are necessary for instrumental performance to adjust appropriately to a change in drive state. Rats were trained to press a lever and pull a chain concurrently, with one action being reinforced by sucrose solution and the other by food pellets. In Experiment 1 for animals that were hungry throughout training the rate of lever pressing in an extinction test under thirst was unaffected by the type of reinforcer produced by this action during training, even though the sucrose solution would maintain a higher rate than the food pellets during training under thirst. In contrast, animals that experienced the instrumental contingencies arranged by the concurrent schedule while thirsty at some point during training pressed the lever more during the extinction test under thirst when this action had been reinforced with the sucrose solution rather than the food pellets. The remaining three experiments demonstrated that for this incentive effect to occur it is sufficient that the sucrose solution be delivered non-contingently under thirst at some stage of training. Thus, it would appear that performance mediated by an instrumental contingency adjusts appropriately to reinforcer revaluation brought about by a drive shift only if the animals have had prior experience with the incentive under the test drive state. This observation was interpreted in terms of Tolman's “cathexis” theory of motivation.     

The role of the instrumental contingency in the motivational control of performance

In four experiments we investigated an irrelevant incentive effect based upon a transition from hunger to thirst. Hungry rats were trained to lever press either for sucrose solution or for food pellets before performance was tested in extinction while they were thirsty. Reinforcer-specific motivational control was found in the first experiment in that the animals pressed the lever more on tests following training with the sucrose solution rather than with food pellets. Moreover, this effect was seen only when testing was conducted following water, but not following food deprivation. The outcome of the remaining experiments suggests that this motivational control is not mediated by the instrumental contingency between lever pressing and the sucrose reinforcer during training. In these studies lever pressing and chain pulling were reinforced concurrently, one with sucrose and the other with food pellets, in order to equate the noninstrumental functions of the incentives. Following this training, lever pressing in extinction under thirst was unaffected by the type of incentive used as its reinforcer during training.     

Instrumental responding following reinforcer devaluation

In two experiments, hungry rats were given instrumental lever-press training for an appetitive reinforcer and, in addition, were exposed to another type of food which was not contingent on lever pressing. In the first experiment, exposure to each type of food was on separate days, whereas in the second experiment rats were exposed to each type of food in strict alternation within each session. Subsequently, a food aversion was conditioned to the reinforcer for the experimental group and to the non-contingent food for the control group. In both experiments, animals with an aversion to the reinforcer responded less in an extinction test than animals with an aversion to the non-contingent food. Subsequent reacquisition tests confirmed that the aversion to the non-contingent food in the control group was of comparable strength with that to the reinforcer in the experimental group. The results were discussed in terms of whether the reinforcer is encoded in the associative structure set up by exposure to an instrumental contingency.     

Post-Conditioning Devaluation of an Instrumental Reinforcer has no Effect on Extinction Performance

The extent to which a representation of the reinforcer controls an instrumental response can be assessed by studying the effect of post-conditioning changes in the reinforcer value. In the first experiment rats were trained to press a lever for sucrose pellets on a variable-interval (VI) schedule. The sucrose was subsequently devalued by pairing with Lithium Chloride (LiCl). This had no effect on lever pressing in extinction, although it profoundly reduced reacquisition responding and consumption. In Experiment II rats were trained to shuttle between the two distinctive chambers of a choice-box, in which lever pressing was reinforced in one chamber by sucrose and in the other chamber by food pellets programmed on independent VI schedules. A LiCl-induced taste-aversion was conditioned to the sucrose, and although this markedly affected reacquisition, extinction responding in the sucrose chamber and chamber preference were unaffected. These results indicate that instrumental performance can be independent of the current value of the reinforcer, and are discussed with reference to stimulus-response theory and second-order Pavlovian conditioning.     

Pavlovian processes in the motivational control of instrumental performance

Hungry rats were rewarded for pressing a lever on a multiple schedule. During one component the reward was a sucrose solution, whereas food pellets acted as the reward during another component. Lever pressing was never rewarded during the third component. When the drive state was switched from hunger to thirst and the animals tested in extinction, they pressed more in the presence of the component stimulus that had been associated with the sucrose reward during training. A similar effect was observed during the extinction test of a second study in which the component stimuli had signalled non-contingent presentations of either the sucrose or pellet rewards in the absence of the lever. This suggests that the instrumental irrelevant incentive effect observed in the first experiment was due, at least in part, to the Pavlovian relationship between the component stimuli and the reinforcers during training. In fact, when the size of the effects controlled by purely Pavlovian and supposedly instrumental contingencies was compared directly in the final study, no difference could be detected.     

Contingency effects with maintained instrumental reinforcement

In three experiments we investigated the effect on the performance of thirsty rats of varying the instrumental contingency between lever pressing and the delivery of a saccharin reinforcer. In Experiment 1, the subjects performed more slowly in a non-contingent condition, in which the momentary probability of reinforcement was unaffected by whether or not the animals pressed, than in a contingent condition in which the reinforcer was never presented except following a lever press. This was true of performance under both random ratio and interval schedules in which the function determining the probability of reinforcement following a lever press remained the same across the contingent and non-contingent conditions. Experiment 2 demonstrated that instrumental performance was less affected when the contingency was degraded by the introduction of free reinforcers if these reinforcers were signalled. In Experiment 3, lever pressing was reinstated to some degree after non-contingent training by giving non-reinforced exposure to the operant chamber in the absence of the lever. These results suggest that free reinforcers depress instrumental behaviour through a performance mechanism engaged by their ability to support conditioning of the contextual cues.     

Free-operant acquisition with delayed reinforcement

The acquisition of free-operant lever pressing by hungry rats was investigated under a schedule in which the first lever press in each second programmed a reinforcer delivery after a fixed delay. In two studies acquisition was observed under programmed delays ranging between 0 and 32 sec, although animals trained with a delay of 64 sec pressed no more than yoked, non-contingent controls. In the final study an attempt was made to enhance sensitivity to the instrumental contingency under a 64-sec delay by exposing the animals to the operant chamber in the absence of the lever prior to each training session. Only animals receiving such exposure pressed significantly more than their non-contingent controls.     

Instrumental Outcome Devaluation Is Attenuated by the Anti-emetic Ondansetron

In three experiments we assessed the effect of an anti-emetic, the selective S-HT antagonist ondansetron, on (1) the conditioning of a taste aversion using lithium chloride (LiCl); (2) the expression of that aversion; and (3) instrumental outcome-devaluation effects. In Experiment 1 it was found that ondansetron reduced the aversion induced by LiCl when administered prior to the LiCl injection and also attenuated the expression of that aversion when administered prior to test sessions. In Experiments 2 and 3, thirsty rats were trained, in a single session, to lever press and chain pull for sucrose and saline solutions concurrently before being injected with LiCl. They were then re-exposed to both solutions, one after injection of vehicle and the other after injection of ondansetron. In a choice extinction test on the levers and chains, animals performed more of the action whose training outcome was re-exposed under ondansetron than the other action, whether the test was conducted after an injection of vehicle or after one of ondansetron.     

Mechanisms of resurgence II: Response-contingent reinforcers can reinstate a second extinguished behavior

Three experiments with rat subjects examined resurgence of an extinguished instrumental response using the procedure introduced by Epstein (1983) with pigeons. There were three phases: (1) initial acquisition of pressing on a lever (L1) for pellet reward, (2) extinction of L1, and (3) a test session in which a second lever (L2) was inserted, briefly reinforced, and then extinguished. Experiment 1 confirmed that if pressing L2 delivered 20 pellets followed by extinction, rats would resume L1 responding in the final test. Experiment 2 compared the effects of response-contingent and non-contingent rewards delivered upon insertion of L2. Although insertion of L2 alone did not increase L1 responding, response-contingent and non-contingent rewards led to comparable increases in L1 responding. Experiment 3 found that the delivery of non-contingent pellets during extinction of L1, which would be expected to reduce the ability of pellets to set the occasion for the L1 response, also reduced the effects of both response-contingent and non-contingent rewards during the final test. The results indicate that in this method, the resurgence treatment leads to an increase in L1 pressing due to simple presentation of the pellet; delivering the reinforcer after extinction of L1 reinstates L1 responding by setting the occasion for the L1 response.     

The role of response-contingent incentives in lithium chloride-mediated suppression of an operant response

Three experiments investigated what role a novel incentive plays in the development of operant response suppression mediated by lithium chloride. In all experiments animals were trained to press two levers under concurrent schedules of reinforcement. In Experiment 1 responding on one lever delivered a familiar incentive (food pellets), whereas responding on an alternative lever delivered a novel incentive (sucrose solution) prior to lithium chloride injections. If lithium was administered immediately after the instrumental session, the action associated with the novel, but not with the familiar, incentive was suppressed. By comparison, in a control group for which responding on both levers led to the familiar incentive, both actions were suppressed. Experiment 2 examined whether the novelty, rather than the sensory properties, of the incentive is crucial for observing performance suppression. It was found that animals familiarized with the “target” incentive were insensitive to aversion conditioning by lithium, in that there was no difference in response rates between the action that delivered the familiar incentive from that which earned the “target”. In contrast, if animals were unfamiliar with the “target” incentive at the time of aversion conditioning, they suppressed responding on the lever that was associated with the novel incentive but did not suppress responding on the lever associated with the familiar incentive. Experiment 3 investigated the mechanism underlying instrumental performance suppression. After the completion of concurrent lever press training, novel sucrose was introduced in conjunction with the pellets for responding on one lever; responding on the other lever continued to deliver only familiar pellets. Lithium injections were then administered either immediately following the sessions or several hours after the sessions. It was found that the rate of responding on the lever associated with the contingent delivery of sucrose was suppressed below that of the pellet-alone action. By comparison, if lithium injections were administered several hours following the session, an elevation in responding on the sucrose-plus-pellet lever was observed. The outcomes of all three experiments demonstrate not only that the novelty of an incentive is important in obtaining performance suppression, but also that a novel incentive can punish instrumental responding if it has been associated with toxicosis.     

Alcohol seeking by rats: Action or habit?

In two experiments, we examined the relative susceptibility to outcome devaluation of lever pressing by rats for either a 10% ethanol solution or food pellets. The rats were trained to press different levers for these two reinforcers using a sucrose-substitution procedure. An aversion was then conditioned from either the ethanol solution or the food pellets by pairing consumption with illness induced by lithium chloride. When instrumental performance was subsequently tested in extinction, the rats pressed less on the pellet lever if the pellets, rather than the ethanol, had been devalued by aversion conditioning. By contrast, performance on the ethanol lever was unaffected by whether the ethanol or pellets were devalued. Moreover, noncontingent presentations of the devalued reinforcer had no impact on test performance. The differential resistance to outcome devaluation suggests that, in contrast to food seeking, alcohol seeking is a stimulus-response habit rather than a goal-directed action mediated by a representation of the action-outcome contingency.     

Irrelevant incentive learning during training on ratio and interval schedules

Two experiments investigated the effect of a motivationally-induced change in the value of the training reinforcer on instrumental performance. Initially, thirsty rats were trained to lever press for either a sodium or a potassium solution. Responding in an extinction test was then measured following the induction of sodium appetite. In Experiment I sodium-trained rats responded faster in a test given one day following the end of instrumental training. Furthermore, the relative size of this irrelevant incentive effect did not depend upon whether a ratio or interval schedule was employed during training. Delaying the test for eight days following the end of training abolished the difference between the test performance of sodium- and potassium-trained animals. Experiment II provided a further study of the effect of the training schedule when the introduction of the sodium reinforcer was delayed until responding was well established. Again the relative size of the difference between the performance of sodium- and potassium-trained animals was comparable following training on ratio and interval schedules. The insensitivity of this irrelevant incentive effect to the training contingency is in marked contrast to previous failures to detect an effect of reinforcer revaluation brought about by aversion conditioning following training on an interval schedule (Dickinson, Nicholas and Adams, 1983).     

Discrimination between outcomes in instrumental learning: Effects of preexposure to the reinforcers

In two experiments rats received instrumental training with two response levers, one response being reinforced by sucrose solution and the other by sucrose pellets. Prior to a test session, on which both levers were made available in the absence of reinforcement, the rats were given free access to one of the reinforcers, a procedure known to reduce its value. It was found that the rats responded at a lower rate on the lever that had produced the now-devalued reinforcer, but that this effect was substantial only in rats that had received preexposure to the two reinforcers before instrumental training was begun (Experiment 1). Experiment 2 demonstrated that this effect was obtained only when presentations of the two reinforcers were presented according to an inter-mixed schedule during preexposure. It is suggested that this result constitutes an instance of the perceptual learning effect in which intermixed preexposure to similar events enhances their discriminability.     

Re-examination of the role of the instrumental contingency in the sodium-appetite irrelevant incentive effect

Thirsty rats were trained to perform one action—either lever pressing or chain pulling—for a sodium chloride solution and the other for water on a concurrent random interval schedule. When the experimental groups were tested in extinction under a sodium appetite, the instrumental action trained with the sodium solution was not performed at a significantly higher rate than that reinforced by water relative to control groups tested while thirsty. In a second experiment all animals were tested in extinction under a sodium appetite after training with either a sodium solution and water in the experimental condition or a potassium chloride solution and water in the control condition. The experimental animals chain-pulled more rapidly on test than the control rats, irrespective of whether this action was reinforced with the sodium solution or with water. These results confirm the findings of Dickinson and Nicholas (1983a) that the instrumental contingency plays little role in the irrelevant incentive effect based upon a sodium appetite.