Signalling and incentive processes in instrumental reinforcer devaluation

We have previously reported that conditioning an aversion to the reinforcer using an isotonic lithium chloride (LiCl) solution following instrumental training reduces performance in a subsequent extinction test only if animals are re-exposed to the reinforcer prior to the test. Rescorla (1992), in contrast, reported an immediate devaluation effect using a hypertonic LiCl solution that did not depend upon re-exposure. In two experiments we examined the effect of using a hypertonic LiCl solution to condition the aversion to the reinforcer on subsequent instrumental performance in extinction, with and without re-exposure. In Experiment 1 thirsty rats were trained to press a lever for a sucrose solution before being injected with 0.6 M LiCl either immediately or after a delay. Half of the immediate and delay groups were then re-exposed to the sucrose in the absence of the lever, with the remainder being exposed to water. Contrary to the previously reported effects of isotonic LiCl, a hypertonic solution induced a reinforcer devaluation effect in all the immediately poisoned animals, which did not depend upon re-exposure to the reinforcer. In Experiment 2 the possibility that this devaluation effect was induced by the discomfort associated with the hypertonicity of the solution was assessed by replicating Experiment 1 but, in addition, using two immediately poisoned groups given the LiCl injection under anaesthesia. In the absence of anaesthesia, the devaluation effect observed without re-exposure to the reinforcer in Experiment 1 was replicated. When the injection was given under anaesthesia, however, a reinforcer devaluation effect was observed only in animals that were re-exposed to the reinforcer prior to the extinction test. These results were interpreted as evidence that a reinforcer devaluation effect induced by pairing the reinforcer with illness depends upon a process of incentive learning, whereas a devaluation effect mediated by learning a signalling relationship between the reinforcer and somatic discomfort does not.     

Instrumental performance following reinforcer devaluation depends upon incentive learning

Conditioning an aversion to the reinforcer following instrumental training reduces performance in a subsequent extinction test. Three experiments examined whether this reinforcer-devaluation effect depends upon experience with the devalued reinforcer prior to the extinction test. In Experiments 1 and 2 thirsty rats were trained to press a lever for sucrose solution in a single session. All animals then received an injection of lithium chloride (LiCl) either immediately following the session or after a delay of 6 hr. On the next day either the sucrose solution or water was presented non-contingently either in the operant chamber without the lever present or in a separate drinking cage. In a subsequent extinction test only the animals that had received immediate LiCl and re-exposure to non-contingent sucrose pressed less than those in the delayed-LiCl control groups. Experiment 3 demonstrated that this difference depended, at least in part, on post-conditioning exposure to a contingent reinforcer. Lever pressing and chain pulling were reinforced concurrently with either a sucrose or a sodium chloride solution in a single session immediately before the administration of LiCl. All animals then received non-contingent presentations of one of the reinforcers in the absence of both manipulanda. Finally, performance of both actions was assessed in an extinction test. Re-exposure to a reinforcer produced a relative reduction in the performance of its associated action on test. These results are interpreted as evidence that the instrumental reinforcer devaluation effect depends upon a process of incentive learning.     

Motivational control of instrumental performance following a shift from thirst to hunger

Thirsty rats were trained to press a lever for either a sucrose solution or saline before performance was tested in extinction while the animals were either hungry alone or experiencing both hunger and a sodium appetite. Reinforcer-specific motivational control was observed in that the animals trained with the sucrose solution pressed more than those trained with the saline when they were tested hungry, but not when they were tested under combined hunger and sodium appetite. In order to assess the role of a Pavlovian incentive process in this effect, thirsty animals received non-contingent pairings of one stimulus with the sucrose solution and another with saline in the second experiment. In an extinction test the sucrose stimulus augmented lever pressing relative to the saline stimulus when the animals were hungry, but not when they were thirsty. In the subsequent experiments the contribution of the Pavlovian process was equated by giving concurrent training with both incentives. Lever pressing and chain pulling were reinforced concurrently, one with the sucrose solution and the other with saline, while the animals were thirsty. Once again, the animals pressed more in extinction if this action had been trained with the sucrose solution rather than the saline, but only if they were hungry rather than thirsty. Thus, instrumental performance across a thirst-to-hunger shift can also be controlled by an instrumental incentive process. The direct engagement of the instrumental process by this motivational shift contrasts to the absence of such control following a hunger-to-thirst transition (Dickinson & Dawson, 1987a), a fact attributed to the asymmetrical motivational interactions produced by water and food deprivation.     

Instrumental Outcome Devaluation Is Attenuated by the Anti-emetic Ondansetron

In three experiments we assessed the effect of an anti-emetic, the selective S-HT antagonist ondansetron, on (1) the conditioning of a taste aversion using lithium chloride (LiCl); (2) the expression of that aversion; and (3) instrumental outcome-devaluation effects. In Experiment 1 it was found that ondansetron reduced the aversion induced by LiCl when administered prior to the LiCl injection and also attenuated the expression of that aversion when administered prior to test sessions. In Experiments 2 and 3, thirsty rats were trained, in a single session, to lever press and chain pull for sucrose and saline solutions concurrently before being injected with LiCl. They were then re-exposed to both solutions, one after injection of vehicle and the other after injection of ondansetron. In a choice extinction test on the levers and chains, animals performed more of the action whose training outcome was re-exposed under ondansetron than the other action, whether the test was conducted after an injection of vehicle or after one of ondansetron.     

The role of the instrumental contingency in the motivational control of performance

In four experiments we investigated an irrelevant incentive effect based upon a transition from hunger to thirst. Hungry rats were trained to lever press either for sucrose solution or for food pellets before performance was tested in extinction while they were thirsty. Reinforcer-specific motivational control was found in the first experiment in that the animals pressed the lever more on tests following training with the sucrose solution rather than with food pellets. Moreover, this effect was seen only when testing was conducted following water, but not following food deprivation. The outcome of the remaining experiments suggests that this motivational control is not mediated by the instrumental contingency between lever pressing and the sucrose reinforcer during training. In these studies lever pressing and chain pulling were reinforced concurrently, one with sucrose and the other with food pellets, in order to equate the noninstrumental functions of the incentives. Following this training, lever pressing in extinction under thirst was unaffected by the type of incentive used as its reinforcer during training.     

Motivational control of instrumental performance: The role of prior experience of the reinforcer

In four experiments we investigated the conditions that are necessary for instrumental performance to adjust appropriately to a change in drive state. Rats were trained to press a lever and pull a chain concurrently, with one action being reinforced by sucrose solution and the other by food pellets. In Experiment 1 for animals that were hungry throughout training the rate of lever pressing in an extinction test under thirst was unaffected by the type of reinforcer produced by this action during training, even though the sucrose solution would maintain a higher rate than the food pellets during training under thirst. In contrast, animals that experienced the instrumental contingencies arranged by the concurrent schedule while thirsty at some point during training pressed the lever more during the extinction test under thirst when this action had been reinforced with the sucrose solution rather than the food pellets. The remaining three experiments demonstrated that for this incentive effect to occur it is sufficient that the sucrose solution be delivered non-contingently under thirst at some stage of training. Thus, it would appear that performance mediated by an instrumental contingency adjusts appropriately to reinforcer revaluation brought about by a drive shift only if the animals have had prior experience with the incentive under the test drive state. This observation was interpreted in terms of Tolman's “cathexis” theory of motivation.     

Variations in the sensitivity of instrumental responding to reinforcer devaluation

In five experiments hungry rats were trained to make a lever press response for a sucrose reinforcer. That sucrose was subsequently devalued by conditioning a food-aversion to it, and the ability of the rats to integrate knowledge about the instrumental contingency with that gained from aversion training was assessed in an extinction test. Experiment I showed successful integration following limited but not extended instrumental training. Experiment II suggested that the crucial factor was the spacing of training; successful integration was seen after massed but not distributed training. The third experiment implicated distributed experience with the reinforcer, rather than distributed response practice, in failures of integration. Experiment IV showed that if the distribution of food-aversion learning was dissimilar to that of instrumental training then a failure of integration could result; this finding was able to account for the distribution of training effects seen in previous studies, but not the effect of extended training. Experiment V replicated the extended training effect seen in Experiment I, and provided evidence that this may reflect the degree of exposure to the reinforcer rather than the extent of response practice.     

The effect of the instrumental training contingency on susceptibility to reinforcer devaluation

Two experiments investigated performance of instrumental lever pressing by rats following post-conditioning devaluation of the sucrose reinforcer produced by establishing an aversion to it. In Experiment I rats responded less in an extinction test after being averted from the sucrose following training on a ratio schedule, but not following an equivalent amount of training on an interval schedule. This was true even though the devalued sucrose would not act as an effective reinforcer on either the ratio or interval schedule. Experiment II provided a further investigation of the insensitivity of interval responding to reinforcer devaluation by comparing test performance under simple extinction with responding when the devalued reinforcer was presented on either a response-contingent or non-contingent schedule during the test. Once again simple extinction performance was unaffected by prior reinforcer devaluation. Furthermore, neither non-contingent nor contingent presentations of the devalued reinforcer significantly depressed responding below the level seen in the extinction condition. Ratio, but not interval performance appears to be controlled by knowledge about the instrumental contingency that encodes specific properties of the training reinforcer.     

Irrelevant incentive learning during training on ratio and interval schedules

Two experiments investigated the effect of a motivationally-induced change in the value of the training reinforcer on instrumental performance. Initially, thirsty rats were trained to lever press for either a sodium or a potassium solution. Responding in an extinction test was then measured following the induction of sodium appetite. In Experiment I sodium-trained rats responded faster in a test given one day following the end of instrumental training. Furthermore, the relative size of this irrelevant incentive effect did not depend upon whether a ratio or interval schedule was employed during training. Delaying the test for eight days following the end of training abolished the difference between the test performance of sodium- and potassium-trained animals. Experiment II provided a further study of the effect of the training schedule when the introduction of the sodium reinforcer was delayed until responding was well established. Again the relative size of the difference between the performance of sodium- and potassium-trained animals was comparable following training on ratio and interval schedules. The insensitivity of this irrelevant incentive effect to the training contingency is in marked contrast to previous failures to detect an effect of reinforcer revaluation brought about by aversion conditioning following training on an interval schedule (Dickinson, Nicholas and Adams, 1983).     

Incentive learning and the motivational control of instrumental performance

Hungry rats were trained to press a lever and pull a chain concurrently, with one action being reinforced with a sucrose solution and the other with food pellets. In addition, in the first two experiments all animals experienced non-contingent presentations of the two incentives in the absence of the operant manipulanda while either thirsty or hungry and either before (Experiment 1A) or after (Experiment 1B) the instrumental training. When lever pressing was assessed subsequently in extinction under thirst, the animals pressed at a relatively high rate only if (1) this action had been reinforced with the sucrose solution rather than the food pellets during training and (2) they had received the non-contingent presentations of the sucrose solution and food pellets on days on which they were thirsty rather than hungry. A third experiment demonstrated that non-contingent exposure to the sucrose solution alone, but not to water under thirst was sufficient to bring about this type of motivational control of instrumental performance.     

Learned preference for a hedonically negative flavor is observed after pairings with positive post-ingestion consequences rather than with a palatable flavor

In two experiments, thirsty rats consumed a compound of sucrose and a non-preferred flavor. In Experiment 1, a conditioned preference was observed in the experimental group when animals were tested both thirsty and hungry, but not when they were tested just thirsty. Animals in the control group, which experienced the flavor and the sucrose unpaired, never showed a preference. Experiment 2 replicated the absence of a preference in the experimental group when rats were tested thirsty, but provided evidence that a flavor-taste association had been formed during training. After conditioning, sucrose was paired with LiCl in group Dev whereas it was unpaired in group NonDev. The sucrose devaluation produced a decrease in CS preference in group Dev, and an increment in group NonDev. Taken together, these results show that preference for a non-preferred flavor can be readily observed after pairings with the positive consequences of the US (calories or absence of an expected illness) rather than with a palatable flavor.     

Bidirectional Instrumental Conditioning

Three experiments examined bidirectional instrumental conditioning by training hungry rats to push a pole in one direction for food pellets and in the other for either a sugar or a starch solution. In the first study we examined whether the animals learned about the actionreinforcer relations using a specific satiety procedure. Prefeeding one type of reinforcer before an extinction test selectively depressed the performance of the action that had been paired with this reinforcer during training. The second experiment investigated the sensitivity of the bidirectional actions to variations in the action-reinforcer contingencies. When the instrumental contingency was degraded by presenting unpaired reinforcers, the animals pushed less in the direction that was paired with the reinforcer type that was the same as the non-contiguous one. A third study revealed that increasing the rate of reinforcement for one action enhanced its rate of performance without significantly affecting the performance of the other action. We conclude that the effects of reinforcer devaluation, the action-outcome contingency, and the rate of reinforcement are not mediated by Pavlovian associations between the manipulandum and the reinforcer.     

Irrelevant incentive learning during instrumental conditioning: The role of the drive-reinforcer and response-reinforcer relationships

Four experiments investigated the processes by which a motivationally-induced change in the value of the training reinforcer affects instrumental performance. Initially, thirsty rats were trained to lever press for either a sodium or non-sodium solution. In Experiment I sodium-trained rats responded faster in extinction following the induction of a sodium appetite, but not following either food or water deprivation. Thus, enhanced extinction performance depends upon the relevance of the training reinforcer to the test drive state. The remaining experiments examined the role of the instrumental contingency. Animals received response-contingent presentations of one solution alternated either within (Experiments II and III) or between sessions (Experiment IV) with non-contingent presentations of another solution. Neither procedure yielded convincing evidence that contingent sodium presentations generated more responding in extinction under a sodium appetite than did non-contingent sodium presentations. On the basis of these results, we argue that the instrumental contingency itself does not play a major role in this irrelevant incentive effect.     

Post-Conditioning Devaluation of an Instrumental Reinforcer has no Effect on Extinction Performance

The extent to which a representation of the reinforcer controls an instrumental response can be assessed by studying the effect of post-conditioning changes in the reinforcer value. In the first experiment rats were trained to press a lever for sucrose pellets on a variable-interval (VI) schedule. The sucrose was subsequently devalued by pairing with Lithium Chloride (LiCl). This had no effect on lever pressing in extinction, although it profoundly reduced reacquisition responding and consumption. In Experiment II rats were trained to shuttle between the two distinctive chambers of a choice-box, in which lever pressing was reinforced in one chamber by sucrose and in the other chamber by food pellets programmed on independent VI schedules. A LiCl-induced taste-aversion was conditioned to the sucrose, and although this markedly affected reacquisition, extinction responding in the sucrose chamber and chamber preference were unaffected. These results indicate that instrumental performance can be independent of the current value of the reinforcer, and are discussed with reference to stimulus-response theory and second-order Pavlovian conditioning.     

Extinction of conditioned flavor preferences

In a series of 4 experiments, the effects of extinction on flavor preferences conditioned by mixing flavor cues with a nutrient were examined. In each experiment it was observed that rats preferred a flavor cue that had not undergone extinction to one that had. In addition, this preference was reversed in subjects trained thirsty (Experiments 1 and 2) if the associated nutrient had been devalued prior to the test or the preference for the nonextinguished cue was attenuated by nutrient devaluation in subjects trained hungry (Experiments 3 and 4). The results suggest that extinction may weaken associations between the flavor and the specific sensory properties of the nutrient and, for subjects trained hungry, between the flavor and the motivational components of the nutrient as well.     

Instrumental performance following a shift in primary motivation depends on incentive learning.

In 5 experiments the role of incentive learning in instrumental performance following a shift in primary motivation was examined. In Experiments 1 and 2 rats trained to perform an instrumental action reinforced by either pellets or maltodextrin when in a low-deprivation state were shifted to a high-deprivation state and tested in extinction. This shift in deprivation increased performance only if the animals had been exposed to the reinforcer in the high-deprivation state prior to instrumental training. Experiments 3, 4, and 5 examined the reverse shift training in a high-deprivation state and testing in a low-deprivation state, and found, similarly, that performance was only sensitive to this shift if animals were previously exposed to the reinforcer while in the low-deprivation state. These experiments support the conclusion that instrumental performance following revaluation of the reinforcer depends on a process of incentive learning.     

Instrumental responding following reinforcer devaluation

In two experiments, hungry rats were given instrumental lever-press training for an appetitive reinforcer and, in addition, were exposed to another type of food which was not contingent on lever pressing. In the first experiment, exposure to each type of food was on separate days, whereas in the second experiment rats were exposed to each type of food in strict alternation within each session. Subsequently, a food aversion was conditioned to the reinforcer for the experimental group and to the non-contingent food for the control group. In both experiments, animals with an aversion to the reinforcer responded less in an extinction test than animals with an aversion to the non-contingent food. Subsequent reacquisition tests confirmed that the aversion to the non-contingent food in the control group was of comparable strength with that to the reinforcer in the experimental group. The results were discussed in terms of whether the reinforcer is encoded in the associative structure set up by exposure to an instrumental contingency.     

Alcohol seeking by rats: Action or habit?

In two experiments, we examined the relative susceptibility to outcome devaluation of lever pressing by rats for either a 10% ethanol solution or food pellets. The rats were trained to press different levers for these two reinforcers using a sucrose-substitution procedure. An aversion was then conditioned from either the ethanol solution or the food pellets by pairing consumption with illness induced by lithium chloride. When instrumental performance was subsequently tested in extinction, the rats pressed less on the pellet lever if the pellets, rather than the ethanol, had been devalued by aversion conditioning. By contrast, performance on the ethanol lever was unaffected by whether the ethanol or pellets were devalued. Moreover, noncontingent presentations of the devalued reinforcer had no impact on test performance. The differential resistance to outcome devaluation suggests that, in contrast to food seeking, alcohol seeking is a stimulus-response habit rather than a goal-directed action mediated by a representation of the action-outcome contingency.     

Acquisition of instrumental conditioned reinforcement is resistant to the devaluation of the unconditioned stimulus

The associative mechanisms responsible for the efficacy of Pavlovian stimuli during first- and second-order conditioning have been extensively studied, but little is known about the representations underlying instrumental conditioned reinforcement. The present study investigated the associative structure underlying conditioned reinforcement, by employing an unconditioned stimulus (US) devaluation procedure on a commonly used instrumental task: the acquisition of a new response with conditioned reinforcement. Whilst US-directed behaviour was abolished following devaluation, the conditioned stimulus acting as a conditioned reinforcer supported the acquisition of instrumental responding. In this preparation then, the conditioned reinforcer appears to be impervious to devaluation of its associated US, suggesting that the underlying representation maintaining behaviour is independent of the current value of the US and may reflect the activation of a central appetitive motivational state.     

Resurgence of goal-directed actions and habits

This study investigated how goal-directed and habitual behaviors recover after extinction within the context of the resurgence effect, a form of relapse induced by the removal or worsening of alternative reinforcement. Rats were trained to press a target lever with one reinforcer (O1) for either minimal (4) or extended (16) sessions. An extinction test after the completion of O1 devaluation confirmed that minimal and extended training formed goal-directed and habitual behaviors, respectively. Then, pressing an alternative lever was reinforced with a second reinforcer (O2) while the target response was placed on extinction. When O2 was discontinued, the minimally trained target response resurged with goal-directed status as in the extinction test. However, the extinguished habitual behavior in the extensively trained rats did not recover as a habit but instead with goal-directed status, possibly due to the context specificity of habits or the introduction of a new response–reinforcer contingency. The critical finding that reinforcer devaluation consistently led to less resurgence regardless of the amount of acquisition training provides a clinical implication that coupling differential-reinforcement-of-alternative-behavior (DRA) treatments with the devaluation of the associated reinforcer of problematic behavior could effectively diminish its recurrence.