Instrumental responding following reinforcer devaluation

In two experiments, hungry rats were given instrumental lever-press training for an appetitive reinforcer and, in addition, were exposed to another type of food which was not contingent on lever pressing. In the first experiment, exposure to each type of food was on separate days, whereas in the second experiment rats were exposed to each type of food in strict alternation within each session. Subsequently, a food aversion was conditioned to the reinforcer for the experimental group and to the non-contingent food for the control group. In both experiments, animals with an aversion to the reinforcer responded less in an extinction test than animals with an aversion to the non-contingent food. Subsequent reacquisition tests confirmed that the aversion to the non-contingent food in the control group was of comparable strength with that to the reinforcer in the experimental group. The results were discussed in terms of whether the reinforcer is encoded in the associative structure set up by exposure to an instrumental contingency.     

Signalling and incentive processes in instrumental reinforcer devaluation

We have previously reported that conditioning an aversion to the reinforcer using an isotonic lithium chloride (LiCl) solution following instrumental training reduces performance in a subsequent extinction test only if animals are re-exposed to the reinforcer prior to the test. Rescorla (1992), in contrast, reported an immediate devaluation effect using a hypertonic LiCl solution that did not depend upon re-exposure. In two experiments we examined the effect of using a hypertonic LiCl solution to condition the aversion to the reinforcer on subsequent instrumental performance in extinction, with and without re-exposure. In Experiment 1 thirsty rats were trained to press a lever for a sucrose solution before being injected with 0.6 M LiCl either immediately or after a delay. Half of the immediate and delay groups were then re-exposed to the sucrose in the absence of the lever, with the remainder being exposed to water. Contrary to the previously reported effects of isotonic LiCl, a hypertonic solution induced a reinforcer devaluation effect in all the immediately poisoned animals, which did not depend upon re-exposure to the reinforcer. In Experiment 2 the possibility that this devaluation effect was induced by the discomfort associated with the hypertonicity of the solution was assessed by replicating Experiment 1 but, in addition, using two immediately poisoned groups given the LiCl injection under anaesthesia. In the absence of anaesthesia, the devaluation effect observed without re-exposure to the reinforcer in Experiment 1 was replicated. When the injection was given under anaesthesia, however, a reinforcer devaluation effect was observed only in animals that were re-exposed to the reinforcer prior to the extinction test. These results were interpreted as evidence that a reinforcer devaluation effect induced by pairing the reinforcer with illness depends upon a process of incentive learning, whereas a devaluation effect mediated by learning a signalling relationship between the reinforcer and somatic discomfort does not.     

Amount of training and cue-evoked taste-reactivity responding in reinforcer devaluation

In two experiments, rats received minimal (16) pairings of one auditory conditioned stimulus (CS) cue with a sucrose reinforcer, and extensive (112) pairings of another auditory CS with that reinforcer. After sucrose was devalued by pairing it with lithium chloride in some rats (Devalue groups) but not others (Maintain groups), taste reactivity (TR) and other responses to unflavored water were assessed in the presence of the auditory CSs alone. The minimally trained CS controlled substantially more evaluative TR responses than the extensively trained CS. Those TR responses were hedonic (positive) in the Maintain groups, but aversive (negative) in the Devalue groups. By contrast, food cup entry and other responses thought not to reflect evaluative taste processing were controlled more by the extensively trained cue. These responses were reduced by sucrose devaluation comparably, regardless of the amount of training. The results suggest rapid changes in the content of learning as conditioning proceeds. Early in training, CSs may be capable of activating preevaluative processing of an absent food reinforcer that includes information about its palatability, but that capability is lost as training proceeds.     

Instrumental performance following reinforcer devaluation depends upon incentive learning

Conditioning an aversion to the reinforcer following instrumental training reduces performance in a subsequent extinction test. Three experiments examined whether this reinforcer-devaluation effect depends upon experience with the devalued reinforcer prior to the extinction test. In Experiments 1 and 2 thirsty rats were trained to press a lever for sucrose solution in a single session. All animals then received an injection of lithium chloride (LiCl) either immediately following the session or after a delay of 6 hr. On the next day either the sucrose solution or water was presented non-contingently either in the operant chamber without the lever present or in a separate drinking cage. In a subsequent extinction test only the animals that had received immediate LiCl and re-exposure to non-contingent sucrose pressed less than those in the delayed-LiCl control groups. Experiment 3 demonstrated that this difference depended, at least in part, on post-conditioning exposure to a contingent reinforcer. Lever pressing and chain pulling were reinforced concurrently with either a sucrose or a sodium chloride solution in a single session immediately before the administration of LiCl. All animals then received non-contingent presentations of one of the reinforcers in the absence of both manipulanda. Finally, performance of both actions was assessed in an extinction test. Re-exposure to a reinforcer produced a relative reduction in the performance of its associated action on test. These results are interpreted as evidence that the instrumental reinforcer devaluation effect depends upon a process of incentive learning.     

Effect of reinforcer devaluation on discriminative control of instrumental behavior

Two experiments examined the effect of reinforcer devaluation on the ability of a discriminative stimulus (Sd) to control instrumental behavior in Sprague-Dawley rats. In Experiment 1 reinforcer devaluation reduced, but did not eliminate, the ability of the Sd to control performance of the original response and to transfer its control to a new response trained with the same reinforcer. The effect of devaluation was more complete in Experiment 2, in which the reinforcer was delivered directly into the oral cavity. However, retraining the response with a different reinforcer partially restored the ability of the Sd to control performance of that response. These results suggest that an Sd may not augment its trained responses when the reinforcer has been completely devalued but may promote responses with which it shares a reinforcer, as long as those responses are associated with some reinforcer that retains its value. The implications of these results for the way that discriminative stimuli control instrumental behavior are discussed.     

Variations in the sensitivity of instrumental responding to reinforcer devaluation

In five experiments hungry rats were trained to make a lever press response for a sucrose reinforcer. That sucrose was subsequently devalued by conditioning a food-aversion to it, and the ability of the rats to integrate knowledge about the instrumental contingency with that gained from aversion training was assessed in an extinction test. Experiment I showed successful integration following limited but not extended instrumental training. Experiment II suggested that the crucial factor was the spacing of training; successful integration was seen after massed but not distributed training. The third experiment implicated distributed experience with the reinforcer, rather than distributed response practice, in failures of integration. Experiment IV showed that if the distribution of food-aversion learning was dissimilar to that of instrumental training then a failure of integration could result; this finding was able to account for the distribution of training effects seen in previous studies, but not the effect of extended training. Experiment V replicated the extended training effect seen in Experiment I, and provided evidence that this may reflect the degree of exposure to the reinforcer rather than the extent of response practice.     

Stress associated changes in Pavlovian-instrumental transfer in humans

Predictive learning is known to influence instrumental responding for reward. Cues associated with an instrumental outcome can influence performance in two ways: (a) by selectively promoting actions associated with the outcome predicted by the cue (specific transfer), and (b) by increasing motivation and the vigour of instrumental responding (general transfer). To examine these two distinct processes in humans we developed a novel behavioural task in which participants were able to liberate junk-food snacks from a virtual vending machine. Additionally, the relationship between stress and cue-driven reward seeking was examined using participant scores on the Depression Anxiety and Stress Scale (DASS). Reward-paired cues were found to separately bias action selection and influence the rate of responding for rewards. Furthermore, the effects of reward-paired cues on the rate of responding for reward was influenced by increased stress and anxiety. Increased levels of stress and anxiety were associated particularly with changes in cue-driven response vigour; whereas high levels of stress and anxiety were associated with elevated responding above baseline in the presence of a cue associated with a non-rewarding outcome, participants with low levels of anxiety and stress showed appropriate suppression of responding during this cue. These differences in performance between high and low anxiety and stress participants provides initial evidence that, as has been demonstrated in rodents, stress affects the influence of cue-driven response vigour in humans.     

The effect of the instrumental training contingency on susceptibility to reinforcer devaluation

Two experiments investigated performance of instrumental lever pressing by rats following post-conditioning devaluation of the sucrose reinforcer produced by establishing an aversion to it. In Experiment I rats responded less in an extinction test after being averted from the sucrose following training on a ratio schedule, but not following an equivalent amount of training on an interval schedule. This was true even though the devalued sucrose would not act as an effective reinforcer on either the ratio or interval schedule. Experiment II provided a further investigation of the insensitivity of interval responding to reinforcer devaluation by comparing test performance under simple extinction with responding when the devalued reinforcer was presented on either a response-contingent or non-contingent schedule during the test. Once again simple extinction performance was unaffected by prior reinforcer devaluation. Furthermore, neither non-contingent nor contingent presentations of the devalued reinforcer significantly depressed responding below the level seen in the extinction condition. Ratio, but not interval performance appears to be controlled by knowledge about the instrumental contingency that encodes specific properties of the training reinforcer.     

Localized disruption of Narp in medial prefrontal cortex blocks reinforcer devaluation performance

Neuronal activity regulated pentraxin (Narp) is a secreted protein that regulates α-amino-3-hydroxyl-5-methyl-4-isoxazole-propionate receptors (AMPAR) aggregation and synaptogenesis. Mapping of Narp-positive neurons in brain has revealed it is prominently expressed in several limbic system projection pathways. Consistent with this localization pattern, Narp knockout mice show deficits in using the current value of a reinforcer to guide behavior, a critical function of the limbic system. To help assess whether this behavioral deficit is due to impairment of synaptogenesis during development or in modulating synaptic signaling in the mature brain, we have used a dominant negative Narp viral construct which blocks trafficking of endogenous Narp to axons. Focal injection of this viral construct into the medial prefrontal cortex (mPFC) of adult mice, a region containing Narp-positive projection neurons, blocked reinforcer devaluation. Thus, these results indicate that Narp released from mPFC neurons plays a key role in mediating synaptic changes underlying instrumental reinforcer devaluation.     

Incentive learning following reinforcer devaluation is not conditional upon the motivational state during re-exposure

Three experiments analysed the effect of re-exposure to the reinforcer following aversion conditioning on instrumental performance. In the first experiment, groups of hungry and thirsty rats were trained to press a lever for sucrose, which was then followed by a single injection of lithium chloride (LiCl). On the following day, half the animals in each motivational condition received re-exposure to the sucrose solution; the remaining animals were not re-exposed. In a subsequent extinction test animals that had received re-exposure to the sucrose pressed less than animals that were not re-exposed. Moreover, the effect of re-exposure to the sucrose solution was similar following training under hunger and thirst. In the remaining studies, animals were trained to lever-press for sucrose while either hungry or thirsty. They were then injected with LiCl and re-exposed to the sucrose while either hungry or thirsty, i.e. in the same or different motivational state employed during training, or they were not re-exposed. Lever pressing was then tested in extinction in the training motivational state. As in the first experiment, re-exposure to the reinforcer after aversion conditioning enhanced the magnitude of the reinforcer devaluation effect. More importantly, re-exposure to the sucrose produced a comparable effect on instrumental performance, whether re-exposure was given under the same or different motivational state to that employed during training. These results suggest that the instrumental reinforcer devaluation effect depends upon a process of incentive learning, but that this process is not conditional upon the current motivational state of the animal.     

The effects of acute stress on Pavlovian-instrumental transfer in rats

Pavlovian stimuli invigorate ongoing instrumental action, a phenomenon termed the Pavlovian–instrumental transfer (PIT) effect. Acute stressors can markedly enhance the release of corticotropin-releasing factor (CRF), and CRF injection into the nucleus accumbens increases the PIT effect. However, it is unknown whether acute stressors by themselves would amplify the PIT effect. Here, we examined the effects of acute stressors on PIT. Rats first received Pavlovian and instrumental training, and then the impact of the Pavlovian stimuli on instrumental responding was analyzed in the subsequent PIT test. Acute stressors were applied prior to the PIT test. Because the effects of acute stressors critically depend on stressor type and time of day, we used two acute stressors that involved one or several distinct stressors (denoted here as “single” vs. “multiple” stressors) applied either in the light or the dark period of the light:dark cycle. The results revealed that single and multiple stressors applied in the light period did not alter the PIT effect—that is, the ability of an appetitive Pavlovian stimulus to enhance leverpressing—or the basal leverpress rate. When applied in the dark period, single and multiple stressors also did not alter the PIT effect, but they did markedly reduce the basal leverpress rate. Diazepam pretreatment did not counteract the declines in basal instrumental responding in the PIT test that were induced by either a single or multiple stressors. Our findings suggest that acute stressors were unable to amplify the incentive salience of reward-predictive Pavlovian stimuli to activate instrumental responding, but, depending on the time of day of stressor exposure, they did reduce basal instrumental responding.     

Dopamine D1 and D2 receptors in the nucleus accumbens core and shell mediate Pavlovian-instrumental transfer

Pavlovian stimuli previously paired with food can markedly elevate the rate of food-reinforced instrumental responding. This effect, termed Pavlovian-instrumental transfer (PIT), depends both on general activating and specific cueing properties of Pavlovian stimuli. Recent evidence suggests that the general activating properties of Pavlovian stimuli are mediated by mesoaccumbens dopamine systems; however, the role of NAC dopamine D1 and D2 receptors is still unknown. Here we examined the effects of a selective dopamine D1 and D2 receptor blockade in the shell and core subregion of the NAC on general PIT. Rats were trained to press a single lever for food, and the effect of a single Pavlovian stimulus previously associated with the same food on performance of that lever was measured in extinction. Results reveal that PIT, that is, the increase in instrumental responding during presentation of the Pavlovian stimulus, was reduced by microinjections of the dopamine D1 receptor antagonist SCH-23390 and, less pronounced, by microinjections of the dopamine D2 receptor antagonist raclopride into the NAC core or shell, respectively. Our data suggest that dopamine D1 and D2 receptors in the NAC core and shell mediate the general activating effects of Pavlovian stimuli on instrumental behavior.     

The interaction between stimulus and reinforcer control on remembering.

In a symbolic matching-to-sample task, 6 pigeons obtained food by pecking a red side key when the brighter of two white lights had been presented on the center key and by pecking a green side key when the dimmer of two white lights had been presented on the center key. Across Part 1 and Parts 6 to 10, the delay between sample-stimulus presentation and the availability of the choice keys was varied between 0 s and 25 s. Across Parts 1 to 5, the delay between the emission of a correct choice and the delivery of a reinforcer was varied between 0 s and 30 s. Although increasing both types of delay decreased stimulus discriminability, lengthening the stimulus-choice delay produced a greater decrement in choice accuracy than did lengthening the choice-reinforcer delay. Additionally, the relative reinforcer rate for correct choice was varied across both types of delay. The sensitivity of behavior to the distribution of reinforcers decreased as discriminability decreased under both procedures. These data are consistent with the view, based on the generalized matching law, that sample stimuli and reinforcers interact in their control over remembering.     

Every reinforcer counts: reinforcer magnitude and local preference

Six pigeons were trained on concurrent variable-interval schedules. Sessions consisted of seven components, each lasting 10 reinforcers, with the conditions of reinforcement differing between components. The component sequence was randomly selected without replacement. In Experiment 1, the concurrent-schedule reinforcer ratios in components were all equal to 1.0, but across components reinforcer-magnitude ratios varied from 1:7 through 7:1. Three different overall reinforcer rates were arranged across conditions. In Experiment 2, the reinforcer-rate ratios varied across components from 27:1 to 1:27, and the reinforcer-magnitude ratios for each alternative were changed across conditions from 1:7 to 7:1. The results of Experiment 1 replicated the results for changing reinforcer-rate ratios across components reported by Davison and Baum (2000, 2002): Sensitivity to reinforcer-magnitude ratios increased with increasing numbers of reinforcers in components. Sensitivity to magnitude ratio, however, fell short of sensitivity to reinforcer-rate ratio. The degree of carryover from component to component depended on the reinforcer rate. Larger reinforcers produced larger and longer postreinforcer preference pulses than did smaller reinforcers. Similar results were found in Experiment 2, except that sensitivity to reinforcer magnitude was considerably higher and was greater for magnitudes that differed more from one another. Visit durations following reinforcers measured either as number of responses emitted or time spent responding before a changeover were longer following larger than following smaller reinforcers, and were longer following sequences of same reinforcers than following other sequences. The results add to the growing body of research that informs model building at local levels.     

Concurrent schedules: Interaction of reinforcer frequency and reinforcer duration

Six pigeons were trained on concurrent variable-interval schedules with unequal reinforcer durations for the two responses. The schedules arranged on the two keys were kept equal while they were varied in absolute size. As the overall reinforcer rate was increased, both response-allocation and time-allocation measures of choice showed a trend toward indifference, and measures of sensitivity to reinforcer-duration ratios significantly decreased. Recent reports have shown that the generalized matching law cannot describe the changes in behavior allocation under constant delay-, duration-, or rate-ratios when changes are made in the absolute levels of each of these variables. The present results complement these findings by demonstrating that the concatenated generalized matching law cannot describe the interactions of two reinforcer variables on behavior allocation.     

Tests of transitivity in choices between fixed and variable reinforcer delays.

This experiment tested for transitivity in pigeons' choices between variable-time (VT) and fixed-time (FT) schedules. In a discrete-trials procedure, a subject chose between two alternatives by making a single key peck. Each choice was between a "standard alternative," which was the same schedule throughout a condition, and an "adjusting alternative," in which the delay to reinforcement was systematically increased or decreased many times a session. These adjustments enabled an approximate indifference point to be identified--the value of the adjusting delay at which the subject chose each alternative about equally often. Each test of transitivity involved four conditions. In one, the standard alternative was a variable-time schedule with a 2-s reinforcer, and the adjusting alternative also delivered a 2-s reinforcer. A second condition was similar except that the adjusting alternative delivered a 5-s reinforcer. The indifference point from each of these conditions was then converted to a fixed-time schedule for subsequent comparisons in the third and fourth conditions, respectively. Each of these last two conditions compared one of the fixed-time schedules (based upon the previous conditions and including their different reinforcer durations) with an adjusting schedule that delivered the alternative reinforcer duration, to determine whether the obtained indifference points would be those predicted from the prior alternative-duration comparisons with the VT schedule. There was little evidence for intransitivity of choice: Averaged across subjects and replications, the obtained indifference points deviated from perfect transitivity by less than 8%, and these deviations were not statistically significant. These results contrast with those of Navarick and Fantino (1972), who found frequent violations of transitivity between periodic and aperiodic schedules using a concurrent-chains procedure with variable-interval schedules in the initial links.     

Sensitization and habituation regulate reinforcer effectiveness

We argue that sensitization and habituation occur to the sensory properties of reinforcers when those reinforcers are presented repeatedly or for a prolonged time. Sensitization increases, and habituation decreases, the ability of a reinforcer to control behavior. Supporting this argument, the rate of operant responding changes systematically within experimental sessions even when the programmed rate of reinforcement is held constant across the session. These within-session changes in operant responding are produced by repeated delivery of the reinforcer, and their empirical characteristics correspond to the characteristics of behavior undergoing sensitization and habituation. Two characteristics of habituation (dishabituation, stimulus specificity) are particularly useful in separating habituation from alternative explanations. Arguing that habituation occurs to reinforcers expands the domain of habituation. The argument implies that habituation occurs to biologically important, not just to neutral, stimuli. The argument also implies that habituation may be observed in “voluntary” (operant), not just in reflexive, behavior. Expanding the domain of habituation has important implications for understanding operant and classical conditioning. Habituation may also contribute to the regulation of motivated behaviors. Habituation provides a more accurate and a less cumbersome explanation for motivated behaviors than homeostasis. Habituation also has some surprising, and easily testable, implications for the control of motivated behaviors.     

Effects of response type on pigeons' sensitivity to variation in reinforcer amount and reinforcer delay

Twelve pigeons, divided into two groups, responded on concurrent nonindependent variable-interval schedules to obtain access to grain by either pecking keys or pressing treadles. Either the amount of grain or the delay to the receipt of grain was varied in separate conditions to determine the sensitivity of relative responding to variation in reinforcer amount (s_A), the sensitivity to variation in reinforcer delay (s_D), and s_A/s_D, a measure related to self-control. There were no significant differences between the two groups in the values of s_A, s_D, and s_A/s_D. These results suggest that the values of s_A, s_D, and s_A/s_D for pigeons may be similar across these two types of responses.