Instrumental learning in hyperdopaminergic mice

In two experiments we investigated the effects of elevated dopaminergic tone on instrumental learning and performance using dopamine transporter knockdown (DAT KD) mice. In Experiment 1, we showed that both DAT KD mice and wild-type controls were similarly sensitive to outcome devaluation induced by sensory specific satiety, indicating normal action-outcome learning in both groups. In Experiment 2, we used a Pavlovian-to-instrumental transfer procedure to assess the potentiation of instrumental responding by Pavlovian conditional stimuli (CS). Although during the Pavlovian training phase the DAT KD mice entered the food magazine more frequently in the absence of the CS, when tested later both groups showed outcome-selective PIT. These results suggest that the elevated dopaminergic tone reduced the selectivity of stimulus control over conditioned behavior, but did not affect instrumental learning.     

Signalling and incentive processes in instrumental reinforcer devaluation

We have previously reported that conditioning an aversion to the reinforcer using an isotonic lithium chloride (LiCl) solution following instrumental training reduces performance in a subsequent extinction test only if animals are re-exposed to the reinforcer prior to the test. Rescorla (1992), in contrast, reported an immediate devaluation effect using a hypertonic LiCl solution that did not depend upon re-exposure. In two experiments we examined the effect of using a hypertonic LiCl solution to condition the aversion to the reinforcer on subsequent instrumental performance in extinction, with and without re-exposure. In Experiment 1 thirsty rats were trained to press a lever for a sucrose solution before being injected with 0.6 M LiCl either immediately or after a delay. Half of the immediate and delay groups were then re-exposed to the sucrose in the absence of the lever, with the remainder being exposed to water. Contrary to the previously reported effects of isotonic LiCl, a hypertonic solution induced a reinforcer devaluation effect in all the immediately poisoned animals, which did not depend upon re-exposure to the reinforcer. In Experiment 2 the possibility that this devaluation effect was induced by the discomfort associated with the hypertonicity of the solution was assessed by replicating Experiment 1 but, in addition, using two immediately poisoned groups given the LiCl injection under anaesthesia. In the absence of anaesthesia, the devaluation effect observed without re-exposure to the reinforcer in Experiment 1 was replicated. When the injection was given under anaesthesia, however, a reinforcer devaluation effect was observed only in animals that were re-exposed to the reinforcer prior to the extinction test. These results were interpreted as evidence that a reinforcer devaluation effect induced by pairing the reinforcer with illness depends upon a process of incentive learning, whereas a devaluation effect mediated by learning a signalling relationship between the reinforcer and somatic discomfort does not.     

Negative emotional appraisal selectively disrupts retrieval of expected outcome values required for goal-directed instrumental choice*

Stress induction reduces people's ability to modify their instrumental choices following changes in the value of outcomes, but the mechanisms underpinning this effect have not been specified because previous studies have lacked crucial control conditions. To address this, the current study had participants learn two instrumental responses for food and water, respectively, before water was devalued by specific satiety. Choice between these two responses was then measured in extinction, reacquisition and Pavlovian to instrumental transfer (PIT) tests. Concurrently during these tests, a negative emotional appraisal group evaluated aversive images (stress induction), whereas a control group evaluated neutral images, at the same time as choosing between the two instrumental responses. Negative emotional appraisal abolished the impact of water devaluation on instrumental choice in the extinction test, but did not affect instrumental choice in the reacquisition or PIT tests. These findings suggest that negative emotional appraisal selectively impaired participants’ ability to retrieve the expected value of outcomes required to make goal-directed instrumental choices in the extinction test, and that this effect was not due to task disengagement, nullification of the devaluation treatment or impaired knowledge of response-outcome relationships.     

Effect of reinforcer devaluation on discriminative control of instrumental behavior

Two experiments examined the effect of reinforcer devaluation on the ability of a discriminative stimulus (Sd) to control instrumental behavior in Sprague-Dawley rats. In Experiment 1 reinforcer devaluation reduced, but did not eliminate, the ability of the Sd to control performance of the original response and to transfer its control to a new response trained with the same reinforcer. The effect of devaluation was more complete in Experiment 2, in which the reinforcer was delivered directly into the oral cavity. However, retraining the response with a different reinforcer partially restored the ability of the Sd to control performance of that response. These results suggest that an Sd may not augment its trained responses when the reinforcer has been completely devalued but may promote responses with which it shares a reinforcer, as long as those responses are associated with some reinforcer that retains its value. The implications of these results for the way that discriminative stimuli control instrumental behavior are discussed.     

Improving conditional discrimination learning and memory in five-year-old children: Differential outcomes effect using different types of reinforcement

Previous studies have demonstrated that discriminative learning is facilitated when a particular outcome is associated with each relation to be learned. When this training procedure is applied (the differential outcomes procedure; DOP), learning is faster and better than when the typical common outcomes procedure or nondifferential outcomes (NDO) is used. Our primary purpose in the two experiments reported here was to assess the potential advantage of DOP in 5-year-old children using three different strategies of reinforcement in which (a) children received a reinforcer following a correct choice (“ + ”), (b) children lost a reinforcer following an incorrect choice (“ ? ”), or (c) children received a reinforcer following a correct choice and lost one following an incorrect choice (“ + / ? ”). In Experiment 1, we evaluated the effects of the presence of DOP and different types of reinforcement on learning and memory of a symbolic delayed matching-to-sample task using secondary and primary reinforcers. Experiment 2 was similar to the previous one except that only primary reinforcers were used. The results from these experiments indicated that, in general, children learned the task faster and showed higher performance and persistence of learning whenever differential outcomes were arranged independent of whether it was differential gain, loss, or combinations. A novel finding was that they performed the task better when they lost a reinforcer following an incorrect choice (type of training “ ? ”) in both experiments. A further novel finding was that the advantage of the DOP over the nondifferential outcomes training increased in a retention test.     

The effect of the instrumental training contingency on susceptibility to reinforcer devaluation

Two experiments investigated performance of instrumental lever pressing by rats following post-conditioning devaluation of the sucrose reinforcer produced by establishing an aversion to it. In Experiment I rats responded less in an extinction test after being averted from the sucrose following training on a ratio schedule, but not following an equivalent amount of training on an interval schedule. This was true even though the devalued sucrose would not act as an effective reinforcer on either the ratio or interval schedule. Experiment II provided a further investigation of the insensitivity of interval responding to reinforcer devaluation by comparing test performance under simple extinction with responding when the devalued reinforcer was presented on either a response-contingent or non-contingent schedule during the test. Once again simple extinction performance was unaffected by prior reinforcer devaluation. Furthermore, neither non-contingent nor contingent presentations of the devalued reinforcer significantly depressed responding below the level seen in the extinction condition. Ratio, but not interval performance appears to be controlled by knowledge about the instrumental contingency that encodes specific properties of the training reinforcer.     

Toxicosis affects instrumental behaviour in rats

Three experiments examined the effect of toxicosis on instrumental responding. These studies were prompted by Morrison and Collyer's (1974, Experiment 1) finding that the induction of toxicosis after an instrumental conditioning session produces greater response suppression if the response is reinforced by a novel saccharin solution rather than familiar water during conditioning. Experiments 1 and 2 investigated whether this suppression was mediated by the Pavlovian contingency between the contextual cues and the saccharin solution or the instrumental relationship between the response and the reward. A role for the instrumental contingency was indicated by the greater suppression of the response producing novel saccharin rather than water when the context of both responses was equally associated with the saccharin and illness. Experiment 3 found that extinction of the aversion to a novel reinforcer following aversive conditioning would re-establish an action previously associated with that reinforcer, in contrast to an action whose reinforcer remained aversive. This result was a further indication that the instrumental contingency between the response and reward contributes to response suppression.     

Instrumental performance following reinforcer devaluation depends upon incentive learning

Conditioning an aversion to the reinforcer following instrumental training reduces performance in a subsequent extinction test. Three experiments examined whether this reinforcer-devaluation effect depends upon experience with the devalued reinforcer prior to the extinction test. In Experiments 1 and 2 thirsty rats were trained to press a lever for sucrose solution in a single session. All animals then received an injection of lithium chloride (LiCl) either immediately following the session or after a delay of 6 hr. On the next day either the sucrose solution or water was presented non-contingently either in the operant chamber without the lever present or in a separate drinking cage. In a subsequent extinction test only the animals that had received immediate LiCl and re-exposure to non-contingent sucrose pressed less than those in the delayed-LiCl control groups. Experiment 3 demonstrated that this difference depended, at least in part, on post-conditioning exposure to a contingent reinforcer. Lever pressing and chain pulling were reinforced concurrently with either a sucrose or a sodium chloride solution in a single session immediately before the administration of LiCl. All animals then received non-contingent presentations of one of the reinforcers in the absence of both manipulanda. Finally, performance of both actions was assessed in an extinction test. Re-exposure to a reinforcer produced a relative reduction in the performance of its associated action on test. These results are interpreted as evidence that the instrumental reinforcer devaluation effect depends upon a process of incentive learning.     

Associative and Normative Accounts of Negative Transfer

Subjects were given a prediction task in which they had to learn that one cue, P (positive), was followed by the outcome, and a compound of P and another cue, N (negative), was not followed by the outcome. Next, N was tested in compound with a transfer cue, T, which had signalled the outcome but had never been compounded with N. Experiment 1 confirmed an important assumption of the Rescorla-Wagner model (Wagner & Rescorla, 1972) that negation of T should depend on the specific P cue compound with N being positively contingent. Experiments 2 and 3 confirmed the model's prediction that no decrement in negative transfer should be observed following postlearning devaluation of P. Unfortunately, the model did not anticipate that a large proportion of devaluation trials relative to learning trials would attenuate negative transfer (Experiment 4), nor did it predict that negative transfer would occur when P was neutral during the learning stage and was only later made positive (Experiment 3). The results can be accommodated by the Rescorla-Wagner model if one assumes that absent cues have their associative strengths altered.     

Reinforcer devaluation in palatability-based learned flavor preferences

Rats exposed to simultaneous compounds of 1 neutral flavor with dilute (2%) sucrose and a 2nd flavor with dilute (2%) maltodextrin subsequently consumed both flavors in preference to a 3rd flavor that was never paired with a palatable taste. Brief training exposure under ad lib food and water minimized the post-ingestive effects of nutrients, emphasizing the contribution of palatability to these preferences. Devaluation of sucrose or maltodextrin by pairing with illness (Experiment 1) or sensory-specific satiety (Experiment 2) selectively reduced the preference for the flavor previously paired with the devalued reinforcer. Such reinforcer-specific devaluation effects suggest that palatability-based learned flavor preferences are under-pinned by a Pavlovian process whereby the cue flavor is associated with the taste of the concurrently consumed palatable reinforcer.     

Relations between Pavlovian-instrumental transfer and reinforcer devaluation

Relations between posttraining reinforcer devaluation and Pavlovian-instrumental transfer were examined in 2 experiments. When a single reinforcer was used, extended training of the instrumental response increased transfer but reduced devaluation effects. When multiple instrumental reinforcers were used, both reinforcer-specific transfer and devaluation effects were less influenced by the amount of instrumental training. Finally, although reinforcer devaluation decreased both Pavlovian conditioned responses and baseline instrumental responding, it had no effect on either single-reinforcer or reinforcer-specific transfer. These results indicate that transfer and reinforcer devaluation can reflect different aspects of associative learning and that the nature of associative learning can be influenced by parameters such as the amount of training and the use of multiple reinforcers.     

THE INFLUENCE OF BEHAVIOR PRECEDING A REINFORCED RESPONSE ON BEHAVIOR CHANGE IN THE CLASSROOM1

The influence of behavior that immediately precedes a reinforced target response on the effectiveness of a reinforcement contingency was examined in two experiments with mentally retarded children in a special-education classroom. Two reinforcement schedules were examined in each experiment. For each schedule, a prespecified period of attentive behavior served as the target response. The schedules differed in whether inattentive or attentive behavior was required immediately to precede the target response. These schedules were examined with one child in a simultaneous treatment design using praise as the reinforcer (Experiment I), and with two children in separate reversal designs using tokens as the reinforcer (Experiment II). While attentive behavior increased under each schedule, the increase was greater when attentive rather than inattentive behavior preceded the reinforced response. The results indicated that the effect of a contingency may be determined not only by the specific response reinforced but also by the behavior that immediately precedes that response.     

Aftereffects of the surprising presentation and omission of appetitive reinforcers on key-pecking performance in pigeons

The reinforcement-omission effect (ROE), also known as frustration effect, refers to greater response strength immediately after nonreinforcement (N) than reinforcement (R). The ROE was traditionally interpreted as transient invigoration after N induced by primary frustration. Pigeons demonstrate similar ROEs whether outcomes are surprising (partial R) or expected (discrimination training) in runway (Experiment 1) and Skinner box situations (Experiments 2-3). Variations in the interval between N and the opportunity to respond indicate that the ROE results from an aftereffect of food consumption (Experiment 4). Increasing reinforcer magnitude increased the after-R effect, without modifying the after-N function (Experiment 5). These results are reviewed in the context of comparative research on spaced-trial successive negative contrast and related phenomena that have failed to appear in experiments involving nonmammalian vertebrates.     

The effects of satiation and reinforcer develuation on signal-centered behavior in the rat

Two experiments are described which investigate the effects of satiation and reinforcer devaluation on signal-centered behavior in rats. In Experiment 1 lever contacts were established in hungry rats by pairing retractable lever presentations (CS) with response-independent food (UCS). Subsequently, food satiating these subjects significantly reduced the level of CS contact during an extinction test and, in particular, suppressed CS-directed licking and pawing. In Experiment 2 lever contacts which were established by lever-food pairings were suppressed when the food reinforcer was paired with lithium chloride (LiCl) induced illness. In particular, CS-directed licking, sniffing, and orienting were significantly suppressed by these food-LiCl pairings. These results suggest that signal-centered behavior (i) is not simply a manifestation of “conditioned hunger,” (ii) is determined to some extent by the animal's current need state, and (iii) is influenced by the status of specific reinforcer representations.     

PREFERENCE FOR A STIMULUS THAT FOLLOWS A RELATIVELY AVERSIVE EVENT: CONTRAST OR DELAY REDUCTION?

Several types of contrast effects have been identified including incentive contrast, anticipatory contrast, and behavioral contrast. Clement, Feltus, Kaiser, and Zentall (2000) proposed a type of contrast that appears to be different from these others and called it within-trial contrast. In this form of contrast the relative value of a reinforcer depends on the events that occur immediately prior to the reinforcer. Reinforcers that follow relatively aversive events are preferred over those that follow less aversive events. In many cases the delay reduction hypothesis proposed by Fantino (1969) also can account for such effects. The current experiments provide a direct test of the delay reduction and contrast hypotheses by manipulating the schedule of reinforcement while holding trial duration constant. In Experiment 1, preference for fixed-interval (FI) versus differential-reinforcement-of-other-behavior (DRO) schedules of reinforcement was assessed. Some pigeons preferred one schedule over the other while others demonstrated a position (side) preference. Thus, no systematic preference was found. In Experiment 2, a simultaneous color discrimination followed the FI or DRO schedule, and following training, preference was assessed by presenting the two positive stimuli simultaneously. Consistent with the contrast hypothesis, pigeons showed a significant preference for the positive stimulus that in training had followed their less preferred schedule.     

Reward value effects on timing in the peak procedure

Three experiments examined the effect of motivational variables on timing in the peak procedure. In Experiment 1, rats received a 60-s peak procedure that was coupled with long-term, between-phase changes in reinforcer magnitude. Increases in reinforcer magnitude produced a leftward shift in the peak that persisted for 20 sessions of training. In a final phase, the rats received lithium chloride-induced aversion prior to testing and a rightward shift in the peak was observed. Experiment 2 confirmed the rightward shift in the peak under lithium chloride devaluation and induced a comparable shift with satiety devaluation. The degree of rightward shift was neither additive nor multiplicative, suggesting that two processes may have contributed. Experiment 3 examined the effect of extinction on peak responding, revealing a decrease in response rate, but no evidence of any change in the timing of responding. The implications of the results for contemporary timing theories are discussed.     

Induction when rats lick for 1% liquid-sucrose reinforcement

Previous research reported that rats responding for 1% liquid-sucrose reinforcement when 32% sucrose reinforcement is upcoming will decrease their response rate (contrast) if licking is the dependent measure and increase their response rate (induction) if pressing a lever is the dependent measure. The present study investigated whether induction could be observed when licking served as the dependent measure and whether induction in lever pressing and contrast in licking behaviour could be concurrently observed. Experiment 1 found induction when rats licked to earn the rewards but consumed them at a location separate from the spout licked to earn them. Experiment 2 also found induction when rats earned (and consumed) rewards by licking the same spout throughout the session. Experiment 3 separately measured instrumental lever pressing for sucrose rewards and licking the sucrose during the reward period. We found that both measures increased for 1% sucrose when 32% sucrose reinforcement was upcoming. The present results indicate that the type of response is not the sole determinant of whether contrast or induction is observed. Rather, they suggest that other procedural details, such as the location of reinforcer delivery, influence which effect is observed. The results also indicate that associative processes underlie the appearance of induction in responding for 1% sucrose.     

Induction when rats lick for 1% liquid-sucrose reinforcement

Previous research reported that rats responding for 1% liquid-sucrose reinforcement when 32% sucrose reinforcement is upcoming will decrease their response rate (contrast) if licking is the dependent measure and increase their response rate (induction) if pressing a lever is the dependent measure. The present study investigated whether induction could be observed when licking served as the dependent measure and whether induction in lever pressing and contrast in licking behaviour could be concurrently observed. Experiment 1 found induction when rats licked to earn the rewards but consumed them at a location separate from the spout licked to earn them. Experiment 2 also found induction when rats earned (and consumed) rewards by licking the same spout throughout the session. Experiment 3 separately measured instrumental lever pressing for sucrose rewards and licking the sucrose during the reward period. We found that both measures increased for 1% sucrose when 32% sucrose reinforcement was upcoming. The present results indicate that the type of response is not the sole determinant of whether contrast or induction is observed. Rather, they suggest that other procedural details, such as the location of reinforcer delivery, influence which effect is observed. The results also indicate that associative processes underlie the appearance of induction in responding for 1% sucrose.     

Response-outcome contingency: Behavioral and judgmental effects of appetitive and aversive outcomes with college students

In two experiments, positive, negative, and zero response-outcome contingencies were responded to and rated by college students under a free-operant procedure. In Experiment 1, outcomes were either neutral or were associated with point gain. In Experiment 2, subjects were administered different outcome treatments: neutral outcomes, outcomes associated with money gain, or outcomes associated with money loss. In both experiments, subjects' judgments of response-outcome contingency and their operant responses were each strong linear functions of ΔP, the difference between the probability of an outcome given a response and the probability of an outcome given no response. Appetitive and aversive outcomes produced opposite and symmetrical response patterns. In Experiment 1, no differences in ratings occurred with neutral or appetitive outcomes; however, in Experiment 2, more potent appetitve outcomes led to somewhat more extreme ratings than either neutral or aversive outcomes. Increasing outcome probability produced only a slight bias in ratings of noncontingent problems in Experiment 1 and no bias in Experiment 2. Contrary to predictions derived from an analysis of superstitious behavior, increasing outcome probability in noncontingent problems decreased operant responding when outcomes were appetitive and increased operant responding when outcomes were aversive. Trend analyses revealed that Δ P was superior to several other metrics in predicting subjects' estimates of contingency and the behavioral effects of contingency. Operant responding was in closer accord with matching predictions than with maximizing predictions.     

Asymmetrical effects of Pavlovian excitatory and inhibitory aversive transfer on Pavlovian appetitive responding and acquisition

Predictions of a theory of Pavlovian motivational transfer, which incorporates principles of both the theory of reciprocal inhibition and the Rescorla-Wagner model, were tested in several Pavlovian aversive to Pavlovian appetitive transfer tasks. As predicted, the presence of a signal for an aversive event, conditioned stimulus (AV CS+), reliably suppressed performance of appetitive conditioned responses (CRs) whether imposed during acquisition or on independently established responding. Acquisition of appetitive responding to a novel CS reinforced in compound with an AV CS+, however, was enhanced (“superconditioning”). This observation suggests that the effects of a discrepancy between expectation and actual outcome on a conditioning trial are influenced by the affective value of both the expectation and the reinforcer. These transfer effects were not symmetrical for an inhibitory aversive stimulus (AV CS?). An AV CS? did not enhance appetitive responding compared to a random control condition, nor did the AV CS? reduce (i.e., block) appetitive conditioning to a novel CS when appetitive reinforcement occurred in the presence of the AV CS?. Comparison of the two shock-exposed conditions with a naive control condition suggests that previous results that were apparently consistent with inhibitory aversive enhancement and blocking of appetitive conditioning may have been due to aversive context conditioning.