Prediction and uncertainty in human Pavlovian to instrumental transfer

Attentional capture and behavioral control by conditioned stimuli have been dissociated in animals. The current study assessed this dissociation in humans. Participants were trained on a Pavlovian schedule in which 3 visual stimuli, A, B, and C, predicted the occurrence of an aversive noise with 90%, 50%, or 10% probability, respectively. Participants then went on to separate instrumental training in which a key-press response canceled the aversive noise with a .5 probability on a variable interval schedule. Finally, in the transfer phase, the 3 Pavlovian stimuli were presented in this instrumental schedule and were no longer differentially predictive of the outcome. Observing times and gaze dwell time indexed attention to these stimuli in both training and transfer. Aware participants acquired veridical outcome expectancies in training--that is, A > B > C, and these expectancies persisted into transfer. Most important, the transfer effect accorded with these expectancies, A > B > C. By contrast, observing times accorded with uncertainty--that is, they showed B > A = C during training, and B < A = C in the transfer phase. Dwell time bias supported this association between attention and uncertainty, although these data showed a slightly more complicated pattern. Overall, the study suggests that transfer is linked to outcome prediction and is dissociated from attention to conditioned stimuli, which is linked to outcome uncertainty.     

Pavlovian to instrumental transfer of control in a human learning task

Pavlovian learning tasks have been widely used as tools to understand basic cognitive and emotional processes in humans. The present studies investigated one particular task, Pavlovian-to-instrumental transfer (PIT), with human participants in an effort to examine potential cognitive and emotional effects of Pavlovian cues upon instrumentally trained performance. In two experiments, subjects first learned two separate instrumental response-outcome relationships (i.e., R1-O1 and R2-O2) and then were exposed to various stimulus-outcome relationships (i.e., S1-O1, S2-O2, S3-O3, and S4-) before the effects of the Pavlovian stimuli on instrumental responding were assessed during a non-reinforced test. In Experiment 1, instrumental responding was established using a positive-reinforcement procedure, whereas in Experiment 2, a quasi-avoidance learning task was used. In both cases, the Pavlovian stimuli exerted selective control over instrumental responding, whereby S1 and S2 selectively elevated the instrumental response with which it shared an outcome. In addition, in Experiment 2, S3 exerted a nonselective transfer of control effect, whereby both responses were elevated over baseline levels. These data identify two ways, one specific and one general, in which Pavlovian processes can exert control over instrumental responding in human learning paradigms, suggesting that this method may serve as a useful tool in the study of basic cognitive and emotional processes in human learning.     

Reactivation-dependent amnesia for appetitive memories is determined by the contingency of stimulus presentation

Previously acquired aversive and appetitive memories are not stable and permanent. The reactivation of such memories by re-exposure to training stimuli renders them vulnerable to disruption by amnestic agents such as the noncompetitive N-methyl-D-aspartate receptor antagonist (+)-5-methyl-10,11-dihydro-SH-dibenzo{a,d}cyclohepten-5,10-imine maleate (MK-801). However, relatively little is known about the parameters that influence the reactivation process. Here, we show that the method of stimulus presentation during memory reactivation is of great importance. Male Lister Hooded rats were trained to acquire a lever press response that delivered a sucrose reward paired with a light conditioned stimulus (CS). The CS-sucrose association was then reactivated through re-exposure to the CS, either contingently upon the lever press response, or noncontingently in the absence of instrumental responding. Systemic administration of MK-801 (0.1 mg/kg) at the time of memory reactivation resulted in amnesia, and hence a reduction in subsequent sucrose seeking induced by, and dependent upon, presentation of the CS, only when the memory was reactivated contingently. Therefore, stimuli may have to be presented in the same manner at memory reactivation as during learning in order to render a previously acquired memory vulnerable to disruption. These results have important implications for the potential translational use of glutamatergic treatments in conjunction with targeted memory reactivation.     

Pavlovian processes in the motivational control of instrumental performance

Hungry rats were rewarded for pressing a lever on a multiple schedule. During one component the reward was a sucrose solution, whereas food pellets acted as the reward during another component. Lever pressing was never rewarded during the third component. When the drive state was switched from hunger to thirst and the animals tested in extinction, they pressed more in the presence of the component stimulus that had been associated with the sucrose reward during training. A similar effect was observed during the extinction test of a second study in which the component stimuli had signalled non-contingent presentations of either the sucrose or pellet rewards in the absence of the lever. This suggests that the instrumental irrelevant incentive effect observed in the first experiment was due, at least in part, to the Pavlovian relationship between the component stimuli and the reinforcers during training. In fact, when the size of the effects controlled by purely Pavlovian and supposedly instrumental contingencies was compared directly in the final study, no difference could be detected.     

Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus

Three experiments examined appetitive Pavlovian-instrumental interactions by presenting separately trained conditioned stimuli (CSs) during reinforced instrumental responding in rabbits. Intra-oral reinforcement was used to minimize interference from peripheral responses such as magazine approach. In experiment 1, the rabbits were first trained to perform an instrumental head-raising response for sucrose reward. A conditioned jaw movement response was then established to a 2-sec CS by pairing it with sucrose; a control stimulus was unpaired with sucrose. Instrumental responding maintained by a variable-interval 40-sec schedule was enhanced during 10-sec presentations of the paired, but not the unpaired, CS. Responding on a variable-ratio 15 schedule was unaffected except on trials on which the pre-CS baseline response rate was low; in such cases the paired CS caused a long-lasting acceleration of responding. Noncontingent presentation of the sucrose reinforcer itself briefly suppressed responding but had no long-term effect. In Experiment 2, a CS that had been conditioned at a 10-sec duration produced the same pattern of effects as in the first study, indicating that facilitation resulted from CS presentation rather than from the frustrative effects of non-reinforcement of the CS. In Experiment 3 an inhibitory CS blocked facilitation by the excitatory CS but did not itself affect instrumental responding. These results support the view that Pavlovian processes play a positive role in instrumental performance and suggest that previous findings of suppression by a short-duration CS reflect peripheral interference. The dependence of facilitation on the baseline level of responding is discussed in terms of associative and motivational theories of Pavlovian mediation.     

The role of nucleus accumbens dopamine in outcome encoding in instrumental and Pavlovian conditioning

Considerable evidence suggests that dopamine in the core subregion of the nucleus accumbens is not only involved in Pavlovian conditioning but also supports instrumental performance. However, it is largely unknown whether NAc dopamine is required for outcome encoding which plays an important role both in Pavlovian stimulus-outcome learning and instrumental action-outcome learning. Therefore, we tested rats with 6-hydroxydopamine (6-OHDA) induced dopamine depletion of the NAc core for their sensitivity to outcome devaluation in a Pavlovian and an instrumental task. Results indicate that 6-OHDA-lesioned animals were sensitive to outcome devaluation in an instrumental task. This finding provides support to the notion that NAc core dopamine may not be crucial in encoding action-outcome associations. However, during instrumental conditioning lever pressing rates in 6-OHDA-lesioned animals were markedly lower which could reflect an impaired behavioral activation. By contrast, after outcome-specific devaluation in a Pavlovian task, performance in 6-OHDA-lesioned animals was impaired, i.e. their magazine-directed responding was non-selectively reduced. One possibility to explain non-selective responding is that NAc core DA depletion impaired the ability of conditioned stimuli to activate the memory of the current value of the reinforcer.     

Pavlovian conditioned stimulus effects upon instrumental choice behavior are reinforcer specific

In a transfer-of-control experiment with rats, Pavlovian CSs were tested for the specificity of their effects. The instrumental behavior consisted of a discriminative, conditional two-lever choice task in which qualitatively different appetitive reinforcers were contingent upon the two correct choices. In a Pavlovian phase, subjects experienced conditioning to establish either a CS⁺ or CS^? for one reinforcer or a CS⁺ or CS^? for the other reinforcer. Finally, in a test, these CSs were presented when there was the opportunity to make choice responses. The CS⁺s evoked choices of the lever which had eventuated in the reinforcer that had served as the Pavlovian US, while the CS^?s showed only a slight tendency to evoke the other choice responses. When the CSs were compounded with the original S^Ds, the CS⁺s had little effect upon the vigor of responding while the CS^?s reduced the vigor of responding to the S^D for the reinforcer that was the same as the US used in establishing the CS^?. The results are discussed in terms of associative mediational theory and the reinforcer specificity of Pavlovian conditioned excitation and inhibition.     

Asymmetrical effects of Pavlovian excitatory and inhibitory aversive transfer on Pavlovian appetitive responding and acquisition

Predictions of a theory of Pavlovian motivational transfer, which incorporates principles of both the theory of reciprocal inhibition and the Rescorla-Wagner model, were tested in several Pavlovian aversive to Pavlovian appetitive transfer tasks. As predicted, the presence of a signal for an aversive event, conditioned stimulus (AV CS+), reliably suppressed performance of appetitive conditioned responses (CRs) whether imposed during acquisition or on independently established responding. Acquisition of appetitive responding to a novel CS reinforced in compound with an AV CS+, however, was enhanced (“superconditioning”). This observation suggests that the effects of a discrepancy between expectation and actual outcome on a conditioning trial are influenced by the affective value of both the expectation and the reinforcer. These transfer effects were not symmetrical for an inhibitory aversive stimulus (AV CS?). An AV CS? did not enhance appetitive responding compared to a random control condition, nor did the AV CS? reduce (i.e., block) appetitive conditioning to a novel CS when appetitive reinforcement occurred in the presence of the AV CS?. Comparison of the two shock-exposed conditions with a naive control condition suggests that previous results that were apparently consistent with inhibitory aversive enhancement and blocking of appetitive conditioning may have been due to aversive context conditioning.     

Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling

Here we attempted to clarify the role of dopamine signaling in reward seeking. In Experiment 1, we assessed the effects of the dopamine D(1)/D(2) receptor antagonist flupenthixol (0.5 mg/kg i.p.) on Pavlovian incentive motivation and found that flupenthixol blocked the ability of a conditioned stimulus to enhance both goal approach and instrumental performance (Pavlovian-to-instrumental transfer). In Experiment 2 we assessed the effects of flupenthixol on reward palatability during post-training noncontingent re-exposure to the sucrose reward in either a control 3-h or novel 23-h food-deprived state. Flupenthixol, although effective in blocking the Pavlovian goal approach, was without effect on palatability or the increase in reward palatability induced by the upshift in motivational state. This noncontingent re-exposure provided an opportunity for instrumental incentive learning, the process by which rats encode the value of a reward for use in updating reward-seeking actions. Flupenthixol administered prior to the instrumental incentive learning opportunity did not affect the increase in subsequent off-drug reward-seeking actions induced by that experience. These data suggest that although dopamine signaling is necessary for Pavlovian incentive motivation, it is not necessary for changes in reward experience, or for the instrumental incentive learning process that translates this experience into the incentive value used to drive reward-seeking actions, and provide further evidence that Pavlovian and instrumental incentive learning processes are dissociable.     

Stimulus generalization of suppression in rats following aversively motivated instrumental or Pavlovian training.

Following 100 or 300 avoidance training trials, instrumental subjects and their yoked Pavlovian counterparts were tested for generalization of lick suppression along the frequency dimension of the avoidance conditioned stimulus. Gradients of stimulus control were evident after 300 instrumental avoidance training trials, and additional intradimensional Pavlovian discrimination training further sharpened the gradients. After 100 trials, the yoked Pavlovian subjects suppressed more than their instrumental counterparts. However, with increased Pavlovian training, flatter gradients with decreased suppression were obtained. Results from a second experiment revealed that, whereas Pavlovian experience decreased suppression to the tone, subjects suppressed drinking in the presence of static, environmental cues. Data from both experiments supported interpretations that stress the role of response control over environmental events.     

Relative validity effects with either one or two more valid cues in Pavlovian and instrumental conditioning

Two experiments investigated the relative validity effect with either 1 or 2 continuously reinforced cues in Wistar rats using appetitive Pavlovian and instrumental preparations. Discrimination training involved 3 compound cues containing a common element (1AX: 1BX: 2CX). In the first true-discrimination group (TD-1), CX was followed by food, but AX and BX were not. In the second true-discrimination group (TD-2), AX and BX but not CX were followed by food. In the third, pseudodiscrimination group (PD), food followed 50% of each compound. Compared with the PD group, there were lower levels of responding to X in Groups TD-1 and TD-2, which did not differ. That is, both TD treatments showed equivalent relative validity effects. There was evidence for a relative validity effect on the context. The Rescorla-Wagner model incorrectly predicts a smaller relative validity effect after the TD-2 than the TD-1 treatment. Comparator theory predicts these results.     

Operant and Pavlovian factors in cross-response transfer of inhibitory stimulus control

Experiment 1 used a within-response procedure to compare inhibitory stimulus control of the extinction cue in multiple variable interval-extinction (mult VI EXT) and the variable time cue in multiple variable interval-variable time (mult VI VT). Experiment 2A made the same comparison using a crossresponse procedure. In addition, Experiment 2A also tested the extinction cue in multiple variable time-extinction (mult VT EXT), and Experiment 2B tested the variable time cue in mult VT EXT. Unlike the mult VI EXT and mult VI VT conditions, the mult VT EXT condition was administered in the absence of a response manipulandum. The main findings of the present study were: (1) cross-response suppression by the variable time cue in mult VI VT; (2) greater response suppression by the extinction cue in mult VI EXT than by either the variable time cue in mult VI VT or the extinction cue in mult VT EXT; and (3) response facilitation by the variable time cue in mult VT EXT. These findings indicate that inhibitory stimulus control based on an operant source (i.e., absence of an R-S contingency) does transfer across responses, and that such transfer cannot be explained in terms of competing responses. These findings also indicate that operant and Pavlovian sources of stimulus control summate algebraically to determine the inhibitory strength of the extinction cue in mult VI EXT. Finally, the third finding is not consistent with notions that emphasize the importance of location and localization of cues explicitly paired with positive events.     

Evaluation of the relation between Pavlovian occasion-setting and instrumental discriminative stimuli: a blocking analysis

Experiments with rat subjects used a blocking design (A+, then AB+) to assess the relation between Pavlovian occasion-setting and instrumental discriminative stimuli. Prior conditioning of both associative and occasion-setting functions of A in a serial feature-positive procedure blocked acquisition of an instrumental discriminative function by a novel stimulus (B) trained in compound with A. However, neither prior conditioning of only an A-US (unconditioned stimulus) association nor prior conditioning of Stimulus A using a Pavlovian simultaneous feature-positive procedure, which does not endow A with an occasion-setting function, blocked acquisition of an instrumental discriminative function by B. Prior acquisition of an instrumental discriminative function by A blocked acquisition of a Pavlovian occasion-setting function by a novel stimulus (B) trained in compound with A, but did not block acquisition of an association between B and the US. These outcomes indicate that Pavlovian occasion-setting and instrumental discriminative properties of a stimulus are functionally equivalent and that both Pavlovian occasion-setting and instrumental discriminative properties of a stimulus are functionally independent of simple associative relations between that stimulus and other events.     

Instrumental and Pavlovian incentive processes have dissociable effects on components of a heterogeneous instrumental chain

Two experiments compared the effects of Pavlovian stimuli and incentive learning on the performance of a heterogeneous chain of instrumental actions. Using a Pavlovian-instrumental transfer design, the authors found that only a stimulus paired with the same outcome as that earned by performance of the chain produced positive transfer, an effect that was restricted to the action in the chain most proximal to reward delivery. In contrast, after a shift to a nondeprived state, only animals that had previously consumed the instrumental outcome when they were nondeprived decreased instrumental performance. Furthermore, this effect of the incentive learning treatment was limited to performance of the distal action. Together these data suggest that Pavlovian and instrumental incentive manipulations have dissociable effects on instrumental performance.     

How transitions from nonrewarded to rewarded trials regulate responding in Pavlovian and instrumental learning following extensive acquisition training

In both discrimination learning and partial reinforcement, transitions may occur from nonrewarded to rewarded trials (NR transition). In discrimination learning, NR transitions may occur in two different stimulus alternatives (NR different transitions). In partial reward, NR transitions may occur in a single stimulus alternative (NR same transitions). Available instrumental learning data indicate that resistance to extinction is increased by both types of NR transitions following limited acquisition training. Following more extensive acquisition training, resistance to extinction appears to be increased by NR same transitions but not by NR different transitions. In Experiment 1, it was shown for the first time that following extensive acquisition training the effects of the two types of transitions are the same in the Pavlovian situation as in the instrumental situation. This finding indicates that on the current trial the rat remembered the conditioned stimulus (CS) and the unconditioned stimulus (US) from the prior trial and those memories along with the CS on the current trial become the signal for the current US. Experiments 2 and 3, which tested hypotheses about instrumental learning, identified why NR different transitions lose their capacity to promote vigorous responding following extensive acquisition training. This is because cues occurring on the rewarded trials of NR different transitions more validly signal reward than other situational cues and thus overshadow them. Finally, some implications of the present findings for understanding the role of NR different transitions in discrimination learning situations were discussed.     

Aversive CS control of instrumental avoidance as a function of selected parameters and method of Pavlovian conditioning

Effects of aversive CSs upon dogs' instrumental avoidance responding were assessed as a function of the Pavlovian conditioning parameters using off-baseline (Experiment 1) and on-baseline (Experiment 2) conditioning procedures. In each experiment, three UCS intensitites (2, 6, and 10 mA) were crossed factorially with two CS-UCS intervals (10 and 90 sec) yielding six groups. After 4 days of conditioning, the CS-produced effects on avoidance were determined primarily by an interaction between Pavlovian procedure (off- versus on-baseline) and UCS intensity. With off-baseline conditioning, CS-produced facilitation of avoidance was an increasing function of the UCS intensity. However, with on-baseline conditioning, only the CS paired with weak UCS facilitated avoidance, and the CS paired with the strong UCS suppressed responding. This reversing parametric function poses problems for two-process motivational theories of avoidance.     

Defending the wide-scope approach to instrumental reason

The Wide-Scope approach to instrumental reason holds that the requirement to intend the necessary means to your ends should be understood as a requirement to either intend the means, or else not intend the end. In this paper I explain and defend a neglected version of this approach. I argue that three serious objections to Wide-Scope accounts turn on a certain assumption about the nature of the reasons that ground the Wide-Scope requirement. The version of the Wide-Scope approach defended here allows us to reject this assumption, and so defuse the objections.