Goal-directed and transfer-cue-elicited drug-seeking are dissociated by pharmacotherapy: evidence for independent additive controllers

According to contemporary learning theory, drug-seeking behavior reflects the summation of 2 dissociable controllers. Whereas goal-directed drug-seeking is determined by the expected current incentive value of the drug, stimulus-elicited drug-seeking is determined by the expected probability of the drug independently of its current incentive value, and these 2 controllers contribute additively to observed drug-seeking. One applied prediction of this model is that smoking cessation pharmacotherapies selectively attenuate tonic but not cue-elicited craving because they downgrade the expected incentive value of the drug but leave expected probability intact. To test this, the current study examined whether nicotine replacement therapy (NRT) nasal spray would modify goal-directed tobacco choice in a human outcome devaluation procedure, but leave cue-elicited tobacco choice in a Pavlovian to instrumental transfer (PIT) procedure intact. Smokers (N= 96) first underwent concurrent choice training in which 2 responses earned tobacco or chocolate points, respectively. Participants then ingested either NRT nasal spray (1 mg) or chocolate (147 g) to devalue 1 outcome. Concurrent choice was then tested again in extinction to measure goal-directed control of choice, and in a PIT test to measure the extent to which tobacco and chocolate stimuli enhanced choice of the same outcome. It was found that NRT modified tobacco choice in the extinction test but not the extent to which the tobacco stimulus enhanced choice of the tobacco outcome in the PIT test. This dissociation suggests that the propensity to engage in drug-seeking is determined independently by the expected value and probability of the drug, and that pharmacotherapy has partial efficacy because it selectively effects expected drug value.     

Alcohol seeking by rats: Action or habit?

In two experiments, we examined the relative susceptibility to outcome devaluation of lever pressing by rats for either a 10% ethanol solution or food pellets. The rats were trained to press different levers for these two reinforcers using a sucrose-substitution procedure. An aversion was then conditioned from either the ethanol solution or the food pellets by pairing consumption with illness induced by lithium chloride. When instrumental performance was subsequently tested in extinction, the rats pressed less on the pellet lever if the pellets, rather than the ethanol, had been devalued by aversion conditioning. By contrast, performance on the ethanol lever was unaffected by whether the ethanol or pellets were devalued. Moreover, noncontingent presentations of the devalued reinforcer had no impact on test performance. The differential resistance to outcome devaluation suggests that, in contrast to food seeking, alcohol seeking is a stimulus-response habit rather than a goal-directed action mediated by a representation of the action-outcome contingency.     

Impaired goal-directed behavioural control in human impulsivity

Two dissociable learning processes underlie instrumental behaviour. Whereas goal-directed behaviour is controlled by knowledge of the consequences, habitual behaviour is elicited directly by antecedent Pavlovian stimuli without knowledge of the consequences. Predominance of habitual control is thought to underlie psychopathological conditions associated with corticostriatal abnormalities, such as impulsivity and drug dependence. To explore this claim, smokers were assessed for nicotine dependence, impulsivity, and capacity for goal-directed control over instrumental performance in an outcome devaluation procedure. Reduced goal-directed control was selectively associated with the Motor Impulsivity factor of Barrett's Impulsivity Scale (BIS), which reflects propensity for action without thought. These data support the claim that human impulsivity is marked by impaired use of causal knowledge to make adaptive decisions. The predominance of habit learning may play a role in psychopathological conditions that are associated with trait impulsivity.     

Cue-elicited food seeking is eliminated with aversive outcomes following outcome devaluation

In outcome-selective Pavlovian-to-instrumental transfer (PIT), stimuli that are predictive of specific outcomes prime instrumental responses that are associated with these outcomes. Previous human studies yielded mixed evidence in respect to whether the PIT effect is affected by a posttraining devaluation of an outcome, with the PIT effect being preserved after a devaluation of a primary reinforcer (food, drugs) but not following the devaluation of a secondary reinforcer (money). The present research examined whether outcome-selective transfer is eliminated when the devaluation of a primary (liquid) reinforcer is strong and aversive. Experiment 1 confirmed these expectations following a devaluation with bad tasting Tween 20. However, outcome-selective transfer was still observed when the earned (devalued) outcome was not consumed immediately after each test (Experiment 2). These results suggest that the capacity of a Pavlovian cue to motivate a specific response is affected by the incentive value of the shared outcome only when the devaluation yields an aversive outcome that is consumed immediately.     

The temporal structure of goal-directed and habitual operant behavior

Operant behavior can reflect the influence of goal-directed and habitual processes. These can be distinguished by changes to response rate following devaluation of the reinforcing outcome. Whether a response is goal directed or habitual depends on whether devaluation affects response rate. Response rate can be decomposed into frequencies of bouts and pauses by analyzing the distribution of interresponse times. This study sought to characterize goal-directed and habitual behaviors in terms of bout-initiation rate, within-bout response rate, bout length, and bout duration. Data were taken from three published studies that compared sensitivity to devaluation following brief and extended training with variable-interval schedules. Analyses focused on goal-directed and habitual responding, a comparison of a habitual response to a similarly trained response that had been converted back to goal-directed status after a surprising event, and a demonstration of contextual control of habit and goal direction in the same subjects. Across experiments and despite responses being clearly distinguished as goal directed and habitual by total response rate, analyses of bout-initiation rate, within-bout rate, bout length, and bout duration did not reveal a pattern that distinguished goal-directed from habitual responding.     

The role of drug expectancy in the control of human drug seeking

Human drug seeking may be goal directed in the sense that it is mediated by a mental representation of the drug or habitual in the sense that it is elicited by drug-paired cues directly. To test these 2 accounts, the authors assessed whether a drug-paired stimulus (S+) would transfer control to an independently trained drug-seeking response. Smokers were trained on an instrumental discrimination that established a tobacco S+ in Experiment 1 and a tobacco and a money S+ in Experiment 2 that elicited an expectancy of their respective outcomes. Participants then learned 2 new instrumental responses, 1 for each outcome, in the absence of these stimuli. Finally, in the transfer test, each S+ was found to augment performance of the new instrumental response that was trained with the same outcome. This outcome-specific transfer effect indicates that drug-paired stimuli controlled human drug seeking via a representation or expectation of the drug rather than through a direct stimulus-response association.     

Stimulus control of habits: Evidence for both stimulus specificity and devaluation insensitivity in a dual-response task

Goal-directed and habitual actions are clearly defined by their associative relations. Whereas goal-directed control can be confirmed via tests of outcome devaluation and contingency-degradation sensitivity, a comparable criterion for positively detecting habits has not been established. To confirm habitual responding, a test of control by the stimulus–response association is required while also ruling out goal-directed control. Here we describe an approach to developing such a test in rats using two discriminative stimuli that set the occasion for two different responses that then earn the same outcome. Performance was insensitive to outcome devaluation and showed stimulus–response specificity, indicative of stimulus-controlled behavior. The reliance of stimulus–response associations was further supported by a lack of sensitivity during the single extinction test session used here. These results demonstrate that two concurrently trained responses can come under habitual control when they share a common outcome. By reducing the ability of one stimulus to signal its corresponding response–outcome association, we found evidence for goal-directed control that can be dissociated from habits. Overall, these experiments provide evidence that tests assessing specific stimulus–response associations can be used to investigate habits.     

Negative emotional appraisal selectively disrupts retrieval of expected outcome values required for goal-directed instrumental choice*

Stress induction reduces people's ability to modify their instrumental choices following changes in the value of outcomes, but the mechanisms underpinning this effect have not been specified because previous studies have lacked crucial control conditions. To address this, the current study had participants learn two instrumental responses for food and water, respectively, before water was devalued by specific satiety. Choice between these two responses was then measured in extinction, reacquisition and Pavlovian to instrumental transfer (PIT) tests. Concurrently during these tests, a negative emotional appraisal group evaluated aversive images (stress induction), whereas a control group evaluated neutral images, at the same time as choosing between the two instrumental responses. Negative emotional appraisal abolished the impact of water devaluation on instrumental choice in the extinction test, but did not affect instrumental choice in the reacquisition or PIT tests. These findings suggest that negative emotional appraisal selectively impaired participants’ ability to retrieve the expected value of outcomes required to make goal-directed instrumental choices in the extinction test, and that this effect was not due to task disengagement, nullification of the devaluation treatment or impaired knowledge of response-outcome relationships.     

Resurgence of goal-directed actions and habits

This study investigated how goal-directed and habitual behaviors recover after extinction within the context of the resurgence effect, a form of relapse induced by the removal or worsening of alternative reinforcement. Rats were trained to press a target lever with one reinforcer (O1) for either minimal (4) or extended (16) sessions. An extinction test after the completion of O1 devaluation confirmed that minimal and extended training formed goal-directed and habitual behaviors, respectively. Then, pressing an alternative lever was reinforced with a second reinforcer (O2) while the target response was placed on extinction. When O2 was discontinued, the minimally trained target response resurged with goal-directed status as in the extinction test. However, the extinguished habitual behavior in the extensively trained rats did not recover as a habit but instead with goal-directed status, possibly due to the context specificity of habits or the introduction of a new response–reinforcer contingency. The critical finding that reinforcer devaluation consistently led to less resurgence regardless of the amount of acquisition training provides a clinical implication that coupling differential-reinforcement-of-alternative-behavior (DRA) treatments with the devaluation of the associated reinforcer of problematic behavior could effectively diminish its recurrence.     

The control of instrumental action following outcome devaluation in young children aged between 1 and 4 years

To determine the role of action-outcome learning in the control of young children's instrumental behavior, the authors trained 18- to 48-month-olds to manipulate visual icons on a touch-sensitive display to obtain different types of video clips as outcomes. Subsequently, one of the outcomes was devalued by repeated exposure, and children's propensity to perform the trained actions was tested in extinction. On test, children with a mean age greater than 2.5 years performed the action trained with the devalued outcome less than those trained with the still-valued outcome, thereby demonstrating that their actions were mediated by action-outcome learning. By contrast, the instrumental responses of younger children (mean age<2 years) were resistant to outcome devaluation and may have been elicited directly by the icons associated with each response, rather than mediated by a specific action-outcome expectation.     

Resolution of outcome-induced response conflict by humans after extended training

Studies of incongruent discrimination learning, where the outcome event of one response acts as the discriminative stimulus for the opposite response, suggest that humans rely on habitual stimulus–response (S–R) associations when outcome–response (O–R) associations would cause response conflict. Here, two experiments were conducted to investigate the robustness of this habitual strategy. In Experiment 1, we found that extensive instrumental discrimination training supported learning about the incongruent R → O contingencies, as assessed by an outcome devaluation test. Differential representations of the stimulus and the (associatively retrieved) outcome may have allowed for goal-directed incongruent performance. Experiment 2 failed to provide evidence for this possibility; direct presentation as well as associative retrieval of the incongruent events (by Pavlovian stimuli) activated the response that was associated with each event in its role of stimulus as opposed to outcome. We did find that participants successfully acquired explicit knowledge of the incongruent contingencies, which raises the possibility that propositional encoding allowed them to overcome the response conflict caused by O–R associations. Alternative associative and propositional accounts of successful goal-directed incongruent performance with extensive training will be discussed.     

Effect of reinforcer devaluation on discriminative control of instrumental behavior

Two experiments examined the effect of reinforcer devaluation on the ability of a discriminative stimulus (Sd) to control instrumental behavior in Sprague-Dawley rats. In Experiment 1 reinforcer devaluation reduced, but did not eliminate, the ability of the Sd to control performance of the original response and to transfer its control to a new response trained with the same reinforcer. The effect of devaluation was more complete in Experiment 2, in which the reinforcer was delivered directly into the oral cavity. However, retraining the response with a different reinforcer partially restored the ability of the Sd to control performance of that response. These results suggest that an Sd may not augment its trained responses when the reinforcer has been completely devalued but may promote responses with which it shares a reinforcer, as long as those responses are associated with some reinforcer that retains its value. The implications of these results for the way that discriminative stimuli control instrumental behavior are discussed.     

Domestic hens have declarative representations

It is generally considered that information can be stored either as a procedural or as a declarative representation. A devaluation technique was used to determine whether hens have declarative representations. Individual hens (Gallus gallus domesticus) were fed in an enclosure with two containers, each with a new food type. One of the food types was devalued by pre-feeding with that food, after which the hens were tested with empty food containers. The pre-feeding should only affect the choice of the hens if they have learned where a particular food type was (declarative representation) rather than “go left when coming into the enclosure” (procedural representation). A significant proportion of the hens went to the location previously occupied by the non-devalued food (seven out of eight). This supports the hypothesis that domestic hens can form declarative representations. Received: 6 March 1999 / Accepted after revision: 1 November 2000     

Full Preservation of a Response-Outcome Association through Training with a Second Outcome

In two experiments rats were trained to make instrumental responses, with each response earning first one and then a second outcome. Subsequent devaluation of either the first or the second earned outcome had a depressive effect on responding. The magnitude of the effect was similar, whichever outcome was devalued. This result suggests that initially trained response-outcome associations are fully preserved despite the same response being trained with a second outcome.     

The effect of the instrumental training contingency on susceptibility to reinforcer devaluation

Two experiments investigated performance of instrumental lever pressing by rats following post-conditioning devaluation of the sucrose reinforcer produced by establishing an aversion to it. In Experiment I rats responded less in an extinction test after being averted from the sucrose following training on a ratio schedule, but not following an equivalent amount of training on an interval schedule. This was true even though the devalued sucrose would not act as an effective reinforcer on either the ratio or interval schedule. Experiment II provided a further investigation of the insensitivity of interval responding to reinforcer devaluation by comparing test performance under simple extinction with responding when the devalued reinforcer was presented on either a response-contingent or non-contingent schedule during the test. Once again simple extinction performance was unaffected by prior reinforcer devaluation. Furthermore, neither non-contingent nor contingent presentations of the devalued reinforcer significantly depressed responding below the level seen in the extinction condition. Ratio, but not interval performance appears to be controlled by knowledge about the instrumental contingency that encodes specific properties of the training reinforcer.     

Reinforcement value and fixed‐interval performance

The concept of reinforcement value summarizes the effect of different variables, such as reinforcement delay, reinforcement magnitude, and deprivation level, on behavior. In the present set of experiments, we evaluated the effect of reinforcement devaluation on performance under FI schedules. The literature on timing and reinforcement value suggests that devaluation generates longer expected times to reinforcement than the same intervals trained under control conditions. We devalued reinforcement with delay in Experiments 1A, 1B, and 2, and diminished deprivation in Experiments 3A and 3B. Devaluation reduced response rates, increased the number of one‐response intervals, and lengthened postreinforcement pauses, but had inconsistent effects on other timing measures such as quarter life and breakpoint. The results of delayed reinforcement and diminished deprivation manipulations are well summarized as reinforcement devaluation effects. These results suggest that devaluation may reduce stimulus control. In addition, we argue that the process by which delayed reinforcement affects behavior might also explain some effects observed in other devaluation procedures through the concept of reinforcement value.     

Impaired acquisition of goal-directed action in healthy aging

According to dual-system theories, instrumental learning is supported by dissociable goal-directed and habitual systems. Previous investigations of the dual-system balance in healthy aging have yielded mixed results. To further investigate this issue, we compared performance of young (17–24 years) and older (69–84 years) adults on an instrumental learning task. Following the initial learning phase, the behavioral autonomy of the motivational significance of the instrumental outcome was assessed with an outcome-devaluation test and slips-of-action test. The present study provides evidence for a disrupted dual-system balance in healthy aging, as reflected in reduced outcome-induced conflict during acquisition, as well as in impaired performance during the test stage, during which participants had to flexibly adjust their actions to changes in the current desirability of the behavioral outcome. These findings will be discussed in relation to previous aging studies into habitual and goal-directed control, as well as other cognitive impairments, challenges that older adults may face in everyday life, and to the neurobiological basis of the developmental pattern of goal-directed action across the lifespan.     

The propositional basis of cue-controlled reward seeking

Two experiments examined the role of propositional and automatic (ideomotor) processes in cue-elicited responding for rewarding outcomes (beer and chocolate). In a training phase, participants earned either chocolate or beer points by making one of two button-press responses. Rewards were indicated by the presentation of chocolate and beer pictures. On test, each trial began with a picture of beer or chocolate, or a blank screen, and choice of the beer versus chocolate response was assessed in the presence of these three pictures. Participants tended to choose the beer and chocolate response in the presence of the beer and chocolate pictures, respectively. In Experiment 1, instructions signalling that the pictures did not indicate which response would be rewarded significantly reduced the priming effect. In Experiment 2, instructions indicating that the pictures signified which response would not be rewarded resulted in a reversed priming effect. Finally, in both experiments, the priming effect correlated with self-reported beliefs that the cues signalled which response was more likely to be reinforced. These results suggest that cue-elicited response selection is mediated by a propositional belief regarding the efficacy of the response–outcome relationship, rather than an automatic ideomotor mechanism.     

Habit and persistence

Voluntary behaviors (operants) can come in two varieties: Goal-directed actions, which are emitted based on the remembered value of the reinforcer, and habits, which are evoked by antecedent cues and performed without the reinforcer's value in active memory. The two are perhaps most clearly distinguished with the reinforcer-devaluation test: Goal-directed actions are suppressed when the reinforcer is separately devalued and responding is tested in extinction, and habitual behaviors are not. But what is the function of habit learning? Habits are often thought to be strong and unusually persistent. The present selective review examines this idea by asking whether habits identified by the reinforcer-devaluation test are more resistant to extinction, resistant to the effects of other contingency change, vulnerable to relapse, resistant to the weakening effects of context change, or permanently in place once they are learned. Surprisingly little evidence supports the idea that habits are permanent or more persistent. Habits are more context-specific than goal-directed actions are. Methods that make behavior persistent do not necessarily work by encouraging habit. The function of habit learning may not be to make a behavior strong or more persistent but to make it automatic and efficient in a particular context.     

Unconditioned stimulus devaluation effects in nutrient-conditioned flavor preferences

Experiments with different temporal relations between the conditioned stimulus (CS) and the unconditioned stimulus (US) in conditioning assessed whether US devaluation effects can be obtained after nutrient-conditioned flavor preference learning. One flavor (CScarb) was paired with a carbohydrate, Polycose; a 2nd flavor (CSprot) was paired with a protein, casein; and a 3rd flavor (CS-) was presented by itself. Following conditioning, one of the nutrients was devalued through pairings with lithium chloride in the absence of the CS flavors. In a subsequent 2-bottle test, rats preferred CScarb over CSprot; however, this preference was smaller when the carbohydrate was devalued than when the protein was devalued. Results suggest that CS flavors are able to form associations with the sensory features of nutrient USs under a wide variety of circumstances.