Involvement of the rat anterior cingulate cortex in control of instrumental responses guided by reward expectancy

The anterior cingulate cortex (ACC) plays a critical role in stimulus-reinforcement learning and reward-guided selection of actions. Here we conducted a series of experiments to further elucidate the role of the ACC in instrumental behavior involving effort-based decision-making and instrumental learning guided by reward-predictive stimuli. In Experiment 1, rats were trained on a cost-benefit T-maze task in which they could either choose to climb a barrier to obtain a high reward (four pellets) in one arm or a low reward (two pellets) in the other with no barrier present. In line with previous studies, our data reveal that rats with quinolinic acid lesions of the ACC selected the response involving less work and smaller reward. Experiment 2 demonstrates that breaking points of instrumental performance under a progressive ratio schedule were similar in sham-lesioned and ACC-lesioned rats. Thus, lesions of the ACC did not interfere with the effort a rat is willing to expend to obtain a specific reward in this test. In a subsequent task, we examined effort-based decision-making in a lever-press task where rats had the choice between pressing a lever to receive preferred food pellets under a progressive ratio schedule, or free feeding on a less preferred food, i.e. lab chow. Results show that sham- and ACC-lesioned animals had similar breaking points and ingested comparable amounts of less-preferred food. Together, the results of Experiment 1 and 2 suggest that the ACC plays a role in evaluating how much effort to expend for reward; however, the ACC is not necessary in all situations requiring an assessment of costs and benefits. In Experiment 3 we investigated learning and reversal learning of instrumental responses guided by reward predictive stimuli. A reaction time (RT) task demanding conditioned lever release was used in which the upcoming reward magnitude (five vs. one food pellet) was signalled in advance by discriminative visual stimuli. Results revealed that rats with ACC lesions were able to discriminate reward magnitude-predictive stimuli and to adapt instrumental behavior to reversed stimulus-reward magnitude contingencies. Thus, in a simple discrimination task as used here, the ACC appears not to be required to discriminate reward magnitude-predictive stimuli and to use the learned significance of the stimuli to guide instrumental behavior.     

Visual Reinforcement Signals Interfere with the Effects of Reinforcer Magnitude Manipulations

Three experiments examined the effect of reinforcement magnitude on free-operant response rates. In Experiment 1, rats that received four food pellets responded faster than rats that received one pellet on a variable ratio 30 schedule. However, when the food hopper was illuminated during reinforcer delivery, there was no difference between the rates of response produced by the two magnitudes of reward. In Experiment 2, there was no difference in response rates emitted by rats receiving either one or four pellets of food as reward on a random interval (RI) 60-s schedule. In Experiment 3, rats responding on an RI 30-s schedule did so at a lower rate with four pellets as reinforcement than with one pellet. This effect was abolished by the illumination of the food hopper during reinforcement delivery. These results indicate that the influence of magnitude is obscured by manipulations which signal the delivery of reinforcement.     

Differential reinforcement of lever-press durations: Effects of deprivation level and reward magnitude

Three experiments investigated the performance of rats on a task involving differential reinforcement of lever-press durations. Experiment 1, which employed a discrete-trials procedure, manipulated deprivation level between subjects and reward magnitude within subjects. The minimum lever-press duration which would result in reward was varied from .4 to 6.4 sec. It was found that low deprivation resulted in longer mean durations and less response variability at the higher criterial values than did high deprivation. The magnitude of reward was not found to affect performance. Experiment 2 manipulated reward magnitude between subjects while holding deprivation level constant, and used the same general procedures as in Experiment 1. Small reward resulted in longer mean lever-press durations and less variability in responding than did large reward at the higher criterial values. The intertrial intervals were omitted in Experiment 3 in which deprivation level was varied between subjects and reinforcement was delivered only for response durations extending between 6.0 and 7.6 sec. Low deprivation resulted in longer mean lever-press durations and less response variability than did high deprivation, but the probability of a rewarded press duration did not differ between groups. The results overall are consistent with the hypothesis that low deprivation and small reward magnitude lead to weaker goal-approach responses and, hence, to less competition with lever holding. The deprivation and reward magnitude manipulations did not appear to influence lever holding performance by affecting the ability of animals to form temporal discriminations.     

Probabilistic numerical discrimination in mice

Previous studies showed that both human and non-human animals can discriminate between different quantities (i.e., time intervals, numerosities) with a limited level of precision due to their endogenous/representational uncertainty. In addition, other studies have shown that subjects can modulate their temporal categorization responses adaptively by incorporating information gathered regarding probabilistic contingencies into their time-based decisions. Despite the psychophysical similarities between the interval timing and nonverbal counting functions, the sensitivity of count-based decisions to probabilistic information remains an unanswered question. In the current study, we investigated whether exogenous probabilistic information can be integrated into numerosity-based judgments by mice. In the task employed in this study, reward was presented either after few (i.e., 10) or many (i.e., 20) lever presses, the last of which had to be emitted on the lever associated with the corresponding trial type. In order to investigate the effect of probabilistic information on performance in this task, we manipulated the relative frequency of different trial types across different experimental conditions. We evaluated the behavioral performance of the animals under models that differed in terms of their assumptions regarding the cost of responding (e.g., logarithmically increasing vs. no response cost). Our results showed for the first time that mice could adaptively modulate their count-based decisions based on the experienced probabilistic contingencies in directions predicted by optimality.     

Effects of magnitude of food reinforcement on free-operant response rates

In Experiment 1 rats were trained to press a lever on a variable-ratio schedule of food presentation and were then exposed to progressively increasing magnitudes of food reinforcement. Response running rates (rates exclusive of the postreinforcement pause) were found to increase as a function of increasing reinforcement magnitudes. The effect of reinforcement magnitude on response rates inclusive of the postreinforcement pause, however, was less pronounced. Increases in the magnitude of reinforcement were also found to increase the length of the postreinforcement pause. Rats in Experiment 2 were trained to respond on a chained differential-reinforcement-of-low-rate variable-ratio schedule, and were exposed to increasing magnitudes of reinforcement as in Experiment 1. Response running rates increased in the variable-ratio component but decreased in the other component of the schedule. The results are discussed with reference to incentive accounts of reinforcement and the action of reinforcement on the response units generated by the operative contingencies.     

Hippocampal associative cellular responses: dissociation with behavioral responses revealed by a transfer-of-control technique

Multiunit activity was recorded in the CA3 field of the dorsal hippocampus in freely moving rats during classical conditioning and subsequent presentation of the CS on operant baselines for food reward as well as shock avoidance. Rats were first trained in a nonsignaled bar-pressing-dependent shock omission task and in a food-motivated lever-pressing task (60-s VI). Five sessions with presentations of a previously habituated tone as a CS paired with footshock as a US were then given. Testing was carried out by presenting the CS alone while behavioral responses were maintained by reinforcement in both instrumental tasks on alternate sessions. As expected, the CS induced a marked suppression of lever pressing for food reward and a marked enhancement of bar-pressing for shock avoidance. The analysis of the frequency of multiunit discharges to the CS revealed that the hippocampal cellular responses established during classical conditioning were maintained while two different behavioral responses were exhibited to the CS. The results showed that the associative response of hippocampal neurons may be dissociated from the Pavlovian conditioned responses the CS elicits. They support the hypothesis that hippocampal cellular responses represent a neural index of the acquired CS-US associative representation.     

Roles of nucleus accumbens and basolateral amygdala in autoshaped lever pressing

Initially-neutral cues paired with rewards are thought to acquire motivational significance, as if the incentive motivational value of the reward is transferred to the cue. Such cues may serve as secondary reinforcers to establish new learning, modulate the performance of instrumental action (Pavlovian-instrumental transfer, PIT), and be the targets of approach and other cue-directed behaviors. Here we examined the effects of lesions of the ventral striatal nucleus accumbens (ACb) and the basolateral amygdala (BLA) on the acquisition of discriminative autoshaped lever-pressing in rats. Insertion of one lever into the experimental chamber was reinforced by sucrose delivery, but insertion of another lever was not reinforced. Although sucrose was delivered independently of the rats' behavior, sham-lesioned rats rapidly came to press the reinforced but not the nonreinforced lever. Bilateral ACb lesions impaired the initial acquisition of sign-tracking but not its terminal levels. In contrast, BLA lesions produced substantial deficits in terminal levels of sign-tracking. Furthermore, whereas ACb lesions primarily affected the probability of lever press responses, BLA lesions mostly affected the rate of responding once it occurred. Finally, disconnection lesions that disrupted communication between ACb and BLA produced both sets of deficits. We suggest that ACb is important for initial acquisition of consummatory-like responses that incorporate hedonic aspects of the reward, while BLA serves to enhance such incentive salience once it is acquired.     

Failure to exploit learned spatial value information during visual search

ABSTRACT

In recent years there has been rapid proliferation of studies demonstrating how reward learning guides visual search. However, most of these studies have focused on feature-based reward, and there has been scant evidence supporting the learning of space-based reward. We raise the possibility that the visual search apparatus is impenetrable to spatial value contingencies, even when such contingencies are learned and represented online in a separate knowledge domain. In three experiments, we interleaved a visual choice task with a visual search task in which one display quadrant produced greater monetary rewards than the remaining quadrants. We found that participants consistently exploited this spatial value contingency during the choice task but not during the search task – even when these tasks were interleaved within the same trials and when rewards were contingent on response speed. These results suggest that the expression of spatial value information is task specific and that the visual search apparatus could be impenetrable to spatial reward information. Such findings are consistent with an evolutionary framework in which the search apparatus has little to gain from spatial value information in most real world situations.     

Neural coding of reward magnitude in the orbitofrontal cortex of the rat during a five-odor olfactory discrimination task

The orbitofrontal cortex (OBFc) has been suggested to code the motivational value of environmental stimuli and to use this information for the flexible guidance of goal-directed behavior. To examine whether information regarding reward prediction is quantitatively represented in the rat OBFc, neural activity was recorded during an olfactory discrimination “go”/“no-go” task in which five different odor stimuli were predictive for various amounts of reward or an aversive reinforcer. Neural correlates related to both actual and expected reward magnitude were observed. Responses related to reward expectation occurred during the execution of the behavioral response toward the reward site and within a waiting period prior to reinforcement delivery. About one-half of these neurons demonstrated differential firing toward the different reward sizes. These data provide new and strong evidence that reward expectancy, regardless of reward magnitude, is coded by neurons of the rat OBFc, and are indicative for representation of quantitative information concerning expected reward. Moreover, neural correlates of reward expectancy appear to be distributed across both motor and nonmotor phases of the task.     

The role of the GABA(A) agonist muscimol on memory performance: reward contingencies determine the nature of the deficit

A matching-to-position (MTP) paradigm was altered to influence the type of associations a rat would use to solve the task. Our main behavioral manipulation was the application of the differential outcomes procedure (DOP). The DOP involves correlating each to-be-remembered event with a distinct reward condition. This procedure results in the development of unique reward expectancies that enhance and guide choice behavior. Such distinct reward expectancies are not formed when either a common or random assignment of reward is used (a non-differential outcomes procedure [NOP]). Intracerebroventricular infusions of the amnestic agent muscimol (GABA(A) agonist) or aCSF were delivered to male rats trained on a delayed MTP task that implemented either the DOP or the NOP. Muscimol impaired performance in a dose dependent fashion in both groups--but the nature of the deficit differed as a function of reinforcement contingencies. Rats trained with the DOP displayed a non-mnemonic delay-independent impairment: performance at all delay intervals was disrupted. In contrast, NOP-trained rats displayed a delay-dependent impairment demonstrating that muscimol can also have memory-disrupting effects. The difference in pattern of impairment appears to be a function of the associations formed during training and the type of cognitive strategies involved in maintaining behavior on a conditional delayed discrimination task when reinforcement contingencies are varied. Thus, these results demonstrate that increasing GABA(A) receptor activation impairs a range of associative and memory functions.     

The effects of hippocampal lesions on learning, memory, and reward expectancies

The hippocampus appears to be critical for the formation of certain types of memories. Hippocampal-lesioned animals fail to exhibit some spatial, contextual, and relational associations. After aspiration lesions of the hippocampus and/or cortex, male rats were allowed to recover for three weeks before being trained on a matching-to-position task. The matching-to-position task was altered to influence the type of cognitive strategies a subject would use to solve the task. The main behavioral manipulation was the reinforcement contingency assignment: Use of a differential outcomes procedure (DOP) or a nondifferential outcomes procedure (NOP). The DOP involves correlating each to-be-remembered event with a distinct reward condition via Pavlovian trace conditioning, whereas the NOP results in random reward contingency. We found that hippocampal lesions did retard learning the matching rule, regardless of the reinforcement contingency assignment. However, when delay intervals were added to the task memory performance of subjects with hippocampal lesions was dramatically impaired--if subjects were not trained with the DOP. When subjects were trained with the DOP, the hippocampal lesion had a marginal effect on delayed memory performance. These findings demonstrate two important points regarding lesions of the hippocampus: (1) hippocampal lesions have a minimal effect on the on the ability of rats to use reward information to solve a delayed discrimination task; (2) rats with hippocampal lesions have the ability to learn about reward information using Pavlovian trace conditioning procedures.     

Reward magnitude effects on temporal discrimination

Changes in reward magnitude or value have been reported to produce effects on timing behavior, which have been attributed to changes in the speed of an internal pacemaker in some instances and to attentional factors in other cases. The present experiments therefore aimed to clarify the effects of reward magnitude on timing processes. In Experiment 1, rats were trained to discriminate a short (2 s) vs. a long (8 s) signal followed by testing with intermediate durations. Then, the reward on short or long trials was increased from 1 to 4 pellets in separate groups. Experiment 2 measured the effect of different reward magnitudes associated with the short vs. long signals throughout training. Finally, Experiment 3 controlled for satiety effects during the reward magnitude manipulation phase. A general flattening of the psychophysical function was evident in all three experiments, suggesting that unequal reward magnitudes may disrupt attention to duration.     

Aged rats are impaired on an attentional set-shifting task sensitive to medial frontal cortex damage in young rats

Normal aging is associated with disruption of neural systems that subserve different aspects of cognitive function, particularly in the hippocampus and frontal cortex. Abnormalities in hippocampal function have been well investigated in rodent models of aging, but studies of frontal cortex function in aged rodents are few. We tested young (4–5 mo old) and aged (27–28 mo old) male Long-Evans rats on an attentional set-shifting task modified slightly from previous publication. After training on two problems in which the reward was consistently associated with the same stimulus dimension, and a reversal of one problem, a new problem was presented in which the reward was consistently associated with the previously irrelevant stimulus dimension (extradimensional shift [EDS]). Aged rats as a group were significantly impaired on the EDS, although some individual aged rats performed as well as young rats on this phase. In addition, some aged rats were impaired on the reversal, although a group effect did not reach significance in this phase. Impairment in neither reversal nor EDS was associated with impairments in spatial learning in the Morris water maze. Young rats with neurotoxic lesions of medial frontal cortex are also selectively impaired on the EDS. These results indicate that normal aging in rats is associated with impaired medial frontal cortex function. Furthermore, age-related declines in frontal cortex function are independent of those in hippocampal function. These results provide a possible basis for correlating age-related changes in neurobiological markers in frontal cortex with cognitive decline.     

Intact risk-based decision making in rats with prefrontal or accumbens dopamine depletion

The medial prefrontal cortex (mPFC) and the core region of the nucleus accumbens (AcbC) are key regions of a neural system that subserves risk-based decision making. Here, we examined whether dopamine (DA) signals conveyed to the mPFC and AcbC are critical for risk-based decision making. Rats with 6-hydroxydopamine or vehicle infusions into the mPFC or AcbC were examined in an instrumental task demanding probabilistic choice. In each session, probabilities of reward delivery after pressing one of two available levers were signaled in advance in forced trials followed by choice trials that assessed the animal??s preference. The probabilities of reward delivery associated with the large/risky lever declined systematically across four consecutive blocks but were kept constant within four subsequent daily sessions of a particular block. Thus, in a given session, rats need to assess the current value associated with the large/risky versus small/certain lever and adapt their lever preference accordingly. Results demonstrate that the assessment of within-session reward probabilities and probability discounting across blocks were not altered in rats with mPFC and AcbC DA depletions, relative to sham controls. These findings suggest that the capacity to evaluate the magnitude and likelihood of rewards associated with alternative courses of action seems not to rely on intact DA transmission in the mPFC or AcbC.     

Punishment and the potential for negative reinforcement with histamine injection

The present study examined punishment of responding with histamine injection, and its potential to generate avoidance of punishment. Sprague–Dawley rats were trained under concurrent schedules in which responses on one lever (the punishment lever) produced food under a variable‐interval schedule, and under some conditions intermittent injections of histamine, which suppressed behavior. Responses on a second (avoidance) lever prevented histamine injections scheduled on the punishment lever. After stabilization of punished responding, a variable‐interval 15‐s schedule of cancellation of histamine (avoidance) was added for responding on the second/avoidance lever, without subsequent acquisition of responding on that lever. Progressive decreases in the length of the punishment variable‐interval schedule increased suppression on the punishment lever without increases in response rates on the avoidance lever. Exchanging contingencies on the levers ensured that response rates on the avoidance lever were sufficiently high to decrease the histamine injection frequency; nonetheless response rates on the avoidance lever decreased over subsequent sessions. Under no condition was responding maintained on the avoidance lever despite continued punishing effectiveness of histamine throughout. The present results suggest that avoidance conditioning is not a necessary condition for effective punishment, and confirm the importance of empirical rather than presumed categorization of behavioral effects of stimulus events.     

Duration-reduction of avoidance sessions as negative reinforcement

Five rats were exposed to a shock-postponement procedure in which responses on each of two levers initially had equivalent effects. After an initial training sequence that ensured at least some responding on each lever, an additional consequence was made conjointly operative on the previously less-preferred lever for each animal. Each response on this lever continued to postpone shock, but also reduced the session duration by one minute. The conjoint contingencies were operative until, through session-shortening responses and the passage of time, the session was scheduled to end in two minutes; during the final two minutes the session-shortening contingency was disabled while the shock-postponement contingency continued to be operative on both levers. When responding shifted to a predominance on the session-shortening lever, the conjoint contingency was shifted to the other lever; for four of the five rats this reversal was followed by two additional reversals. Two of the rats' responding showed clear, strong, and unambiguous sensitivity to the session-shortening contingency. The responding of two others was also systematically controlled by that contingency, but the effects were less clearcut. The fifth animal showed an initial shift when session-shortening was introduced, but its subsequent behavior proved insensitive to reversals of procedure. The results clearly indicate a sensitivity of behavior to events on a time scale quite distinct from that of immediate consequences. They also support an interpretation of avoidance sessions, considered in their entirety, as events whose contingent relationship to behavior can affect that behavior—even in the absence of stimuli that delineate those relationships. Finally, these results support an interpretation of aversively based conditioning within a broader context, analogous to the “open versus closed economy” interpretation of appetitively controlled behavior.     

Extinction of a running response as a function of reinforcement magnitude

Four groups of rats were trained on different sucrose solutions in a straight runway. Terminal running speed was a monotonic function of reinforcement magnitude. After training each group was subdivided, one subgroup being extinguished under spaced, the other under massed conditions. In spaced extinction the animals trained on non extreme reward magnitudes showed most resistance to extinction. It was concluded that resistance to extinction is an inverted U-shaped function of reinforcement magnitude found in training. The massed extinction trials were conducted with a very short inter-trial interval. The animals showed an immediate drop in running speed followed by a gradual recovery and a subsequent decline. The number of trials taken to reach the peak recovery speed was a function of reinforcement magnitude found in training. Results on both massed and spaced extinction trials were interpreted in terms of the facilitatory and inhibitory effects of momentary and conditioned frustration.     

The roles of expectancy and aftereffects in alternation

In each of three experiments rats were pretreated in straight alleys to discriminate the brightness of an alley as signal for the magnitude of reward. Then the rats were tested in E-shaped mazes in which reward varied across trials and in which the brightness of the stem signaled the magnitude of reward. In all three experiments the rats alternated more on trials following a small reward on the previous trial, and in the third experiment the rats alternated more on those trials where a small reward was signaled. Rats choice responses on Trial n were also affected by the stimulus/reward and choice on Trial n − 2 and by the relationship between stimuli/rewards on Trials n − 2 and n. These results were interpreted as a tendency by the rat to change a less satisfactory situation by varying its response, and to retain a more satisfactory situation by perseverating in its choice response.     

Runway acquisition and extinction as a joint function of magnitude of reward and percentage of rewarded acquisition trials

To determine the joint effects of partial reward and reward magnitude on acquisition and extinction rates, and on acquisition and extinction asymptotes, 215 Wistar albino rats were trained in a Hunter straight runway. The experimental design was a 4 × 4 × 2 factorial combining four reward magnitudes, four reward percentages, and two experimenters. The data revealed that the acquisition rate was an increasing function of both percentage and magnitude of reward and that neither reward magnitude nor percentage of reward significantly affected acquisition asymptote. For extinction, it was found that, for continuous schedules, the larger the reward magnitude the less the resistance to extinction and, for partial schedules, the larger the reward magnitude the greater the resistance to extinction. These results were interpreted within the framework of the sequential effects hypothesis (Capaldi, 1966).     

Learned preferences induced by electrical stimulation of a food-related area of the parabrachial complex: effects of naloxone

Electrical stimulation of the External Lateral Parabrachial Subnucleus (LPBe), a food-related area, induced behavioral preferences for associated stimuli in a taste discrimination learning task. Although this stimulation appeared to be ineffective to elicit standard lever press self-stimulation, it induced place preference for one of two training compartments of a rectangular maze in which animals (adult male Wistar rats) received concurrent electrical brain stimulation. In subjects that consistently showed a preference behavior in different trials, administration of the opioid antagonist naloxone (4 mg/ml/kg) blocked concurrent learning when the test was made in a new maze but not in the same maze in which animals had learned the task. These results are discussed in terms of the possible participation of the LPBe subnucleus in different natural and artificial brain reward systems.