Involvement of the rat anterior cingulate cortex in control of instrumental responses guided by reward expectancy

The anterior cingulate cortex (ACC) plays a critical role in stimulus-reinforcement learning and reward-guided selection of actions. Here we conducted a series of experiments to further elucidate the role of the ACC in instrumental behavior involving effort-based decision-making and instrumental learning guided by reward-predictive stimuli. In Experiment 1, rats were trained on a cost-benefit T-maze task in which they could either choose to climb a barrier to obtain a high reward (four pellets) in one arm or a low reward (two pellets) in the other with no barrier present. In line with previous studies, our data reveal that rats with quinolinic acid lesions of the ACC selected the response involving less work and smaller reward. Experiment 2 demonstrates that breaking points of instrumental performance under a progressive ratio schedule were similar in sham-lesioned and ACC-lesioned rats. Thus, lesions of the ACC did not interfere with the effort a rat is willing to expend to obtain a specific reward in this test. In a subsequent task, we examined effort-based decision-making in a lever-press task where rats had the choice between pressing a lever to receive preferred food pellets under a progressive ratio schedule, or free feeding on a less preferred food, i.e. lab chow. Results show that sham- and ACC-lesioned animals had similar breaking points and ingested comparable amounts of less-preferred food. Together, the results of Experiment 1 and 2 suggest that the ACC plays a role in evaluating how much effort to expend for reward; however, the ACC is not necessary in all situations requiring an assessment of costs and benefits. In Experiment 3 we investigated learning and reversal learning of instrumental responses guided by reward predictive stimuli. A reaction time (RT) task demanding conditioned lever release was used in which the upcoming reward magnitude (five vs. one food pellet) was signalled in advance by discriminative visual stimuli. Results revealed that rats with ACC lesions were able to discriminate reward magnitude-predictive stimuli and to adapt instrumental behavior to reversed stimulus-reward magnitude contingencies. Thus, in a simple discrimination task as used here, the ACC appears not to be required to discriminate reward magnitude-predictive stimuli and to use the learned significance of the stimuli to guide instrumental behavior.     

Size and reversal learning in the beagle dog as a measure of executive function and inhibitory control in aging

Several studies converge on the idea that executive processes age earlier than other cognitive processes. As part of a larger effort to investigate age-related changes in executive processes in the dog, inhibitory control was measured in young, middle-aged, old, and senior dogs using size discrimination learning and reversal procedures. Compared to young and middle-aged dogs, old and senior dogs were impaired on both the initial learning of the size task and the reversal of original reward contingencies. Impaired performance in the two aged groups was characterized as a delay in learning the correct stimulus-reward contingencies and, among the senior dogs in particular, an increase in perseverative responding. These separate patterns of reversal impairments in the old and senior dogs may reflect different rates of aging in subregions of the frontal cortex.     

Nonspatial attentional capture by previously rewarded scene semantics

Recent research has shown that reward influences visual perception and cognition in a way that is distinct from the well-documented goal-directed mechanisms. In the current study we explored how task-irrelevant stimulus-reward associations affect processing of stimuli when attention is constrained and reward no longer delivered. During a training phase, participants performed a choice game, exposing them to different reward schedules for different semantic categories of natural scene pictures. In a separate test session in which no additional reward was provided, the differentially rewarded scene categories were used as task-irrelevant non-targets in a rapid serial visual presentation (RSVP) task. Participants performed a detection task on a previously non-rewarded target category. Results show that participants' sensitivity index (d′) for target detection decreased as a function of whether the reward was coupled to the distractor category during training. The semantic category of natural scenes associated with high reward caused more interference in target detection than the semantic category associated with a low reward. The same was found when new, unseen distractor pictures of the same semantic categories were used. The present findings suggest that reward can be selectively associated with high level scene semantics. We argue that learned stimulus-reward associations persistently bias perceptual processing independently of spatial shifts in attention and the immediate prospect of reward.     

The role of the GABA(A) agonist muscimol on memory performance: reward contingencies determine the nature of the deficit

A matching-to-position (MTP) paradigm was altered to influence the type of associations a rat would use to solve the task. Our main behavioral manipulation was the application of the differential outcomes procedure (DOP). The DOP involves correlating each to-be-remembered event with a distinct reward condition. This procedure results in the development of unique reward expectancies that enhance and guide choice behavior. Such distinct reward expectancies are not formed when either a common or random assignment of reward is used (a non-differential outcomes procedure [NOP]). Intracerebroventricular infusions of the amnestic agent muscimol (GABA(A) agonist) or aCSF were delivered to male rats trained on a delayed MTP task that implemented either the DOP or the NOP. Muscimol impaired performance in a dose dependent fashion in both groups--but the nature of the deficit differed as a function of reinforcement contingencies. Rats trained with the DOP displayed a non-mnemonic delay-independent impairment: performance at all delay intervals was disrupted. In contrast, NOP-trained rats displayed a delay-dependent impairment demonstrating that muscimol can also have memory-disrupting effects. The difference in pattern of impairment appears to be a function of the associations formed during training and the type of cognitive strategies involved in maintaining behavior on a conditional delayed discrimination task when reinforcement contingencies are varied. Thus, these results demonstrate that increasing GABA(A) receptor activation impairs a range of associative and memory functions.     

Pavlovian reward learning underlies value driven attentional capture

Recent evidence shows that distractors that signal high compared to low reward availability elicit stronger attentional capture, even when this is detrimental for task-performance. This suggests that simply correlating stimuli with reward administration, rather than their instrumental relationship with obtaining reward, produces value-driven attentional capture. However, in previous studies, reward delivery was never response independent, as only correct responses were rewarded, nor was it completely task-irrelevant, as the distractor signaled the magnitude of reward that could be earned on that trial. In two experiments, we ensured that associative reward learning was completely response independent by letting participants perform a task at fixation, while high and low rewards were automatically administered following the presentation of task-irrelevant colored stimuli in the periphery (Experiment 1) or at fixation (Experiment 2). In a following non-reward test phase, using the additional singleton paradigm, the previously reward signaling stimuli were presented as distractors to assess truly task-irrelevant value driven attentional capture. The results showed that high compared to low reward-value associated distractors impaired performance, and thus captured attention more strongly. This suggests that genuine Pavlovian conditioning of stimulus-reward contingencies is sufficient to obtain value-driven attentional capture. Furthermore, value-driven attentional capture can occur following associative reward learning of temporally and spatially task-irrelevant distractors that signal the magnitude of available reward (Experiment 1), and is independent of training spatial shifts of attention towards the reward signaling stimuli (Experiment 2). This confirms and strengthens the idea that Pavlovian reward learning underlies value driven attentional capture.     

Orbital prefrontal cortex and guidance of instrumental behavior of rats by visuospatial stimuli predicting reward magnitude

The orbital prefrontal cortex (OPFC) is part of a circuitry mediating the perception of reward and the initiation of adaptive behavioral responses. We investigated whether the OPFC is involved in guidance of the speed of instrumental behavior by visuospatial stimuli predictive of different reward magnitudes. Unoperated rats, sham-lesioned rats, and rats with bilateral lesions of the OPFC by N-methyl-D-aspartate (NMDA) were trained in a visuospatial discrimination task. The task required a lever press on the illuminated lever of two available to obtain a food reward. Different reward magnitudes were permanently assigned to lever presses to respective sides of the operant chamber; that is, responses to one lever (e.g., the left one) were always rewarded with one pellet and responses to the other lever with five pellets. On each trial, the position of the illuminated lever was pseudorandomly determined in advance. Results revealed that OPFC lesions did not impair acquisition of the task, as the speed of conditioned responses was significantly shorter with expectancy of a high reward magnitude. In addition, during reversal, shift and reshift of lever position–reward magnitude contingencies and under extinction conditions, performance of the OPFC-lesioned and control groups did not differ. It is concluded that the OPFC in rats might not be critical for adapting behavioral responses to changes of stimulus–reward magnitude contingencies signaled by visuospatial cues.     

Failure to exploit learned spatial value information during visual search

ABSTRACT

In recent years there has been rapid proliferation of studies demonstrating how reward learning guides visual search. However, most of these studies have focused on feature-based reward, and there has been scant evidence supporting the learning of space-based reward. We raise the possibility that the visual search apparatus is impenetrable to spatial value contingencies, even when such contingencies are learned and represented online in a separate knowledge domain. In three experiments, we interleaved a visual choice task with a visual search task in which one display quadrant produced greater monetary rewards than the remaining quadrants. We found that participants consistently exploited this spatial value contingency during the choice task but not during the search task – even when these tasks were interleaved within the same trials and when rewards were contingent on response speed. These results suggest that the expression of spatial value information is task specific and that the visual search apparatus could be impenetrable to spatial reward information. Such findings are consistent with an evolutionary framework in which the search apparatus has little to gain from spatial value information in most real world situations.     

Lack of inhibition: a motivational deficit in children with attention deficit/hyperactivity disorder and children with traumatic brain injury.

Recent research has demonstrated that both brain-injured children and children with attention deficit/hyperactivity disorder (ADHD) suffer from response inhibition deficits. To investigate whether these deficits can be influenced by motivational factors, the stop-signal task was performed with and without reward contingencies for successful inhibition. Three groups of children between 8 and 12 years of age, participated in the study: 31 children with ADHD, 37 with traumatic brain injuries (TBI), and 26 normal controls. Results indicated that, although all groups showed comparable learning effects, reward contingencies had different effects on the groups. Whereas the performance of children with ADHD under reward contingencies were brought up to the performance level of normal controls, rewards were found less effective at improving response inhibition in children with TBI. The results further support a motivational/energetic explanation of the inhibitory deficit in children with ADHD, and of a primary response inhibition deficit due to structural brain damage in children with TBI.     

Lack of Inhibition: A Motivational Deficit in Children With Attention Deficit/Hyperactivity Disorder and Children With Traumatic Brain Injury*

Recent research has demonstrated that both brain-injured children and children with attention deficit/hyperactivity disorder (ADHD) suffer from response inhibition deficits. To investigate whether these deficits can be influenced by motivational factors, the stop-signal task was performed with and without reward contingencies for successful inhibition. Three groups of children between 8 and 12 years of age, participated in the study: 31 children with ADHD, 37 with traumatic brain injuries (TBI), and 26 normal controls. Results indicated that, although all groups showed comparable learning effects, reward contingencies had different effects on the groups. Whereas the performance of children with ADHD under reward contingencies were brought up to the performance level of normal controls, rewards were found less effective at improving response inhibition in children with TBI. The results further support a motivational/energetic explanation of the inhibitory deficit in children with ADHD, and of a primary response inhibition deficit due to structural brain damage in children with TBI.     

场景在价值驱动注意捕获中的作用

本研究采用训练-测试范式考察场景信息在价值驱动注意捕获中的作用。实验1训练阶段将唯一性的场景信息与奖励建立联结,测试阶段发现只有当分心刺激出现在高奖励场景中才能捕获注意;实验2训练阶段将场景和颜色信息联合但分别独立预测奖励,结果发现高奖励颜色出现在任何场景中都能捕获注意。研究结果表明:(1)场景-价值联结能够引发注意捕获效应;(2)与场景相比,颜色特征在价值驱动的注意捕获中具有优势效应。     

Don’t let it distract you: how information about the availability of reward affects attentional selection

Previous research has shown that attentional selection is affected by reward contingencies: previously selected and rewarded stimuli continue to capture attention even if the reward contingencies are no longer in place. In the current study, we investigated whether attentional selection also is affected by stimuli that merely signal the magnitude of reward available on a given trial but, crucially, have never had instrumental value. In a series of experiments, we show that a stimulus signaling high reward availability captures attention even when that stimulus is and was never physically salient or part of the task set, and selecting it is harmful for obtaining reward. Our results suggest that irrelevant reward-signaling stimuli capture attention, because participants have learned about the relationship between the stimulus and reward. Importantly, we only observed learning after initial attentional prioritization of the reward signaling stimulus. We conclude that nonsalient, task-irrelevant but reward-signaling stimuli can affect attentional selection above and beyond top-down or bottom-up attentional control, however, only after such stimuli were initially prioritized for selection.     

Contingency awareness,partial reinforcement effects,and curiosity motivation in children

Third- and fourth-grade children were given a two-choice discrimination learning task, designed to associate each of three syllables with a different reward schedule: 100% reward, 50% reward, and 0% reward. Subsequent to the conditioning phase, measures were made of the subject's awareness of the reward contingencies associated with the syllables, as well as the pleasantness and curiosity which these syllables had acquired during the conditioning. The results showed that a partially rewarded cue was evaluated as somewhat less “pleasant” than a continuously rewarded cue, but evoked more “curiosity” than either the continuous or nonrewarded cues. These effects emerged only for the contingency aware groups.     

Reward anticipation enhances brain activation during response inhibition

The chance to achieve a reward starts up the required neurobehavioral mechanisms to adapt our thoughts and actions in order to accomplish our objective. However, reward does not equally reinforce everybody but depends on interindividual motivational dispositions. Thus, immediate reward contingencies can modulate the cognitive process required for goal achievement, while individual differences in personality can affect this modulation. We aimed to test the interaction between inhibition-related brain response and motivational processing in a stop signal task by reward anticipation and whether individual differences in sensitivity to reward (SR) modulate such interaction. We analyzed the cognitive–motivational interaction between the brain pattern activation of the regions involved in correct and incorrect response inhibition and the association between such brain activations and SR scores. We also analyzed the behavioral effects of reward on both reaction times for the “go” trials before and after correct and incorrect inhibition in order to test error prediction performance and postinhibition adjustment. Our results show enhanced activation during response inhibition under reward contingencies in frontal, parietal, and subcortical areas. Moreover, activation of the right insula and the left putamen positively correlates with the SR scores. Finally, the possibility of reward outcome affects not only response inhibition performance (e.g., reducing stop signal reaction time), but also error prediction performance and postinhibition adjustment. Therefore, reward contingencies improve behavioral performance and enhance brain activation during response inhibition, and SR is related to brain activation. Our results suggest the conditions and factors that subserve cognitive control strategies in cognitive motivational interactions during response inhibition.     

Histidine-decarboxylase knockout mice show deficient nonreinforced episodic object memory, improved negatively reinforced water-maze performance, and increased neo- and ventro-striatal dopamine turnover

The brain's histaminergic system has been implicated in hippocampal synaptic plasticity, learning, and memory, as well as brain reward and reinforcement. Our past pharmacological and lesion studies indicated that the brain's histamine system exerts inhibitory effects on the brain's reinforcement respective reward system reciprocal to mesolimbic dopamine systems, thereby modulating learning and memory performance. Given the close functional relationship between brain reinforcement and memory processes, the total disruption of brain histamine synthesis via genetic disruption of its synthesizing enzyme, histidine decarboxylase (HDC), in the mouse might have differential effects on learning dependent on the task-inherent reinforcement contingencies. Here, we investigated the effects of an HDC gene disruption in the mouse in a nonreinforced object exploration task and a negatively reinforced water-maze task as well as on neo- and ventro-striatal dopamine systems known to be involved in brain reward and reinforcement. Histidine decarboxylase knockout (HDC-KO) mice had higher dihydrophenylacetic acid concentrations and a higher dihydrophenylacetic acid/dopamine ratio in the neostriatum. In the ventral striatum, dihydrophenylacetic acid/dopamine and 3-methoxytyramine/dopamine ratios were higher in HDC-KO mice. Furthermore, the HDC-KO mice showed improved water-maze performance during both hidden and cued platform tasks, but deficient object discrimination based on temporal relationships. Our data imply that disruption of brain histamine synthesis can have both memory promoting and suppressive effects via distinct and independent mechanisms and further indicate that these opposed effects are related to the task-inherent reinforcement contingencies.     

Stimulus-driven and goal-driven effects on Pavlovian associative reward learning

It has been shown that pure Pavlovian associative reward learning can elicit value-driven attentional capture. However, in previous studies, task-irrelevant and response-independent reward-signalling stimuli hardly competed for visual selective attention. Here we put Pavlovian reward learning to the test by manipulating the extent to which bottom-up (Experiment 1) and top-down (Experiment 2) processes were involved in this type of learning. In Experiment 1, the stimulus, the colour of which signalled the magnitude of the reward given, was presented simultaneously with another randomly coloured stimulus, so that it did not capture attention in a stimulus-driven manner. In Experiment 2, observers performed an attentionally demanding RSVP-task at the centre of the screen to largely tax goal-driven attentional resources, while a task-irrelevant and response-independent stimulus in the periphery signalled the magnitude of the reward given. Both experiments showed value-driven attentional capture in a non-reward test phase, indicating that the reward-signalling stimuli were imbued with value during the Pavlovian reward conditioning phases. This suggests that pure Pavlovian reward conditioning can occur even when (1) competition prevents attention being automatically allocated to the reward-signalling stimulus in a stimulus-driven manner, and (2) attention is occupied by a demanding task, leaving little goal-driven attentional resources available to strategically select the reward-signalling stimulus. The observed value-driven attentional capture effects appeared to be similar for observers who could and could not explicitly report the stimulus–reward contingencies. Altogether, this study provides insight in the conditions under which mere stimulus–reward contingencies in the environment can be learned to affect future behaviour.     

Unconscious reward facilitates motion perceptual learning

Reward signal plays an important role in guiding human learning behaviour. Recent studies have provided evidence that reward signal modulates perceptual learning of basic visual features. Typically, the reward effects on perceptual learning were accompanied with consciously presented reward during the learning process. However, whether an unconsciously presented reward signal that minimizes the contribution of attentional and motivational factors can facilitate perceptual learning remains less well understood. We trained human subjects on a visual motion detection task and subliminally delivered a monetary reward for correct response during the training. The results showed significantly larger learning effect for high reward-associated motion direction than low reward-associated motion direction. Importantly, subjects could neither discriminate the relative values of the subliminal monetary reward nor correctly report the reward-direction contingencies. Our findings suggest that reward signal plays an important modulatory role in perceptual learning even if the magnitude of the reward was not consciously perceived.     

Differential effects of social and non‐social reward on response inhibition in children and adolescents

An important issue in the field of clinical and developmental psychopathology is whether cognitive control processes, such as response inhibition, can be specifically enhanced by motivation. To determine whether non‐social (i.e. monetary) and social (i.e. positive facial expressions) rewards are able to differentially improve response inhibition accuracy in typically developing children and adolescents, an ‘incentive’ go/no‐go task was applied with reward contingencies for successful inhibition. In addition, the impact of children's personality traits (such as reward seeking and empathy) on monetary and social reward responsiveness was assessed in 65 boys, ages 8 to 12 years. All subjects were tested twice: At baseline, inhibitory control was assessed without reward, and then subjects were pseudorandomly assigned to one of four experimental conditions, including (1) social reward only, (2) monetary reward only, (3) mixed social and monetary reward, or (4) a retest condition without reward. Both social and non‐social reward significantly improved task performance, although larger effects were observed for monetary reward. The higher the children scored on reward seeking scales, the larger was their improvement in response inhibition, but only if monetary reward was used. In addition, there was a tendency for an association between empathic skills and benefits from social reward. These data suggest that social incentives do not have an equally strong reinforcing value as compared to financial incentives. However, different personality traits seem to determine to what extent a child profits from different types of reward. Clinical implications regarding probable hyposensitivity to social reward in subjects with autism and dysregulated reward‐seeking behaviour in children with attention‐deficit/hyperactivity disorder (ADHD) are discussed.     

The effects of information-processing demands and social-situational factors on the social decision making of children

Previous research has consistently demonstrated age differences in cooperative, competitive, and individualistic social decision making. The present studies were designed to demonstrate that these age differences may result from the development of the requisite information-processing capabilities and social/situational factors. It was hypothesized that the limited information-processing systems of young children place constraints upon the social decisions available, but making a specific social decision was dependent upon sociocultural factors which influence the salience of aspects of the social situation or the expected contingencies of each social decision. Study 1 assessed the social decision making of 3- to 11-year-olds in two forms of a decision-making task (designed to manipulate the information processing demands of key social decisions) under two instructional sets (designed to manipulate the expected reward contingencies of key social decisions). Study 2 assessed the social decision making of 3- to 12-year-olds toward close friends or acquaintances (designed to manipulate the expected social contingencies) in the same two forms of the decision-making task. Age, the task form manipulation, and the social/situational manipulation influenced social decision making in each study in a manner consistent with the theoretical perspective A cognitive social learning interpretation, and the implications for the cognitive developmental influence upon social behavior, are discussed.     

Analogical reasoning and the differential outcome effect: transitory bridging of the conceptual gap for rhesus monkeys (Macaca mulatta)

Monkeys, unlike chimpanzees and humans, have a marked difficulty acquiring relational matching-to-sample (RMTS) tasks that likely reflect the cognitive foundation upon which analogical reasoning rests. In the present study, rhesus monkeys (Macaca mulatta) completed a categorical (identity and nonidentity) RMTS task with differential reward (pellet ratio) and/or punishment (timeout ratio) outcomes for correct and incorrect choices. Monkeys in either differential reward-only or punishment-only conditions performed at chance levels. However, the RMTS performance of monkeys experiencing both differential reward and punishment conditions was significantly better than chance. Subsequently when all animals experienced nondifferential outcomes tests, their RMTS performance levels were at chance. These results indicate that combining differential reward and punishment contingencies provide an effective, albeit transitory, scaffolding for monkeys to judge analogical relations-between-relations.     

The modulation of reward priority by top-down knowledge

Reward-associated features capture attention automatically and continue to do so even when the reward contingencies are removed. This profile has led to the hypothesis that rewards belong to a separate class of attentional biases that is neither typically top-down nor bottom-up. The goal of these experiments was to understand the degree to which top-down knowledge can modulate value-driven attentional capture within (a) the timecourse of a single trial and (b) when the reward contingencies change explicitly over trials. The results suggested that top-down knowledge does not affect the size of value-driven attentional capture within a single trial. There were clear top-down modulations in the magnitude of value-driven capture when reward contingencies explicitly changed, but the original reward associations continued to have a persistent bias on attention. These results contribute to a growing body of evidence that reward associations bias attention through mechanisms separate from other top-down and bottom-up attentional biases.