Dopamine D1 receptors in the anterior cingulate cortex regulate effort-based decision making

The anterior cingulate cortex (ACC) has been implicated in encoding whether or not an action is worth performing in view of the expected benefit and the cost of performing the action. Dopamine input to the ACC may be critical for this form of effort-based decision making; however, the role of distinct ACC dopamine receptors is yet unknown. Therefore, we examined in rats the effects of an intra-ACC D1 and D2 receptor blockade on effort-based decision making tested in a T-maze cost-benefit task. In this task, subjects could either choose to climb a barrier to obtain a high reward in one arm or a low reward in the other arm without a barrier. Unlike vehicle-treated rats, rats with intra-ACC infusion of the D1 receptor antagonist SCH23390 exhibited a reduced preference for the high-cost- high-reward response option when having the choice to obtain a low reward with little effort. In contrast, in rats with intra-ACC infusion of the D2 receptor antagonist eticlopride, the preference for the high-cost-high-reward response option was not altered relative to vehicle-treated rats. These data provide the first evidence that D1 receptors in the ACC regulate effort-based decision making.     

Reward-based contextual learning supported by anterior cingulate cortex

The anterior cingulate cortex (ACC) is commonly associated with cognitive control and decision making, but its specific function is highly debated. To explore a recent theory that the ACC learns the reward values of task contexts (Holroyd & McClure in Psychological Review, 122, 54–83, 2015; Holroyd & Yeung in Trends in Cognitive Sciences, 16, 122–128, 2012), we recorded the event-related brain potentials (ERPs) from participants as they played a novel gambling task. The participants were first required to select from among three games in one “virtual casino,” and subsequently they were required to select from among three different games in a different virtual casino; unbeknownst to them, the payoffs for the games were higher in one casino than in the other. Analysis of the reward positivity, an ERP component believed to reflect reward-related signals carried to the ACC by the midbrain dopamine system, revealed that the ACC is sensitive to differences in the reward values associated with both the casinos and the games inside the casinos, indicating that participants learned the values of the contexts in which rewards were delivered. These results highlight the importance of the ACC in learning the reward values of task contexts in order to guide action selection.     

The Role of Consistent Context in Rapid Movement Planning: Suboptimal Endpoint Adjustment to Changing Rewards

The authors' purpose was to determine if participants adjusted their endpoint during a rapid aiming task in the context of changing rewards and whether participants needed consistent feedback to do so. Participants aimed to a target that was overlapped by a penalty region. Participants gained points for hitting the target but lost points for hitting the penalty region. The reward value associated with target contact either changed trial to trial, reducing consistent feedback (variable condition) or changed between blocks of trials (blocked condition) with the repetition of reward value within a block increasing the consistency of feedback. Participants adjusted their endpoints with changing reward value in the blocked but not variable condition indicating consistent feedback is needed to adjust endpoint to changing rewards.     

The relationship between outcome prediction and cognitive fatigue: A convergence of paradigms

Cognitive fatigue is common after strenuous cognitive effort. A large body of literature has implicated a network of brain areas in fatigue, including the basal ganglia and cortical areas including ventro-medal prefrontal cortex and anterior cingulate cortex (ACC). Furthermore, the ACC has been shown to be involved in processes such as error and conflict monitoring, outcome prediction, and effort processing. Thus, the ACC appears to be one common denominator between clinical work on fatigue and research on outcome prediction and effort. In the present study, we examined whether the same region of the ACC is activated during the processing of errors and fatigue. Cognitive fatigue was induced by having subjects perform a difficult working memory task, during which they rated on-task fatigue. Activation associated with error processing was determined by using error trials on the working memory task. After localizing the region engaged in error processing, we evaluated whether there was a relationship between BOLD activation of that region and on-task fatigue scores. The results showed that as subjects became more fatigued, they responded with longer latencies and increased accuracy for the more difficult task. Moreover, the ACC areas that were activated by error processing were also associated with fatigue. These results suggest that cognitive fatigue may be related to changes in effort and reward. We speculate that as the brain detects these changes, cognitive fatigue is generated as a way for the brain to signal itself that the effort required for the task no longer merits the rewards received for performing it.     

Involvement of the rat anterior cingulate cortex in control of instrumental responses guided by reward expectancy

The anterior cingulate cortex (ACC) plays a critical role in stimulus-reinforcement learning and reward-guided selection of actions. Here we conducted a series of experiments to further elucidate the role of the ACC in instrumental behavior involving effort-based decision-making and instrumental learning guided by reward-predictive stimuli. In Experiment 1, rats were trained on a cost-benefit T-maze task in which they could either choose to climb a barrier to obtain a high reward (four pellets) in one arm or a low reward (two pellets) in the other with no barrier present. In line with previous studies, our data reveal that rats with quinolinic acid lesions of the ACC selected the response involving less work and smaller reward. Experiment 2 demonstrates that breaking points of instrumental performance under a progressive ratio schedule were similar in sham-lesioned and ACC-lesioned rats. Thus, lesions of the ACC did not interfere with the effort a rat is willing to expend to obtain a specific reward in this test. In a subsequent task, we examined effort-based decision-making in a lever-press task where rats had the choice between pressing a lever to receive preferred food pellets under a progressive ratio schedule, or free feeding on a less preferred food, i.e. lab chow. Results show that sham- and ACC-lesioned animals had similar breaking points and ingested comparable amounts of less-preferred food. Together, the results of Experiment 1 and 2 suggest that the ACC plays a role in evaluating how much effort to expend for reward; however, the ACC is not necessary in all situations requiring an assessment of costs and benefits. In Experiment 3 we investigated learning and reversal learning of instrumental responses guided by reward predictive stimuli. A reaction time (RT) task demanding conditioned lever release was used in which the upcoming reward magnitude (five vs. one food pellet) was signalled in advance by discriminative visual stimuli. Results revealed that rats with ACC lesions were able to discriminate reward magnitude-predictive stimuli and to adapt instrumental behavior to reversed stimulus-reward magnitude contingencies. Thus, in a simple discrimination task as used here, the ACC appears not to be required to discriminate reward magnitude-predictive stimuli and to use the learned significance of the stimuli to guide instrumental behavior.     

Impaired reward processing by anterior cingulate cortex in children with attention deficit hyperactivity disorder

Decades of research have examined the neurocognitive mechanisms of cognitive control, but the motivational factors underlying task selection and performance remain to be elucidated. We recently proposed that anterior cingulate cortex (ACC) utilizes reward prediction error signals carried by the midbrain dopamine system to learn the value of tasks according to the principles of hierarchical reinforcement learning. According to this position, disruption of the ACC–dopamine interface can disrupt the selection and execution of extended, task-related behaviors. To investigate this issue, we recorded the event-related brain potential (ERP) from children with attention deficit hyperactivity disorder (ADHD), which is strongly associated with ACC–dopamine dysfunction, and from typically developing children while they navigated a simple “virtual T-maze” to find rewards. Depending on the condition, the feedback stimuli on each trial indicated that the children earned or failed to earn either money or points. We found that the reward positivity, an ERP component proposed to index the impact of dopamine-related reward signals on ACC, was significantly larger with money feedback than with points feedback for the children with ADHD, but not for the typically developing children. These results suggest that disruption of the ACC–dopamine interface may underlie the impairments in motivational control observed in childhood ADHD.     

Visual selective attention and the effects of monetary rewards

Outcomes of actions, in the form of rewards and punishments, are known to shape behavior. For example, an action followed by reward will be more readily elicited on subsequent encounters with the same stimuli and context -- a phenomenon known as the law of effect. These consequences of rewards (and punishments) are important because they reinforce adaptive behaviors at the expense of competing ones, thus increasing fitness of the organism in its environment. However, it is unknown whether similar influences regulate covert mental processes, such as visual selective attention. Visual selective attention allows privileged processing of task-relevant information, while inhibiting distracting contextual elements. Using variable monetary rewards as arbitrary feedback on performance, we tested whether acts of attentional selection, and in particular the resulting aftereffects, can be modulated by their consequences. Results show that the efficacy of visual selective attention can be sensibly adjusted by external feedback. Specifically, although lingering inhibition of distractors is robust after highly rewarded selections, it is eliminated after poorly rewarded selections. This powerful feature of visual selective attention provides attentive processes with both flexibility and self-regulation properties.     

Posterior cingulate cortex: adapting behavior to a changing world

When has the world changed enough to warrant a new approach? The answer depends on current needs, behavioral flexibility and prior knowledge about the environment. Formal approaches solve the problem by integrating the recent history of rewards, errors, uncertainty and context via Bayesian inference to detect changes in the world and alter behavioral policy. Neuronal activity in posterior cingulate cortex - a key node in the default network - is known to vary with learning, memory, reward and task engagement. We propose that these modulations reflect the underlying process of change detection and motivate subsequent shifts in behavior.     

Cooperative problem solving by tufted capuchin monkeys (Cebus apella): spontaneous division of labor, communication, and reciprocal altruism

Using an experimentally induced cooperation task, the authors investigated whether tufted capuchin monkeys (Cebus apella) share the following 3 characteristics of cooperation with humans: division of labor, communication, and reciprocal altruism. In Experiment 1, the authors trained individual monkeys to perform the necessary sequence of actions for rewards and tested them in pairs to assess whether they could solve the task by spontaneously dividing the sequence of actions. All pairs solved this task. In Experiment 2, monkeys worked in the cooperation task and a task requiring no partner help. They looked at the partner significantly longer in the former task than in the latter, but communicative intent could not be determined. In Experiment 3, only 1 of 2 participants obtained a reward on each trial. Monkeys maintained cooperation when their roles were reversed on alternate trials. Their cooperative performances demonstrated division of labor; results suggest task-related communication and reciprocal altruism.     

Post-learning infusion of anisomycin into the anterior cingulate cortex impairs instrumental acquisition through an effect on reinforcer valuation

The integrity of the rodent anterior cingulate cortex (ACC) is essential for various aspects of instrumental behavior, but it is not clear if the ACC is important for the acquisition of a simple instrumental response. Here, it was demonstrated that post-session infusions of anisomycin into the rat ACC completely prevented the acquisition of instrumental responding. The experimental use of post-session intracranial infusions of plasticity inhibitors is assumed to affect local consolidation of plasticity, but not behavioral task performance. However, in associative appetitive conditioning, post-session intracranial infusion of pharmaco-active compounds could actually interfere with subsequent task performance indirectly through retrospective effects on the valuation of ingested rewards. Thus, it was subsequently demonstrated that the intracranial infusion of anisomycin into the ACC after sucrose pellet consumption significantly reduced subsequent pellet consumption, suggesting that the infusion of anisomycin into the ACC produced conditioned taste avoidance. In the third experiment, an innovative procedure was introduced that dissociated the effects of intracranial infusions after conditioning sessions on task-learning and unconditioned stimulus valuation. With this procedure, the infusion of anisomycin into the ACC after instrumental sessions did not affect instrumental reinforcer valuation or the acquisition of instrumental responding, suggesting that plasticity in the ACC is not necessary for the acquisition of instrumental behavior.The anterior cingulate cortex (ACC) in the rodent brain is the area of association cortex that is most intimately associated with movement control, as it has direct corticospinal projections to motor neurons (Miller 1987) and is intricately connected with motor and premotor cortex (Brecht et al. 2004; Wang et al. 2008), and movement-related discharges in motivated tasks have been observed in the ACC (Jung et al. 1998; Kargo et al. 2007). On the basis of these findings, the ACC has been described as the limbic motor cortex (Craig 2003). Indeed, it has been demonstrated that neural activity in the ACC codes for performed actions and observed appetitive outcomes (Lapish et al. 2008), suggesting that the ACC could be important for instrumental conditioning. However, it is currently unclear whether the role of the ACC in instrumental conditioning is restricted to relatively complex instrumental tasks with response conflict (Lapish et al. 2008) and high response efforts (Rudebeck et al. 2006), or whether it extends generally to the acquisition of a simple appetitively motivated instrumental response as well. For example, it has been reported that the acquisition of instrumental responding under variable ratio 2 schedule of reinforcement was impaired after presession intra-ACC infusions of the N-methyl-d-aspartate (NMDA) receptor antagonist AP-5 (McKee et al. 2007).However, the practical investigation of the neural basis of learning is complicated by what can be considered as “side effects” of manipulations that are intended to interfere selectively with learning mechanisms. Both pretraining lesions and presession infusions of pharmacological agents into specific brain regions can produce interference with basic task performance in addition to impairing task-related learning. This problem can be overcome by the use of infusions of pharmacological agents after a behavioral conditioning session, specifically to target the consolidation of learning. Positive effects of post-session intracranial infusions of protein synthesis inhibitors such as anisomycin (Rosenblum et al. 1993), or NMDA, and dopamine receptor antagonists such as AP-5 or SCH 23390 (Dalley et al. 2005) on the acquisition of conditioned responses can be interpreted more confidently in terms of interference with learning, at least when combined with negative effects in delayed-infusion control groups. Certainly, positive effects of post-session infusions on learning in the absence of effects of delayed control infusions cannot be attributed to interference with task performance.Nevertheless, post-session intracranial infusions may still cause confounding effects on conditioned behavior in appetitive conditioning experiments, by changing the valuation of the unconditioned stimulus (US), or reward, in subsequent sessions. Such manipulations could reduce the valuation of rewards either through interference with consolidation or reconsolidation of the sensory or motivational properties of the US (Wang et al. 2005; Pedroza-Llinás et al. 2009), which would lead to persistent neophobia, or through retrospective devaluation of the task reward through the post-ingestion induction of an aversive state. Neither effect would be expected to be apparent in delayed-infusion control conditions. Furthermore, considering that animals typically earn far fewer rewards in conditioning sessions than they would maximally consume if rewards were freely available, such effects would not necessarily be visible in terms of consumption of earned rewards and unaffected latencies to collect rewards.In view of these considerations, the effect of the infusion of anisomycin into the rat ACC after instrumental sessions on the acquisition of conditioned responding was investigated. Based on the obtained results, it was further investigated whether the infusion of anisomycin into the ACC after free consumption of a novel reward would affect subsequent pellet consumption. Finally, a third experiment was conducted that introduced an innovative procedure that was designed to dissociate the effects of intracranial infusion of protein synthesis inhibitors after conditioning sessions on task-learning and US valuation.     

A test of the reward-value hypothesis

Rats retain source memory (memory for the origin of information) over a retention interval of at least 1 week, whereas their spatial working memory (radial maze locations) decays within approximately 1 day. We have argued that different forgetting functions dissociate memory systems. However, the two tasks, in our previous work, used different reward values. The source memory task used multiple pellets of a preferred food flavor (chocolate), whereas the spatial working memory task provided access to a single pellet of standard chow-flavored food at each location. Thus, according to the reward-value hypothesis, enhanced performance in the source memory task stems from enhanced encoding/memory of a preferred reward. We tested the reward-value hypothesis by using a standard 8-arm radial maze task to compare spatial working memory accuracy of rats rewarded with either multiple chocolate or chow pellets at each location using a between-subjects design. The reward-value hypothesis predicts superior accuracy for high-valued rewards. We documented equivalent spatial memory accuracy for high- and low-value rewards. Importantly, a 24-h retention interval produced equivalent spatial working memory accuracy for both flavors. These data are inconsistent with the reward-value hypothesis and suggest that reward value does not explain our earlier findings that source memory survives unusually long retention intervals.     

Effects of anonymous peer observation on adolescents' preference for immediate rewards

Research suggests that the presence of peers influences adolescent risk‐taking by increasing the perceived reward value of risky decisions. While prior work has involved observation of participants by their friends, the current study examined whether observation by an anonymous peer could elicit similarly increased reward sensitivity. Late adolescent participants completed a delay discounting task either alone or under the belief that performance was being observed from a neighboring room by an unknown viewer of the same gender and age. Even in this limited social context, participants demonstrated a significantly increased preference for smaller, immediate rewards when they believed that they were being watched. This outcome challenges several intuitive accounts of the peer effect on adolescent risk‐taking, and indicates that the peer influence on reward sensitivity during late adolescence is not dependent on familiarity with the observer. The findings have both theoretical and practical implications for our understanding of social influences on adolescents' risky behavior.     

Rewards and creative performance: a meta-analytic test of theoretically derived hypotheses

Although many scholars and practitioners are interested in understanding how to motivate individuals to be more creative, whether and how rewards affect creativity remain unclear. We argue that the conflicting evidence may be due to differences between studies in terms of reward conditions and the context in which rewards are offered. Specifically, we examine 5 potential moderators of the rewards-creative performance relationship: (a) the reward contingency, (b) the extent to which participants are provided information about their past or current creative performance, (c) the extent to which the reward and context offer choice or impose control, (d) the extent to which the context serves to enhance task engagement, and (e) the extent to which the performance tasks are complex. Using random-effects models, we meta-analyzed 60 experimental and nonexperimental studies (including 69 independent samples) that examined the rewards-creativity relationship with children or adults. Our results suggest that creativity-contingent rewards tend to increase creative performance-and are more positively related to creative performance when individuals are given more positive, contingent, and task-focused performance feedback and are provided more choice (and are less controlled). In contrast, performance-contingent or completion-contingent rewards tend to have a slight negative effect on creative performance.     

With age comes wisdom: decision making in younger and older adults

In two experiments, younger and older adults performed decision-making tasks in which reward values available were either independent of or dependent on the previous sequence of choices made. The choice-independent task involved learning and exploiting the options that gave the highest rewards on each trial. In this task, the stability of the expected reward for each option was not influenced by the previous choices participants made. The choice-dependent task involved learning how each choice influenced future rewards for two options and making the best decisions based on that knowledge. Younger adults performed better when rewards were independent of choice, whereas older adults performed better when rewards were dependent on choice. These findings suggest a fundamental difference in the way in which younger adults and older adults approach decision-making situations. We discuss the results in the context of prominent decision-making theories and offer possible explanations based on neurobiological and behavioral changes associated with aging.     

Positive affect and learning: exploring the “Eureka Effect” in dogs

Animals may experience positive affective states in response to their own achievements. We investigated emotional responses to problem-solving in dogs, separating these from reactions to rewards per se using a yoked control design. We also questioned whether the intensity of reaction would vary with reward type. We examined the response (behavior and heart rate) of dogs as they learned to gain access to different rewards: (1) food (2) human contact, and (3) dog contact. Twelve beagles were assigned to matched pairs, and each dog served as both an experimental and a control animal during different stages of the experiment. We trained all dogs to perform distinct operant tasks and exposed them to additional devices to which they were not trained. Later, dogs were tested in a new context. When acting as an experimental dog, access to the reward was granted immediately upon completion of trained operant tasks. When acting as a control, access to the reward was independent of the dog’s actions and was instead granted after a delay equal to their matched partner’s latency to complete their task. Thus, differences between the two situations could be attributed to experimental dogs having the opportunity to learn to control access to the reward. Experimental dogs showed signs of excitement (e.g., increased tail wagging and activity) in response to their achievements, whereas controls showed signs of frustration (e.g., chewing of the operant device) in response to the unpredictability of the situation. The intensity of emotional response in experimental dogs was influenced by the reward type, i.e., greatest response to food and least to another dog. Our results suggest that dogs react emotionally to problem-solving opportunities and that tail wagging may be a useful indicator of positive affective states in dogs.     

Domestic dogs (Canis familiaris) coordinate their actions in a problem-solving task

Cooperative hunting is a cognitively challenging activity since individuals have to coordinate movements with a partner and at the same time react to the prey. Domestic dogs evolved from wolves, who engage in cooperative hunting regularly, but it is not clear whether dogs have kept their cooperative hunting skills. We presented pairs of dogs with a reward behind a fence with two openings in it. A sliding door operated by the experimenter could block one opening but not both simultaneously. The dogs needed to coordinate their actions, so that each was in front of a different opening, if one of them was to cross through and get food. All 24 dog pairs solved the problem. In study 1, we demonstrated that dogs understood how the apparatus worked. In study 2, we found that, although the performance of the pairs did not depend on the divisibility of the reward, pairs were quicker at coordinating their actions when both anticipated rewards. However, the dogs did not monitor one another, suggesting that their solutions were achieved by each individual attempting to maximize for itself.     

Comparison of the distortion of probability information in decision under risk and an equivalent visual task

Decision makers typically overweight small probabilities and underweight large probabilities. However, there are recent reports that when probability is presented in the form of relative frequencies, this typical pattern reverses. We tested this hypothesis by comparing decision making in two tasks: In one task, probability was stated numerically, and in the other task, it was conveyed through a visual representation. In the visual task, participants chose whether a "stochastic bullet" should be fired at either a large target for a small reward or a small target for a large reward. Participants' knowledge of probability in the visual task was the result of extensive practice firing bullets at targets. In the classical numerical task, participants chose between pairs of lotteries with probabilities and rewards matched to the probabilities and rewards in the visual task. We found that participants' probability-weighting functions were significantly different in the two tasks, but the pattern for the visual task was the typical, not the reversed, pattern.     

Opposing pupil responses to offered and anticipated reward values

Previous studies have shown that the pupils dilate more in anticipation of larger rewards. This finding raises the possibility of a more general association between reward amount and pupil size. We tested this idea by characterizing macaque pupil responses to offered rewards during evaluation and comparison in a binary choice task. To control attention, we made use of a design in which offers occurred in sequence. By looking at pupil responses after choice but before reward, we confirmed the previously observed positive association between pupil size and anticipated reward values. Surprisingly, however, we find that pupil size is negatively correlated with the value of offered gambles before choice, during both evaluation and comparison stages of the task. These results demonstrate a functional distinction between offered and anticipated rewards and present evidence against a narrow version of the simulation hypothesis; the idea that we represent offers by reactivating states associated with anticipating them. They also suggest that pupil size is correlated with relative, not absolute, values of offers, suggestive of an accept–reject model of comparison.     

Dissociating affective evaluation and social cognitive processes in the ventral medial prefrontal cortex

In recent studies, various regions of the ventral medial prefrontal cortex (vmPFC) have been implicated in at least two potentially different mental functions: reasoning about the minds of other people (social cognition) and processing reward related information (affective evaluation). In this study, we test whether the activation in a specific area of the vmPFC, the para-anterior cingulate cortex (PACC), correlates with the reward value of stimuli in general or is specifically associated with social cognition. Participants performed a time estimation task with trial-to-trial feedback in which reward and social context were manipulated separately. Reward was manipulated by giving either positive or negative feedback in the form of small squirts of fluid delivered orally. Social context was manipulated by instructing participants that positive and negative feedback was determined by another person or a computer. The data demonstrate a main effect of feedback, but not social context, in the PACC, suggesting that this area of the vmPFC serves a general function in evaluating and/or representing reward value. In addition, activity in a more anterior subregion of the vmPFC demonstrated reward-related sensitivity only in the social context. Another area that showed a similar interaction was the subgenual cingulate, but this region was only sensitive to negative feedback in the social condition. These findings suggest that, within the vmPFC, the PACC subserves primarily an affective function, whereas in other regions social context can modulate affective responses.