Reinforcement learning in the brain

A wealth of research focuses on the decision-making processes that animals and humans employ when selecting actions in the face of reward and punishment. Initially such work stemmed from psychological investigations of conditioned behavior, and explanations of these in terms of computational models. Increasingly, analysis at the computational level has drawn on ideas from reinforcement learning, which provide a normative framework within which decision-making can be analyzed. More recently, the fruits of these extensive lines of research have made contact with investigations into the neural basis of decision making. Converging evidence now links reinforcement learning to specific neural substrates, assigning them precise computational roles. Specifically, electrophysiological recordings in behaving animals and functional imaging of human decision-making have revealed in the brain the existence of a key reinforcement learning signal, the temporal difference reward prediction error. Here, we first introduce the formal reinforcement learning framework. We then review the multiple lines of evidence linking reinforcement learning to the function of dopaminergic neurons in the mammalian midbrain and to more recent data from human imaging experiments. We further extend the discussion to aspects of learning not associated with phasic dopamine signals, such as learning of goal-directed responding that may not be dopamine-dependent, and learning about the vigor (or rate) with which actions should be performed that has been linked to tonic aspects of dopaminergic signaling. We end with a brief discussion of some of the limitations of the reinforcement learning framework, highlighting questions for future research.     

The Dopamine Receptor D4 Gene (<Emphasis Type="Italic">DRD4</Emphasis>) Moderates Family Environmental Effects on ADHD

Attention-Deficit/Hyperactivity Disorder (ADHD) is a prime candidate for exploration of gene-by-environment interaction (i.e., G x E), particularly in relation to dopamine system genes, due to strong evidence that dopamine systems are dysregulated in the disorder. Using a G x E design, we examined whether the DRD4 promoter 120-bp tandem repeat polymorphism, previously associated with ADHD, moderated the effects of inconsistent parenting and marital conflict on ADHD or Oppositional-Defiant Disorder (ODD). Participants were 548 children with ADHD and non-ADHD comparison children and their parents. Homozygosity for the DRD4 promoter 120-bp tandem repeat insertion allele increased vulnerability for ADHD and ODD only in the presence of inconsistent parenting and appeared to increase susceptibility to the influence of increased child self-blame for marital conflict on ADHD inattention. DRD4 genotypes may interact with these proximal family environmental risk factors by increasing the individual’s responsivity to environmental contingencies.     

Reward prediction error signals by reticular formation neurons

As a key part of the brain’s reward system, midbrain dopamine neurons are thought to generate signals that reflect errors in the prediction of reward. However, recent evidence suggests that “upstream” brain areas may make important contributions to the generation of prediction error signals. To address this issue, we recorded neural activity in midbrain reticular formation (MRNm) while rats performed a spatial working memory task. A large proportion of these neurons displayed positive and negative reward prediction error-related activity that was strikingly similar to that observed in dopamine neurons. Therefore, MRNm may be a source of reward prediction error signals.The capacity of an organism to respond appropriately to environmental stimuli depends on the ability to detect changes in the outcome of its behavior. The mesocorticolimbic dopamine system is thought to be central to this function (Wise 2004; Fields et al. 2007). Dopamine neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) increase activity relative to the presentation of cues that predict rewards and rewards of greater value than expected, and decrease activity relative to rewards of less value than predicted (Nakahara et al. 2004; Bayer and Glimcher 2005; Pan et al. 2005; Tobler et al. 2005). This activity is thought to be involved in a computation about errors in the prediction of reward (Schultz and Dickinson 2000) that can be used to correct behavior. A central issue relevant to the behavioral and computational interpretation of dopamine signals is whether prediction error signals are generated by dopamine neurons, per se, or by cells in “upstream” brain areas.Recent data suggest that brain areas afferent to dopamine neurons generate, or participate in, reward prediction error computations. Lateral habenula, which has been shown to inhibit the activity of VTA and SNc dopamine neurons (Herkenham and Nauta 1979; Christoph et al. 1986), has recently been identified as a potential source of reward prediction error signals (Matsumoto and Hikosaka 2007). A similar finding has been demonstrated in the pedunculopontine tegmental nucleus (PPTg), which is an important regulator of dopamine neuron activity (Floresco et al. 2003). PPTg neural responses varied according to whether or not the animal received expected rewards (Kobayashi and Okada 2007).As part of a larger study investigating the role of VTA in context-dependent spatial working memory (C.B. Puryear, M.J. Kim, and S.J.Y. Mizumori, in prep.), we recorded the activity of neurons in the magnocellular region of the midbrain reticular formation (MRNm according to the method of Swanson [2003]) at the level of the diencephalon. The reticular formation is thought to be important for modulating arousal and vigilance levels necessary for attending to and acting upon salient stimuli (Pragay et al. 1978; Mesulam 1981), and it has recently been shown that this portion of the reticular formation provides glutamatergic input to VTA (Geisler et al. 2007). Thus, MRNm is in a prime position to modulate the activity of VTA dopamine neurons when the outcome of behavior does not meet expectations and therefore may be a source of reward prediction error signals. Accordingly, we investigated whether MRNm neurons exhibited reward related activity, and whether this activity was related to the ability to predict acquisition of reward.Four male Long-Evans rats (4–6 mo old from Simonson Laboratory, Gilroy, CA) were housed individually in Plexiglas cages in a temperature- and humidity-controlled environment (12:12 h light:dark). Rats were provided with food and water ad libitum for 5 d prior to being handled daily and reduced to 85% of ad libitum feeding weights. Animal care and use was conducted according to University of Washington’s Institutional Animal Care and Use Committee guidelines.Rats were habituated to the testing environment and trained to perform a differential reward, win-shift spatial working memory task using radial maze procedures reported previously (Pratt and Mizumori 2001; C.B. Puryear, M.J. Kim, and S.J.Y. Mizumori, in prep.). Briefly, prior to the start of each trial, the end of each of the eight maze arms was baited with either a large (five drops) or small (one drop) amount of chocolate milk on alternating arms. Maze arms containing large or small amounts of reward were counterbalanced across rats and held constant throughout training. Trials started with a sample phase by presenting four maze arms (two large and two small reward arms; individually and randomly selected) to the rat. Immediately after presentation of the fourth arm, a test phase began by making all maze arms accessible so the rat could collect the remaining rewards. The trial ended once all arms were visited. The rat was then confined to the center of the maze for a 2-min intertrial interval. Arm re-entries were counted as errors. Once the rat was able to perform at 15 trials in less than 1 h for seven consecutive days, recording electrodes were surgically implanted.Details concerning the construction of recording electrodes and microdrives and surgical procedures can be found in previous works (McNaughton et al. 1983; Puryear et al. 2006; C.B. Puryear, M.J. Kim, and S.J.Y. Mizumori, in prep.). Briefly, rats were chronically implanted with either eight stereotrodes (four/hemisphere) or four tetrodes (two/hemisphere) made from 25-μm lacquer-coated tungsten wire (California Fine Wire), centered on the following coordinates relative to bregma: −5.25 mm posterior, 0.7 mm lateral, 6 mm ventral. One week of free feeding was allowed for rats to recover from surgery before recording experiments began.Recordings were performed as described previously (Puryear et al. 2006; C.B. Puryear, M.J. Kim, and S.J.Y. Mizumori, in prep.). If no clear spontaneous neural activity was encountered, electrodes were lowered in ∼25-μm increments (up to 175 μm/d) until unambiguous, isolatable units were observed. Single units were isolated from multiunit records using standard cluster-cutting software (MClust; A.D. Redish, University of Wisconsin). A template-matching algorithm was also used to facilitate separation of unique spike waveforms. We only included cells that had a high signal-to-noise ratio (>3:1), exhibited stable clusters throughout the recording session, and had clear refractory periods in the interspike interval histograms following cluster cutting.The final position of each stereotrode was marked by passing a 25-μA current through each recording wire for 25 sec while rats were under 5% isofluorane anesthesia. Rats were then given an overdose of sodium pentobarbital and transcardially perfused (0.9% buffered saline, followed by 10% formalin). Electrodes were retracted, and the brain was removed and allowed to sink in 30% sucrose-formalin. Coronal sections (40 μm) were sliced with a cryostat and stained with cresyl violet. Recording locations were verified by comparing depth measurements and reconstructions of the electrode tracks. Only cells determined to be located in MRNm (Swanson 2003) were considered for analysis.Rats were well trained on the spatial working memory task, committing 0.86 ± 0.2 (mean ± SEM) errors per trial during the first five trials (baseline trials). Importantly, rats demonstrated the ability to discriminate large and small reward locations. There was a significant negative correlation between the first four test phase arm choices (i.e., first, second, third, and fourth arm choice) and the probability that the arm chosen contained a large reward (Fig. 1B; Spearman’s ρ = −0.65, P < 0.001), indicating that rats reliably visited large reward arms before small reward arms during the test phase of each trial.Open in a separate windowFigure 1.Histology and basic firing properties of MRNm neurons. (A) Distribution of cells localized to MRNm. Each dot may represent the location of more than one neuron. Coronal slices adapted from Swanson (2003) (reprinted with permission from Academic Press ©2003). (B) Rats displayed preference for arms that contained large amounts of reward. Plotted is the average probability of choosing a large reward arm during the first four arm choices of the test phase of each trial. Error bars represent SEM. (C) Distribution of average firing rates and spike duration of MRNm neurons. Most cells fired less than 10 spikes/sec and exhibited waveform durations between 1.5 and 2.0 msec. (D) Examples of two MRNm neurons. Top row shows their average waveform on each wire of the tetrode. Scale bar = 1 msec. Middle and bottom rows depict their interspike interval and autocorrelation histograms, respectively.A total of 18 cells localized to MRNm were recorded while rats performed the task. Of these, one cell was omitted from analysis due to a very low average firing rate (∼0.2 spikes/sec), yielding 17 cells included in the following analyses. Figure 1A depicts the distribution of cells localized to MRNm. These cells exhibited a range of average firing rates, spike durations (defined as the time from the start to the end of the action potential) (Fig. 1C), and firing patterns (for representative interspike interval and autocorrelation histograms, see Fig. 1D).Reward-related neural activity was obtained by placing rewards in small metal cups mounted to the end of each maze arm and connected to the recording equipment (custom designed by Neuralynx, Inc.), which served as “lick detectors.” An event marker was automatically inserted into the data stream when the rat licked the cup, providing an instantaneous measurement of the time the rat first obtained reward.In order to determine whether MRNm neurons exhibited significant reward-related activity, peri-event time histograms (PETHs) were constructed (50 msec bins, ±2.5 sec around each reward event). A cell was considered to have a significant excitatory reward response if it passed the following two criteria: (1) The cell had a peak firing rate within ±150 msec of reward acquisition and (2) the peak rate was >150% of its average firing rate for the block of trials. These criteria were applied to PETHs collapsed across reward amounts and separately for large and small reward events. Overall, 47% (eight of 17) of MRNm neurons were found to exhibit significant excitatory responses upon acquisition of reward (Fig. 2D). Of these cells, most (88%, seven of eight) were found to fire relative to acquisition of only large rewards (e.g., Fig. 2A–C), while the remaining neuron fired relative to acquisition of both reward amounts. No cells were found to fire preferentially to acquisition of only small rewards. Aspects of animals’ movement (e.g., velocity) were not found to be a major contributor to the firing patterns of MRNm neurons during performance of the spatial working memory task (data not shown). Therefore, it appears that MRNm unit activity is predominantly biased to represent higher reward values.Open in a separate windowFigure 2.Reward-related activity of MRNm neurons. (A) Peri-event time histograms of one cell that exhibited a short-latency, excitatory response upon acquisition of rewards (t₍₀₎, bin width = 50 msec). Left histogram shows only a modest excitatory response when considering all rewards together. However, top and bottom right histograms show that the reward-related firing occurred upon acquisition of large and not small rewards. Gray-shaded areas indicate time periods analyzed for significant increases in firing rate. (B) Population summary of the proportion of MRNm neurons that demonstrated significant reward-related activity.In order to determine whether MRNm reward-related activity was associated with reward prediction, we tested unit responses to unexpected alterations of reward outcome or elimination of visuospatial information important for reward prediction. To do this, we allowed the rat to perform a second block of five trials with either the locations of large and small rewards switched (reward location switch condition), with two rewards (one large and one small, randomly selected) omitted from the study phase of each trial (reward omission condition), or with the maze room lights extinguished (darkness condition). Importantly, each of these manipulations created situations in which reward prediction errors likely occurred. Overall, these three testing conditions created the following situations, respectively: a mismatch between the locations of large and small rewards, a decreased probability of obtaining a reward, and a situation in which rats are not able to discriminate between arms associated with large and small amounts of reward (C.B. Puryear, M.J. Kim, and S.J.Y. Mizumori, in prep.). Therefore, positive prediction errors could occur when the animal received a large amount of reward on an arm previously associated with a small amount, when the rat retrieved rewards after visiting arms in which reward had been omitted, and when the rat obtained a large reward in darkness. Negative reward prediction errors could occur when the animal received a small amount reward on a maze arm previously associated with a large amount, when the rat visited an arm that did not contain a reward, and when the rat obtained a small reward in darkness.Eight cells with significant responses to large rewards were recorded during these tests (two during reward location switch, four during reward omission conditions, and two during darkness conditions). In order to determine whether the reward manipulations affected the reward-related activity of these cells, a reward activity value (RA) was first calculated, which was the average firing rate in the ±150 msec around the time of acquisition of rewards, expressed as a percentage in change relative to the cell’s average firing rate for each block of trials. These values were normalized to the maximum RA value observed, yielding a normalized RA value for the first and second block of trials (RA_n1 and RA_n2, respectively). These calculations were made for large and small rewards separately, and for non-rewarded arms in the reward omission condition.We then created scatterplots of RA_n’s for each block of trials. If reward-related activity was independent of the expectation of the reward received, RA_n1 and RA_n2 should be similar in each block of trials. As can been seen in Figure 3A, the reward-related activity was consistently higher when rats received more reward than expected in the second block of trials. Conversely, Figure 3B clearly shows that neural activity was consistently suppressed when rats received less reward than expected. These differences in reward-related firing were quantified by calculating the distance of each data point to the diagonal (i.e., the reward activity change index, or RACI): Directionality of the change in reward activity was taken into account in order to discern between increases and decreases in firing rate. A one-sample t-test (α = 0.05) indicated that average RACI values were significantly increased when rats received more reward than expected (t₍₇₎ = −2.73, P < 0.03) and were significantly decreased when rats received less reward than expected (t₍₇₎ = −4.88, P < 0.001). These results are consistent with positive and negative reward prediction error signals, respectively. An example of an MRNm neuron that exhibited both positive and negative prediction error-related activity in the reward location switch condition is depicted in Figure 3D.Open in a separate windowFigure 3.Reward prediction errors in MRNm neurons. (A) Plotted is each neuron’s normalized large reward activity (RA_N, defined in text) for each block of trials. The reward activity during block 2 (y-axis) represents activity at times when more reward than expected was obtained. Note that reward-related activity during these times is consistently more robust than during times in which the rat received the expected reward (block 1), indicating that positive reward prediction errors occurred. (B) Plotted are RA_N values for rewarded and devalued arms in block 2 (x- and y-axes, respectively). Devalued arms include arms associated with a large amount of reward but baited with a small amount of reward, arms in which reward was omitted, and arms containing small rewards visited in darkness. Note that reward-related activity on devalued arms is consistently suppressed, indicating that negative reward prediction errors occurred. (Symbols in A,B: ● indicates cell recorded in darkness condition; o, cell recorded in reward omission condition; and x, cell recorded in reward location switch condition.) (C) Average changes in reward activity (RACI, defined in text) for times in which the rat obtained more and less reward than expected. Asterisks indicate significant differences (P < 0.05). Error bars indicate SEM. (D) An example of one neuron that did not respond to acquisition of rewards during the first block of trials. When the locations of large and small rewards were switched, however, the cell developed an excitatory response to acquisition of large rewards on arms previously associated with small amounts of reward (positive reward prediction error). Furthermore, the firing of the cell was inhibited upon acquisition of small rewards on arms previously associated with large amounts of reward (negative reward prediction error).We demonstrate here that a large proportion of MRNm neurons may be involved in computations about reward acquisition. Similar to dopamine neurons (Tobler et al. 2005), the majority of reward-related MRNm neurons preferentially fired relative to acquisition of large amounts of reward. To our knowledge, this is the first demonstration of discriminative reward responses of reticular formation neurons and highlights a novel role for the reticular formation in reward value representations. Furthermore, these data suggest that MRNm neurons represent similar reward prediction error signals as dopamine neurons (Nakahara et al. 2004; Bayer and Glimcher 2005; Pan et al. 2005; Tobler et al. 2005). It is important to note that in this initial sample, there was remarkable overall consistency and reliability of the positive and negative reward prediction error signals by MRNm neurons. This is similar to the homogeneity of dopamine neuron responses, suggesting that reward prediction may be a major function of the overall population of MRNm reward-related neurons. Nevertheless, further parametric studies are necessary to determine whether MRNm neural activity conforms to the same basic firing profiles that have been well-described for dopamine neurons (i.e., predictive cues and reward probabilities).The reticular formation has traditionally been thought to be important for initiating general arousal states. This is in part due to initial reports of changes in unit activity during transitions from sleep to wakefulness (Huttenlocher 1961; Kasamatsu 1970; Manohar et al. 1972). In addition, a more specific role for the reticular formation in attention has been described in primates performing visual discrimination tasks (Pragay et al. 1978; Fabre et al. 1983). This is consistent with reports of sensory neglect following reticular formation lesions (Watson et al. 1974). Together, these foundational data suggest that reticular formation may function to enhance the overall level of arousal and vigilance necessary for attending to and acting upon salient stimuli (Mesulam 1981). Accordingly, changes in reward-related MRNm neuronal activity could provide an important signal indicating that the contingencies of recently executed behaviors have changed.The striking similarity of the reward prediction error signals of MRNm neurons reported here suggests that MRNm, along with brain regions such as lateral habenula (Matsumoto and Hikosaka 2007), pedunculopontine nucleus (PPTg) (Kobayashi and Okada 2007), and dorsal raphé nucleus (Nakamura et al. 2008) may contribute to the generation of reward prediction error signals observed in dopamine neurons. Furthermore, these data suggest the possibility that such signals are a general property of a large network of midbrain structures. Given that the projections from the reticular formation to VTA are glutamatergic (Geisler et al. 2007), it is possible that the changes in reward-related activity of MRNm neurons, in concert with PPTg, could provide an excitatory component of the reward prediction error signal. In combination with inhibitory inputs from lateral habenula, this may then selectively activate dopamine neurons to initiate the coordinated selection of appropriate behaviors in response to changes in reward outcome (Humphries et al. 2007).     

Etiologic Subtypes of Attention-Deficit/Hyperactivity Disorder: Brain Imaging, Molecular Genetic and Environmental Factors and the Dopamine Hypothesis

Multiple theories of Attention-Deficit/Hyper- activity Disorder (ADHD) have been proposed, but one that has stood the test of time is the dopamine deficit theory. We review the narrow literature from recent brain imaging and molecular genetic studies that has improved our understanding of the role of dopamine in manifestation of symptoms of ADHD, performance deficits on neuropsychological tasks, and response to stimulant medication that constitutes the most common treatment of this disorder. First, we consider evidence of the presence of dopamine deficits based on the recent literature that (1) confirms abnormalities in dopamine-modulated frontal-striatal circuits, reflected by size (smaller-than-average components) and function (hypoactivation); (2) clarifies the agonist effects of stimulant medication on dopaminergic mechanisms at the synaptic and circuit level of analysis; and (3) challenges the most-widely accepted ADHD-related neural abnormality in the dopamine system (higher-than-normal dopamine transporter [DAT] density). Second, we discuss possible genetic etiologies of dopamine deficits based on recent molecular genetic literature, including (1) multiple replications that confirm the association of ADHD with candidate genes related to the dopamine receptor D4 (DRD4) and the DAT; (2) replication of differences in performance of neuropsychological tasks as a function of the DRD4 genotype; and (3) multiple genome-wide linkage scans that demonstrate the limitations of this method when applied to complex disorders but implicate additional genes that may contribute to the genetic basis of ADHD. Third, we review possible environmental etiologies of dopamine deficits based on recent studies of (1) toxic substances that may affect the dopamine system in early development and contribute substantially to the etiology of ADHD; (2) fetal adaptations in dopamine systems in response to stress that may alter early development with lasting effects, as proposed by the developmental origins of health and disease hypothesis; and (3) gene-environment interactions that may moderate selective damage or adaptation of dopamine neurons. Based on these reviews, we identify critical issues about etiologic subtypes of ADHD that may involve dopamine, discuss methods that could be used to address these issues, and review old and new theories that may direct research in this area in the future.     

Association of Dopamine D2 Receptor Gene with Creative Ideation

Although several studies suggest that dopamine D2 receptor (DRD2) gene may contribute to creativity, the relationship between DRD2 and creativity still needs to be further validated. To further test the relevance of DRD2 and creativity, this study explored the association between DRD2 and creative ideation in 483 unrelated healthy Chinese undergraduate students. A total of 15 single nucleotide polymorphisms (SNPs) covering the DRD2 were genotyped, and creative ideation was assessed by the Runco Ideational Behavior Scale (RIBS). Single SNP analysis showed that 2 SNPs (rs4648317 and rs4938019) were nominally associated with fluency, 4 SNPs (rs4648317, rs4938019, rs4648319, and rs1800497) were nominally associated with flexibility, and 1 SNP (rs4648317) was nominally associated with originality. Haplotype analysis showed several haplotypes were nominally associated with various creative ideation indexes. However, none of these nominal associations survived correction for multiple testing. Overall, this study provides suggestive evidence for the genetic impact of DRD2 on creative ideation and supports the assumption that the genotype variations in DRD2 contribute to creativity.     

Individual identity and affective valence in marmoset calls: in vivo brain imaging with vocal sound playback

As with humans, vocal communication is an important social tool for nonhuman primates. Common marmosets (Callithrix jacchus) often produce whistle-like ‘phee’ calls when they are visually separated from conspecifics. The neural processes specific to phee call perception, however, are largely unknown, despite the possibility that these processes involve social information. Here, we examined behavioral and whole-brain mapping evidence regarding the detection of individual conspecific phee calls using an audio playback procedure. Phee calls evoked sound exploratory responses when the caller changed, indicating that marmosets can discriminate between caller identities. Positron emission tomography with [¹⁸F] fluorodeoxyglucose revealed that perception of phee calls from a single subject was associated with activity in the dorsolateral prefrontal, medial prefrontal, orbitofrontal cortices, and the amygdala. These findings suggest that these regions are implicated in cognitive and affective processing of salient social information. However, phee calls from multiple subjects induced brain activation in only some of these regions, such as the dorsolateral prefrontal cortex. We also found distinctive brain deactivation and functional connectivity associated with phee call perception depending on the caller change. According to changes in pupillary size, phee calls from a single subject induced a higher arousal level compared with those from multiple subjects. These results suggest that marmoset phee calls convey information about individual identity and affective valence depending on the consistency or variability of the caller. Based on the flexible perception of the call based on individual recognition, humans and marmosets may share some neural mechanisms underlying conspecific vocal perception.     

Value, pleasure and choice in the ventral prefrontal cortex

Rapid advances have recently been made in understanding how value-based decision-making processes are implemented in the brain. We integrate neuroeconomic and computational approaches with evidence on the neural correlates of value and experienced pleasure to describe how systems for valuation and decision-making are organized in the prefrontal cortex of humans and other primates. We show that the orbitofrontal and ventromedial prefrontal (VMPFC) cortices compute expected value, reward outcome and experienced pleasure for different stimuli on a common value scale. Attractor networks in VMPFC area 10 then implement categorical decision processes that transform value signals into a choice between the values, thereby guiding action. This synthesis of findings across fields provides a unifying perspective for the study of decision-making processes in the brain.     

Eye blink rate predicts reward decisions in adolescents

The ventral striatum displays hyper‐responsiveness to reward in adolescents relative to other age groups, and animal research on the developmental trajectory of the dopaminergic system suggests that dopamine may underlie adolescent sensitivity to reward. However, practical limitations prevent the direct measurement of dopamine in healthy adolescents. Eye blink rate (EBR) shows promise as a proxy measure of striatal dopamine D2 receptor function. We investigated developmental differences in the relationship between EBR and reward‐seeking behavior on a risky decision‐making task. Increasing EBR was associated with greater reward maximization on the task for adolescent but not adult participants. Furthermore, adolescents demonstrated greater sensitivity to reward value than adults, as evinced by shifts in decision patterns based on increasing potential reward. These findings suggest that previously observed adolescent behavioral and neural hypersensitivity to reward may in fact be due to greater dopamine receptor activity, as represented by the relationship of blink rate and reward‐seeking behavior. They also demonstrate the feasibility and utility of using EBR as a proxy for dopamine in healthy youth in whom direct measurements of dopamine are prohibitively invasive.     

When is an error not a prediction error? An electrophysiological investigation

A recent theory holds that the anterior cingulate cortex (ACC) uses reinforcement learning signals conveyed by the midbrain dopamine system to facilitate flexible action selection. According to this position, the impact of reward prediction error signals on ACC modulates the amplitude of a component of the event-related brain potential called the error-related negativity (ERN). The theory predicts that ERN amplitude is monotonically related to the expectedness of the event: It is larger for unexpected outcomes than for expected outcomes. However, a recent failure to confirm this prediction has called the theory into question. In the present article, we investigated this discrepancy in three trial-and-error learning experiments. All three experiments provided support for the theory, but the effect sizes were largest when an optimal response strategy could actually be learned. This observation suggests that ACC utilizes dopamine reward prediction error signals for adaptive decision making when the optimal behavior is, in fact, learnable.     

<Emphasis Type="Italic">DRD4</Emphasis> Variants Moderate the Impact of Parental Characteristics on Child Attention-Deficit Hyperactivity Disorder: Exploratory Evidence from a Multiplex Family Design

Parental ADHD symptomatology and related impairments have been robustly associated with youth ADHD across decades of work. Notably, these factors may impede typical development of child self-regulation capabilities through both neurobiological and interpersonal processes. High heritability of estimates for the disorder further suggest that these effects are likely genetically-mediated, at least in part. Variation within the dopamine D4 receptor gene (DRD4) has been shown to moderate parental influences on youth ADHD. Use of a multiplex family design (i.e., samples of families that included multiple affected members) may facilitate identification of additional gene variants of interest and advance understanding of gene-environment interplay in regard to parenting. Thirty multiplex families consisting of 114 individuals (66 youth, 48 parents) completed a multi-stage, multi-informant diagnostic and neurocognitive assessment, measures of parenting, and provided saliva samples for DNA analyses. Sanger sequencing of the DRD4 gene yielded 16 rare variants; a polygenic risk score was computed for both parents and youth. Generalized estimating equations (GEE) examined the predictive effects of parental ADHD symptoms, parental neurocognitive functioning, and poor parenting dimensions on youth ADHD as well as moderation of these effects by parental and youth DRD4 variants. Findings indicated that parental DRD4 variants moderated the impact of parental ADHD and neurocognitive functioning on youth ADHD symptoms. Youth DRD4 variants moderated the impact of parental inconsistent discipline on child ADHD. In all cases, stronger associations were observed for those individuals with more risk variants. These exploratory findings highlight the potential utility of a multiplex family design for examining the interplay between parent and child characteristics in predicting youth outcomes.     

Linking Gene, Brain, and Behavior: DRD4, Frontal Asymmetry, and Temperament

ABSTRACT— Gene-environment interactions involving exogenous environmental factors are known to shape behavior and personality development. Although gene-environment interactions involving endogenous environmental factors are hypothesized to play an equally important role, this conceptual approach has not been empirically applied in the study of early-developing temperament in humans. Here we report evidence for a gene- endo environment (i.e., resting frontal brain electroencephalogram, EEG, asymmetry) interaction in predicting child temperament. The dopamine D4 receptor (DRD4) gene (long allele vs. short allele) moderated the relation between resting frontal EEG asymmetry (left vs. right) at 9 months and temperament at 48 months. Children who exhibited left frontal EEG asymmetry at 9 months and who possessed the DRD4 long allele were significantly more soothable at 48 months than other children. Among children with right frontal EEG asymmetry at 9 months, those with the DRD4 long allele had significantly more difficulties focusing and sustaining attention at 48 months than those with the DRD4 short allele. Resting frontal EEG asymmetry did not influence temperament in the absence of the DRD4 long allele. We discuss how the interaction of genetic and endoenvironmental factors may confer risk and protection for different behavioral styles in children.     

Experimental evidence for differential susceptibility: dopamine D4 receptor polymorphism (DRD4 VNTR) moderates intervention effects on toddlers' externalizing behavior in a randomized controlled trial

In a randomized controlled trial we tested the role of genetic differences in explaining variability in intervention effects on child externalizing behavior. One hundred fifty-seven families with 1- to 3-year-old children screened for their relatively high levels of externalizing behavior participated in a study implementing Video-feedback Intervention to promote Positive Parenting and Sensitive Discipline (VIPP-SD), with six 1.5-hr intervention sessions focusing on maternal sensitivity and discipline. A moderating role of the dopamine D4 receptor (DRD4) variable-number tandem repeat (VNTR) exon III polymorphism was found: VIPP-SD proved to be effective in decreasing externalizing behavior in children with the DRD4 7-repeat allele, a polymorphism that is associated with motivational and reward mechanisms and Attention Deficit Hyperactivity Disorder (ADHD) in children. VIPP-SD effects were largest in children with the DRD4 7-repeat allele whose parents showed the largest increase in the use of positive discipline. The findings of this first experimental test of (measured) gene by (observed) environment interaction in human development indicate that children may be differentially susceptible to intervention effects depending on genetic differences.     

Selection for reduced aggressiveness towards man and dopaminergic activity in Norway rats

The brain dopaminergic system is involved in the process of long-term selection for reduced aggressive reaction towards man in Norway rats. The dopamine levels in the striatum as well as the nucleus accumbens with the tuberculum olfactorium were significantly lower in domesticated rats than in their wild counterparts. A substantial decrease was found in homovanillic acid level in the n. accumbens and tuberculum olfactorium. Specific binding of [³H]spiperone which labels D-2 dopamine receptors was higher in the mesolimbic structure of tame rats, whereas binding of [³H]SCH 23390 (D-1 receptors) was unchanged in this area. No substantial differences were detected in D-1 and D-2 binding in striatum. Apomorphine (0.3 mg/kg) elicited less locomotion in tame animals, reflecting a decrease of sensitivity of postsynaptic dopamine receptors. Tame rats showed fewer aggressive contacts in a foot-shock test than wild rats and the D-2 receptor antagonist sulpiride (25 mg/kg) significantly decreased the foot-shock aggression only in wild rats. Therefore, domestication, which diminishes defensive behavior and emotional reactivity of animals, is associated with decreases of dopamine level in the striatum, changed metabolism of dopamine in mesolimbic system, and an alteration in density and senstivity of D-2 receptors.     

Training cooperation in the Prisoner's Dilemma

Female Ss' choices in two types of mixed-motive game situations were used to select Ss who had predominantly Own Gain and Ss who had predominantly Relative Gain goals. On the basis of simple reward notions, it was predicted that the former but not the latter would change from competitive to cooperative responding in decomposed Prisoner's Dilemma game situations when interacting with a conditionally cooperative other. Corresponding yoked controls were not expected to become cooperative. The availability of social comparison with an outcome which was smaller than the mutually cooperative outcome but larger than the mutually competitive out-come was expected to lead Own Gain Ss to more rapid learning of cooperation but not to affect the responses of corresponding yoked controls or of Relative Gain Ss. A 2 × 2 × 2 × 5 factorial design was used in which the factors were goal orientation (Own Gain vs Relative Gain), strategy of the other (conditionally cooperative vs yoked control), the availability vs unavailability of the social comparison, and trials. The results strongly supported each of the expectations. The results were discussed in terms of how the operation of the reward mechanism would be affected by the operation of some other social psychological processes.     

Adaptive scaling of reward in episodic memory: a replication study

Reward is thought to enhance episodic memory formation via dopaminergic consolidation. Bunzeck, Dayan, Dolan, and Duzel [(2010). A common mechanism for adaptive scaling of reward and novelty. Human Brain Mapping, 31, 1380–1394] provided functional magnetic resonance imaging (fMRI) and behavioural evidence that reward and episodic memory systems are sensitive to the contextual value of a reward—whether it is relatively higher or lower—as opposed to absolute value or prediction error. We carried out a direct replication of their behavioural study and did not replicate their finding that memory performance associated with reward follows this pattern of adaptive scaling. An effect of reward outcome was in the opposite direction to that in the original study, with lower reward outcomes leading to better memory than higher outcomes. There was a marginal effect of reward context, suggesting that expected value affected memory performance. We discuss the robustness of the reward memory relationship to variations in reward context, and whether other reward-related factors have a more reliable influence on episodic memory.     

Dopamine D1 receptors in the anterior cingulate cortex regulate effort-based decision making

The anterior cingulate cortex (ACC) has been implicated in encoding whether or not an action is worth performing in view of the expected benefit and the cost of performing the action. Dopamine input to the ACC may be critical for this form of effort-based decision making; however, the role of distinct ACC dopamine receptors is yet unknown. Therefore, we examined in rats the effects of an intra-ACC D1 and D2 receptor blockade on effort-based decision making tested in a T-maze cost-benefit task. In this task, subjects could either choose to climb a barrier to obtain a high reward in one arm or a low reward in the other arm without a barrier. Unlike vehicle-treated rats, rats with intra-ACC infusion of the D1 receptor antagonist SCH23390 exhibited a reduced preference for the high-cost- high-reward response option when having the choice to obtain a low reward with little effort. In contrast, in rats with intra-ACC infusion of the D2 receptor antagonist eticlopride, the preference for the high-cost-high-reward response option was not altered relative to vehicle-treated rats. These data provide the first evidence that D1 receptors in the ACC regulate effort-based decision making.     

Joint effect of dopaminergic genes on likelihood of smoking following treatment with bupropion SR.

OBJECTIVE: To determine the relationship between joint variation in 2 dopaminergic genes and the likelihood of nonsmoking following treatment with bupropion sustained release (SR). DESIGN: Three hundred twenty-three participants in a bupropion SR smoking cessation effectiveness trial with 12-month follow-up were genotyped for variants of dopamine receptor gene DRD2 and dopamine transporter SLC6A3. MAIN OUTCOME MEASURES: Self-reported 7-day point prevalence of nonsmoking. RESULTS: Neither genotype alone was associated with 7-day point-prevalent nonsmoking at the 12-month follow-up. However, in the presence of the DRD2 A1 allele, SLC6A3 status was significantly associated with the likelihood of nonsmoking at the 12-month follow-up (individuals with DRD2 A1+ and SLC6A3 9- were more likely to be smoking). In the absence of the DRD2 A1 allele, the association between SLC6A3 status and nonsmoking was nonsignificant. CONCLUSION: Although these results are suggestive, a more compelling test is needed of the hypothesis that dopaminergic gene interaction underlies, in part, the likelihood of smoking following treatment with bupropion SR. Most likely this will come from larger studies involving prospective randomization to treatment based on genotype.     

DRD4基因与亲社会行为的关联及其潜在脑机制

DRD4基因是亲社会行为的重要候选基因,且与环境交互影响亲社会行为的发生发展。通过梳理既有研究,本文从性别差异、亲社会行为的不同类型及发展动态性等角度探讨了亲社会行为遗传研究存在分歧的原因,并在此基础上探索了DRD4基因作用于亲社会行为的潜在脑机制。未来研究应采用纵向设计探究DRD4基因影响亲社会行为的发展动态性问题,并深入探索其性别差异;采用多质多法分析考察不同类型亲社会行为遗传机制的差异性;采用影像遗传学设计揭示“DRD4基因—脑—亲社会行为”作用机制。     

Adolescents' Reward‐related Neural Activation: Links to Thoughts of Nonsuicidal Self‐Injury

Adolescence is a critical developmental period marked by an increase in risk behaviors, including nonsuicidal self‐injury (NSSI). Heightened reward‐related brain activation and relatively limited recruitment of prefrontal regions contribute to the initiation of risky behaviors in adolescence. However, neural reward processing has not been examined among adolescents who are at risk for future engagement for NSSI specifically, but who have yet to actually engage in this behavior. In the current fMRI study (N = 71), we hypothesized that altered reward processing would be associated with adolescents' thoughts of NSSI. Results showed that NSSI youth exhibited heightened activation in the bilateral putamen in response to a monetary reward. This pattern of findings suggests that heightened neural sensitivity to reward is associated with thoughts of NSSI in early adolescence. Implications for prevention are discussed.     

Reinvestigating the Effects of Promised Reward on Creativity

In a pivotal study, Eisenberger and Rhoades (2001 Eisenberger , R. , & Rhoades , L. ( 2001 ). Incremental effects of reward on creativity . Journal of Personality and Social Psychology , 81 , 728 – 741 .[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]) recently put forth evidence that promised reward enhances creativity when the reward is clearly contingent upon creative, as opposed to conventional performance. The present study reinvestigated this finding, additionally examining the role of reward framing (i.e., gain vs. non-gain) as well as that of expected competence information and self-determination. In both experiments, only non-gain framed rewards reliably enhanced creativity, suggesting that that the mental construal of a reward (as either a potential gain or non-gain) is an important moderator of its effects on creative generation. Results also indicated that the facilitative influence of promised reward on creativity is due to the offering of an incentive per se and not the concomitant prospect of receiving normative competence feedback. Finally, although it was influenced by reward framing, perceived self-determination did not mediate the effect of promised reward on creativity.