Maintenance of grip-induced muscle tension: a behavioral marker of schizophrenia.

A grip-induced muscle tension maintenance task distinguished between schizophrenic patients, regardless of medication or hospitalization status, and both normals and controls with affective (unipolar and bipolar) disorders. Unaffected first-degree relatives of schizophrenic patients also showed a grip deficit. Coupling the grip task with a visual discrimination task that divided attention through instruction or reinforcement contingency, increased grip error times for all groups. No group differences appeared on the discrimination task, regardless of difficulty level, and the tone used to provide corrective feedback was not implicated in the grip deficit. The results suggest that the grip task is tapping, in a systematic and reliable manner, a motor-control abnormality that may be useful as a behavioral marker of schizophrenia.     

Adolescent-specific patterns of behavior and neural activity during social reinforcement learning

Humans are sophisticated social beings. Social cues from others are exceptionally salient, particularly during adolescence. Understanding how adolescents interpret and learn from variable social signals can provide insight into the observed shift in social sensitivity during this period. The present study tested 120 participants between the ages of 8 and 25 years on a social reinforcement learning task where the probability of receiving positive social feedback was parametrically manipulated. Seventy-eight of these participants completed the task during fMRI scanning. Modeling trial-by-trial learning, children and adults showed higher positive learning rates than did adolescents, suggesting that adolescents demonstrated less differentiation in their reaction times for peers who provided more positive feedback. Forming expectations about receiving positive social reinforcement correlated with neural activity within the medial prefrontal cortex and ventral striatum across age. Adolescents, unlike children and adults, showed greater insular activity during positive prediction error learning and increased activity in the supplementary motor cortex and the putamen when receiving positive social feedback regardless of the expected outcome, suggesting that peer approval may motivate adolescents toward action. While different amounts of positive social reinforcement enhanced learning in children and adults, all positive social reinforcement equally motivated adolescents. Together, these findings indicate that sensitivity to peer approval during adolescence goes beyond simple reinforcement theory accounts and suggest possible explanations for how peers may motivate adolescent behavior.     

A learning rule that explains how rewards teach attention

Many theories propose that top-down attentional signals control processing in sensory cortices by modulating neural activity. But who controls the controller? Here we investigate how a biologically plausible neural reinforcement learning scheme can create higher order representations and top-down attentional signals. The learning scheme trains neural networks using two factors that gate Hebbian plasticity: (1) an attentional feedback signal from the response-selection stage to earlier processing levels; and (2) a globally available neuromodulator that encodes the reward prediction error. We demonstrate how the neural network learns to direct attention to one of two coloured stimuli that are arranged in a rank-order. Like monkeys trained on this task, the network develops units that are tuned to the rank-order of the colours and it generalizes this newly learned rule to previously unseen colour combinations. These results provide new insight into how individuals can learn to control attention as a function of reward contingency.     

不同类型反馈对儿童学习效果的影响及性别差异

儿童阶段是对外部反馈最敏感的阶段。以往研究发现,不同类型反馈（积极/消极反馈）和不同性质的强化物（物质/社会性强化）对儿童学习效果存在交互作用,并且对不同性别儿童的影响有所不同。本研究通过两个实验,采用联结学习范式以考察不同类型反馈对8~10岁儿童学习效果的影响。首先考察了积极反馈和消极反馈对儿童学习效果的影响,进而在积极、消极反馈的基础上加入了物质、社会性强化物,探究物质、社会性强化条件下积极、消极反馈对儿童学习效果影响的性别差异。结果表明,对于8~10岁儿童来说,消极反馈比积极反馈对儿童学习效果的影响更大。并且,物质、社会性强化物对儿童反馈学习效果的影响存在性别差异,对于男孩来说,在物质性强化条件下,消极反馈更能促进其学习;而对于女孩来说,在社会性强化物下,消极反馈更能促进她们的学习。该研究为如何运用反馈促进儿童学习提供了实证依据。     

Adult age differences in frontostriatal representation of prediction error but not reward outcome

Emerging evidence from decision neuroscience suggests that although younger and older adults show similar frontostriatal representations of reward magnitude, older adults often show deficits in feedback-driven reinforcement learning. In the present study, healthy adults completed reward-based tasks that did or did not depend on probabilistic learning, while undergoing functional neuroimaging. We observed reductions in the frontostriatal representation of prediction errors during probabilistic learning in older adults. In contrast, we found evidence for stability across adulthood in the representation of reward outcome in a task that did not require learning. Together, the results identify changes across adulthood in the dynamic coding of relational representations of feedback, in spite of preserved reward sensitivity in old age. Overall, the results suggest that the neural representation of prediction error, but not reward outcome, is reduced in old age. These findings reveal a potential dissociation between cognition and motivation with age and identify a potential mechanism for explaining changes in learning-dependent decision making in old adulthood.     

Impaired reward processing by anterior cingulate cortex in children with attention deficit hyperactivity disorder

Decades of research have examined the neurocognitive mechanisms of cognitive control, but the motivational factors underlying task selection and performance remain to be elucidated. We recently proposed that anterior cingulate cortex (ACC) utilizes reward prediction error signals carried by the midbrain dopamine system to learn the value of tasks according to the principles of hierarchical reinforcement learning. According to this position, disruption of the ACC–dopamine interface can disrupt the selection and execution of extended, task-related behaviors. To investigate this issue, we recorded the event-related brain potential (ERP) from children with attention deficit hyperactivity disorder (ADHD), which is strongly associated with ACC–dopamine dysfunction, and from typically developing children while they navigated a simple “virtual T-maze” to find rewards. Depending on the condition, the feedback stimuli on each trial indicated that the children earned or failed to earn either money or points. We found that the reward positivity, an ERP component proposed to index the impact of dopamine-related reward signals on ACC, was significantly larger with money feedback than with points feedback for the children with ADHD, but not for the typically developing children. These results suggest that disruption of the ACC–dopamine interface may underlie the impairments in motivational control observed in childhood ADHD.     

Handwriting movement analyses for monitoring drug-induced motor side effects in schizophrenia patients treated with risperidone

Epidemiologic studies indicate that nearly 60% of schizophrenia (SZ) patients treated with conventional antipsychotic drugs develop extrapyramidal side effects (EPS) such as parkinsonism and tardive dyskinesia. Although the prevalence of EPS has decreased due to the newer antipsychotics, EPS continue to limit the effectiveness of these medicines. Ongoing monitoring of EPS is likely to improve treatment outcome or compliance and reduce the frequency of re-hospitalization. A quantitative analysis of handwriting kinematics was used to evaluate effects of antipsychotic medication type and dose in schizophrenia patients. Twenty-seven schizophrenia patients treated with risperidone, six schizophrenia patients who received no antipsychotic medication and 47 healthy comparison participants were enrolled. Participants performed a 20-min handwriting task consisting of loops of various sizes and a sentence. Data were captured and analyzed using MovAlyzeR software. Results indicated that risperidone-treated participants exhibited significantly more dysfluent handwriting movements than either healthy or untreated SZ participants. Risperidone-treated participants exhibited lower movement velocities during production of simple loops compared to unmedicated patients. Handwriting dysfluency during sentence writing increased with dose. A 3-factor model consisting of kinematic variables derived from sentence writing accounted for 83% (r = .91) of the variability in medication dose. In contrast, we found no association between observer-based EPS severity ratings and medication dose. These findings support the importance of handwriting-based measures to monitor EPS in medicated schizophrenia patients.     

Reinforcement learning, conditioning, and the brain: Successes and challenges

The field of reinforcement learning has greatly influenced the neuroscientific study of conditioning. This article provides an introduction to reinforcement learning followed by an examination of the successes and challenges using reinforcement learning to understand the neural bases of conditioning. Successes reviewed include (1) the mapping of positive and negative prediction errors to the firing of dopamine neurons and neurons in the lateral habenula, respectively; (2) the mapping of model-based and model-free reinforcement learning to associative and sensorimotor cortico-basal ganglia-thalamo-cortical circuits, respectively; and (3) the mapping of actor and critic to the dorsal and ventral striatum, respectively. Challenges reviewed consist of several behavioral and neural findings that are at odds with standard reinforcement-learning models, including, among others, evidence for hyperbolic discounting and adaptive coding. The article suggests ways of reconciling reinforcement-learning models with many of the challenging findings, and highlights the need for further theoretical developments where necessary. Additional information related to this study may be downloaded from http://cabn.psychonomic-journals.org/content/supplemental.     

Genetic polymorphisms of the dopamine and serotonin systems modulate the neurophysiological response to feedback and risk taking in healthy humans

Genetic differences in the dopamine and serotonin systems have been suggested as potential factors underlying interindividual variability in risk taking and in brain activation during the processing of feedback. Here, we studied the effects of dopaminergic (dopamine transporter [DAT1], catecholamine-O-methyltransferase val158met [COMT]) and serotonergic (serotonin transporter [5HTTLPR]) polymorphisms on risk taking and brain responses following feedback in 60 healthy female subjects. The subjects completed a well-established experimental gambling paradigm while an electroencephalogram was recorded. During the task, risk-taking behavior and prefrontal brain responses (feedback-related negativity [FRN]) following monetary gains and losses were assessed. FRN amplitudes were enhanced for nine-repeat-allele carriers of the DAT1 and short-allele carriers of 5HTTLPR, which are both presumably linked to less transporter activity and higher neurotransmitter levels. Moreover, nine-repeat DAT1 carriers displayed a trend toward increased risk taking in general, whereas 5HTTLPR short-allele carriers showed decreased risk taking following gains. COMT val158met genotype was unrelated to FRN amplitude and average risk taking. However, COMT met/met carriers showed a pronounced feedback P3 amplitude independent of valence, and a gradual increase in risk taking during the gambling task. In sum, the present findings underline the importance of genetic variability in the dopamine and serotonin systems regarding the neurophysiology of feedback processing.     

Reinforcement learning in the brain

A wealth of research focuses on the decision-making processes that animals and humans employ when selecting actions in the face of reward and punishment. Initially such work stemmed from psychological investigations of conditioned behavior, and explanations of these in terms of computational models. Increasingly, analysis at the computational level has drawn on ideas from reinforcement learning, which provide a normative framework within which decision-making can be analyzed. More recently, the fruits of these extensive lines of research have made contact with investigations into the neural basis of decision making. Converging evidence now links reinforcement learning to specific neural substrates, assigning them precise computational roles. Specifically, electrophysiological recordings in behaving animals and functional imaging of human decision-making have revealed in the brain the existence of a key reinforcement learning signal, the temporal difference reward prediction error. Here, we first introduce the formal reinforcement learning framework. We then review the multiple lines of evidence linking reinforcement learning to the function of dopaminergic neurons in the mammalian midbrain and to more recent data from human imaging experiments. We further extend the discussion to aspects of learning not associated with phasic dopamine signals, such as learning of goal-directed responding that may not be dopamine-dependent, and learning about the vigor (or rate) with which actions should be performed that has been linked to tonic aspects of dopaminergic signaling. We end with a brief discussion of some of the limitations of the reinforcement learning framework, highlighting questions for future research.     

预期错误在恐惧记忆更新中的作用与机制

依据错误驱动的学习理论,行为预期结果与实际结果之间的不匹配即预期错误(Predictionerror,PE)是学习产生的驱动因素。作为显著性信息中的一种,预期错误和物理显著性、惊讶、新异性等存在信息加工阶段的不同,与记忆更新的关系也有差异。近年来,记忆再巩固干预范式(reconsolidation interference)被证明可用于人类条件性恐惧记忆的更新,其中记忆提取激活阶段所包含的预期错误起到了引发记忆“去稳定”、开启记忆再巩固的关键作用。在促进恐惧记忆更新的行为机制上,PE被认为是记忆去稳定的必要非充分条件。记忆提取必须包含适量的PE,但其引发的是记忆去稳定、消退还是中间状态,还需结合记忆本身性质确定。在促进恐惧记忆更新的神经机制上,杏仁核、导水管周围灰质(PAG)、海马均在PE探测和计算过程中具有重要作用;前额叶皮层(PFC)及其亚区在PE开启记忆再巩固过程中扮演了重要角色。上述过程又受到神经系统中特定神经递质的重要调节,尤其是多巴胺能和谷氨酸能。未来研究应进一步探索基于PE计算模型的量化研究,整合PE与其他边界条件的交互作用,考察不同类型显著性在记忆再巩固中的作用等;并亟...     

Neural mechanism underlying self-controlled feedback on motor skill learning

The present study investigated the neural mechanisms of self-controlled (SC) feedback underlying its learning advantages. Forty-two participants, including 24 females (16.43 ± 2.61 years) and 18 males (17.56 ± 0.86 years), were randomly assigned to a SC or yoked (YK) group. The 6-key-pressing task with a goal movement time was adopted as the experimental task. The behavioral results showed that the SC group demonstrated superior performance in transfer; however, the differences in retention did not reach statistical significance. Event-related potential analyses revealed that the SC group exhibited larger post-stimulus and post-feedback P3 amplitudes than the YK group in the frontal regions; these amplitudes were larger in the YK group in the parietal regions. The post-response error positivity amplitude was found to be larger in the YK group than in the SC group. These results suggest that SC feedback may allow the learner to more actively process the task stimuli and feedback information, and contributes to enhancing the learner’s motivation and attachment to the task being practiced. The present study provides a neurophysiological explanation for why SC feedback is effective in learning a new motor skill.     

The study of sequence learning in individuals with schizophrenia: A critical review of the literature

The serial reaction time task (SRTT) has been used extensively to study implicit sequence learning. A number of studies have used the SRTT to examine sequence learning in schizophrenia patients. Despite these studies, it remains unclear whether sequence learning is impaired in patients, whether antipsychotic medications affect sequence learning, and what types of sequential information patients might have difficulty learning. Methodological limitations have made it difficult to obtain good answers to these questions. Methodological innovations from the general SRTT literature that have not yet been adopted in the schizophrenia literature could provide better answers.     

The neural basis of human error processing: reinforcement learning,dopamine, and the error-related negativity

The authors present a unified account of 2 neural systems concerned with the development and expression of adaptive behaviors: a mesencephalic dopamine system for reinforcement learning and a "generic" error-processing system associated with the anterior cingulate cortex. The existence of the error-processing system has been inferred from the error-related negativity (ERN), a component of the event-related brain potential elicited when human participants commit errors in reaction-time tasks. The authors propose that the ERN is generated when a negative reinforcement learning signal is conveyed to the anterior cingulate cortex via the mesencephalic dopamine system and that this signal is used by the anterior cingulate cortex to modify performance on the task at hand. They provide support for this proposal using both computational modeling and psychophysiological experimentation.     

Histidine-decarboxylase knockout mice show deficient nonreinforced episodic object memory, improved negatively reinforced water-maze performance, and increased neo- and ventro-striatal dopamine turnover

The brain's histaminergic system has been implicated in hippocampal synaptic plasticity, learning, and memory, as well as brain reward and reinforcement. Our past pharmacological and lesion studies indicated that the brain's histamine system exerts inhibitory effects on the brain's reinforcement respective reward system reciprocal to mesolimbic dopamine systems, thereby modulating learning and memory performance. Given the close functional relationship between brain reinforcement and memory processes, the total disruption of brain histamine synthesis via genetic disruption of its synthesizing enzyme, histidine decarboxylase (HDC), in the mouse might have differential effects on learning dependent on the task-inherent reinforcement contingencies. Here, we investigated the effects of an HDC gene disruption in the mouse in a nonreinforced object exploration task and a negatively reinforced water-maze task as well as on neo- and ventro-striatal dopamine systems known to be involved in brain reward and reinforcement. Histidine decarboxylase knockout (HDC-KO) mice had higher dihydrophenylacetic acid concentrations and a higher dihydrophenylacetic acid/dopamine ratio in the neostriatum. In the ventral striatum, dihydrophenylacetic acid/dopamine and 3-methoxytyramine/dopamine ratios were higher in HDC-KO mice. Furthermore, the HDC-KO mice showed improved water-maze performance during both hidden and cued platform tasks, but deficient object discrimination based on temporal relationships. Our data imply that disruption of brain histamine synthesis can have both memory promoting and suppressive effects via distinct and independent mechanisms and further indicate that these opposed effects are related to the task-inherent reinforcement contingencies.     

Impact of Parkinson’s disease and dopaminergic medication on adaptation to explicit and implicit visuomotor perturbations

The capacity to learn new visuomotor associations is fundamental to adaptive motor behavior. Evidence suggests visuomotor learning deficits in Parkinson’s disease (PD). However, the exact nature of these deficits and the ability of dopamine medication to improve them are under-explored. Previous studies suggested that learning driven by large and small movement errors engaged distinct neural mechanisms. Here, we investigated whether PD patients have a generalized impairment in visuomotor learning or selective deficits in learning from large explicit errors which engages cognitive strategies or small imperceptible movement errors involving primarily implicit learning processes. Visuomotor learning skills of non-medicated and medicated patients were assessed in two reaching tasks in which the size of visuospatial errors experienced during learning was manipulated using a novel three-dimensional virtual reality environment. In the explicit perturbation task, the visuomotor perturbation was applied suddenly resulting in large consciously detected initial spatial errors, whereas in the implicit perturbation task, the perturbation was gradually introduced in small undetectable steps such that subjects never experienced large movement errors. A major finding of this study was that PD patients in non-medicated and medicated conditions displayed slower learning rates and smaller adaptation magnitudes than healthy subjects in the explicit perturbation task, but performance similar to healthy controls in the implicit perturbation task. Also, non-medicated patients showed an average reduced deadaptation relative to healthy controls when exposed to the large errors produced by the sudden removal of the perturbation in both the explicit and implicit perturbation tasks. Although dopaminergic medication consistently improved motor signs, it produced a variable impact on learning the explicit perturbation and deadaptation and unexpectedly worsened performance in some patients. Considered together, these results indicate that PD selectively impairs the ability to learn from large consciously detected visuospatial errors. This finding suggests that basal ganglia-related circuits are important neural structures for adaptation to sudden perturbations requiring awareness and high-cost action selection. Dopaminergic treatment may selectively compromise the ability to learn from large explicit movement errors for reasons that remain to be elucidated.     

Ventral tegmental area and substantia nigra neural correlates of spatial learning

The ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) may provide modulatory signals that, respectively, influence hippocampal (HPC)- and striatal-dependent memory. Electrophysiological studies investigating neural correlates of learning and memory of dopamine (DA) neurons during classical conditioning tasks have found DA neural activity in VTA and SNc to be tightly coupled with reinforcement expectations. Also, VTA integrity and DA in HPC have been found to regulate the encoding of HPC-dependent memories. Therefore, to determine the nature of the neural code HPC may receive from midbrain DA regions, the present study investigated VTA and SNc neural activity as navigating rats engaged in new spatial learning and experienced changes in expected goal locations. VTA and SNc cells were differentially engaged during training to a series of three novel goal locations. During task acquisition, the peak firing rates of VTA neurons decreased at the time of reward and shifted to time points before reward retrieval, whereas the peak firing rates of SNc neurons remained elevated at the time of reward during training to all three goal locations. Both VTA and SNc egocentric coding was strongest during training to the first goal location, which coincided with the time subjects learned the behavioral rules specific to the task. These data imply that VTA and SNc play complementary yet distinct roles in spatial learning to optimize adaptive behavior.     

反馈负波及其理论解释

反馈负波（feedback related negativity, FRN）是由代表行为错误或失去金钱等负性反馈刺激诱发的一种脑电波成分,出现在刺激呈现后250~300ms左右,偶极子源定位发现这一成分产生于前扣带回附近。对于FRN的功能意义的解释主要有强化学习理论和情绪动机假说,前者认为FRN反映的是神经系统对反馈刺激的认知加工过程,而后者认为FRN反映的是对情绪动机意义的评价过程。两种理论框架下还有一些目前研究仍待解决的问题,可能是未来研究的方向     

Hierarchical Error Evaluation: The Role of Medial-Frontal Cortex in Postural Control

Motor error evaluation appears to be a hierarchically organized process subserved by 2 distinct systems: a higher level system within medial-frontal cortex responsible for movement outcome evaluation (high-level error evaluation) and a lower level posterior system(s) responsible for the mediation of within-movement errors (low-level error evaluation). While a growing body of evidence suggests that a reinforcement learning system within medial-frontal cortex plays a crucial role in the evaluation of high-level errors made during discrete reaching movements and continuous motor tracking, the role of this system in postural control is currently unclear. Participants learned a postural control task via a feedback-driven trial-and-error shaping process. In line with previous findings, electroencephalographic recordings revealed that feedback about movement outcomes elicited a feedback error–related negativity: a component of the human event-related brain potential associated with high-level outcome evaluation within medial-frontal cortex. Thus, the data provide evidence that a high-level error-evaluation system within medial-frontal cortex plays a key role in learning to control our body posture.     

Reduced susceptibility to confirmation bias in schizophrenia

Patients with schizophrenia (SZ) show cognitive impairments on a wide range of tasks, with clear deficiencies in tasks reliant on prefrontal cortex function and less consistently observed impairments in tasks recruiting the striatum. This study leverages tasks hypothesized to differentially recruit these neural structures to assess relative deficiencies of each. Forty-eight patients and 38 controls completed two reinforcement learning tasks hypothesized to interrogate prefrontal and striatal functions and their interaction. In each task, participants learned reward discriminations by trial and error and were tested on novel stimulus combinations to assess learned values. In the task putatively assessing fronto-striatal interaction, participants were (inaccurately) instructed that one of the stimuli was valuable. Consistent with prior reports and a model of confirmation bias, this manipulation resulted in overvaluation of the instructed stimulus after its true value had been experienced. Patients showed less susceptibility to this confirmation bias effect than did controls. In the choice bias task hypothesized to more purely assess striatal function, biases in endogenously and exogenously chosen actions were assessed. No group differences were observed. In the subset of participants who showed learning in both tasks, larger group differences were observed in the confirmation bias task than in the choice bias task. In the confirmation bias task, patients also showed impairment in the task conditions with no prior instruction. This deficit was most readily observed on the most deterministic discriminations. Taken together, these results suggest impairments in fronto-striatal interaction in SZ, rather than in striatal function per se.