Electrophysiological measures reveal the role of anterior cingulate cortex in learning from unreliable feedback

Although a growing number of studies have investigated the neural mechanisms of reinforcement learning, it remains unclear how the brain responds to feedback that is unreliable. A recent theory proposes that the reward positivity (RewP) component of the event-related brain potential (ERP) and frontal midline theta (FMT) power reflect separate feedback-related processing functions of anterior cingulate cortex (ACC). In the present study, the electroencephalogram (EEG) was recorded from participants as they engaged in a time estimation task in which feedback reliability was manipulated across conditions. After each response, they received a cue that indicated that the following feedback stimulus was 100%, 75%, or 50% reliable. The results showed that participants’ time estimates adjusted linearly according to the feedback reliability. Moreover, presentation of the cue indicating 100% reliability elicited a larger RewP-like ERP component than the other cues did, and feedback presentation elicited a RewP of approximately equal amplitude for all of the three reliability conditions. By contrast, FMT power elicited by negative feedback decreased linearly from the 100% condition to 75% and 50% condition, and only FMT power predicted behavioral adjustments on the following trials. In addition, an analysis of Beta power and cross-frequency coupling (CFC) of Beta power with FMT phase suggested that Beta-FMT communication modulated motor areas for the purpose of adjusting behavior. We interpreted these findings in terms of the hierarchical reinforcement learning account of ACC, in which the RewP and FMT are proposed to reflect reward processing and control functions of ACC, respectively.     

Goals and task difficulty expectations modulate striatal responses to feedback

The striatum plays a critical role in learning from reward, and it has been implicated in learning from performance-related feedback as well. Positive and negative performance-related feedback is known to engage the striatum during learning by eliciting a response similar to the reinforcement signal for extrinsic rewards and punishments. Feedback is an important tool used to teach new skills and promote healthful lifestyle changes, so it is important to understand how motivational contexts can modulate its effectiveness at promoting learning. While it is known that striatal responses scale with subjective factors influencing the desirability of rewards, it is less clear how expectations and goals might modulate the striatal responses to cognitive feedback during learning. We used functional magnetic resonance imaging to investigate the effects of task difficulty expectations and achievement goals on feedback processing during learning. We found that individuals who scored high in normative goals, which reflect a desire to outperform other students academically, showed the strongest effects of our manipulation. High levels of normative goals were associated with greater performance gains and exaggerated striatal sensitivity to positive versus negative feedback during blocks that were expected to be more difficult. Our findings suggest that normative goals may enhance performance when difficulty expectations are high, while at the same time modulating the subjective value of feedback as processed in the striatum.     

Self-control of feedback during motor learning: accounting for the absolute amount of feedback using a yoked group with self-control over feedback

A traditional control group yoked to a group that self-controls their reception of feedback receives feedback in the same relative and absolute manner. This traditional control group typically does not learn the task as well as the self-control group. Although the groups are matched for the amount of feedback they receive, the information is provided on trials in which the individual may not request feedback if he or she were provided the opportunity. Similarly, individuals may not receive feedback on trials for which it would be a beneficial learning experience. Subsequently, the mismatch between the provision of feedback and the potential learning opportunity leads to a decrement in retention. The present study was designed to examine motor learning for a yoked group with the same absolute amount of feedback, but who could self-control when they received feedback. Increased mental processing of error detection and correction was expected for the participants in the yoked self-control group because of their choice to employ a limited resource in the form of a decreasing amount of feedback opportunities. Participants in the yoked with self-control group committed fewer errors than the self-control group in retention and the traditional yoked group in both the retention and time transfer blocks. The results suggest that the yoked with self-control group was able to produce efficient learning effects and can be a viable control group for further motor learning studies.     

Optimal classifier feedback improves cost-benefit but not base-rate decision criterion learning in perceptual categorization

Unequal payoffs engender separate reward- and accuracy-maximizing decision criteria; unequal base rates do not. When payoffs are unequal, observers place greater emphasis on accuracy than is optimal. This study compares objective classifier (the objectively correct response) with optimal classifier feedback (the optimal classifier's response) when payoffs or base rates are unequal. It provides a critical test of Maddox and Bohil's (1998) competition between reward and accuracy maximization (COBRA) hypothesis, comparing it with a competition between reward and probability matching (COBRM) and a competition between reward and equal response frequencies (COBRE) hypothesis. The COBRA prediction that optimal classifier feedback leads to better decision criterion leaning relative to objective classifier feedback when payoffs are unequal, but not when base rates are unequal, was supported. Model-based analyses suggested that the weight placed on accuracy was reduced for optimal classifier feedback relative to objective classifier feedback. In addition, delayed feedback affected learning of the reward-maximizing decision criterion.     

Learning from delayed feedback: neural responses in temporal credit assignment

When feedback follows a sequence of decisions, relationships between actions and outcomes can be difficult to learn. We used event-related potentials (ERPs) to understand how people overcome this temporal credit assignment problem. Participants performed a sequential decision task that required two decisions on each trial. The first decision led to an intermediate state that was predictive of the trial outcome, and the second decision was followed by positive or negative trial feedback. The feedback-related negativity (fERN), a component thought to reflect reward prediction error, followed negative feedback and negative intermediate states. This suggests that participants evaluated intermediate states in terms of expected future reward, and that these evaluations supported learning of earlier actions within sequences. We examine the predictions of several temporal-difference models to determine whether the behavioral and ERP results reflected a reinforcement-learning process.     

Medial frontal event-related potentials and reward prediction: do responses matter?

Medial frontal event-related potentials (ERPs) following rewarding feedback index outcome evaluation. The majority of studies examining the feedback related medial frontal negativity (MFN) employ active tasks during which participants’ responses impact their feedback, however, the MFN has been elicited during passive tasks. Many of the studies examining the MFN show enhanced effects when an error in reward prediction occurs (i.e. expected rewards are not delivered). To clarify the roles of reward prediction error and active responding in producing the MFN, the current study employed a reward prediction design with active and passive task blocks. Following the presentation of a reward predictor, participants (active task) or the computer (passive task) indicated whether participants would receive the outcome associated with a stimulus presented on the left or right of the reward predictor. The MFN was largest when the trial outcome was worse than predicted and this effect was enhanced when the participant, rather than the computer, made the choice. These results show that both reward prediction error and active choice impact the neural system of outcome monitoring with the largest MFN when the individual’s decision led to the negative outcome.     

On the role of reduced auditory feedback and kinesic self-stimulation during Stroop Color-Word performance1

Feedback from one's own voice provides important vocal–motor cues for effective cognitive processing. Reduction of such feedback is known to disturb such functioning. Work in our laboratory has shown that kinesic self-stimulation also plays an important role in cognition, and appears to regulate the focusing of attention under conditions of distraction. The present study investigated the effects of both auditory feedback and kinesic self-stimulation in the regulation of cognitive interference during performance of the Stroop Color-Word Task. Twelve subjects were tested on the Stroop task under conditions of normal and occluded hearing. Kinesic self-stimulation and response errors during color-word performance were recorded on video tape. The findings indicated that not only did self-stimulation increase when voice feedback was reduced, but that this increase was associated with a reduction in specific types of color-word performance errors. Individual differences revealed that high kinesic responders made significantly fewer errors in task performance than did low kinesic responders. Results were interpreted as revealing a kinesic feedback mechanism which has adaptive significance in regard to self-editing when auditory feedback is reduced.     

Antipsychotic dose modulates behavioral and neural responses to feedback during reinforcement learning in schizophrenia

Schizophrenia is characterized by an abnormal dopamine system, and dopamine blockade is the primary mechanism of antipsychotic treatment. Consistent with the known role of dopamine in reward processing, prior research has demonstrated that patients with schizophrenia exhibit impairments in reward-based learning. However, it remains unknown how treatment with antipsychotic medication impacts the behavioral and neural signatures of reinforcement learning in schizophrenia. The goal of this study was to examine whether antipsychotic medication modulates behavioral and neural responses to prediction error coding during reinforcement learning. Patients with schizophrenia completed a reinforcement learning task while undergoing functional magnetic resonance imaging. The task consisted of two separate conditions in which participants accumulated monetary gain or avoided monetary loss. Behavioral results indicated that antipsychotic medication dose was associated with altered behavioral approaches to learning, such that patients taking higher doses of medication showed increased sensitivity to negative reinforcement. Higher doses of antipsychotic medication were also associated with higher learning rates (LRs), suggesting that medication enhanced sensitivity to trial-by-trial feedback. Neuroimaging data demonstrated that antipsychotic dose was related to differences in neural signatures of feedback prediction error during the loss condition. Specifically, patients taking higher doses of medication showed attenuated prediction error responses in the striatum and the medial prefrontal cortex. These findings indicate that antipsychotic medication treatment may influence motivational processes in patients with schizophrenia.     

Positional cues in serial learning: The spin-list technique

To test the hypothesis that serial learning depends largely on the encoding and retrieval of position-to-item associations, we examined whether people can learn spin lists on which starting position is randomly varied across successive learning trials. By turning positional information from a reliable cue into a source of intertrial interference, we expected learning to be greatly impaired. Contrary to this hypothesis, we found that participants were only slightly worse at serial learning under spin conditions and that this impairment reflects a substantial increase in initiation errors coupled with a small increase in intertrial forgetting. These data show that participants can effectively use nonpositional cues when positional cues are unreliable.     

People newly in love are more responsive to positive feedback

Passionate love is associated with increased activity in dopamine-rich regions of the brain. Increased dopamine in these regions is associated with a greater tendency to learn from reward in trial-and-error learning tasks. This study examined the prediction that individuals who were newly in love would be better at responding to reward (positive feedback). In test trials, people who were newly in love selected positive outcomes significantly more often than their single (not in love) counterparts but were no better at the task overall. This suggests that people who are newly in love show a bias toward responding to positive feedback, which may reflect a general bias towards reward-seeking.     

A feedback model of perceptual learning and categorization

Top-down, feedback, influences are known to have significant effects on visual information processing. Such influences are also likely to affect perceptual learning. This paper employs a computational model of the cortical region inter-actions underlying visual perception to investigate possible influences of top-down information on learning. The results suggest that feedback could bias the way in which perceptual stimuli are categorized and could also facilitate the learning of subordinate level representations suitable for object identification and perceptual expertise.     

Motor skill acquisition and retention as a function of average feedback, summary feedback, and performance variability

Summary feedback involves withholding feedback from subjects until the last trial in a block is completed, and then presenting feedback about each trial. A variation of this method, called average feedback (Young & Schmidt, 1992), presents subjects with only the mean of the trial block. We investigated whether these methods have similar effects on acquisition and retention of a simple motor skill. Five groups of subjects (n = 16 per group) performed 60 acquisition trials of an aiming task involving both spatial and temporal accuracy. We presented average and summary feedback based on either 5-trial blocks or 15-trial blocks and compared these schedules with every-trial feedback. During acquisition, all groups improved with practice, with a slight tendency for the every-trial condition to have less absolute error than the longer summary and average conditions. Analysis of delayed no-feedback retention tests, however, revealed a strong advantage for the 5-trial summary and average conditions compared with the every-trial condition. In addition, we found that for long blocks of acquisition trials without augmented feedback, the performance variability of those trials was associated with retention performance. Results are discussed in terms of how these different manipulations may make feedback less useful during acquisition, but foster the use of certain information processing activities that enhance overall learning.     

Prediction Errors and Attention in the Presence and Absence of Feedback

ABSTRACT— Contemporary theories of learning typically assume that learning is driven by prediction errors—in other words, that we learn more when our predictions turn out to be incorrect than we do when our predictions are correct. Results from the recording of electrical brain activity suggest one mechanism by which this might happen; we seem to direct visual attention toward the likely causes of previous prediction errors. This can happen very rapidly—within less than 200 milliseconds of the error-causing object being presented. It is tempting to infer that if learning is driven by prediction errors, then little can be learned in the absence of feedback. Such a conclusion is unwarranted. In fact, the substantial learning that is sometimes the result of simple exposure to objects can also be explained by processes of directing attention toward the likely causes of previous prediction errors.     

Warning signal termination and a feedback signal may not serve the same function

Three experiments were conducted to test the possibility that a feedback signal (FS) and warning signal termination (WST), while equally reinforcing in the avoidance learning situation, reinforce through different underlying mechanisms. The first experiment showed that the reinforcing properties of an FS are reduced more than those of WST when these stimulus changes are made unreliable by the presence of shock following a CR on specified trials throughout acquisition. Experiment 2 confirmed this effect on avoidance performance when only a few punishment trials were administered following asymptotic avoidance acquisition. Experiment 3 demonstrated this effect during avoidance extinction with and without the presentation of punishment trials between acquisition and extinction performance. The results provided no support for the expectancy explanation of avoidance learning and were interpreted as consistent with the assumption that WST reinforces by permitting fear to dissipate and that the FS reinforces through fear inhibition.     

Adult age differences in frontostriatal representation of prediction error but not reward outcome

Emerging evidence from decision neuroscience suggests that although younger and older adults show similar frontostriatal representations of reward magnitude, older adults often show deficits in feedback-driven reinforcement learning. In the present study, healthy adults completed reward-based tasks that did or did not depend on probabilistic learning, while undergoing functional neuroimaging. We observed reductions in the frontostriatal representation of prediction errors during probabilistic learning in older adults. In contrast, we found evidence for stability across adulthood in the representation of reward outcome in a task that did not require learning. Together, the results identify changes across adulthood in the dynamic coding of relational representations of feedback, in spite of preserved reward sensitivity in old age. Overall, the results suggest that the neural representation of prediction error, but not reward outcome, is reduced in old age. These findings reveal a potential dissociation between cognition and motivation with age and identify a potential mechanism for explaining changes in learning-dependent decision making in old adulthood.     

Age differences in the neural response to negative feedback

Affective processing is one domain that remains relatively intact in healthy aging. Investigations into the neural responses associated with reward anticipation have revealed that older and younger adults recruit the same midbrain reward regions, but other evidence suggests this recruitment may differ depending on the valence (gain, loss) of the incentive cue. The goal of the current study was to examine functional covariance during gain and loss feedback in younger and healthy older adults. A group of 15 older adults (mean age = 68.5) and 16 younger adults (mean age = 25.4) completed a revised Monetary Incentive Delay task (rMID; Knutson, Westdorp, Kaiser, &; Hommer, 2000) while in the fMRI scanner. The rMID is a reaction time task where successful performance, either gaining a reward or avoiding a loss, is defined by hitting a button during the brief presentation of a visual target. Participants receive gain and loss anticipation cues before each trial and feedback after each trial with four possible outcomes: +$5.00, +0.00, -$5.00, and -Affective processing is one domain that remains relatively intact in healthy aging. Investigations into the neural responses associated with reward anticipation have revealed that older and younger adults recruit the same midbrain reward regions, but other evidence suggests this recruitment may differ depending on the valence (gain, loss) of the incentive cue. The goal of the current study was to examine functional covariance during gain and loss feedback in younger and healthy older adults. A group of 15 older adults (mean age = 68.5) and 16 younger adults (mean age = 25.4) completed a revised Monetary Incentive Delay task (rMID; Knutson, Westdorp, Kaiser, & Hommer, 2000) while in the fMRI scanner. The rMID is a reaction time task where successful performance, either gaining a reward or avoiding a loss, is defined by hitting a button during the brief presentation of a visual target. Participants receive gain and loss anticipation cues before each trial and feedback after each trial with four possible outcomes: +$5.00, +0.00, -$5.00, and -$0.00. Using seed-voxel partial least squares analyses, with seed voxels in the caudate and ventromedial prefrontal cortex, whole-brain functional covariance revealed that younger and older adults engage the same network of regions to support general feedback processing. However, older adults engaged two additional networks to support processing of negative feedback, gain_miss (+0), loss_miss (-$5), and loss_hit (?0), specifically. These findings are in line with theories of a positivity effect in aging and may have implications for reward-stimulus learning and decision making following performance-contingent negative feedback.     

Learning words from reliable and unreliable speakers

Three studies examined whether 3- and 4-year olds would trust a reliable speaker over an unreliable speaker when learning a new word and whether that trust would be reversed, and the word mapping revised, when a trusted speaker later proved unreliable. Study 1 indicated that 3- and 4-year olds trusted a reliable speaker over an unreliable speaker. Study 2 indicated that some 4-year olds reversed trust and revised a word mapping when a trusted speaker later proved unreliable. Study 3 indicated that those 4-year olds who reversed trust and revised the word mapping were likely to maintain the revision and tended to favor the previously reliable speaker over time. These results are discussed in terms of the role of speaker reliability in young children's word learning.     

Empathy and feedback processing in active and observational learning

The feedback-related negativity (FRN) and the P300 have been related to the processing of one’s own and other individuals’ feedback during both active and observational learning. The aim of the present study was to elucidate the role of trait-empathic responding with regard to the modulation of the neural correlates of observational learning in particular. Thirty-four healthy participants completed an active and an observational learning task. On both tasks, the participants’ aim was to maximize their monetary gain by choosing from two stimuli the one that showed the higher probability of reward. Participants gained insight into the stimulus–reward contingencies according to monetary feedback presented after they had made an active choice or by observing the choices of a virtual partner. Participants showed a general improvement in learning performance on both learning tasks. P200, FRN, and P300 amplitudes were larger during active, as compared with observational, learning. Furthermore, nonreward elicited a significantly more negative FRN than did reward in the active learning task, while only a trend was observed for observational learning. Distinct subcomponents of trait cognitive empathy were related to poorer performance and smaller P300 amplitudes for observational learning only. Taken together, both the learning performance and event-related potentials during observational learning are affected by different aspects of trait cognitive empathy, and certain types of observational learning may actually be disrupted by a higher tendency to understand and adopt other people’s perspectives.     

Cognitive demands of error processing

This study used a dual-task methodology to assess attention demands associated with error processing during an anticipation-timing task. A difference was predicted in attention demands during feedback on trials with correct responses and errors. This was addressed by requiring participants to respond to a probe reaction-time stimulus after augmented feedback presentation. 16 participants (8 men, 8 women) completed two phases, the reaction time task only and the anticipation-timing task with the probe RT task. False feedback indicating error and a financial reward manipulation were used to increase relevance of errors. Data supported the hypothesis that error processing is associated with higher cognitive demands than processing feedback denoting a correct response. Individuals responded with quicker probe reaction times during presentation of feedback on correct trials than on error trials. These results are discussed with respect to the cognitive processes which might occur during error processing and their role in motor learning.     

Anger and attitudinal reactions to negative feedback: The effects of emotional instability and power

Feedback is a basic tool that is used to stimulate learning and performance at all organizational levels. However, negative feedback can sometimes evoke defensive responses such as feelings of anger or the repudiation of the feedback. In two experiments we explored whether people’s negating responses to feedback are grounded in their emotional instability, and if this effect is stronger for those who hold more power. The findings from Study 1 (N = 84) showed that in response to negative feedback more emotionally unstable individuals experienced more anger. In Study 2 (N = 47) we indicated that anger mediated the negative effects of emotional instability and power on liking of the feedback provider, perceived ability of the feedback provider, and feedback acceptance. Our findings indicate that power strengthens the influence of emotional instability on responses to negative feedback and point to the importance of anger as the underlying factor influencing crucial attitudinal feedback reactions.