A computational model of sequential movement learning with a signal mimicking dopaminergic neuron activities

We present a computational model of approach learning in a simulated maze environment. Our maze environment and training method mimics those used in the experimental literature. We show that our model learns the correct sequence of six decisions that lead to the location of positive reinforcement and in a manner consistent with experimental observations. Our model exhibits many properties that are characteristic of animal learning in maze environments including delay conditioning, secondary conditioning, and backward chaining. Finally, we map our model to the basal ganglia and show that a signal in our model that is responsible for learning has the same temporal properties as dopamine, the neurotransmitter believed to play an important part in learning decision sequences.     

Neural dynamics of object-based multifocal visual spatial attention and priming: object cueing, useful-field-of-view, and crowding

How are spatial and object attention coordinated to achieve rapid object learning and recognition during eye movement search? How do prefrontal priming and parietal spatial mechanisms interact to determine the reaction time costs of intra-object attention shifts, inter-object attention shifts, and shifts between visible objects and covertly cued locations? What factors underlie individual differences in the timing and frequency of such attentional shifts? How do transient and sustained spatial attentional mechanisms work and interact? How can volition, mediated via the basal ganglia, influence the span of spatial attention? A neural model is developed of how spatial attention in the where cortical stream coordinates view-invariant object category learning in the what cortical stream under free viewing conditions. The model simulates psychological data about the dynamics of covert attention priming and switching requiring multifocal attention without eye movements. The model predicts how “attentional shrouds” are formed when surface representations in cortical area V4 resonate with spatial attention in posterior parietal cortex (PPC) and prefrontal cortex (PFC), while shrouds compete among themselves for dominance. Winning shrouds support invariant object category learning, and active surface-shroud resonances support conscious surface perception and recognition. Attentive competition between multiple objects and cues simulates reaction-time data from the two-object cueing paradigm. The relative strength of sustained surface-driven and fast-transient motion-driven spatial attention controls individual differences in reaction time for invalid cues. Competition between surface-driven attentional shrouds controls individual differences in detection rate of peripheral targets in useful-field-of-view tasks. The model proposes how the strength of competition can be mediated, though learning or momentary changes in volition, by the basal ganglia. A new explanation of crowding shows how the cortical magnification factor, among other variables, can cause multiple object surfaces to share a single surface-shroud resonance, thereby preventing recognition of the individual objects.     

Learning to avoid in older age

The dopamine hypothesis of aging suggests that a monotonic dopaminergic decline accounts for many of the changes found in cognitive aging. The authors tested 44 older adults with a probabilistic selection task sensitive to dopaminergic function and designed to assess relative biases to learn more from positive or negative feedback. Previous studies demonstrated that low levels of dopamine lead to avoidance of those choices that lead to negative outcomes, whereas high levels of dopamine result in an increased sensitivity to positive outcomes. In the current study, age had a significant effect on the bias to avoid negative outcomes: Older seniors showed an enhanced tendency to learn from negative compared with positive consequences of their decisions. Younger seniors failed to show this negative learning bias. Moreover, the enhanced probabilistic integration of negative outcomes in older seniors was accompanied by a reduction in trial-to-trial learning from positive outcomes, thought to rely on working memory. These findings are consistent with models positing multiple neural mechanisms that support probabilistic integration and trial-to-trial behavior, which may be differentially impacted by older age.     

Individual differences in insular sensitivity during loss anticipation predict avoidance learning

The anterior insula has been implicated in both the experience and the anticipation of negative outcomes. Although individual differences in insular sensitivity have been associated with self-report measures of chronic anxiety, previous research has not examined whether individual differences in insular sensitivity predict learning to avoid aversive stimuli. In the present study, insular sensitivity was assessed as participants anticipated monetary losses while undergoing functional magnetic resonance imaging. We found that insular responsiveness to anticipated losses predicted participants' ability to learn to avoid losses (but not to approach gains) in a behavioral test several months later. These findings suggest that in addition to correlating with self-reported anxiety, heightened insular sensitivity may promote learning to avoid loss.     

A Biologically Plausible Action Selection System for Cognitive Architectures: Implications of Basal Ganglia Anatomy for Learning and Decision‐Making Models

Several attempts have been made previously to provide a biological grounding for cognitive architectures by relating their components to the computations of specific brain circuits. Often, the architecture's action selection system is identified with the basal ganglia. However, this identification overlooks one of the most important features of the basal ganglia—the existence of a direct and an indirect pathway that compete against each other. This characteristic has important consequences in decision‐making tasks, which are brought to light by Parkinson's disease as well as genetic differences in dopamine receptors. This paper shows that a standard model of action selection in a cognitive architecture (ACT‐R) cannot replicate any of these findings, details an alternative solution that reconciles action selection in the architecture with the physiology of the basal ganglia, and extends the domain of application of cognitive architectures. The implication of this solution for other architectures and existing models are discussed.     

The role of the basal ganglia in learning and memory: insight from Parkinson's disease

It has long been known that memory is not a single process. Rather, there are different kinds of memory that are supported by distinct neural systems. This idea stemmed from early findings of dissociable patterns of memory impairments in patients with selective damage to different brain regions. These studies highlighted the role of the basal ganglia in non-declarative memory, such as procedural or habit learning, contrasting it with the known role of the medial temporal lobes in declarative memory. In recent years, major advances across multiple areas of neuroscience have revealed an important role for the basal ganglia in motivation and decision making. These findings have led to new discoveries about the role of the basal ganglia in learning and highlighted the essential role of dopamine in specific forms of learning. Here we review these recent advances with an emphasis on novel discoveries from studies of learning in patients with Parkinson's disease. We discuss how these findings promote the development of current theories away from accounts that emphasize the verbalizability of the contents of memory and towards a focus on the specific computations carried out by distinct brain regions. Finally, we discuss new challenges that arise in the face of accumulating evidence for dynamic and interconnected memory systems that jointly contribute to learning.     

A modular neural-network model of the basal ganglia’s role in learning and selecting motor behaviours

This work presents a modular neural-network model (based on reinforcement-learning actor–critic methods) that tries to capture some of the most relevant known aspects of the role that basal ganglia play in learning and selecting motor behavior related to different goals. The model uses a mixture of experts network for the critic and a hierarchical network with two levels for the actor. Some simulations with the model show that basal ganglia select ‘chunks’ of behavior whose ‘details’ are specified by direct sensory-motor pathways, and how emergent modularity can help to deal with tasks with asynchronous multiple goals. A ‘top-down’ approach is adopted that first analyses some adaptive non-trivial interaction of a whole (simulated) organism with the environment, and its capacity to learn, and then attempts to implement these functions with neural architectures and mechanisms that have an empirical neuroanatomical and neurophysiological foundation.     

Well‐being and the accessibility of pleasant and unpleasant concepts

The trait–congruency hypothesis predicts that persons high in positive or negative trait affect more readily process pleasant or unpleasant stimuli, respectively. In two studies, participants were administered measures of personality and affect. Moreover, a yes/no lexical decision task with pleasant, unpleasant and neutral words was administered in Study 1, whereas a go/no‐go task was used in Study 2. Several methods to increase reliabilities of differences in reaction times are explored. Correlations of measures of personality and trait affect with decision times were mostly consistent with the trait–congruency hypothesis, particularly for decision times in the go/no‐go task that measured individual differences in valence‐specific decision times more reliably. The findings suggest that trait‐related concept accessibility is one source of trait congruity. Copyright © 2006 John Wiley & Sons, Ltd.     

Aging, emotion, and health-related decision strategies: motivational manipulations can reduce age differences

According to socioemotional selectivity theory, age-related constraints on time horizons are associated with motivational changes that increasingly favor goals related to emotional well-being. Such changes have implications for emotionally taxing tasks such as making decisions, especially when decisions require consideration of unpleasant information. This study examined age differences in information acquisition and recall in the health care realm. Using computer-based decision scenarios, 60 older and 60 young adults reviewed choice criteria that contained positive, negative, and neutral information about different physicians and health care plans. As predicted, older adults reviewed and recalled a greater proportion of positive than of negative information compared with young adults. Age differences were eliminated when motivational manipulations elicited information-gathering goals or when time perspective was controlled statistically. Implications for improving decision strategies in older adults are discussed.     

Defensive copers show a deficit in passive avoidance learning on Newman's go/no-go task: implications for self-deception and socialization

The present study investigated whether passive avoidance learning was retarded by defensive coping strategies designed to minimize exposure to negatively valenced stimuli. High-anxious individuals, low-anxious individuals, and defensive copers completed a computerized go/no-go task, in which they learned when to press or not to press a button, in response to contingent positive and negative feedback. The duration that feedback remained onscreen was self-regulated. Defensive copers showed preferential reflection away from negative feedback, committed more passive-avoidance errors, and were characterized by impaired learning, overall. Further, the ratio of reflection on negative feedback to reflection on positive feedback directly mediated both passive-avoidance errors and overall learning. Defensive coping strategies, therefore, appear to interfere with passive avoidance learning, thereby fostering perseverative, dysfunctional action patterns by reducing knowledge gained from previous mistakes. Implications for the learning of effective socialization strategies, and for psychopathy-which is commonly characterized by similar passive-avoidance deficits-are subsequently considered.     

Aging, working memory, and discrimination learning

Older adults easily learn probabilistic relationships between cues and outcomes when the predictive event is the occurrence of a cue, but have greater difficulty when the predictive event is the nonoccurrence of a cue (Mutter & Pliske, 1996; Mutter & Plumlee, 2004; Mutter & Williams, 2004). This study explored whether this age-related deficit occurs in a simpler learning context and whether it might be related to working memory (WM) decline. We gave younger and older adults simultaneous discrimination tasks that allowed us to compare their ability to learn deterministic relationships when either the occurrence (feature positive; FP) or the nonoccurrence (feature negative; FN) of a distinctive feature predicted reinforcement. We also included a group of younger adults who performed the discrimination tasks under a concurrent WM load. Both age and WM load had a detrimental effect on initial FP and FN discrimination; however, these effects persisted only in FN discrimination after additional learning experience. Learning predictive relationships requires inductive reasoning processes that apparently do not operate as efficiently in individuals with reduced WM capacity. The impact of WM decline may ultimately be greater for negative cue-outcome relationships because learning these relationships requires more difficult inductive reasoning processes, which place greater demands on WM.     

Age differences in financial decision making: The benefits of more experience and less negative emotions

The emerging literature on aging and decision making posits that decision‐making competence changes with age, as a result of age differences in various cognitive and noncognitive individual‐differences characteristics. In a national life‐span sample from the United Kingdom (N = 926), we examined age differences in financial decisions, including performance measures of sunk cost and credit card repayment decisions, and self‐report measures of money management and financial decision outcomes. Participants also completed four individual‐differences characteristics that have been proposed as relevant to financial decision making, including two cognitive ones (numeracy and experience‐based knowledge) and two noncognitive ones (negative emotions about financial decisions). First, we examined how age was related to the four financial decision‐making measures and the four individual‐differences characteristics. Older age was correlated to better scores on each of the four financial decision‐making measures, more experience‐based knowledge, less negative emotions about financial decisions, whereas numeracy and motivation were not significantly correlated with age. Second, we found that considering both the two cognitive and the two noncognitive individual‐differences characteristics increased predictions of financial decision making, as compared with considering either alone. Third, we examined how these four individual‐differences characteristics contributed to age differences in financial decision making. Older adults' higher levels of experience‐based knowledge and lower levels of negative emotions seemed to especially benefit their financial decision making. We discuss implications for theories on aging and decision making, as well as for interventions targeting financial decisions.     

Aging,working memory,and discrimination learning

Older adults easily learn probabilistic relationships between cues and outcomes when the predictive event is the occurrence of a cue, but have greater difficulty when the predictive event is the nonoccurrence of a cue (Mutter & Pliske, 1996; Mutter & Plumlee, 2004; Mutter & Williams, 2004). This study explored whether this age-related deficit occurs in a simpler learning context and whether it might be related to working memory (WM) decline. We gave younger and older adults simultaneous discrimination tasks that allowed us to compare their ability to learn deterministic relationships when either the occurrence (feature positive; FP) or the nonoccurrence (feature negative; FN) of a distinctive feature predicted reinforcement. We also included a group of younger adults who performed the discrimination tasks under a concurrent WM load. Both age and WM load had a detrimental effect on initial FP and FN discrimination; however, these effects persisted only in FN discrimination after additional learning experience. Learning predictive relationships requires inductive reasoning processes that apparently do not operate as efficiently in individuals with reduced WM capacity. The impact of WM decline may ultimately be greater for negative cue–outcome relationships because learning these relationships requires more difficult inductive reasoning processes, which place greater demands on WM.     

Individual differences in the anterior insula are associated with the likelihood of financially helping versus harming others

The neural basis of individual differences in positive and negative social decisions and behaviors in healthy populations is yet undetermined. Recent work has focused on the potential role of the anterior insula in guiding social and nonsocial decision making, but the specific nature of its activation during such decision making remains unclear. To identify the neural regions mediating individual differences in helpful and harmful decisions and to assess the nature of insula activation during such decisions, in the present study we used a novel fMRI task featuring intentional and unintentional decisions to financially harm or help persons in need. Based on a whole-brain, unbiased approach, our findings indicate that individual differences in dorsal anterior insula, anterior cingulate cortex (ACC), and right temporo-parietal junction activation are associated with behavioral tendencies to financially harm or help another. Furthermore, activity in the dorsal anterior insula and ACC was greatest during unintended outcomes, whether these were gains or losses for a charity or for oneself, supporting models of the role of these regions in salience prediction error signaling. Together, the results suggest that individual differences in risk anticipation, as reflected in the dorsal anterior insula and dorsal ACC, guide social decisions to refrain from harming others.     

A neural model of hippocampal–striatal interactions in associative learning and transfer generalization in various neurological and psychiatric patients

Building on our previous neurocomputational models of basal ganglia and hippocampal region function (and their modulation by dopamine and acetylcholine, respectively), we show here how an integration of these models can inform our understanding of the interaction between the basal ganglia and hippocampal region in associative learning and transfer generalization across various patient populations. As a common test bed for exploring interactions between these brain regions and neuromodulators, we focus on the acquired equivalence task, an associative learning paradigm in which stimuli that have been associated with the same outcome acquire a functional similarity such that subsequent generalization between these stimuli increases. This task has been used to test cognitive dysfunction in various patient populations with damages to the hippocampal region and basal ganglia, including studies of patients with Parkinson’s disease (PD), schizophrenia, basal forebrain amnesia, and hippocampal atrophy. Simulation results show that damage to the hippocampal region—as in patients with hippocampal atrophy (HA), hypoxia, mild Alzheimer’s (AD), or schizophrenia—leads to intact associative learning but impaired transfer generalization performance. Moreover, the model demonstrates how PD and anterior communicating artery (ACoA) aneurysm—two very different brain disorders that affect different neural mechanisms—can have similar effects on acquired equivalence performance. In particular, the model shows that simulating a loss of dopamine function in the basal ganglia module (as in PD) leads to slow acquisition learning but intact transfer generalization. Similarly, the model shows that simulating the loss of acetylcholine in the hippocampal region (as in ACoA aneurysm) also results in slower acquisition learning. We argue from this that changes in associative learning of stimulus–action pathways (in the basal ganglia) or changes in the learning of stimulus representations (in the hippocampal region) can have similar functional effects.     

Avoiding negative vs. achieving positive outcomes in hard and prosperous economic times

Three studies suggest that business cycle fluctuations trigger distinct motivational orientations that selectively affect economic judgment and decision making. Economic contractions induce avoidance motivation and affect negative economic sentiment, but leave approach motivation and positive economic sentiment unaffected. In contrast, economic expansions induce approach motivation and positive economic sentiment, but do not affect avoidance motivation or negative economic sentiment (study 1). Moreover, economic contractions induce risk aversion for negative outcomes, but not for positive outcomes, while economic expansions instigate risk seeking for positive outcomes, but not for negative outcomes (study 2). A time-series study based on consumer spending over eight decades mirrors the findings of the experimental studies: The consumption of products associated with avoiding negative outcomes increases during economic contractions, but not during expansions. In contrast, the consumption of products associated with achieving positive outcomes increases in expansions, but is unaffected by contractions (study 3).     

Reaction to punishment, reflectivity, and passive avoidance learning in extraverts

Passive avoidance learning occupies a central role in accounts of disinhibited behavior, ranging from psychopaths' persistent criminality (Hare 1970) to extraverts' gregariousness (Gray, 1972). To explore the mechanism underlying passive avoidance deficits, we assessed the relation of extraversion, neuroticism, and response latency after punishment to passive avoidance learning by using two successive go/no go discrimination tasks. The tasks were designed to examine two aspects of subjects' reactions to punishment: response speed on trials immediately following punishment (Experiment 1) and time to terminate punishment feedback between successive trials (i.e., reflectivity; Experiment 2). Consistent with previous findings, the results of Experiment 1 showed that extraverts commit more passive avoidance errors than introverts do (Newman, Widom, & Nathan, 1985) and fail to pause following punished errors (Nichols & Newman, 1986). In Experiment 2, only neurotic extraverts displayed this pattern of performance differences. In both experiments, longer pausing following punishment predicted better learning from punishment for both introverts and extraverts. These results suggest that, in the presence of salient cues for reward, extraverts' characteristic reaction to punishment interferes with processing punished errors and may contribute to their more general propensity for impulsive, nonreflective action.     

What you want (and do not want) affects what you see (and do not see): avoidance social goals and social events

Two studies examined the influence of approach and avoidance social goals on memory for and evaluation of ambiguous social information. Study 1 found that individual differences in avoidance social goals were associated with greater memory of negative information, negatively biased interpretation of ambiguous social cues, and a more pessimistic evaluation of social actors. Study 2 experimentally manipulated social goals and found that individuals high in avoidance social motivation remembered more negative information and expressed more dislike for a stranger in the avoidance condition than in the approach condition. Results suggest that avoidance social goals are associated with emphasizing potential threats when making sense of the social environment.     

The development of children's ideal and ought self-guides: parenting, temperament, and individual differences in guide strength

Regulatory focus theory (RFT; Higgins, 1997) predicts that individual differences in the strength of promotion (ideal) and prevention (ought) orientations emerge from patterns of parent/child interactions that emphasize making good things happen versus keeping bad things from happening. This article examines the development of individual differences in the strength of children's promotion and prevention goals and presents selected findings from three studies exploring the origins of regulatory focus. We found a three-factor structure for parenting behaviors that differentiated between the presence/absence of positive outcomes versus the presence/absence of negative outcomes in two different data sets and validated that factor structure by examining its associations with maternal temperament. In turn, the parenting factors predicted individual differences in children's orientations to ideal and ought guides, and those associations were moderated by individual differences in child temperament.