Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective

Research on human and animal behavior has long emphasized its hierarchical structure—the divisibility of ongoing behavior into discrete tasks, which are comprised of subtask sequences, which in turn are built of simple actions. The hierarchical structure of behavior has also been of enduring interest within neuroscience, where it has been widely considered to reflect prefrontal cortical functions. In this paper, we reexamine behavioral hierarchy and its neural substrates from the point of view of recent developments in computational reinforcement learning. Specifically, we consider a set of approaches known collectively as hierarchical reinforcement learning, which extend the reinforcement learning paradigm by allowing the learning agent to aggregate actions into reusable subroutines or skills. A close look at the components of hierarchical reinforcement learning suggests how they might map onto neural structures, in particular regions within the dorsolateral and orbital prefrontal cortex. It also suggests specific ways in which hierarchical reinforcement learning might provide a complement to existing psychological models of hierarchically structured behavior. A particularly important question that hierarchical reinforcement learning brings to the fore is that of how learning identifies new action routines that are likely to provide useful building blocks in solving a wide range of future problems. Here and at many other points, hierarchical reinforcement learning offers an appealing framework for investigating the computational and neural underpinnings of hierarchically structured behavior.     

Synaptic plasticity in the lateral amygdala: a cellular hypothesis of fear conditioning

Fear conditioning is a form of associative learning in which subjects come to express defense responses to a neutral conditioned stimulus (CS) that is paired with an aversive unconditioned stimulus (US). Considerable evidence suggests that critical neural changes mediating the CS-US association occur in the lateral nucleus of the amygdala (LA). Further, recent studies show that associative long-term potentiation (LTP) occurs in pathways that transmit the CS to LA, and that drugs that interfere with this LTP also disrupt behavioral fear conditioning when infused into the LA, suggesting that associative LTP in LA might be a mechanism for storing memories of the CS-US association. Here, we develop a detailed cellular hypothesis to explain how neural responses to the CS and US in LA could induce LTP-like changes that store memories during fear conditioning. Specifically, we propose that the CS evokes EPSPs at sensory input synapses onto LA pyramidal neurons, and that the US strongly depolarizes these same LA neurons. This depolarization, in turn, causes calcium influx through NMDA receptors (NMDARs) and also causes the LA neuron to fire action potentials. The action potentials then back-propagate into the dendrites, where they collide with CS-evoked EPSPs, resulting in calcium entry through voltage-gated calcium channels (VGCCs). Although calcium entry through NMDARs is sufficient to induce synaptic changes that support short-term fear memory, calcium entry through both NMDARs and VGCCs is required to initiate the molecular processes that consolidate synaptic changes into a long-term memory.     

Pavlov as a psychologist

American psychologists are informed on Pavlov’s work on conditional reflexes but not on the full development of his theory of higher nervous activity. This article shows that Pavlov’s theory of higher nervous activity dealt with concepts that concerned contemporary psychologists. Pavlov used the conditioning of the salivary reflex for methodological purposes. Pavlov’s theory of higher nervous activity encompassed overt behavior, neural processes, and the conscious experience. The strong Darwinian element of Pavlov’s theory, with its stress on the higher organisms’ adaptation, is described. With regard to learning, Pavlov, at the end of his scholarly career, proposed that although all learning involves the formation of associations, the organism’s adaptation to the environment is established through conditioning, but the accumulation of knowledge is established by trial and error.     

Dysfunctional Behavior among Sales Representatives: The Effect of Supervisory Trust,Participation, and Information Controls

This study investigates the efficacy of supervisory trust, participation, and information controls in curbing dysfunctional salesperson behavior so that salesperson actions are in line with organizational goals. Using a sample of 210 salespeople, we develop and test a model incorporating supervisory trust, participation, information controls (output information, activity information, and capability information), and dysfunctional behavior. Output and activity information controls directly affect dysfunctional behavior, whereas capability information controls work positively through trust in the supervisor to reduce dysfunctional behavior. Providing sales representatives with information about their capabilities appears to enhance the supervisor–salesperson trust relationship. Results also indicate that salespeople’s supervisory participation is an effective lever for reducing dysfunctional salesperson behavior through the intervening role of trust in the supervisor.     

Verbal conditioning of stimulus significance in the orienting response context

The manipulation of stimulus significance, by instructions from the experimenter, may be taken as an example of verbal conditioning. Consideration of such a mechanism suggested that personality effects previously found in conditioning studies should be apparent in instructional manipulations of significance in a study of the orienting response (OR) to words. Because of recent changes in dimensioning of the personality structure, some of the items originally used to define Eysenck’s extraversion (E) dimension are now used to assess the new dimension of psychoticism (P), suggesting that at least some of the established effects of E upon conditioning may be associated now with P. Hence the P scale was focused on in this study. Words differing on the evaluative dimension of the semantic differential were presented in three blocks, the first under indifferent instructions, the second under instructions to rate the words for their affective impact, and the third under indifferent instructions again. These blocks correspond to baseline, conditioning, and extinction conditions respectively. Electrodermal activity indicated enhanced conditioning, together with greater carry-over effects in the extinction phase, for low-P compared with high-P subjects. The results indicate the importance of personality effects in studies of stimulus significance and illustrate the value of the verbal conditioning mechanism in this area of the OR field. They also suggest the need to re-examine previously obtained E-effects in conditioning studies in light of changing personality tests.     

Transgenic inhibition of neuronal calcineurin activity in the forebrain facilitates fear conditioning, but inhibits the extinction of contextual fear memories

It is unclear whether protein phosphatases, which counteract the actions of protein kinases, play a beneficial role in the formation and extinction of previously acquired fear memories. In this study, we investigated the role of the calcium/calmodulin dependent phosphatase 2B, also known as calcineurin (CaN) in the formation of contextual fear memory and extinction of previously acquired contextual fear. We used a temporally regulated transgenic approach, that allowed us to selectively inhibit neuronal CaN activity in the forebrain either during conditioning or only during extinction training leaving the conditioning undisturbed. Reducing CaN activity through the expression of a CaN inhibitor facilitated contextual fear conditioning, while it impaired the extinction of previously formed contextual fear memory. These findings give the first genetic evidence that neuronal CaN plays an opposite role in the formation of contextual fear memories and the extinction of previously formed contextual fear memories.     

Responding to sexual harassment complaints: Effects of a dissolved workplace romance on decision-making standards

We introduce and provide support for an ethical decision-making framework as an explanation for the social–cognitive process through which observers make decisions about a sexual harassment complaint that stems from a prior workplace romance. We conducted two experiments to examine effects of features of a dissolved hierarchical workplace romance and subsequent harassing behavior on raters' responses to a sexual harassment complaint. In Experiment 1, results based on a sample of 217 employees indicate that their attributions of responsibility for the harassment mediated the link between their knowledge of features of the romance and three recommended personnel actions. In Experiment 2, results based on a sample of 258 members of the Society for Human Resource Management indicate that their degree of recognition of the accused's social–sexual behavior as immoral mediated the link between their knowledge of features of the romance and harassment and their attributions of responsibility. Raters' attributions of responsibility, in turn, predicted three recommended personnel actions. We discuss theoretical and practical implications from an ethical decision-making perspective.     

身体活动的双系统理论:一种强化学习的视角

以理性决策为基础的锻炼行为理论被认为是理解身体活动的主导体系, 它提供了与身体活动相关的认知构念作为有价值的信息。基于社会生态模型设计的行为干预措施, 因表现出了更好的效果而备受关注。近期研究表明, 积极的运动认知和当前体育环境都没能很好地促进个人锻炼习惯的养成, 因此有必要探索新的理论体系来阐明个人锻炼习惯的形成机制。解释身体活动的最新体系是双系统理论, 由于其考虑了身体活动的无意识和快乐决定因素, 有望提供一个更广泛的动机视角。一方面, 多个有代表性的身体活动双系统模型, 从简单的自发路径, 到情境线索与锻炼习惯, 再到突出自动情感评价作用的复杂概念模型, 阐明了系统1的构建, 结合锻炼行为理论所关注的系统2, 为模型的构建提供了依据。另一方面, 通过对双系统的竞争、协调和层级控制原则的分析, 为模型的控制提供了建议。经典的强化学习框架解释了双系统模型的构建与控制原则:在模型的构建方面, 无模型与基于模型的强化学习分别表示系统1和系统2。在模型的控制方面, Dyna协作架构与分层强化学习, 为身体活动可能是一种相互协作、分层执行的复杂行动组合提供了合理解释。最后提出强化学习视角下锻炼者-体育环境的互动模式, 试图从一个全新的角度探讨锻炼行为。     

Overt behavior and ultrasonic vocalization in a fear conditioning paradigm: a dose-response study in the rat

The behavioral analysis of laboratory rats is usually confined to the level of overt behavior, like locomotion, behavioral inhibition, instrumental responses, and others. Apart from such visible outcome, however, behaviorally relevant information can also be obtained when analyzing the animals' ultrasonic vocalization, which is typically emitted in highly motivational situations, like 22-kHz calls in response to acute or conditioned threat. To further investigate such vocalizations and their relationship with overt behavior, we tested male Wistar rats in a paradigm of Pavlovian fear conditioning, where a tone stimulus (CS) was preceding an aversive foot-shock (US) in a distinct environment. Importantly, the shock dose was varied between groups (0-1.1 mA), and its acute and conditioned outcome were determined. The analysis of visible behavior confirms the usefulness of immobility as a measure of fear conditioning, especially when higher shock doses were used. Rearing and grooming, on the other hand, were more useful to detect conditioned effects with lower shock levels. Ultrasonic vocalization occurred less consistently than changes in overt behavior; however, dose-response relationships were also observed during the phase of conditioning, for example, in latency, call rate and lengths, intervals between calls, and sound amplitude. Furthermore, total calling time (and rate) were highly correlated with overt behavior, namely behavioral inhibition as measured through immobility. These correlations were observed during the phase of fear conditioning, and the subsequent tests. Importantly, conditioned effects in overt behavior were observed, both, to the context and to the CS presented in this context, whereas conditioned vocalization to the context was not observed (except for one rat). In support and extent of previous results, the present data show that a detailed analysis of ultrasonic vocalization can substantially broaden and refine the spectrum of analysis in behavioral work with rats, since it can provide information about situational-, state-, and subject-dependent factors which are partly distinct from what is visible to the experimenter.     

Impaired appetitively as well as aversively motivated behaviors and learning in PDE10A-deficient mice suggest a role for striatal signaling in evaluative salience attribution

Phosphodiesterase 10A (PDE10A) hydrolyzes both cAMP and cGMP, and is a key element in the regulation of medium spiny neuron (MSN) activity in the striatum. In the present report, we investigated the effects of targeted disruption of PDE10A on spatial learning and memory as well as aversive and appetitive conditioning in C57BL/6 J mice. Because of its putative role in motivational processes and reward learning, we also determined the expression of the immediate early gene zif268 in striatum and anterior cingulate cortex. Animals showed decreased response rates in scheduled appetitive operant conditioning, as well as impaired aversive conditioning in a passive avoidance task. Morris water maze performance revealed not-motor related spatial learning and memory deficits. Anxiety and social explorative behavior was not affected in PDE10A-deficient mice. Expression of zif268 was increased in striatum and anterior cingulate cortex, which suggests alterations in the neural connections between striatum and anterior cingulate cortex in PDE10A-deficient mice. The changes in behavior and plasticity in these PDE10A-deficient mice were in accordance with the proposed role of striatal MSNs and corticostriatal connections in evaluative salience attribution.     

Pavlov and the foundation of behavior therapy

The foundation, achievements, and proliferation of behavior therapy have largely been fueled by the movement's foundation in behavioral principles and theories. Although behavioral accounts of the genesis and treatment of psychopathology differ in the extent to which they emphasize classical or operant conditioning, the mediation of cognitive factors, and the role of biological variables, Pavlov's discovery of conditioning principles was essential to the founding of behavior therapy in the 1950s, and continues to be central to modern behavior therapy. Pavlov's reliance on a physiological model of the nervous system, sensible in the context of an early science of neurology, has had an implication for behavior therapists interested in the study of personality types. However, Pavlov's major legacy to behavior therapy was his discovery of "experimental neuroses," shown by his students Eroféeva and Shenger-Krestovnikova, to be produced and eliminated through the principles of conditioning and counter-conditioning. This discovery laid the foundation for the first empirically-validated behavior therapy procedure, systematic desensitization, pioneered by Wolpe. The Pavlovian origins of behavior therapy are analyzed in this paper, and the relevance of conditioning principles to modern behavior therapy is demonstrated. It is shown that Pavlovian conditioning represents far more than a systematic basic learning paradigm. It is also an essential theoretical foundation for the theory and practice of behavior therapy.     

Formulation of a behavior system for sexual conditioning

Empirical and conceptual developments that led to the formulation of a behavior system for the sexual conditioning of male Japanese quail are described. Initial efforts concentrated on conditioning with localized conditioned stimuli and on identifying behavioral indices of conditioning. Later, learning about species-typical cues and about contextual cues was also explored, and it became evident that different types of cues control different aspects of sexual behavior. The results were used to formulate a behavior system containing both response and stimulus dimensions. In this system, contextual cues and local cues are assumed to elicit only general search behavior unconditionally. In contrast, unconditioned responses to species-typical cues of a female quail include general search, focal search, and copulatory behavior. General search, focal search, and copulatory behavior can become conditioned to local cues. Conditioning can also modify focal search behavior elicited by species-typical cues and can result in various modulatory influences between different types of stimuli. The behavior system approach provides a framework for organizing the diverse sexual conditioning effects and suggests future directions for investigation.     

Reward, dopamine and the control of food intake: implications for obesity

The ability to resist the urge to eat requires the proper functioning of neuronal circuits involved in top-down control to oppose the conditioned responses that predict reward from eating the food and the desire to eat the food. Imaging studies show that obese subjects might have impairments in dopaminergic pathways that regulate neuronal systems associated with reward sensitivity, conditioning and control. It is known that the neuropeptides that regulate energy balance (homeostatic processes) through the hypothalamus also modulate the activity of dopamine cells and their projections into regions involved in the rewarding processes underlying food intake. It is postulated that this could also be a mechanism by which overeating and the resultant resistance to homoeostatic signals impairs the function of circuits involved in reward sensitivity, conditioning and cognitive control.     

Classical fear conditioning in the anxiety disorders: a meta-analysis

Fear conditioning represents the process by which a neutral stimulus comes to evoke fear following its repeated pairing with an aversive stimulus. Although fear conditioning has long been considered a central pathogenic mechanism in anxiety disorders, studies employing lab-based conditioning paradigms provide inconsistent support for this idea. A quantitative review of 20 such studies, representing fear-learning scores for 453 anxiety patients and 455 healthy controls, was conducted to verify the aggregated result of this literature and to assess the moderating influences of study characteristics. Results point to modest increases in both acquisition of fear learning and conditioned responding during extinction among anxiety patients. Importantly, these patient-control differences are not apparent when looking at discrimination studies alone and primarily emerge from studies employing simple, single-cue paradigms where only danger cues are presented and no inhibition of fear to safety cues is required.     

An Analysis of Feedback from a Behavior Analytic Perspective

The present paper presents a systematic analysis from a behavior analytic perspective of procedures termed feedback. Although feedback procedures are widely reported in the discipline of psychology, including in the field of behavior analysis, feedback is neither consistently defined nor analyzed. Feedback is frequently treated as a principle of behavior; however, its effects are rarely analyzed in terms of well-established principles of learning and behavior analysis. On the assumption that effectiveness of feedback procedures would be enhanced when their use is informed by these principles, we sought to provide a conceptually systematic account of feedback effects in terms of operant conditioning principles. In the first comprehensive review of this type, we compare feedback procedures with those of well-defined operant procedures. We also compare the functional relations that have been observed between parameters of consequence delivery and behavior under both feedback and operant procedures. The similarities observed in the preceding analyses suggest that processes revealed in operant conditioning procedures are sufficient to explain the phenomena observed in studies on feedback.     

Comparing excitatory backward and forward conditioning

Three Pavlovian lick suppression studies with rats were conducted to compare the role of the conditioning context in excitatory backward and forward conditioning. The experiments explored the possibility that excitatory backward conditioning, but not forward conditioning, is mediated by the context. That is, in excitatory backward conditioning, the conditioning context may function as an excitatory mediator, which supports second-order conditioning of the target cue. This possibility contrasts with traditional accounts, which suggests that common processes underlie excitatory backward and forward conditioning. Experiment 1 found that conditioned responding following backward conditioning was attenuated as a result of posttraining extinction of the training context, but the same manipulation elevated responding after forward conditioning. Experiments 2 and 3 found that posttraining and pretraining associative inflation of the context (presenting unsignalled USs) increased conditioned responding to the target of a backward conditioning procedure but either had no effect or reduced responding to the target of a forward conditioning procedure. Thus, excitatory backward and forward conditioning appear to differ in their dependence on the status of the conditioning context.     

Impulsivity, aggression, and serotonin: a molecular psychobiological perspective

The expression of aggressiveness, which constitutes many facets of behavior, is influenced by a complex interaction of biologic, psychologic, and social variables. Even though individual differences in impulsivity and the behavioral consequences, such as aggression, addiction, and suicidality, are substantially heritable, they ultimately result from an interplay between genetic variations and environmental factors. While formation and integration of multiple neural networks is dependent on the actions of neurotransmitters, such as serotonin (5HT), converging lines of evidence indicate that genetically determined variability in serotonergic gene expression influences complex traits including that of inappropriately aggressive behavior. This contribution reviews studies of major gene effects in inbred and knockout strains of mice with increased aggression-related behavior and discusses the relevance of several serotonergic gene variations in humans which include high aggressiveness as part of the phenotype. Although special emphasis is given to the molecular psychobiology of 5HT in aggression-related behavior in rodents, nonhuman primates, and humans, relevant conceptual and methodological issues in the search for candidate genes for impulsivity and aggressiveness and for the development of mouse models of aggressive and antisocial behavior in humans are also considered.     

BDNF-induced synaptic delivery of AMPAR subunits is differentially dependent on NMDA receptors and requires ERK

Previous studies using an in vitro model of eyeblink classical conditioning in turtles suggest that increased numbers of synaptic AMPARs supports the acquisition and expression of conditioned responses (CRs). Brain-derived neurotrophic factor (BDNF) and its associated receptor tyrosine kinase, TrkB, is also required for acquisition of CRs. Bath application of BDNF alone induces synaptic delivery of GluR1- and GluR4-containing AMPARs that is blocked by coapplication of the receptor tyrosine kinase inhibitor K252a. The molecular mechanisms involved in BDNF-induced AMPAR trafficking remain largely unknown. The aim of this study was to determine whether BDNF-induced synaptic AMPAR incorporation utilizes similar cellular mechanisms as AMPAR trafficking that occurs during in vitro classical conditioning. Using pharmacological blockade and confocal imaging, the results show that synaptic delivery of GluR1 subunits during conditioning or BDNF application does not require activity of NMDARs but is mediated by extracellular signal-regulated kinase (ERK). In contrast, synaptic delivery of GluR4-containing AMPARs during both conditioning and BDNF application is NMDAR- as well as ERK-dependent. These findings indicate that BDNF application mimics AMPAR trafficking observed during conditioning by activation of some of the same intracellular signaling pathways and suggest that BDNF is a key signal transduction element in postsynaptic events that mediate conditioning.     

Tie my hands, tie my eyes

Previous studies have demonstrated that motor abilities allow us not only to execute our own actions and to predict their consequences, but also to predict others' actions and their consequences. But just how deeply are motor abilities implicated in action observation? If an observer is prevented from acting while witnessing others' actions, will this impact on their making sense of others' behavior? We recorded proactive eye movements while participants observed an actor grasping objects. The participants' hands were either freely resting on the table or tied behind their back. Proactivity of gaze behavior was dramatically impaired when participants observed others' actions with their hands tied. Since we don't literally perceive actions with our hands, the effect may be explained by the hypothesis that effective observation of action depends not only on motor abilities but on being in a position to exercise them. This suggests, for the first time, that actions are observed best when we are actually in the position to perform them.