Catastrophic forgetting in connectionist networks

All natural cognitive systems, and, in particular, our own, gradually forget previously learned information. Plausible models of human cognition should therefore exhibit similar patterns of gradual forgetting of old information as new information is acquired. Only rarely does new learning in natural cognitive systems completely disrupt or erase previously learned information; that is, natural cognitive systems do not, in general, forget ‘catastrophically’. Unfortunately, though, catastrophic forgetting does occur under certain circumstances in distributed connectionist networks. The very features that give these networks their remarkable abilities to generalize, to function in the presence of degraded input, and so on, are found to be the root cause of catastrophic forgetting. The challenge in this field is to discover how to keep the advantages of distributed connectionist networks while avoiding the problem of catastrophic forgetting. In this article the causes, consequences and numerous solutions to the problem of catastrophic forgetting in neural networks are examined. The review will consider how the brain might have overcome this problem and will also explore the consequences of this solution for distributed connectionist networks.     

Comparing representations and computations in single neurons versus neural networks

Single-neuron-level explanations have been the gold standard in neuroscience for decades. Recently, however, neural-network-level explanations have become increasingly popular. This increase in popularity is driven by the fact that the analysis of neural networks can solve problems that cannot be addressed by analyzing neurons independently. In this opinion article, I argue that while both frameworks employ the same general logic to link physical and mental phenomena, in many cases the neural network framework provides better explanatory objects to understand representations and computations related to mental phenomena. I discuss what constitutes a mechanistic explanation in neural systems, provide examples, and conclude by highlighting a number of the challenges and considerations associated with the use of analyses of neural networks to study brain function.     

Capturing advanced human cognitive abilities with deep neural networks

How can artificial neural networks capture the advanced cognitive abilities of pioneering scientists? I suggest they must learn to exploit human-invented tools of thought and human-like ways of using them, and must engage in explicit goal-directed problem solving as exemplified in the activities of scientists and mathematicians and taught in advanced educational settings.     

Greatest surprise reduction semantics: an information theoretic solution to misrepresentation and disjunction

Causal theories of content, a popular family of approaches to defining the content of mental states, commonly run afoul of two related and serious problems that prevent them from providing an adequate theory of mental content—the misrepresentation problem and the disjunction problem. In this paper, I present a causal theory of content, built on information theoretic tools, that solves these problems and provides a viable model of mental content. This is the greatest surprise reduction theory of content, which identifies the content of a signal as the event the surprisal of which is most reduced by that signal. Conceptually, this amounts to the claim that the content of a signal is the event the probability of which has increased by the largest proportion, or the event that the signal makes the most less surprising to us. I develop the greatest surprise reduction theory of content in four stages. First, I introduce the general project of causal theories of content, and the challenges presented to this project by the misrepresentation and disjunction problems. Next, I review two recent and prominent causal theories of content and demonstrate the serious challenges faced by these approaches, both clarifying the need for a solution to the misrepresentation and disjunction problems and providing a conceptual background for the greatest surprise reduction theory. Then, I develop the greatest surprise reduction theory of content, demonstrate its ability to resolve the misrepresentation and disjunction problems, and explore some additional applications it may have. Finally, I conclude with a discussion of a particularly difficult challenge that remains to be addressed—the partition problem—and sketch a path to a potential solution.

     

Is it worth generating rules from neural network ensembles?

Although many authors generated comprehensible models from individual networks, much less work has been done in the explanation of ensembles. DIMLP is a special neural network model from which rules are generated at the level of a single network and also at the level of an ensemble of networks. We applied ensembles of 25 DIMLP networks to several datasets of the public domain and a classification problem related to post-translational modifications of proteins. For the classification problems of the public domain, the average predictive accuracy of rulesets extracted from ensembles of neural networks was significantly better than the average predictive accuracy of rulesets generated from ensembles of decision trees. By varying the architectures of DIMLP networks we found that the average predictive accuracy of rules, as well as their complexity were quite stable. The comparison to other rule extraction techniques applied to neural networks showed that rules generated from DIMLP ensembles gave very good results. In the last problem related to bioinformatics, the best result obtained by ensembles of DIMLP networks was also significantly better than the best result obtained by ensembles of decision trees. Thus, although neural networks take much longer to train than decision trees and also rules are generated at a greater computational cost (however, still polynomial), at least for several classification problems it was worth using neural network ensembles, as extracted rules were more accurate, on average. The DIMLP software is available for PC-Linux under http://us.expasy.org/people/Guido.Bologna.html.     

Abstract noun classification: using a neural network to match word context and word meaning

Psychologists have used artificial neural networks for a few decades to simulate perception, language acquisition, and other cognitive processes. This paper discusses the use of artificial neural networks in research on semantics—in particular, in the investigation of abstract noun meanings. It is widely acknowledged that a word’s meaning varies with its contexts of use, but it is a complex task to identify which context elements are relevant to a word’s meaning. The present study illustrates how connectionist networks can be used to examine this problem. A simple feedforward network learned to distinguish among six abstract nouns, on the basis of characteristics of their contexts, in a corpus of randomly selected naturalistic sentences.     

Twitter sentiment analysis with a deep neural network: An enhanced approach using user behavioral information

Sentiment analysis on social media such as Twitter has become a very important and challenging task. Due to the characteristics of such data—tweet length, spelling errors, abbreviations, and special characters—the sentiment analysis task in such an environment requires a non-traditional approach. Moreover, social media sentiment analysis is a fundamental problem with many interesting applications. Most current social media sentiment classification methods judge the sentiment polarity primarily according to textual content and neglect other information on these platforms. In this paper, we propose a neural network model that also incorporates user behavioral information within a given document (tweet). The neural network used in this paper is a Convolutional Neural Network (CNN). The system is evaluated on two datasets provided by the SemEval-2016 Workshop. The proposed model outperforms current baseline models (including Naive Bayes and Support Vector Machines), which shows that going beyond the content of a document (tweet) is beneficial in sentiment classification, because it provides the classifier with a deep understanding of the task.     

The Content of Deduction

For deductive reasoning to be justified, it must be guaranteed to preserve truth from premises to conclusion; and for it to be useful to us, it must be capable of informing us of something. How can we capture this notion of information content, whilst respecting the fact that the content of the premises, if true, already secures the truth of the conclusion? This is the problem I address here. I begin by considering and rejecting several accounts of informational content. I then develop an account on which informational contents are indeterminate in their membership. This allows there to be cases in which it is indeterminate whether a given deduction is informative. Nevertheless, on the picture I present, there are determinate cases of informative (and determinate cases of uninformative) inferences. I argue that the model I offer is the best way for an account of content to respect the meaning of the logical constants and the inference rules associated with them without collapsing into a classical picture of content, unable to account for informative deductive inferences.     

Attention genes

A major problem for developmental science is understanding how the cognitive and emotional networks important in carrying out mental processes can be related to individual differences. The last five years have seen major advances in establishing links between alleles of specific genes and the neural networks underlying aspects of attention. These findings have the potential of illuminating important aspects of normal development and its pathologies. We need to learn how genes and experience combine to influence the structure of neural networks and the efficiency with which they are exercised. Methods for addressing these issues are central to progress in the decade ahead.     

The relationship between consciousness,understanding, and rationality

The purpose of the present article is to explore the relationship between consciousness and understanding. To do so, I first briefly review recent work on the nature of both understanding and consciousness within philosophy and psychology. Building off of this work, I then defend the thesis that if one is conscious of a given content then one also understands that content. I argue that this conclusion can be drawn from (1) the fact that understanding is associated with rational intention formation and (2) the fact that conscious access appears to involve the selective routing/broadcasting of representational content to neural systems that integrate information in order to select cognitive/behavioral intentions in conjunction with goals. Based on these premises I illustrate how a disruption to the rationality of a representation’s influence on intention formation (when it becomes consciously accessible) would also remove any evidence that a person was conscious of the content of that representation. I therefore suggest that conscious content (and associated phenomenology) may be determined by the rational, content-appropriate influences an accessed representation has on intention formation (i.e., the influences associated with understanding). I conclude by offering replies to several potential objections to this thesis.     

Skill theory v2.0: dispositions,emulation, and spatial perception

An attempt is made to defend a general approach to the spatial content of perception, an approach according to which perception is imbued with spatial content in virtue of certain kinds of connections between perceiving organism’s sensory input and its behavioral output. The most important aspect of the defense involves clearly distinguishing two kinds of perceptuo-behavioral skills—the formation of dispositions, and a capacity for emulation. The former, the formation of dispositions, is argued to by the central pivot of spatial content. I provide a neural information processing interpretation of what these dispositions amount to, and describe how dispositions, so understood, are an obvious implementation of Gareth Evans’ proposal on the topic. Furthermore, I describe what sorts of contribution are made by emulation mechanisms, and I also describe exactly how the emulation framework differs from similar but distinct notions with which it is often unhelpfully confused, such as sensorimotor contingencies and forward models.     

Temperament, Development, and Personality

ABSTRACT— Understanding temperament is central to our understanding of development, and temperament constructs are linked to individual differences in both personality and underlying neural function. In this article, I review findings on the structure of temperament, its relation to the Big Five traits of personality, and its links to development and psychopathology. In addition, I discuss the relation of temperament to conscience, empathy, aggression, and the development of behavior problems, and describe the relation between effortful control and neural networks of executive attention. Finally, I present research on training executive attention.     

Revision: is visual perception a requisite for visual imagery?

Vision is the most highly developed sense in man and represents the doorway through which most of our knowledge of the external world arises. Visual imagery can be defined as the representation of perceptual information in the absence of visual input. Visual imagery has been shown to complement vision in this acquisition of knowledge--it is used in memory retrieval, problem solving, and the recognition of properties of objects. The processes underlying visual imagery have been assimilated to those of the visual system and are believed to share a neural substrate. However, results from studies in congenitally and cortically blind subjects have opposed this hypothesis. Here I review the currently available evidence.     

Identification and integration of sensory modalities: Neural basis and relation to consciousness

A key question in studying consciousness is how neural operations in the brain can identify streams of sensory input as belonging to distinct modalities, which contributes to the representation of qualitatively different experiences. The basis for identification of modalities is proposed to be constituted by self-organized comparative operations across a network of unimodal and multimodal sensory areas. However, such network interactions alone cannot answer the question how sensory feature detectors collectively account for an integrated, yet phenomenally differentiated experiential content. This problem turns out to be different from, although related to, the binding problem. It is proposed that the neural correlate of an enriched, multimodal experience is constituted by the attractor state of a dynamic associative network. Within this network, unimodal and multimodal sensory maps continuously interact to influence each other’s attractor state, so that a feature change in one modality results in a fast re-coding of feature information in another modality. In this scheme, feature detection is coded by firing-rate, whereas firing phase codes relational aspects.     

Short-term memory for serial order: a recurrent neural network model

Despite a century of research, the mechanisms underlying short-term or working memory for serial order remain uncertain. Recent theoretical models have converged on a particular account, based on transient associations between independent item and context representations. In the present article, the authors present an alternative model, according to which sequence information is encoded through sustained patterns of activation within a recurrent neural network architecture. As demonstrated through a series of computer simulations, the model provides a parsimonious account for numerous benchmark characteristics of immediate serial recall, including data that have been considered to preclude the application of recurrent neural networks in this domain. Unlike most competing accounts, the model deals naturally with findings concerning the role of background knowledge in serial recall and makes contact with relevant neuroscientific data. Furthermore, the model gives rise to numerous testable predictions that differentiate it from competing theories. Taken together, the results presented indicate that recurrent neural networks may offer a useful framework for understanding short-term memory for serial order.     

创造性问题解决的动态神经加工模式

先前研究基于功能特化的思想, 基本完成了创造性问题解决相关的各个关键脑区的功能定位, 但并未揭示这些关键脑区在创造活动中的动态神经活动以及它们之间的相互作用关系。本研究拟从动态的功能整合的思想出发, 采用时间序列分析和有效连通性分析方法, 对语义类问题的创造性解决中的信息选择和新颖联结形成等关键子过程的大脑动态加工模式进行研究。本研究不仅能丰富并发展创造性问题解决神经基础的研究方法, 而且能够从系统的层面, 从动态信息加工的角度加深对创造性问题解决脑机制的认识, 推动其神经理论的发展。     

Personal Dualism and the Argument from Differential Vagueness

Abstract

In Causing Actions, Pietroski defends a distinctive view of the relationship between mind and body which he calls Personal Dualism. Central to his defence is the Argument from Differential Vagueness. It moves from the claim that mental events have different vagueness of spatiotemporal boundaries from neural events to the claim that mental events are not identical to neural events. In response, I argue that this presupposes an ontological account of vagueness that there is no reason to believe in this context. I further argue that Pietroski's reasons for rejecting the possibility that mental events are vaguely constituted from neural events are inadequate. I go on to show how Pietroski's Personal Dualism is ill-equipped to deal with the problem of mental causation because of its apparently necessary appeal to ceteris paribus laws.     

Forebrain circuits and control of feeding by learned cues

Professor Richard F. Thompson and his highly influential work on the brain substrates of associative learning and memory have critically shaped my research interests and scientific approach. I am tremendously grateful and thank Professor Thompson for the support and influence on my research and career. The focus of my research program is on associative learning and its role in the control of fundamental, motivated behaviors. My long-term research goal is to understand how learning enables environmental cues to control feeding behavior. We use a combination of behavioral studies and neural systems analysis approach in two well-defined rodent models to study how learned cues are integrated with homeostatic signals within functional forebrain networks, and how these networks are modulated by experience. Here, I will provide an overview of the two behavioral models and the critical neural network components mapped thus far, which include areas in the forebrain, the amygdala and prefrontal cortex, critical for associative learning and decision-making, and the lateral hypothalamus, which is an integrator for feeding, reward and motivation.     

Explainability of neural networks for child language: Agent-First strategy in comprehension of Korean active transitive construction

This study investigates how neural networks address the properties of children's linguistic knowledge, with a focus on the Agent-First strategy in comprehension of an active transitive construction in Korean. We develop various neural-network models and measure their classification performance on the test stimuli used in a behavioural experiment involving scrambling and omission of sentential components at varying degrees. Results show that, despite some compatibility of these models’ performance with the children's response patterns, their performance does not fully approximate the children's utilisation of this strategy, demonstrating by-model and by-condition asymmetries. This study's findings suggest that neural networks can utilise information about formal co-occurrences to access the intended message to a certain degree, but the outcome of this process may be substantially different from how a child (as a developing processor) engages in comprehension. This implies some limits of neural networks on revealing the developmental trajectories of child language.

Research Highlights

• This study investigates how neural networks address properties of child language.
• We focus on the Agent-First strategy in comprehension of Korean active transitive.
• Results show by-model/condition asymmetries against children's response patterns.
• This implies some limits of neural networks on revealing properties of child language.