Tools for connectionist modeling: The dynamical systems methodology

Some understanding of dynamical systems is essential to achieving competency in connectionist models. This mathematical background can be acquired either through a rigorous set of upper undergraduate and/or graduate formal courses or via disciplined self-teaching. As part of developing a course in connectionism, we feel that although certain very basic mathematical tools are most appropriately learned in their “pure” form (i.e., from mathematics textbooks and courses), more advanced exposure to dynamical systems theory can be given in the context of an introduction to connectionism. Students thus learn to write connectionist simulations by first writing programs for simulating arbitrary dynamical systems, then using them to learn some aspects of dynamical systems in general by simulating some special cases, and finally applying this technique to connectionist models of increasing complexity.     

Dynamical Models of Sentence Processing

We suggest that the theory of dynamical systems provides a revealing general framework for modeling the representations and mechanism underlying syntactic processing. We show how a particular dynamical model, the Visitation Set Gravitation model of Tabor, Juliano, and Tanenhaus (1997), develops syntactic representations and models a set of contingent frequency effects in parsing that are problematic for other models. We also present new simulations showing how the model accounts for semantic effects in parsing, and propose a new account of the distinction between syntactic and semantic incongruity. The results show how symbolic structures useful in parsing arise as emergent properties of connectionist dynamical systems.     

Learning and encoding higher order rules in neural networks

Some researchers state that whereas neural networks are fine for pattern recognition and categorization, complex rule formation requires a separate “symbolic” level. However, the human brain is a connectionist system and, however imperfectly, does complex reasoning and inference. Familiar modeling principles (e.g., Hebbian or associative learning, lateral inhibition, opponent processing, neuromodulation) could recur, in different combinations, in architectures that can learn diverse rules. These rules include, for example, “go to the most novel object,” “alternate between two given objects,” and “touch three given objects, without repeats, in any order.” Frontal lobe damage interferes with learning all three of those rules. Hence, network models of rule learning and encoding should include a module analogous to the prefrontal cortex. They should also include modules analogous to the hippocampus for episode setting and analogous to the amygdala for emotional evaluation.     

Learning new faces in an interactive activation and competition model

Abstract

Certain contemporary accounts of object and face recognition use connectionist networks with local representations. This paper describes and extends one such account: an interactive activation and competition (IAC) model of face recognition. In contrast to many networks with distributed representations, IAC models do not incorporate a learning mechanism. This limits their use in psychological modelling. This paper describes how a learning mechanism can be built into an IAC model. The mechanism automatically learns new representations and appears to have many of the desirable properties traditionally associated with distributed networks. Some simulations that produce results consistent with our knowledge of human face learning are reported. Finally, the relation between this work and current theories of visual object recognition is discussed.     

Learning to divide the labor: an account of deficits in light and heavy verb production

Theories of sentence production that involve a convergence of activation from conceptual‐semantic and syntactic‐sequential units inspired a connectionist model that was trained to produce simple sentences. The model used a learning algorithm that resulted in a sharing of responsibility (or “division of labor”) between syntactic and semantic inputs for lexical activation according to their predictive power. Semantically rich, or “heavy”, verbs in the model came to rely on semantic cues more than on syntactic cues, whereas semantically impoverished, or “light”, verbs relied more on syntactic cues. When the syntactic and semantic inputs were lesioned, the model exhibited patterns of production characteristic of agrammatic and anomic aphasic patients, respectively. Anomic models tended to lose the ability to retrieve heavy verbs, whereas agrammatic models were more impaired in retrieving light verbs. These results obtained in both sentence production and single‐word naming simulations. Moreover, simulated agrammatic lexical retrieval was more impaired overall in sentences than in single‐word tasks, in agreement with the literature. The results provide a demonstration of the division‐of‐labor principle, as well as general support for the claim that connectionist learning principles can contribute to the understanding of non‐transparent neuropsychological dissociations.     

Hidden meaning of an early loss. The common ground of attachment and social character assessments and their clinical applications

All contemporary psychoanalytical theorists are concerned with the common problem of how to account for the preeminent importance of relations. John Bowlby, the founder of attachment theory, suggests that “instinctive” behavioral systems underlie much of the emotional life of man and have developed because they were necessary for survival. The system with which he was almost exclusively concerned was the multitude of behaviors and experiences constituting the child's “attachment” to the mother. This “strikingly strong tie, evident particularly when disrupted”, has systematically been observed by attachment researchers, through the development of a series of instruments that gauge interpersonal communication. These interpersonal communications have on their part been found to “contain traces of developmental history”.

Bowlby's theory is based on clinical accounts of cases of important loss experiences. A transcendental role is given in Bowlby's theory to the experiences of loss. It stresses that the construction of mourning processes can be seen as a manifestation of search and as a general gradual mental reorientation. The paper introduces the methodological perspectives, which are observable derivations of Bowlby's psycho‐ethological ideas. We will argue that becoming acquainted with these attachment‐research tools and with socio‐psychoanalytic assessment can enhance the development of the clinicians' observational skills, their insight and their scientific research practices. A clinical vignette seen through the lenses of the attachment assessment of loss is presented. It points in addition to the socio‐cultural‐ethnical basis that serves as an underlying structure for the development of meaning.     

Rethinking the explanatory power of dynamical models in cognitive science

In this paper I offer an interventionist perspective on the explanatory structure and explanatory power of (some) dynamical models in cognitive science: I argue that some “pure” dynamical models – ones that do not refer to mechanisms at all – in cognitive science are “contextualized causal models” and that this explanatory structure gives such models genuine explanatory power. I contrast this view with several other perspectives on the explanatory power of “pure” dynamical models. One of the main results is that dynamical models need not refer to underlying mechanisms in order to be explanatory. I defend and illustrate this position in terms of dynamical models of the A-not-B error in developmental psychology as elaborated by Thelen and colleagues, and dynamical models of unintentional interpersonal coordination developed by Richardson and colleagues.     

Extended Cognition & the Causal‐Constitutive Fallacy: In Search for a Diachronic and Dynamical Conception of Constitution

Philosophical accounts of the constitution relation have been explicated in terms of synchronic relations between higher‐ and lower‐level entities. Such accounts, I argue, are temporally austere or impoverished, and are consequently unable to make sense of the diachronic and dynamic character of constitution in dynamical systems generally and dynamically extended cognitive processes in particular. In this paper, my target domain is extended cognition based on insights from nonlinear dynamics. Contrariwise to the mainstream literature in both analytical metaphysics and extended cognition, I develop a nonstandard, alternative conception of constitution, which I call “diachronic process constitution”. It will be argued that only a diachronic and dynamical conception of constitution is consistent with the nature of constitution in distributed cognitive processes.     

Waves,streams, states,and self: An outline of an integral psychology 1

Abstract

Although far from unanimous, there seems to be a general consensus that neither mind nor brain can be reduced without remainder to the other. This essay argues that indeed both mind and brain need to be included in a nonreductionistic way in any genuinely integral theory of consciousness. In order to facilitate such integration, this essay presents the results of an extensive cross‐cultural literature search on the “mind” side of the equation, suggesting that the mental phenomena that need to be considered in any integral theory include developmental levels or waves of consciousness, developmental lines or streams of consciousness, states of consciousness, and the self (or self‐system). A “master template” of these various phenomena, culled from over one‐hundred psychological systems East and West, is presented. It is suggested that this master template represents a general summary of the “mind” side of the brain‐mind integration. The essay concludes with reflections on the “hard problem,” or how the mind‐side can be integrated with the brain‐side to generate a more integral theory of consciousness.     

Prototypes,Location, and Associative Networks (PLAN): Towards a Unified Theory of Cognitive Mapping

An integrated representation of large-scale space, or cognitive map, colled PLAN, is presented that attempts to address a broader spectrum of issues than has been previously attempted in a single model. Rather than examining way-finding as a process separate from the rest of cognition, one or the fundamental goals of this work is to examine how the wayfinding process is integrated into general cognition. One result of this approach is that the model is “heads-up,” or scene-based, because it takes advantage of the properties of the human visual system and, particularly, the visual system's split into two pathways. The emphasis on the human location or “where” system is new to cognitive mapping and is port of an attempt to synthesize prototype theory, associative networks and location together in a connectionist system. Not all of PLAN is new, however. Many of its parts have analogues in one or another preexisting theory. What makes PLAN unique is integrating the various components into a coherent whole, and the capacity of this resulting system to speak to a wide range of constraints. Our approach emphasizes adaptiveness; thus, our focus on such issues as ease of use and efficiency of learning. The result is a model that has a stronger relationship both to the environment, and to the ways that humans interact with it, compared with previous models. The resulting model is examined in some detail and compared to other systems.     

Appraisal of certain methodologies in cognitive science based on Lakatos’s methodology of scientific research programmes

Attempts to apply the mathematical tools of dynamical systems theory to cognition in a systematic way has been well under way since the early 90s and has been recognised as a “third contender” to computationalist and connectionist approaches (Eliasmith in Philos Psychol 9(4):441–463, 1996). Nevertheless, it was also realised that such an application will not lead to a solid paradigm as straightforwardly as was initially hoped (Eliasmith 1996; van Leeuwen in Minds Mach 15:271–333, 2005). In this paper I explicate a method for assessing such proposals by drawing upon Lakatos’s (in: Lakatos, Musgrave (eds) Criticism and the growth of knowledge, Cambridge University Press, London, pp 91–195, 1970) methodology of scientific research programs (hereafter: “MSRP”). MSRP focuses on the heuristics of a particular field and gauges the model/theory building stratagems by reference to theoretical and empirical progress, on the one hand, and the continuity and the autonomy of the way the field’s heuristic generates its series of models/theories, on the other. The requirement of continuity and autonomy afford distinct senses of ad hoc-ness, which serve as an effective tool to detect various subtleties which may otherwise be missed: the present approach identifies shortcomings missed by Chemero’s (Radical embodied cognitive science, The MIT Press, Cambridge, 2009) radical embodied cognitive science and falsifies Chemero’s claim that the methodological powers of his model-based account is on a par with computationalism. In general, I claim that MSRP is relevant to current methodological issues in cognitive science and can supplement debates regarding “local” assessments of methodologies, such as that between mechanical versus covering-law explanations. MSRP must at least be viewed as a necessary constraint for any methodological considerations in cognitive science.

     

Quantum information in an evolutionary perspective

An approach to quantum phenomena is reviewed that deals with the possibility of their realistic interpretation in the sense that they represent manifestations of hermeneutic circles between quantum “objects” and their experimental boundary conditions. Quantum cybernetics provides an evolutionary perspective in that all higher‐level organizations like molecules, cells, living systems, etc., can be discussed under one and the same systemic viewpoint: a hermeneutic circularity between a “core” (or “nucleus") and a relevant “periphery” (or “environment") which constitutes the systems’ organization and information potential.

Generally, in realistic theories, an individual quantum system is analyzable into a local “particle‐like” nonlinearity of a generally nonlocal “wave‐like” mode of some sub‐quantum structure of the vacuum ("Dirac ether"). In this view, a “particle” can be considered as being “guided” along one specific route by the (generally nonlocal) configurations of superimposed waves, which spread along all possible paths of an experimental setup. Moreover, in the approach of Quantum Cybernetics, an additional focus is given on the fact that the energy and momentum of a particle also determine the wave behavior. Thus, “waves” and “particles” are mutually and self‐consistently defined, and Quantum Cybernetics puts particular emphasis on the circular relationship—mediated by plane waves—between a quantum system and its macroscopically defined boundary conditions.     

Learning Orthographic Structure With Sequential Generative Neural Networks

Learning the structure of event sequences is a ubiquitous problem in cognition and particularly in language. One possible solution is to learn a probabilistic generative model of sequences that allows making predictions about upcoming events. Though appealing from a neurobiological standpoint, this approach is typically not pursued in connectionist modeling. Here, we investigated a sequential version of the restricted Boltzmann machine (RBM), a stochastic recurrent neural network that extracts high‐order structure from sensory data through unsupervised generative learning and can encode contextual information in the form of internal, distributed representations. We assessed whether this type of network can extract the orthographic structure of English monosyllables by learning a generative model of the letter sequences forming a word training corpus. We show that the network learned an accurate probabilistic model of English graphotactics, which can be used to make predictions about the letter following a given context as well as to autonomously generate high‐quality pseudowords. The model was compared to an extended version of simple recurrent networks, augmented with a stochastic process that allows autonomous generation of sequences, and to non‐connectionist probabilistic models (n‐grams and hidden Markov models). We conclude that sequential RBMs and stochastic simple recurrent networks are promising candidates for modeling cognition in the temporal domain.     

A Local Connectionist Account of Consonant Harmony in Child Language

This study represents an attempt to sketch a processing model of phonological development in children acquiring their first language. The investigation is framed within a local connectionist network in which activation spreads between levels and inhibition within levels. Three ways are focused on in which an emergent processing system may diverge from a fully developed one—hypoactivation (too little activation due to underdeveloped links), hyperactivation (too much activation due to overloaded links) and impaired self‐inhibition. With a view to determining how children's productions relate to these three “error mechanisms,” one widespread phonological process, consonant harmony, is subjected to examination. Hypoactivation is found to capture the great majority of harmonic patterns. Hyperactivation and impaired self‐inhibition play a more marginal role and are only needed to account for exceptional cases of harmony. The feasibility of the psycholinguistic model is demonstrated with the help of computer simulations that were run with noise on both fully developed and developing networks. The model makes a number of predictions including the claim that difficult sounds are acquired in coda positions first.     

A knowledge-resonance (KRES) model of category learning

This article introduces a connectionist model of category learning that takes into account the prior knowledge that people bring to new learning situations. In contrast to connectionist learning models that assume a feedforward network and learn by the delta rule or backpropagation, this model, the knowledge-resonance model, or KRES, employs a recurrent network with bidirectional symmetric connection whose weights are updated according to a contrastive Hebbian learning rule. We demonstrate that when prior knowledge is represented in the network, KRES accounts for a considerable range of empirical results regarding the effects of prior knowledge on category learning, including (1) the accelerated learning that occurs in the presence of knowledge, (2) the better learning in the presence of knowledge of category features that are not related to prior knowledge, (3) the reinterpretation of features with ambiguous interpretations in light of error-corrective feedback, and (4) the unlearning of prior knowledge when that knowledge is inappropriate in the context of a particular category.     

Connectionist Sentence Processing in Perspective

The emphasis in the connectionist sentence-processing literature on distributed representation and emergence of grammar from such systems can easily obscure the often close relations between connectionist and symbolist systems. This paper argues that the Simple Recurrent Network (SRN) models proposed by Jordan (1989) and Elman (1990) are more directly related to stochastic Part-of-Speech (POS) Taggers than to parsers or grammars as such, while auto-associative memory models of the kind pioneered by Longuet–Higgins, Willshaw, Pollack and others may be useful for grammar induction from a network-based conceptual structure as well as for structure-building. These observations suggest some interesting new directions for specifically connectionist sentence processing research, including more efficient representations for finite state machines, and acquisition devices based on a distinctively connectionist basis for grounded symbolist conceptual structure.     

Forward Models: Supervised Learning with a Distal Teacher

Internal models of the environment have an important role to play in adaptive systems, in general, and are of particular importance for the supervised learning paradigm. In this article we demonstrate that certain classical problems associated with the notion of the “teacher” in supervised learning can be solved by judicious use of learned internal models as components of the adaptive system. In particular, we show how supervised learning algorithms can be utilized in cases in which an unknown dynamical system intervenes between actions and desired outcomes. Our approach applies to any supervised learning algorithm that is capable of learning in multilayer networks.     

Learning Continuous Probability Distributions with Symmetric Diffusion Networks

In this article we present symmetric diffusion networks, a family of networks that instantiate the principles of continuous, stochastic, adaptive and interactive propagation of information. Using methods of Markovion diffusion theory, we formalize the activation dynamics of these networks and then show that they can be trained to reproduce entire multivariate probability distributions on their outputs using the contrastive Hebbion learning rule (CHL). We show that CHL performs gradient descent on an error function that captures differences between desired and obtained continuous multivariate probability distributions. This allows the learning algorithm to go beyond expected values of output units and to approximate complete probability distributions on continuous multivariate activation spaces. We argue that learning continuous distributions is an important task underlying a variety of real-life situations that were beyond the scope of previous connectionist networks. Deterministic networks, like back propagation, cannot learn this task because they are limited to learning average values of independent output units. Previous stochastic connectionist networks could learn probability distributions but they were limited to discrete variables. Simulations show that symmetric diffusion networks can be trained with the CHL rule to approximate discrete and continuous probability distributions of various types.     

On simulating one-trial learning using morphological neural networks

“Learning once, remembering forever”, this wonderful cognitive phenomenon sometimes occurs in the learning process of human beings. Psychologists call this psychological phenomenon “one-trial learning”. The traditional artificial neural networks can simulate the psychological phenomenon of “implicit learning”, but can’t simulate the cognitive phenomenon of “one-trial learning”. Therefore, cognitive psychology gives a challenge to the traditional artificial neural networks. From two aspects of theory and practice in this paper, the possibility of simulating this kind of psychological phenomenon was explored by using morphological neural networks. This paper takes advantage of morphological associative memory networks to realize the simulation of “one-trial learning” for the first time, and gives 5 simulating practical examples. Theoretical analysis and simulation experiments show that the morphological associative memory networks are a higher effective machine learning method, and can better simulate the cognitive phenomenon of “one-trial learning”, therefore provide a theoretical basis and technological support for the study of intelligent science and cognitive science.