Different is good: connectionism and dynamic systems theory are complementary emergentist approaches to development

Connectionist and dynamic systems approaches to development are similar in that they are both emergentist theories that take a very different perspective from more traditional symbolic systems. Moreover, they are both based on similar mathematical principles. Nevertheless, connectionism and dynamic systems differ in the approach they take to the study of development. We argue that differences between connectionist and dynamic systems approaches in terms of the basic components of the models, what they see as the object of study, how they view the nature of knowledge and their notions of developmental change mean that they each stand to make different and unique contributions to a more complete theory of development. We present an example from our work on how children learn to learn words that illustrates the complementary nature of connectionist and dynamic systems theories.     

Implicit and explicit memory: Implications for the symbol-manipulation versus connectionism controversy

Summary This paper deals with three issues. First, the symbol-manipulation versus connectionism controversy is discussed briefly. Both can be seen as formalisms or languages for describing human behavior, but it is argued that these languages are not fully equivalent. The importance of the ability of connectionist models to learn autonomously, often underestimated, allows them to perform tasks that may defy formalization. Secondly, the structural and functional characteristics of a new connectionist learning model are presented. It avoids some of the psychological and biological implausibilities of currently popular models and solves some of the problems and shortcomings of these models. Thirdly, some simulation results are presented. The model successfully simulates the dissociation between explicit and implicit memory performance as found in patients with anterograde amnesia. This simulation provides an example of the advantages of using the connectionist language in the domain of memory psychology. It is also a good example of how psychological evidence can be used to improve connectionist models.The second author is supported by the Dutch Organization for Scientific Research (NWO)     

The Value of Symbolic Computation

Standard generative linguistic theory, which uses discrete symbolic models of cognition, has some strengths and weaknesses. It is strong on providing a network of outposts that make scientific travel in the jungles of natural language feasible. It is weak in that it currently depends on the elaborate and unformalized use of intuition to develop critical supporting assumptions about each data point. In this regard, it is not in a position to characterize natural language systems in the lawful terms that ecological psychologists strive for. Connectionist learning models offer some help: They define lawful relations between linguistic environments and language systems. But our understanding of them is currently weak, especially when it comes to natural language syntax. Fortunately, symbolic linguistic analysis can help connectionism if the two meet via dynamical systems theory. I discuss a case in point: Insights from linguistic explorations of natural language syntax appear to have identified information structures that are particularly relevant to understanding ecologically appealing but analytically mysterious connectionist learning models.     

Language and connectionism: the developing interface

After a difficult initial period in which connectionism was perceived as either irrelevant or antithetical to linguistic theory, connectionist concepts are now beginning to be brought to bear on basic issues concerning the structure, acquisition, and processing of language, both normal and disordered. This article describes some potential points of further contact between connectionism and linguistic theory. I consider how connectionist concepts may be relevant to issues concerning the representation of linguistic knowledge; the role of a priori constraints on acquisition; and the poverty of the stimulus argument. I then discuss whether these models contribute to the development of explanatory theories of language.     

Are rules and modules really necessary for explaining language?

Over the past few years Steven Pinker has argued that although some aspects of language may be more associational, and therefore properly modeled in connectionist networks, for the most part human language is still best characterized as a modularized set of rulesymbol systems. In support of his claim, Pinker garners a broad array of clinical, experimental, and observational data from neurology, psychology, and linguistics. Those data, unfortunately, are not compelling because they do not support his position uniquely. In this paper, I show how each of his arguments is compatible with alternative interpretations. I argue, moreover, that in focusing on certain details of connectionist models Pinker and his colleagues actually overlooked both the most serious deficiencies of the connectionist approach and its most significant theoretical contribution. I conclude by sketching briefly some emerging alternatives to connectionism which avoid those deficiencies while retaining its strengths over the rule-symbol systems of linguistic theory.This research was funded in part by University of Idaho Seed Grant 681-Y304.     

Birth of an Abstraction: A Dynamical Systems Account of the Discovery of an Elsewhere Principle in a Category Learning Task

Human participants and recurrent (“connectionist”) neural networks were both trained on a categorization system abstractly similar to natural language systems involving irregular (“strong”) classes and a default class. Both the humans and the networks exhibited staged learning and a generalization pattern reminiscent of the Elsewhere Condition (Kiparsky, 1973). Previous connectionist accounts of related phenomena have often been vague about the nature of the networks’ encoding systems. We analyzed our network using dynamical systems theory, revealing topological and geometric properties that can be directly compared with the mechanisms of non‐connectionist, rule‐based accounts. The results reveal that the networks “contain” structures related to mechanisms posited by rule‐based models, partly vindicating the insights of these models. On the other hand, they support the one mechanism (OM), as opposed to the more than one mechanism (MOM), view of symbolic abstraction by showing how the appearance of MOM behavior can arise emergently from one underlying set of principles. The key new contribution of this study is to show that dynamical systems theory can allow us to explicitly characterize the relationship between the two perspectives in implemented models.     

Does the eye know calculus? The threshold of representation in classical and connectionist models

The notion of representation lies at the crossroads of questions about the nature of belief and knowledge, meaning, and intentionality. But there is some hope that it might be simpler than all those. If we could understand it clearly, it might then help to explicate those more difficult notions. In this paper, my central aim is to find a principled criterion, along lines that make biological sense, for deciding just when it becomes theoretically plausible to ascribe to some process or state a representational role. I shall be especially concerned with some differences, in this regard, between classical and connectionist models. The relation between ‘standard’ artificial intelligence and connectionism turns out to illustrate a ‘first in, last out’ principle: What we most easily understand (and so can program) is what we have most recently invented; tasks we ourselves perform best, by contrast, are a lot harder to understand. Classical AI has modelled the former; connectionism tries to tackle the latter. I end with some speculations about the possible implications of these considerations for our understanding of understanding.     

Connectionist modelling in psychology: a localist manifesto

Over the last decade, fully distributed models have become dominant in connectionist psychological modelling, whereas the virtues of localist models have been underestimated. This target article illustrates some of the benefits of localist modelling. Localist models are characterized by the presence of localist representations rather than the absence of distributed representations. A generalized localist model is proposed that exhibits many of the properties of fully distributed models. It can be applied to a number of problems that are difficult for fully distributed models, and its applicability can be extended through comparisons with a number of classic mathematical models of behaviour. There are reasons why localist models have been underused, though these often misconstrue the localist position. In particular, many conclusions about connectionist representation, based on neuroscientific observation, can be called into question. There are still some problems inherent in the application of fully distributed systems and some inadequacies in proposed solutions to these problems. In the domain of psychological modelling, localist modelling is to be preferred.     

Grammar-based Connectionist Approaches to Language

This article describes an approach to connectionist language research that relies on the development of grammar formalisms rather than computer models. From formulations of the fundamental theoretical commitments of connectionism and of generative grammar, it is argued that these two paradigms are mutually compatible. Integrating the basic assumptions of the paradigms results in formal theories of grammar that centrally incorporate a certain degree of connectionist computation. Two such grammar formalisms—Harmonic Grammar and Optimality Theory —are briefly introduced to illustrate grammar-based approaches to connectionist language research. The strengths and weaknesses of grammar-based research and more traditional model-based research are argued to be complementary, suggesting a significant role for both strategies in the spectrum of connectionist language research.     

Connectionist Sentence Processing in Perspective

The emphasis in the connectionist sentence-processing literature on distributed representation and emergence of grammar from such systems can easily obscure the often close relations between connectionist and symbolist systems. This paper argues that the Simple Recurrent Network (SRN) models proposed by Jordan (1989) and Elman (1990) are more directly related to stochastic Part-of-Speech (POS) Taggers than to parsers or grammars as such, while auto-associative memory models of the kind pioneered by Longuet–Higgins, Willshaw, Pollack and others may be useful for grammar induction from a network-based conceptual structure as well as for structure-building. These observations suggest some interesting new directions for specifically connectionist sentence processing research, including more efficient representations for finite state machines, and acquisition devices based on a distinctively connectionist basis for grounded symbolist conceptual structure.     

Systematicity redux

One of the main challenges that Jerry Fodor and Zenon Pylyshyn (Cognition 28:3–71, 1988) posed for any connectionist theory of cognitive architecture is to explain the systematicity of thought without implementing a Language of Thought (LOT) architecture. The systematicity challenge presents a dilemma: if connectionism cannot explain the systematicity of thought, then it fails to offer an adequate theory of cognitive architecture; and if it explains the systematicity of thought by implementing a LOT architecture, then it fails to offer an alternative to the LOT hypothesis. Given that thought is systematic, connectionism can offer an adequate alternative to the LOT hypothesis only if it can meet the challenge. Although some critics tried to meet the challenge, others argued that it need not be met since thought is not in fact systematic; and some claimed not to even understand the claim that thought is systematic. I do not here examine attempts to answer the challenge. Instead, I defend the challenge itself by explicating the notion of systematicity in a way that I hope makes clear that thought is indeed systematic, and so that to offer an adequate alternative to the LOT hypothesis, connectionism must meet the challenge.     

Classical and connectionist models: Levels of description

To begin, I introduce an analysis of interlevel relations that allows us to offer an initial characterization of the debate about the way classical and connectionist models relate. Subsequently, I examine a compatibility thesis and a conditional claim on this issue. With respect to the compatibility thesis, I argue that, even if classical and connectionist models are not necessarily incompatible, the emergence of the latter seems to undermine the best arguments for the Language of Thought Hypothesis, which is essential to the former. I attack the conditional claim of connectionism to eliminativism, presented by Ramsey et al. (1990), by discrediting their discrete characterization of common-sense psychological explanations and pointing to the presence of a moderate holistic constraint. Finally, I conclude that neither of the arguments considered excludes the possibility of viewing connectionist models as forming a part of a representational theory of cognition that dispenses with the Language of Thought Hypothesis.     

Challenging the widespread assumption that connectionism and distributed representations go hand-in-hand

One of the central claims associated with the parallel distributed processing approach popularized by D.E. Rumelhart, J.L. McClelland and the PDP Research Group is that knowledge is coded in a distributed fashion. Localist representations within this perspective are widely rejected. It is important to note, however, that connectionist networks can learn localist representations and many connectionist models depend on localist coding for their functioning. Accordingly, a commitment to distributed representations should be considered a specific theoretical claim regarding the structure of knowledge rather than a core principle, as often assumed. In this paper, it is argued that there are fundamental computational and empirical challenges that have not yet been addressed by distributed connectionist theories that are readily accommodated within localist approaches. This is highlighted in the context of modeling word and nonword naming, the domain in which some of the strongest claims have been made. It is shown that current PDP models provide a poor account of naming monosyllable items, and that distributed representations make it difficult for these models to scale up to more complex language phenomena. At the same time, models that learn localist representations are shown to hold promise in supporting many of the core reading and language functions on which PDP models fail. It is concluded that the common rejection of localist coding schemes within connectionist architectures is premature.     

Connectionist representation,semantic compositionality,and the instability of concept structure

Summary In the present paper connectionist approaches to the problem of internal representation and the nature of concepts are discussed. In the first part the concept of representation that underlies connectionist modeling is made explicit. It is argued that the connectionist view of representation relies on a correlational theory of semantic content- i.e., the covariation between internal and external states is taken as the basis for ascribing meaning to internal states. The problems and virtues of such a correlational approach to internal representation are addressed. The second part of the paper is concerned with whether connectionism is capable of accounting for the apparent productivity and systematicity of language and thought. There is an evaluation of the recent arguments of Fodor and Pylyshyn, who claim that systematicity can only be explained if one conceives of mental representations as structured symbols composed of context-free constituents. There is a review of empirical evidence that strongly suggests that concepts are not fixed memory structures and that the meaning of constituent symbols varies, depending on the context in which they are embedded. On the basis of this review it is concluded that the meaning of a complex expression is not computed from the context-free meanings of the constituents, and that strong compositionality, as endorsed by Fodor and Pylyshyn (1988), seems implausible as a process theory for the comprehension of complex concepts. Instead, the hypothesis is endorsed that constraint satisfaction in distributed connectionist networks may allow for an alternative account of weak compositionality compatible with the context sensitivity of meaning. In the final section, it is argued that neither mere implementation of a language of thought in connectionist networks nor radical elimination of symbol systems seems to be a fruitful research strategy, but that it might be more useful to discuss how connectionist systems can develop the capacity to use external symbol systems like language or logic without instantiating symbol systems themselves.     

Chaos breeds autonomy: connectionist design between bias and baby-sitting

In connectionism and its offshoots, models acquire functionality through externally controlled learning schedules. This undermines the claim of these models to autonomy. Providing these models with intrinsic biases is not a solution, as it makes their function dependent on design assumptions. Between these two alternatives, there is room for approaches based on spontaneous self-organization. Structural reorganization in adaptation to spontaneous activity is a well-known phenomenon in neural development. It is proposed here as a way to prepare connectionist models for learning and enhance the autonomy of these models.     

Connectionist models of development

How have connectionist models informed the study of development? This paper considers three contributions from specific models. First, connectionist models have proven useful for exploring nonlinear dynamics and emergent properties, and their role in nonlinear developmental trajectories, critical periods and developmental disorders. Second, connectionist models have informed the study of the representations that lead to behavioral dissociations. Third, connectionist models have provided insight into neural mechanisms, and why different brain regions are specialized for different functions. Connectionist and dynamic systems approaches to development have differed, with connectionist approaches focused on learning processes and representations in cognitive tasks, and dynamic systems approaches focused on mathematical characterizations of physical elements of the system and their interactions with the environment. The two approaches also share much in common, such as their emphasis on continuous, nonlinear processes and their broad application to a range of behaviors.     

Large-Scale Modeling of Wordform Learning and Representation

The forms of words as they appear in text and speech are central to theories and models of lexical processing. Nonetheless, current methods for simulating their learning and representation fail to approach the scale and heterogeneity of real wordform lexicons. A connectionist architecture termed the sequence encoder is used to learn nearly 75,000 wordform representations through exposure to strings of stress-marked phonemes or letters. First, the mechanisms and efficacy of the sequence encoder are demonstrated and shown to overcome problems with traditional slot-based codes. Then, two large-scale simulations are reported that learned to represent lexicons of either phonological or orthographic wordforms. In doing so, the models learned the statistics of their lexicons as shown by better processing of well-formed pseudowords as opposed to ill-formed (scrambled) pseudowords, and by accounting for variance in well-formedness ratings. It is discussed how the sequence encoder may be integrated into broader models of lexical processing.     

Approximate optimal control as a model for motor learning

Current models of psychological development rely heavily on connectionist models that use supervised learning. These models adapt network weights when the network output does not match the target outputs computed by some agent. The authors present a model of motor learning in which the child uses exploration to discover appropriate ways of responding. The model is consistent with what is known about how neural systems evaluate behavior. The authors model the development of reaching and investigate N. Bernstein's (1967) hypotheses about early motor learning. Simulations show the course of learning as well as model the kinematics of reaching by a dynamical arm.