Connectionist Natural Language Processing: The State of the Art

This Special Issue on Connectionist Models of Human Language Processing provides an opportunity for an appraisal both of specific connectionist models and of the status and utility of connectionist models of language in general. This introduction provides the background for the papers in the Special Issue. The development of connectionist models of language is traced, from their intellectual origins, to the state of current research. Key themes that arise throughout different areas of connectionist psycholinguistics are highlighted, and recent developments in speech processing, morphology, sentence processing, language production, and reading are described. We argue that connectionist psycholinguistics has already had a significant impact on the psychology of language, and that connectionist models are likely to have an important influence on future research.     

Are there lexicons?

Many models of the processing of printed or spoken words or objects or faces propose that systems of local representations of the forms of such stimuli—lexicons—exist. This is denied by partisans of the distributed-representation connectionist approach to cognitive modelling. An experimental paradigm of key theoretical importance here is lexical decision and its analogue in the domain of objects, object decision. How does each theoretical camp account for our ability to perform these two tasks? The localists say that the tasks are done by matching or failing to match a stimulus to a local representation in a lexicon. Advocates of distributed representations often do not seek to explain these two tasks; however, when they do, they propose that patterns of activation evoked in a semantic system can be used to discriminate between words and nonwords, or between real objects and false objects. Therefore the distributed-representation account of lexical and object decision tasks predicts that performance on these tasks can never be normal in patients with an impaired semantic system, nor in patients who cannot access semantics normally from the stimulus domain being tested. However, numerous such patients have been reported in the literature, indicating that semantic access is not needed for normal performance on these tasks. Such results support the localist form of modelling rather than the distributed-representation approach.     

Language and connectionism: the developing interface

After a difficult initial period in which connectionism was perceived as either irrelevant or antithetical to linguistic theory, connectionist concepts are now beginning to be brought to bear on basic issues concerning the structure, acquisition, and processing of language, both normal and disordered. This article describes some potential points of further contact between connectionism and linguistic theory. I consider how connectionist concepts may be relevant to issues concerning the representation of linguistic knowledge; the role of a priori constraints on acquisition; and the poverty of the stimulus argument. I then discuss whether these models contribute to the development of explanatory theories of language.     

The brain makes a distinction between hard and easy stimuli: comments on Beretta et al

The Beretta et al. study tested an invalid prediction concerning connectionist models of inflectional morphology and the study exhibits a confound between type of stimulus (regular, irregular) and processing difficulty (easy, hard) that invalidates their conclusions. Harder stimuli produced greater activation across a broader range of brain areas, as in previous studies, but the data have no bearing on the rules vs. connections debate.     

Attitudinal Approaches to Health Behavior: Integrating Expectancy‐Value and Automaticity Accounts

Attitudes are a key construct in health psychology due to their central role in motivating and changing behavior. An expectancy‐value framework has been the dominant conceptual approach for exploring the impact of attitudes on health behavior, applications of which emphasize volition and rational decision making. More recently, attention has focused on automatic attitudes, which are believed to capture reflexive aspects of motivation. Dual‐process models such as the MODE generally treat expectancy‐value and automaticity accounts as representing separate processing pathways. However, both accounts depend on associative learning. Learning history pairs beliefs or features with evaluations; subsequent activation of beliefs or features activates associated evaluations and drives overall attitude. Therefore, a single information processing architecture may accommodate expectancy‐value and automaticity approaches within a unifying framework, and this is provided by neural network (connectionist) accounts. In this paper, we highlight how a greater emphasis on the information processing mechanics of associative learning can provide a more parsimonious model of attitudes, which may also extend to a wider array of memory‐related phenomena of relevance to health psychology.     

Implicit and explicit memory: Implications for the symbol-manipulation versus connectionism controversy

Summary This paper deals with three issues. First, the symbol-manipulation versus connectionism controversy is discussed briefly. Both can be seen as formalisms or languages for describing human behavior, but it is argued that these languages are not fully equivalent. The importance of the ability of connectionist models to learn autonomously, often underestimated, allows them to perform tasks that may defy formalization. Secondly, the structural and functional characteristics of a new connectionist learning model are presented. It avoids some of the psychological and biological implausibilities of currently popular models and solves some of the problems and shortcomings of these models. Thirdly, some simulation results are presented. The model successfully simulates the dissociation between explicit and implicit memory performance as found in patients with anterograde amnesia. This simulation provides an example of the advantages of using the connectionist language in the domain of memory psychology. It is also a good example of how psychological evidence can be used to improve connectionist models.The second author is supported by the Dutch Organization for Scientific Research (NWO)     

Connectionist Sentence Processing in Perspective

The emphasis in the connectionist sentence-processing literature on distributed representation and emergence of grammar from such systems can easily obscure the often close relations between connectionist and symbolist systems. This paper argues that the Simple Recurrent Network (SRN) models proposed by Jordan (1989) and Elman (1990) are more directly related to stochastic Part-of-Speech (POS) Taggers than to parsers or grammars as such, while auto-associative memory models of the kind pioneered by Longuet–Higgins, Willshaw, Pollack and others may be useful for grammar induction from a network-based conceptual structure as well as for structure-building. These observations suggest some interesting new directions for specifically connectionist sentence processing research, including more efficient representations for finite state machines, and acquisition devices based on a distinctively connectionist basis for grounded symbolist conceptual structure.     

Multiple levels of inquiry in cognitive science

Summary Gibsonian ecological psychology, symbolic information processing, and connectionist information processing are frequently construed as three competing paradigms or research traditions, each seeking dominance in experimental psychology and in cognitive science generally. There is an important element of truth in this perspective, and any adequate account of the development of experimental psychology over the past 30 years would have to examine seriously how the various conceptual frameworks, experimental endeavors, and social institutions have figured in this conflict. But the goal of this paper is not to characterize the historical dynamics within experimental psychology and cognitive science; rather, it is to consider what sorts of rapprochement is possible. Rapprochement, however, is not sought simply for its own sake or out of an a priori conviction that scientific enterprises should be unified. Spirited controversy between competing traditions is often an important component of progess (Laudan, 1977). Rapprochement has a purpose when alternative theoretical traditions have reached a point when each confronts serious shortcomings that can best be overcome by incorporating alternative perspectives. In this paper I try to show that this is the situation that exists in experimental psychology and cognitve science generally with respect to the three traditions enumerated above. I first explore how cognitive inquiry directed at internal procedures for processing information could benefit from a detailed study of the context of cognition, including insights provided by the Gibsonian tradition. Second, I examine the current controversy between symbolic and connectionist approaches and address the question of what contributions each offers to the other. Finally, I offer a framework in which multiple levels of inquiry in cognitve science can be related.     

Large-Scale Modeling of Wordform Learning and Representation

The forms of words as they appear in text and speech are central to theories and models of lexical processing. Nonetheless, current methods for simulating their learning and representation fail to approach the scale and heterogeneity of real wordform lexicons. A connectionist architecture termed the sequence encoder is used to learn nearly 75,000 wordform representations through exposure to strings of stress-marked phonemes or letters. First, the mechanisms and efficacy of the sequence encoder are demonstrated and shown to overcome problems with traditional slot-based codes. Then, two large-scale simulations are reported that learned to represent lexicons of either phonological or orthographic wordforms. In doing so, the models learned the statistics of their lexicons as shown by better processing of well-formed pseudowords as opposed to ill-formed (scrambled) pseudowords, and by accounting for variance in well-formedness ratings. It is discussed how the sequence encoder may be integrated into broader models of lexical processing.     

Connectionist natural language parsing

The key developments of two decades of connectionist parsing are reviewed. Connectionist parsers are assessed according to their ability to learn to represent syntactic structures from examples automatically, without being presented with symbolic grammar rules. This review also considers the extent to which connectionist parsers offer computational models of human sentence processing and provide plausible accounts of psycholinguistic data. In considering these issues, special attention is paid to the level of realism, the nature of the modularity, and the type of processing that is to be found in a wide range of parsers.     

Connectionist modelling of the separable processing of consonants and vowels

describe two aphasic patients, with impaired processing of vowels and consonants, respectively. The impairments could not be captured according to the sonority hierarchy or in terms of a feature level analysis. Caramazza et al. claim that this dissociation demonstrates separate representation of the categories of vowels and consonants in speech processing. We present two connectionist models of the management of phonological representations. The models spontaneously develop separable processing of vowels and consonants. The models have two hidden layers and are given as input vowels and consonants represented in terms of their phonological distinctive features. The first model is presented with feature bundles one at a time and the hidden layers have to combine their output to reproduce a unified copy of the feature bundle. In the second model a "fine-coded" layer receives information about feature bundles in isolation, and a "coarse-coded" layer receives information about each feature bundle in the context of the prior and subsequent feature bundle. Coarse-coding facilitated processing of vowels and fine-coding processing of consonants. These models show that separable processing of vowels and consonants is an emergent effect of modular processors operating on feature-based representations. We argue that it is not necessary to postulate an independent level of representation for the consonant/vowel distinction, separate from phonological distinctive features.     

A Connectionist Approach to Knowledge Representation and Limited Inference

Although the connectionist approach has lead to elegant solutions to a number of problems in cognitive science and artificial intelligence, its suitability for dealing with problems in knowledge representation and inference has often been questioned. This paper partly answers this criticism by demonstrating that effective solutions to certain problems in knowledge representation and limited inference can be found by adopting a connectionist approach. The paper presents a connectionist realization of semantic networks, that is, it describes how knowledge about concepts, their properties, and the hierarchical relationship between them may be encoded as an interpreter-free massively parallel network of simple processing elements that can solve an interesting class of inheritance and recognition problems extremely fast—in time proportional to the depth of the conceptual hierarchy. The connectionist realization is based on an evidential formulation that leads to principled solutions to the problems of exceptions and conflicting multiple inheritance situations during inheritance, and the best-match or partial-match computation during recognition. The paper also identifies constraints that must be satisfied by the conceptual structure in order to arrive at an efficient parallel realization.     

Cognition without classical architecture

Fodor and Pylyshyn (1988) argue that any successful model of cognition must use classical architecture; it must depend upon rule-based processing sensitive to constituent structure. This claim is central to their defense of classical AI against the recent enthusiasm for connectionism. Connectionist nets, they contend, may serve as theories of the implementation of cognition, but never as proper theories of psychology. Connectionist models are doomed to describing the brain at the wrong level, leaving the classical view to account for the mind.This paper considers whether recent results in connectionist research weigh against Fodor and Pylyshyn's thesis. The investigation will force us to develop criteria for determining exactly when a net is capable of systematic processing. Fodor and Pylyshyn clearly intend their thesis to affect the course of research in psychology. I will argue that when systematicity is defined in a way that makes the thesis relevant in this way, the thesis is challenged by recent progress in connectionism.     

Connectionist models and their properties

Much of the progress in the fields constituting cognitive science has been based upon the use of explicit information processing models, almost exclusively patterned after conventional serial computers. An extension of these ideas to massively parallel, connectionist models appears to offer a number of advantages. After a preliminary discussion, this paper introduces a general connectionist model and considers how it might be used in cognitive science. Among the issues addressed are: stability and noise-sensitivity, distributed decision-making, time and sequence problems, and the representation of complex concepts.     

ALCOVE: an exemplar-based connectionist model of category learning.

ALCOVE (attention learning covering map) is a connectionist model of category learning that incorporates an exemplar-based representation (Medin & Schaffer, 1978; Nosofsky, 1986) with error-driven learning (Gluck & Bower, 1988; Rumelhart, Hinton, & Williams, 1986). Alcove selectively attends to relevant stimulus dimensions, is sensitive to correlated dimensions, can account for a form of base-rate neglect, does not suffer catastrophic forgetting, and can exhibit 3-stage (U-shaped) learning of high-frequency exceptions to rules, whereas such effects are not easily accounted for by models using other combinations of representation and learning method.     

Connectionist psycholinguistics: capturing the empirical data

Connectionist psycholinguistics is an emerging approach to modeling empirical data on human language processing using connectionist computational architectures. For almost 20 years, connectionist models have increasingly been used to model empirical data across many areas of language processing. We critically review four key areas: speech processing, sentence processing, language production, and reading aloud, and evaluate progress against three criteria: data contact, task veridicality, and input representativeness. Recent connectionist modeling efforts have made considerable headway toward meeting these criteria, although it is by no means clear whether connectionist (or symbolic) psycholinguistics will eventually provide an integrated model of full-scale human language processing.     

Learning to divide the labor: an account of deficits in light and heavy verb production

Theories of sentence production that involve a convergence of activation from conceptual‐semantic and syntactic‐sequential units inspired a connectionist model that was trained to produce simple sentences. The model used a learning algorithm that resulted in a sharing of responsibility (or “division of labor”) between syntactic and semantic inputs for lexical activation according to their predictive power. Semantically rich, or “heavy”, verbs in the model came to rely on semantic cues more than on syntactic cues, whereas semantically impoverished, or “light”, verbs relied more on syntactic cues. When the syntactic and semantic inputs were lesioned, the model exhibited patterns of production characteristic of agrammatic and anomic aphasic patients, respectively. Anomic models tended to lose the ability to retrieve heavy verbs, whereas agrammatic models were more impaired in retrieving light verbs. These results obtained in both sentence production and single‐word naming simulations. Moreover, simulated agrammatic lexical retrieval was more impaired overall in sentences than in single‐word tasks, in agreement with the literature. The results provide a demonstration of the division‐of‐labor principle, as well as general support for the claim that connectionist learning principles can contribute to the understanding of non‐transparent neuropsychological dissociations.     

Shared representations in body perception

The ability to understand events that happen to other people is a characteristic feature of the human mind. Here, we investigate whether the links between mental representation of one's own body and the bodies of other people could form the basis of human social representations. We studied interpersonal body representation (IBR) in a series of behavioural cueing experiments. Subjects responded to tactile events on their own body after a visual event was presented in either the corresponding anatomical location on a model's body, or in a non-corresponding location. We found that reactions were faster when the visual cue was in register with the tactile stimulation. This effect was absent when identical visual events were presented on a non-body control stimulus, suggesting a body specific mechanism for interpersonal registration of purely sensory events. Similar interpersonal systems have been demonstrated previously for the coding of action and emotion, but we believe that our results provide the first behavioural evidence for interpersonal body representation at the purely sensory level. We show that a sensory processing mechanism specific for bodies is automatically activated when viewing another person. Interpersonal body representation may be an important precursor to empathy and theory of mind. In our social world, we understand the percepts of others by registering them against the representations used to perceive our own body, and this mechanism involves an interpersonal somatotopic map.     

Phonology, reading acquisition, and dyslexia: insights from connectionist models.

The development of reading skill and bases of developmental dyslexia were explored using connectionist models. Four issues were examined: the acquisition of phonological knowledge prior to reading, how this knowledge facilitates learning to read, phonological and nonphonological bases of dyslexia, and effects of literacy on phonological representation. Compared with simple feedforward networks, representing phonological knowledge in an attractor network yielded improved learning and generalization. Phonological and surface forms of developmental dyslexia, which are usually attributed to impairments in distinct lexical and nonlexical processing "routes," were derived from different types of damage to the network. The results provide a computationally explicit account of many aspects of reading acquisition using connectionist principles.     

Dissociations in performance on novel versus irregular items: single-route demonstrations with input gain in localist and distributed models

Four pairs of connectionist simulations are presented in which quasi-regular mappings are computed using localist and distributed representations. In each simulation, a control parameter termed input gain was modulated over the only level of representation that mapped inputs to outputs. Input gain caused both localist and distributed models to shift between regularity-based and item-based modes of processing. Performance on irregular items was selectively impaired in the regularity-based modes, whereas performance on novel items was selectively impaired in the item-based modes. Thus, the models exhibited double dissociations without separable processing components. These results are discussed in the context of analogous dissociations found in language domains such as word reading and inflectional morphology.