Connectionist Natural Language Processing: The State of the Art

This Special Issue on Connectionist Models of Human Language Processing provides an opportunity for an appraisal both of specific connectionist models and of the status and utility of connectionist models of language in general. This introduction provides the background for the papers in the Special Issue. The development of connectionist models of language is traced, from their intellectual origins, to the state of current research. Key themes that arise throughout different areas of connectionist psycholinguistics are highlighted, and recent developments in speech processing, morphology, sentence processing, language production, and reading are described. We argue that connectionist psycholinguistics has already had a significant impact on the psychology of language, and that connectionist models are likely to have an important influence on future research.     

A Local Connectionist Account of Consonant Harmony in Child Language

This study represents an attempt to sketch a processing model of phonological development in children acquiring their first language. The investigation is framed within a local connectionist network in which activation spreads between levels and inhibition within levels. Three ways are focused on in which an emergent processing system may diverge from a fully developed one—hypoactivation (too little activation due to underdeveloped links), hyperactivation (too much activation due to overloaded links) and impaired self‐inhibition. With a view to determining how children's productions relate to these three “error mechanisms,” one widespread phonological process, consonant harmony, is subjected to examination. Hypoactivation is found to capture the great majority of harmonic patterns. Hyperactivation and impaired self‐inhibition play a more marginal role and are only needed to account for exceptional cases of harmony. The feasibility of the psycholinguistic model is demonstrated with the help of computer simulations that were run with noise on both fully developed and developing networks. The model makes a number of predictions including the claim that difficult sounds are acquired in coda positions first.     

Connectionist Models of Language Production: Lexical Access and Grammatical Encoding

Theories of language production have long been expressed as connectionist models. We outline the issues and challenges that must be addressed by connectionist models of lexical access and grammatical encoding, and review three recent models. The models illustrate the value of an interactive activation approach to lexical access in production, the need for sequential output in both phonological and grammatical encoding, and the potential for accounting for structural effects on errors and structural priming from learning.     

Connectionist Models and Linguistic Theory: Investigations of Stress Systems in Language

We question the widespread assumption that linguistic theory should guide the formulation of mechanistic accounts of human language processing. We develop a pseudo-linguistic theory for the domain of linguistic stress, based on observation of the learning behavior of a perceptron exposed to a variety of stress patterns. There are significant similarities between our analysis of perception stress learning and metrical phonology, the linguistic theory of human stress. Both approaches attempt to identify salient characteristics of the stress systems under examination without reference to the workings of the underlying processor. Our theory and computer simulations exhibit some strikingly suggestive correspondences with metrical theory. We show, however, that our high-level pseudo-linguistic account bears no causal relation to processing in the perception, and provides little insight into the nature of this processing. Because of the persuasive similarities between the nature of our theory and linguistic theorizing, we suggest that linguistic theory may be in much the same position. Contrary to the usual assumption, it may not provide useful guidance in attempts to identify processing mechanisms underlying human language.     

New Approaches to the Study of Childhood Language Disorders

Not long ago, poor language skills did not necessarily interfere with the quality of a person's life. Many occupations did not require sophisticated language or literacy. Interactions with other people could reasonably be restricted to family members and a few social or business contacts. But in the 21st century, advances in technology and burgeoning population centers have made it necessary for children to acquire high levels of proficiency with at least one language, in both spoken and written form. This situation increases the urgency for us to develop better theoretical accounts of the problems underlying disorders of language, including dyslexia. Empirical investigations of language-learning deficits largely focus on phonological representations and often ask to what extent labeling responses are "categorical." This article describes the history of this approach and presents some relevant findings regarding the perceptual organization of speech signals-findings that should prompt us to expand our investigations of language disorders.     

Logical and Philosophical Ideas in Certain Formal Approaches to Language

This paper reminds, puts in order, sketches and also initiates some researches from the field of logic and philosophy of language. It lays emphasis on the logical-linguistic and ontological developmental lines originated with Polish researchers. The author discusses two opposite orientations of the former line in the process of formalization of language, called here nominalistic and Platonistic. The paper mentions the author's result (1989; 1991) concerning theoretical equivalence of two axiomatic approaches to language syntax which take into consideration these two different philosophical orientations, respectively. It also introduces certain formal-logical foundations of language semantics, discusses and develops the problem of the syntactic and semantic categorial agreement, i.e. the agreement of the syntactic category of each language expression with the ontological category assigned to the reference of this expression. The author, in particular, gives a solution to this problem for quantifiers. This revised version was published online in June 2006 with corrections to the Cover Date.     

Connectionist semantic systematicity

Fodor and Pylyshyn [Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28, 3–71] argue that connectionist models are not able to display systematicity other than by implementing a classical symbol system. This claim entails that connectionism cannot compete with the classical approach as an alternative architectural framework for human cognition. We present a connectionist model of sentence comprehension that does not implement a symbol system yet behaves systematically. It consists in a recurrent neural network that maps sentences describing situations in a microworld, onto representations of these situations. After being trained on particular sentence–situation pairs, the model can comprehend new sentences, even if these describe new situations. We argue that this systematicity arises robustly and in a psychologically plausible manner because it depends on structure inherent in the world.     

Connectionist models of development

How have connectionist models informed the study of development? This paper considers three contributions from specific models. First, connectionist models have proven useful for exploring nonlinear dynamics and emergent properties, and their role in nonlinear developmental trajectories, critical periods and developmental disorders. Second, connectionist models have informed the study of the representations that lead to behavioral dissociations. Third, connectionist models have provided insight into neural mechanisms, and why different brain regions are specialized for different functions. Connectionist and dynamic systems approaches to development have differed, with connectionist approaches focused on learning processes and representations in cognitive tasks, and dynamic systems approaches focused on mathematical characterizations of physical elements of the system and their interactions with the environment. The two approaches also share much in common, such as their emphasis on continuous, nonlinear processes and their broad application to a range of behaviors.     

Connectionist natural language parsing

The key developments of two decades of connectionist parsing are reviewed. Connectionist parsers are assessed according to their ability to learn to represent syntactic structures from examples automatically, without being presented with symbolic grammar rules. This review also considers the extent to which connectionist parsers offer computational models of human sentence processing and provide plausible accounts of psycholinguistic data. In considering these issues, special attention is paid to the level of realism, the nature of the modularity, and the type of processing that is to be found in a wide range of parsers.     

A Connectionist Approach to Knowledge Representation and Limited Inference

Although the connectionist approach has lead to elegant solutions to a number of problems in cognitive science and artificial intelligence, its suitability for dealing with problems in knowledge representation and inference has often been questioned. This paper partly answers this criticism by demonstrating that effective solutions to certain problems in knowledge representation and limited inference can be found by adopting a connectionist approach. The paper presents a connectionist realization of semantic networks, that is, it describes how knowledge about concepts, their properties, and the hierarchical relationship between them may be encoded as an interpreter-free massively parallel network of simple processing elements that can solve an interesting class of inheritance and recognition problems extremely fast—in time proportional to the depth of the conceptual hierarchy. The connectionist realization is based on an evidential formulation that leads to principled solutions to the problems of exceptions and conflicting multiple inheritance situations during inheritance, and the best-match or partial-match computation during recognition. The paper also identifies constraints that must be satisfied by the conceptual structure in order to arrive at an efficient parallel realization.     

Connectionist models of neuropsychological disorders

Artificial neural networks ('connectionist models') embody aspects of real neuronal systems. But does studying the breakdown of performance in such models help us to understand cognitive impairments in humans following brain damage? Here we review recent attempts to capture different neuropsychological disorders using connectionist models with simulated lesions. We show how such lesion studies can be used to evaluate some of the standard assumptions made in neuropsychological research, concerning both double dissociations and associations between patterns of impairment. We also illustrate how lesioned models, like humans, can sometimes be more impaired on the easier of two tasks and demonstrate that connectionist models can incorporate forms of internal structure. Finally we discuss the utility of the models for understanding and predicting the effectiveness of different rehabilitation strategies. Future questions concern the role and possible development of internal structure within these models, whether the models can be generalized to larger-scale simulations, and whether they can accommodate higher-order linguistic disorders.     

Connectionist Modelling of Word Recognition

Connectionist models offer concretemechanisms for cognitive processes. When these modelsmimic the performance of human subjects theycan offer insights into the computationswhich might underlie human cognition. We illustratethis with the performance of a recurrentconnectionist network which produces the meaningof words in response to their spellingpattern. It mimics a paradoxical pattern oferrors produced by people trying to read degradedwords. The reason why the network produces thesurprising error pattern lies in the nature ofthe attractors which it develops as it learns tomap spelling patterns to semantics. The keyrole of attractor structure in the successfulsimulation suggests that the normal adult semanticreading route may involve attractor dynamics, andthus the paradoxical error pattern isexplained.     

Connectionist Models of Word Reading

Abstract— Connectionist models of word reading attempt to explain the computational mechanisms underlying this important skill. The goal of this research is an integrated theory of reading and its brain bases, with the computational model as the interface between the two. The models are governed by computational principles that differ considerably from naive intuitions but nonetheless account for many aspects of normal and impaired (dyslexic) reading.     

CAB: Connectionist Analogy Builder

The ability to make informative comparisons is central to human cognition. Comparison involves aligning two representations and placing their elements into correspondence. Detecting correspondences is a necessary component of analogical inference, recognition, categorization, schema formation, and similarity judgment. Connectionist Analogy Builder (CAB) determines correspondences through a simple iterative computation that matches elements in one representation with elements playing compatible roles in the other representation while simultaneously enforcing structural constraints. CAB shows promise as a process model of comparison as its performance can be related to human performance (e.g., solution trajectory, error patterns, time‐on‐task). Furthermore, CAB's bounded working memory allows it to account for the inherent capacity limitations of human processing. CAB's strengths are its parsimony, transparency of operations, and ability to generate performance predictions. In this paper, CAB is evaluated against benchmark phenomena from the analogy literature.     

Connectionist Sentence Processing in Perspective

The emphasis in the connectionist sentence-processing literature on distributed representation and emergence of grammar from such systems can easily obscure the often close relations between connectionist and symbolist systems. This paper argues that the Simple Recurrent Network (SRN) models proposed by Jordan (1989) and Elman (1990) are more directly related to stochastic Part-of-Speech (POS) Taggers than to parsers or grammars as such, while auto-associative memory models of the kind pioneered by Longuet–Higgins, Willshaw, Pollack and others may be useful for grammar induction from a network-based conceptual structure as well as for structure-building. These observations suggest some interesting new directions for specifically connectionist sentence processing research, including more efficient representations for finite state machines, and acquisition devices based on a distinctively connectionist basis for grounded symbolist conceptual structure.     

Connectionist models and their properties

Much of the progress in the fields constituting cognitive science has been based upon the use of explicit information processing models, almost exclusively patterned after conventional serial computers. An extension of these ideas to massively parallel, connectionist models appears to offer a number of advantages. After a preliminary discussion, this paper introduces a general connectionist model and considers how it might be used in cognitive science. Among the issues addressed are: stability and noise-sensitivity, distributed decision-making, time and sequence problems, and the representation of complex concepts.     

A Distributed Connectionist Production System

DCPS is a connectionist production system interpreter that uses distributed representations. As a connectionist model it consists of many simple, richly interconnected neuron-like computing units that cooperate to solve problems in parallel. One motivation for constructing DCPS was to demonstrate that connectionist models are capable of representing and using explicit rules. A second motivation was to show how "coarse coding" or "distributed representations" can be used to construct a working memory that requires far fewer units than the number of different facts that can potentially be stored. The simulation we present is intended as a detailed demonstration of the feasibility of certain ideas and should not be viewed as a full implementation of production systems. Our current model only has a few of the many interesting emergent properties that we eventually hope to demonstrate: It is damage-resistant, it performs matching and variable binding by massively parallel constraint satisfaction, and the capacity of its working memory is dependent on the similarity of the items being stored.     

汉语词组的先行词联想效应

该研究在两个实验中以眼动控制方法探讨了影响汉语四字词组歧义出现的因素。实验一的结果表明这种观点是错误的:在句子理解过程中偏正词组专科学生中间的两个字符出现了歧义效应,正确的结论是在正常的汉语句子阅读中这种歧义效应不会出现;实验二检验了这种词组出现歧义效应的必要条件即这两个字符作为先行词必须出现在句首,这是一种新近性效应,结果表明这个先行词是汉语四字词组歧义出现的必要条件,这符合单一方向的句法分析假设与先行词联想效应,先行词实际上所发挥的作用就是一种背景效应或者称作先行词联想效应。     

Connectionist modeling of speech perception.

Connectionist models of perception and cognition, including the process of deducing meaningful messages from patterns of acoustic waves emitted by vocal tracts, are developed and refined as human understanding of brain function, psychological processes, and the properties of massively parallel architectures advances. The present article presents several important contributions from diverse points of view in the area of connectionist modeling of speech perception and discusses their relative merits with respect to specific theoretical issues and empirical findings. TRACE, the Elman/Norris net, and Adaptive Resonance Theory constitute pivotal points exemplifying overall modeling success, progress in temporal representation, and plausible modeling of learning, respectively. Other modeling efforts are presented for the specific insights they offer, and the article concludes with a discussion of computational versus dynamic modeling of phonological processes.