Competition and recency in a hybrid network model of syntactic disambiguation

The competitive attachment model of human parsing is a hybrid connectionist architecture consisting of a distributed feature passing method for establishing syntactic relations within the network, and a numeric competition mechanism for resolving ambiguities, which applies to all syntactic relations. Because the approach employs a uniform mechanism for establishing syntactic relations, and a single competition mechanism for disambiguation, the model can capture general behaviors of the human parser that hold across a range of syntactic constructions. In particular, attachment and binding relations are similarly processed and are therefore subject to the very same influences of disambuguation and processing over time. An important influence on the competitive disambiguation process is distance within the network. Decay of numeric activation, along with distributed feature passing through the network structure, has an unavoidable effect on the outcome of attachment and binding competitions. Inherent properties of the model thus lead to a principled explanation of recency effects in the human parsing of both attachment and filler/gap ambiguities.     

Dynamical Models of Sentence Processing

We suggest that the theory of dynamical systems provides a revealing general framework for modeling the representations and mechanism underlying syntactic processing. We show how a particular dynamical model, the Visitation Set Gravitation model of Tabor, Juliano, and Tanenhaus (1997), develops syntactic representations and models a set of contingent frequency effects in parsing that are problematic for other models. We also present new simulations showing how the model accounts for semantic effects in parsing, and propose a new account of the distinction between syntactic and semantic incongruity. The results show how symbolic structures useful in parsing arise as emergent properties of connectionist dynamical systems.     

Connectionist natural language parsing

The key developments of two decades of connectionist parsing are reviewed. Connectionist parsers are assessed according to their ability to learn to represent syntactic structures from examples automatically, without being presented with symbolic grammar rules. This review also considers the extent to which connectionist parsers offer computational models of human sentence processing and provide plausible accounts of psycholinguistic data. In considering these issues, special attention is paid to the level of realism, the nature of the modularity, and the type of processing that is to be found in a wide range of parsers.     

Connectionist psycholinguistics: capturing the empirical data

Connectionist psycholinguistics is an emerging approach to modeling empirical data on human language processing using connectionist computational architectures. For almost 20 years, connectionist models have increasingly been used to model empirical data across many areas of language processing. We critically review four key areas: speech processing, sentence processing, language production, and reading aloud, and evaluate progress against three criteria: data contact, task veridicality, and input representativeness. Recent connectionist modeling efforts have made considerable headway toward meeting these criteria, although it is by no means clear whether connectionist (or symbolic) psycholinguistics will eventually provide an integrated model of full-scale human language processing.     

Symbolically speaking: a connectionist model of sentence production

The ability to combine words into novel sentences has been used to argue that humans have symbolic language production abilities. Critiques of connectionist models of language often center on the inability of these models to generalize symbolically (Fodor & Pylyshyn, 1988; Marcus, 1998). To address these issues, a connectionist model of sentence production was developed. The model had variables (role‐concept bindings) that were inspired by spatial representations (Landau & Jackendoff, 1993). In order to take advantage of these variables, a novel dual‐pathway architecture with event semantics is proposed and shown to be better at symbolic generalization than several variants. This architecture has one pathway for mapping message content to words and a separate pathway that enforces sequencing constraints. Analysis of the model's hidden units demonstrated that the model learned different types of information in each pathway, and that the model's compositional behavior arose from the combination of these two pathways. The model's ability to balance symbolic and statistical behavior in syntax acquisition and to model aphasic double dissociations provided independent support for the dual‐pathway architecture.     

Hybrid preference machines based on inspiration from neuroscience

In the past, a variety of computational problems have been tackled with different connectionist network approaches. However, very little research has been done on a framework which connects neuroscience-inspired models with connectionist models and higher level symbolic processing. In this paper, we outline a preference machine framework which focuses on a hybrid integration of various neural and symbolic techniques in order to address how we may process higher level concepts based on concepts from neuroscience. It is a first hybrid framework which allows a link between spiking neural networks, connectionist preference machines and symbolic finite state machines. Furthermore, we present an example experiment on interpreting a neuroscience-inspired network by using preferences which may be connected to connectionist or symbolic interpretations.     

Abductive reasoning in neural-symbolic systems

Abduction is or subsumes a process of inference. It entertains possible hypotheses and it chooses hypotheses for further scrutiny. There is a large literature on various aspects of non-symbolic, subconscious abduction. There is also a very active research community working on the symbolic (logical) characterisation of abduction, which typically treats it as a form of hypothetico-deductive reasoning. In this paper we start to bridge the gap between the symbolic and sub-symbolic approaches to abduction. We are interested in benefiting from developments made by each community. In particular, we are interested in the ability of non-symbolic systems (neural networks) to learn from experience using efficient algorithms and to perform massively parallel computations of alternative abductive explanations. At the same time, we would like to benefit from the rigour and semantic clarity of symbolic logic. We present two approaches to dealing with abduction in neural networks. One of them uses Connectionist Modal Logic and a translation of Horn clauses into modal clauses to come up with a neural network ensemble that computes abductive explanations in a top-down fashion. The other combines neural-symbolic systems and abductive logic programming and proposes a neural architecture which performs a more systematic, bottom-up computation of alternative abductive explanations. Both approaches employ standard neural network architectures which are already known to be highly effective in practical learning applications. Differently from previous work in the area, our aim is to promote the integration of reasoning and learning in a way that the neural network provides the machinery for cognitive computation, inductive learning and hypothetical reasoning, while logic provides the rigour and explanation capability to the systems, facilitating the interaction with the outside world. Although it is left as future work to determine whether the structure of one of the proposed approaches is more amenable to learning than the other, we hope to have contributed to the development of the area by approaching it from the perspective of symbolic and sub-symbolic integration.

The allure of connectionism reexamined

There is currently a debate over whether cognitive architecture is classical or connectionist in nature. One finds the following three comparisons between classical architecture and connectionist architecture made in the pro-connectionist literature in this debate: (1) connectionist architecture is neurally plausible and classical architecture is not; (2) connectionist architecture is far better suited to model pattern recognition capacities than is classical architecture; and (3) connectionist architecture is far better suited to model the acquisition of pattern recognition capacities by learning than is classical architecture. If true, (1)–(3) would yield a compelling case against the view that cognitive architecture is classical, and would offer some reason to think that cognitive architecture may be connectionist. We first present the case for (1)–(3) in the very words of connectionist enthusiasts. We then argue that the currently available evidence fails to support any of (1)–(3).     

A Connectionist Approach to Knowledge Representation and Limited Inference

Although the connectionist approach has lead to elegant solutions to a number of problems in cognitive science and artificial intelligence, its suitability for dealing with problems in knowledge representation and inference has often been questioned. This paper partly answers this criticism by demonstrating that effective solutions to certain problems in knowledge representation and limited inference can be found by adopting a connectionist approach. The paper presents a connectionist realization of semantic networks, that is, it describes how knowledge about concepts, their properties, and the hierarchical relationship between them may be encoded as an interpreter-free massively parallel network of simple processing elements that can solve an interesting class of inheritance and recognition problems extremely fast—in time proportional to the depth of the conceptual hierarchy. The connectionist realization is based on an evidential formulation that leads to principled solutions to the problems of exceptions and conflicting multiple inheritance situations during inheritance, and the best-match or partial-match computation during recognition. The paper also identifies constraints that must be satisfied by the conceptual structure in order to arrive at an efficient parallel realization.     

Head position and parsing ambiguity

A strongly principle-based model of parsing seeks to employ principles of the competence grammar directly in language processing. Within grammatical theory, the Projection Principle holds that each level of syntactic representation is a uniform projection of the lexical properties of heads. With respect to parsing this suggests that a phrasal node cannot be projected until the occurrence of its head and thus constitutes a strong empirical hypothesis concerning the fundamental nature of human language processing. This paper contrasts some cross-linguistic predictions made by a specific Grammarderived parsing model against those of a well-known top-down model whose functional motivation is decidedly nonlinguistic. This latter Minimal Attachment model is found to predict significant difficulty with respect to the processing of languages such as Japanese, which display rather different surface properties than English. This problem is not encountered in a model which recognizes the crucial role of heads in licensing argument structure with respect to Processing as well as Grammar. Cross-linguistic parsing differences are attributed to the linear and structural positions of licensing heads which constitute the primary locus of the cross-linguistic variation which is therefore ultimately to be ascribed directly to the Projection Principle.     

Real-Time Processing Implications of Enriched Composition at the Syntax–Semantics Interface

This study reports results on the real-time consequences of aspectual coercion. We define aspectual coercion as a combinatorial semantic operation requiring computation over and above that provided by combining lexical items through expected syntactic processes. An experiment is described assessing whether or not parsing of a string requiring coercion—in addition to syntactic composition—is more computationally costly than parsing a syntactically transparent counterpart, a string that provides for an interpretable representation via syntactic composition alone. The prediction of a higher computational cost for this process is borne out by the results.     

Interference in short-term memory: The magical number two (or three) in sentence processing

Many theories have been proposed to explain difficulty with center embedded constructions, most attributing the problem to some kind of limited-capacity short-term memory. However, these theories have developed for the most part independently of more traditional memory research, which has focused on uncovering general principles such as chunking and interference. This article attempts to gain some unification with this research by suggesting that an interesting range of core sentence processing phenomena can be explained as interference effects in a sharply limited syntactic working memory. These include difficult and acceptable embeddings, as well as certain limitations on ambiguity resolution, length effects in garden path structures, and the requirement for locality in syntactic structure. The theory takes the form of an architecture for parsing that can index no more than two constituents under the same syntactic relation. A limitation of two or three items shows up in a variety of other verbal short-term memory tasks as well.Preparation of this paper was supported by a grant from the James S. McDonnell Foundation to the Human Information Processing Group at Princeton University. Many thanks to Martin Chodorow, Terry Langendoen, Thad Polk, and an anonymous reviewer for helpful comments on the paper and research.     

Distinctive features hold a privileged status in the computation of word meaning: Implications for theories of semantic memory

The authors present data from 2 feature verification experiments designed to determine whether distinctive features have a privileged status in the computation of word meaning. They use an attractor-based connectionist model of semantic memory to derive predictions for the experiments. Contrary to central predictions of the conceptual structure account, but consistent with their own model, the authors present empirical evidence that distinctive features of both living and nonliving things do indeed have a privileged role in the computation of word meaning. The authors explain the mechanism through which these effects are produced in their model by presenting an analysis of the weight structure developed in the network during training.     

A new look at the syntax-discourse interface: the use of binding principles in sentence processing

Within Generative Grammar, binding constraints on co-reference are usually defined in syntactic terms. However, some researchers have pointed out examples in which syntactically defined binding constraints do not seem to apply, proposing instead that a complete account of linguistic co-reference needs to consider notions of discourse structure. There have been several proposals in the literature for the division of labor between syntax and discourse in the definition of binding constraints. In this paper, we review these proposals in the context of recent work that applies on-line techniques to explore the roles of syntactic and discourse preferences in terms of the time course with which they become active during sentence comprehension. Some of this research suggests that (syntactic) binding principles may be momentarily applied during processing, even in cases in which the final interpretation suggests otherwise. We end the paper by considering the theoretical and methodological implications of this view.     

Seven principles of surface structure parsing in natural language

In generative grammar there is a traditional distinction between sentence acceptability, having to do with performance, and sentence grammaticality, having to do with competence. The attempt of this paper is to provide a characterization of the notion ‘acceptable sentence’ in English, with some suggestions as to how this characterization might be made universal. The procedure is to outline a set of procedures which are conjectured to be operative in the assignment of a surface structure tree to an input sentence. To some extent, these principles of parsing are modeled on certain parsing techniques formulated by computer scientists for computer languages. These principles account for the high acceptability of right branching structures, outline the role of grammatical function words in sentence perception, describe what seems to be a fixed limit on short-term memory in linguistic processing, and hypothesize the structure of the internal syntactic processing devices. The operation of various classes of transformations with regard to preparing deep structures for input to parsing procedures such as those outlined in the paper is discussed.     

A Probabilistic Model of Lexical and Syntactic Access and Disambiguation

The problems of access—retrieving linguistic structure from some mental grammar —and disambiguation—choosing among these structures to correctly parse ambiguous linguistic input—are fundamental to language understanding. The literature abounds with psychological results on lexical access, the access of idioms, syntactic rule access, parsing preferences, syntactic disambiguation, and the processing of garden-path sentences. Unfortunately, it has been difficult to combine models which account for these results to build a general, uniform model of access and disambiguation at the lexical, idiomatic, and syntactic levels. For example, psycholinguistic theories of lexical access and idiom access and parsing theories of syntactic rule access have almost no commonality in methodology or coverage of psycholinguistic data. This article presents a single probabilistic algorithm which models both the access and disambiguation of linguistic knowledge. The algorithm is based on a parallel parser which ranks constructions for access, and interpretations for disambiguation, by their conditional probability. Low-ranked constructions and interpretations are pruned through beam-search; this pruning accounts, among other things, for the garden-path effect. I show that this motivated probabilistic treatment accounts for a wide variety of psycholinguistic results, arguing for a more uniform representation of linguistic knowledge and for the use of probabilistically-enriched grammars and interpreters as models of human knowledge of and processing of language.     

Knowledge Representation: Two Kinds Of Emergence

Two different but closely related issues in current cognitive science will be considered in this essay. One is the controversial and extensively discussed question of how connectionist and symbolic representations of knowledge are related to each other. The other concerns the notion of connectionist learning and its relevance for the understanding of the distinction between propositional and nonpropositional knowledge. More specifically, I shall give an overview of a result in Rantala and Vadén (1994) establishing a limiting case correspondence between symbolic and connectionist representations and, on the other hand, study the problem, preliminarily investigated in Rantala (1998), of how propositional knowledge may arise from nonpropositional knowledge. I shall also try to point out that on some more or less plausible assumptions, often made by cognitive scientists, these results may have some significance when we try to comprehend the nature of human knowledge representation. Some of these assumptions are rather hypothethical and debatable for the time being and they will become justified in the future only if there will be more progress in the empirical and theoretical research on the brain and on artificial networks. The assumptions concern, besides some questions of the behavior of neural networks, such things as the relevance of pattern recognition for modelling human cognition, in particular, knowledge acquisition, and the relation between emergence and reduction.     

The role of structural context in perception: Syntax in the recognition of algebraic expressions

Two character-identification experiments investigated the function of structural context during the processing of briefly exposed algebraic strings. Neither experiment prodded evidence to support the notion of an algebra-superiority effect, a contextually driven enhancement of the recognition of specific algebraic characters. However, the results of Experiment 2 indicate that the structure of algebra does provide information at the level of a character’s categorical denomination. These findings suggest that the parsing of an algebraic string includes a level of processing in which its structural context places restrictions on the denominations of its symbols. A processing model of algebraic perception is proposed that incorporates these syntactic constraints—constraints that appear to be independent of feature-based character identification processes.