A Connectionist Approach to Knowledge Representation and Limited Inference

Although the connectionist approach has lead to elegant solutions to a number of problems in cognitive science and artificial intelligence, its suitability for dealing with problems in knowledge representation and inference has often been questioned. This paper partly answers this criticism by demonstrating that effective solutions to certain problems in knowledge representation and limited inference can be found by adopting a connectionist approach. The paper presents a connectionist realization of semantic networks, that is, it describes how knowledge about concepts, their properties, and the hierarchical relationship between them may be encoded as an interpreter-free massively parallel network of simple processing elements that can solve an interesting class of inheritance and recognition problems extremely fast—in time proportional to the depth of the conceptual hierarchy. The connectionist realization is based on an evidential formulation that leads to principled solutions to the problems of exceptions and conflicting multiple inheritance situations during inheritance, and the best-match or partial-match computation during recognition. The paper also identifies constraints that must be satisfied by the conceptual structure in order to arrive at an efficient parallel realization.     

Connectionist models of recognition memory: constraints imposed by learning and forgetting functions

Multilayer connectionist models of memory based on the encoder model using the backpropagation learning rule are evaluated. The models are applied to standard recognition memory procedures in which items are studied sequentially and then tested for retention. Sequential learning in these models leads to 2 major problems. First, well-learned information is forgotten rapidly as new information is learned. Second, discrimination between studied items and new items either decreases or is nonmonotonic as a function of learning. To address these problems, manipulations of the network within the multilayer model and several variants of the multilayer model were examined, including a model with prelearned memory and a context model, but none solved the problems. The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning.     

Acquiring contextualized concepts: a connectionist approach

Conceptual knowledge is acquired through recurrent experiences, by extracting statistical regularities at different levels of granularity. At a fine level, patterns of feature co-occurrence are categorized into objects. At a coarser level, patterns of concept co-occurrence are categorized into contexts. We present and test CONCAT, a connectionist model that simultaneously learns to categorize objects and contexts. The model contains two hierarchically organized CALM modules (Murre, Phaf, & Wolters, 1992). The first module, the Object Module, forms object representations based on co-occurrences between features. These representations are used as input for the second module, the Context Module, which categorizes contexts based on object co-occurrences. Feedback connections from the Context Module to the Object Module send activation from the active context to those objects that frequently occur within this context. We demonstrate that context feedback contributes to the successful categorization of objects, especially when bottom-up feature information is degraded or ambiguous.     

Hybrid preference machines based on inspiration from neuroscience

In the past, a variety of computational problems have been tackled with different connectionist network approaches. However, very little research has been done on a framework which connects neuroscience-inspired models with connectionist models and higher level symbolic processing. In this paper, we outline a preference machine framework which focuses on a hybrid integration of various neural and symbolic techniques in order to address how we may process higher level concepts based on concepts from neuroscience. It is a first hybrid framework which allows a link between spiking neural networks, connectionist preference machines and symbolic finite state machines. Furthermore, we present an example experiment on interpreting a neuroscience-inspired network by using preferences which may be connected to connectionist or symbolic interpretations.     

Are there lexicons?

Many models of the processing of printed or spoken words or objects or faces propose that systems of local representations of the forms of such stimuli—lexicons—exist. This is denied by partisans of the distributed-representation connectionist approach to cognitive modelling. An experimental paradigm of key theoretical importance here is lexical decision and its analogue in the domain of objects, object decision. How does each theoretical camp account for our ability to perform these two tasks? The localists say that the tasks are done by matching or failing to match a stimulus to a local representation in a lexicon. Advocates of distributed representations often do not seek to explain these two tasks; however, when they do, they propose that patterns of activation evoked in a semantic system can be used to discriminate between words and nonwords, or between real objects and false objects. Therefore the distributed-representation account of lexical and object decision tasks predicts that performance on these tasks can never be normal in patients with an impaired semantic system, nor in patients who cannot access semantics normally from the stimulus domain being tested. However, numerous such patients have been reported in the literature, indicating that semantic access is not needed for normal performance on these tasks. Such results support the localist form of modelling rather than the distributed-representation approach.     

Competition and recency in a hybrid network model of syntactic disambiguation

The competitive attachment model of human parsing is a hybrid connectionist architecture consisting of a distributed feature passing method for establishing syntactic relations within the network, and a numeric competition mechanism for resolving ambiguities, which applies to all syntactic relations. Because the approach employs a uniform mechanism for establishing syntactic relations, and a single competition mechanism for disambiguation, the model can capture general behaviors of the human parser that hold across a range of syntactic constructions. In particular, attachment and binding relations are similarly processed and are therefore subject to the very same influences of disambuguation and processing over time. An important influence on the competitive disambiguation process is distance within the network. Decay of numeric activation, along with distributed feature passing through the network structure, has an unavoidable effect on the outcome of attachment and binding competitions. Inherent properties of the model thus lead to a principled explanation of recency effects in the human parsing of both attachment and filler/gap ambiguities.     

Conceptual blending of high-level features and data-driven salience computation in melodic generation

Conceptual Blending (CB) theory describes the cognitive mechanisms underlying the way humans process the emergence of new conceptual spaces by blending two input spaces. CB theory has been primarily used as a method for interpreting creative artefacts, while recently it has been utilised in the context of computational creativity for algorithmic invention of new concepts. Examples in the domain of music include the employment of CB interpretatively as a tool to explain musical semantic structures based on lyrics of songs or on the relations between body gestures and music structures. Recent work on generative applications of CB has shown that proper low-level representation of the input spaces allows the generation of consistent and sometimes surprising blends. However, blending high-level features (as discussed in the interpretative studies) of music explicitly, is hardly feasible with mere low-level representation of objects. Additionally, selecting features that are more salient in the context of two input spaces and relevant background knowledge and should, thus, be preserved and integrated in new interesting blends has not yet been tackled in a cognitively pertinent manner. The paper at hand proposes a novel approach to generating new material that allows blending high-level features by combining low-level structures, based on statistically computed salience values for each high-level feature extracted from data. The proposed framework is applied to a basic but, at the same time, complicated field of music, namely melodic generation. The examples presented herein allow an insightful examination of what the proposed approach does, revealing new possibilities and prospects.     

Theory learning as stochastic search in the language of thought

We present an algorithmic model for the development of children's intuitive theories within a hierarchical Bayesian framework, where theories are described as sets of logical laws generated by a probabilistic context-free grammar. We contrast our approach with connectionist and other emergentist approaches to modeling cognitive development. While their subsymbolic representations provide a smooth error surface that supports efficient gradient-based learning, our symbolic representations are better suited to capturing children's intuitive theories but give rise to a harder learning problem, which can only be solved by exploratory search. Our algorithm attempts to discover the theory that best explains a set of observed data by performing stochastic search at two levels of abstraction: an outer loop in the space of theories and an inner loop in the space of explanations or models generated by each theory given a particular dataset. We show that this stochastic search is capable of learning appropriate theories in several everyday domains and discuss its dynamics in the context of empirical studies of children's learning.     

A feature-integration theory of attention

A new hypothesis about the role of focused attention is proposed. The feature-integration theory of attention suggests that attention must be directed serially to each stimulus in a display whenever conjunctions of more than one separable feature are needed to characterize or distinguish the possible objects presented. A number of predictions were tested in a variety of paradigms including visual search, texture segregation, identification and localization, and using both separable dimensions (shape and color) and local elements or parts of figures (lines, curves, etc. in letters) as the features to be integrated into complex wholes. The results were in general consistent with the hypothesis. They offer a new set of criteria for distinguishing separable from integral features and a new rationale for predicting which tasks will show attention limits and which will not.     

An integrated account of generalization across objects and features

Humans routinely make inductive generalizations about unobserved features of objects. Previous accounts of inductive reasoning often focus on inferences about a single object or feature: accounts of causal reasoning often focus on a single object with one or more unobserved features, and accounts of property induction often focus on a single feature that is unobserved for one or more objects. We explore problems where people must make inferences about multiple objects and features, and propose that people solve these problems by integrating knowledge about features with knowledge about objects. We evaluate three computational methods for integrating multiple systems of knowledge: the output combination approach combines the outputs produced by these systems, the distribution combination approach combines the probability distributions captured by these systems, and the structure combination approach combines a graph structure over features with a graph structure over objects. Three experiments explore problems where participants make inferences that draw on causal relationships between features and taxonomic relationships between animals, and we find that the structure combination approach provides the best account of our data.     

Do current connectionist learning models account for reading development in different languages?

Learning to read a relatively irregular orthography, such as English, is harder and takes longer than learning to read a relatively regular orthography, such as German. At the end of grade 1, the difference in reading performance on a simple set of words and nonwords is quite dramatic. Whereas children using regular orthographies are already close to ceiling, English children read only about 40% of the words and nonwords correctly. It takes almost 4 years for English children to come close to the reading level of their German peers. In the present study, we investigated to what extent recent connectionist learning models are capable of simulating this cross-language learning rate effect as measured by nonword decoding accuracy. We implemented German and English versions of two major connectionist reading models, Plaut et al.'s (Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996). Understanding normal and impaired word reading: computational principles in quasi-regular domains. Psychological Review, 103, 56-115) parallel distributed model and Zorzi et al.'s (Zorzi, M., Houghton, G., & Butterworth, B. (1998a). Two routes or one in reading aloud? A connectionist dual-process model. Journal of Experimental Psychology: Human Perception and Performance, 24, 1131-1161); two-layer associative network. While both models predicted an overall advantage for the more regular orthography (i.e. German over English), they failed to predict that the difference between children learning to read regular versus irregular orthographies is larger earlier on. Further investigations showed that the two-layer network could be brought to simulate the cross-language learning rate effect when cross-language differences in teaching methods (phonics versus whole-word approach) were taken into account. The present work thus shows that in order to adequately capture the pattern of reading acquisition displayed by children, current connectionist models must not only be sensitive to the statistical structure of spelling-to-sound relations but also to the way reading is taught in different countries.     

Rats (Rattus norvegicus) encode the shape of an array of discrete objects

To navigate efficiently, a traveler must establish a heading using a frame of reference. A large body of evidence has indicated that humans and a variety of nonhuman animals utilize the geometry, or shape, of enclosed spaces as a frame of reference to determine their heading. An important and yet unresolved question is whether shape information from arrays of discrete objects and enclosed environments are represented, and utilized, in the same way. In the present study, rats were presented with a reference memory task in which they had to find water that was hidden in 1 of 4 discrete and unique objects placed at the vertices of a rectangle. The results indicate that rats could utilize both feature and geometry cues to locate the hidden goal. The rats' performance declined during transformation tests using a triangular array, indicating that the rats may have encoded the primary axis of the object array, rather than local cues, to direct their search.     

A constructivist connectionist model of transitions on false-belief tasks

How do children come to understand that others have mental representations, e.g., of an object’s location? Preschoolers go through two transitions on verbal false-belief tasks, in which they have to predict where an agent will search for an object that was moved in her absence. First, while three-and-a-half-year-olds usually fail at approach tasks, in which the agent wants to find the object, children just under four succeed. Second, only after four do children succeed at tasks in which the agent wants to avoid the object. We present a constructivist connectionist model that autonomously reproduces the two transitions and suggests that the transitions are due to increases in general processing abilities enabling children to (1) overcome a default true-belief attribution by distinguishing false- from true-belief situations, and to (2) predict search in avoidance situations, where there is often more than one correct, empty search location. Constructivist connectionist models are rigorous, flexible and powerful tools that can be analyzed before and after transitions to uncover novel and emergent mechanisms of cognitive development.     

Serial search and comparison of features of imagined and perceived objects

Four experiments studied the spatial information processing involved in making a series of same-different comparisons of features of two objects. When the path between successively compared features on one object was antiparallel with the corresponding path on the other, comparison of a series of features took longer and produced many more errors. These results were observed both when the objects were externally presented and when one object was imagined and the other externally presented. Knowing the location of the next feature seems much more important for effective search than does monitoring the location of the feature used in the preceding comparison. When paths between corresponding features are parallel, search of features of one object may guide search of the other. When the directions between corresponding features are incongruous, search for the next angle may produce a competition for processes or processing resources, or may produce interfering cross-talk between the spatial information processing of the concurrent search tasks. Because of incongruity, as demonstrated in this study, serial search of objects at different orientations is difficult.     

Implicit and explicit memory: Implications for the symbol-manipulation versus connectionism controversy

Summary This paper deals with three issues. First, the symbol-manipulation versus connectionism controversy is discussed briefly. Both can be seen as formalisms or languages for describing human behavior, but it is argued that these languages are not fully equivalent. The importance of the ability of connectionist models to learn autonomously, often underestimated, allows them to perform tasks that may defy formalization. Secondly, the structural and functional characteristics of a new connectionist learning model are presented. It avoids some of the psychological and biological implausibilities of currently popular models and solves some of the problems and shortcomings of these models. Thirdly, some simulation results are presented. The model successfully simulates the dissociation between explicit and implicit memory performance as found in patients with anterograde amnesia. This simulation provides an example of the advantages of using the connectionist language in the domain of memory psychology. It is also a good example of how psychological evidence can be used to improve connectionist models.The second author is supported by the Dutch Organization for Scientific Research (NWO)     

Connectionist representation,semantic compositionality,and the instability of concept structure

Summary In the present paper connectionist approaches to the problem of internal representation and the nature of concepts are discussed. In the first part the concept of representation that underlies connectionist modeling is made explicit. It is argued that the connectionist view of representation relies on a correlational theory of semantic content- i.e., the covariation between internal and external states is taken as the basis for ascribing meaning to internal states. The problems and virtues of such a correlational approach to internal representation are addressed. The second part of the paper is concerned with whether connectionism is capable of accounting for the apparent productivity and systematicity of language and thought. There is an evaluation of the recent arguments of Fodor and Pylyshyn, who claim that systematicity can only be explained if one conceives of mental representations as structured symbols composed of context-free constituents. There is a review of empirical evidence that strongly suggests that concepts are not fixed memory structures and that the meaning of constituent symbols varies, depending on the context in which they are embedded. On the basis of this review it is concluded that the meaning of a complex expression is not computed from the context-free meanings of the constituents, and that strong compositionality, as endorsed by Fodor and Pylyshyn (1988), seems implausible as a process theory for the comprehension of complex concepts. Instead, the hypothesis is endorsed that constraint satisfaction in distributed connectionist networks may allow for an alternative account of weak compositionality compatible with the context sensitivity of meaning. In the final section, it is argued that neither mere implementation of a language of thought in connectionist networks nor radical elimination of symbol systems seems to be a fruitful research strategy, but that it might be more useful to discuss how connectionist systems can develop the capacity to use external symbol systems like language or logic without instantiating symbol systems themselves.     

The Dynamics of Visual Word Recognition

This article provides an overview of a dynamical systems approach to visual word recognition. In this approach, the dynamics of word recognition are characterized in terms of a connectionist network model. According to this model, seeing a word results in changes in the pattern of activation over the nodes in the lexical network such that, over time, the network moves into an attractor state representing the orthographic, phonological, and semantic properties of that word. At a slower timescale, a learning process modifies the strengths of the connections among the nodes in a way that attunes the network to the statistical regularities in its environment. This view of word identification accommodates a wide body of empirical results, a representative sampling of which is discussed here. Finally, the article closes with a discussion of some of the theoretical issues that should be addressed as the dynamical approach continues to develop.     

Perception as interacting psychophysical functions. Could the configuring of features replace a specialised receptor?

This paper illustrates how perception is achieved through interactions among the psychophysical functions of judged features of an object. The theory is that the perceiver places processed features in a multidimensional space of discriminal processes. Each dimension is scaled in units of discrimination performance. The zero coordinate of each feature is its level in an internal standard (norm) established by previous experience of that category of object in context. Experiments are reported which show that one, two, or three concurrent single-featured objects matched the multiple features of another object in two ways. Either stimulation from the two objects had discrimination distances from norm that added, or the stimulation by one object was processed through a concept describing stimulation by the other object. It follows that, in this case, perception via a receptor for the multi-featured object can be replaced by a point of balance among receptors for each single feature. The object with its own receptor is the gustatory stimulant L-glutamic acid as its monosodium salt. The features that stimulate diverse gustatory receptors of their own are sodium chloride, citric acid, sucrose, and caffeine. A more complex approach to dimensional coding was developed earlier for photoreceptors in colour judgments. The present approach is modality independent, mathematically simple, and economical in experimental data.     

Visual search, visual streams, and visual architectures.

Most psychological, physiological, and computational models of early vision suggest that retinal information is divided into a parallel set of feature modules. The dominant theories of visual search assume that these modules form a "blackboard" architecture: a set of independent representations that communicate only through a central processor. A review of research shows that blackboard-based theories, such as feature-integration theory, cannot easily explain the existing data. The experimental evidence is more consistent with a "network" architecture, which stresses that: (1) feature modules are directly connected to one another, (2) features and their locations are represented together, (3) feature detection and integration are not distinct processing stages, and (4) no executive control process, such as focal attention, is needed to integrate features. Attention is not a spotlight that synthesizes objects from raw features. Instead, it is better to conceptualize attention as an aperture which masks irrelevant visual information.