ALCOVE: an exemplar-based connectionist model of category learning.

ALCOVE (attention learning covering map) is a connectionist model of category learning that incorporates an exemplar-based representation (Medin & Schaffer, 1978; Nosofsky, 1986) with error-driven learning (Gluck & Bower, 1988; Rumelhart, Hinton, & Williams, 1986). Alcove selectively attends to relevant stimulus dimensions, is sensitive to correlated dimensions, can account for a form of base-rate neglect, does not suffer catastrophic forgetting, and can exhibit 3-stage (U-shaped) learning of high-frequency exceptions to rules, whereas such effects are not easily accounted for by models using other combinations of representation and learning method.     

Phonology, reading acquisition, and dyslexia: insights from connectionist models.

The development of reading skill and bases of developmental dyslexia were explored using connectionist models. Four issues were examined: the acquisition of phonological knowledge prior to reading, how this knowledge facilitates learning to read, phonological and nonphonological bases of dyslexia, and effects of literacy on phonological representation. Compared with simple feedforward networks, representing phonological knowledge in an attractor network yielded improved learning and generalization. Phonological and surface forms of developmental dyslexia, which are usually attributed to impairments in distinct lexical and nonlexical processing "routes," were derived from different types of damage to the network. The results provide a computationally explicit account of many aspects of reading acquisition using connectionist principles.     

U-shaped learning and frequency effects in a multi-layered perceptron: implications for child language acquisition

A three-layer back-propagation network is used to implement a pattern association task in which four types of mapping are learned. These mappings, which are considered analogous to those which characterize the relationship between the stem and past tense forms of English verbs, include arbitrary mappings, identity mappings, vowel changes, and additions of a suffix. The degree of correspondence between parallel distributed processing (PDP) models which learn mappings of this sort (e.g., Rumelhart & McClelland, 1986, 1987) and children's acquisition of inflectional morphology has recently been at issue in discussions of the applicability of PDP models to the study of human cognition and language (Pinker & Mehler, 1989; Bever, in press). In this paper, we explore the capacity of a network to learn these types of mappings, focusing on three major issues. First, we compare the performance of a single-layered perceptron similar to the one used by Rumelhart and McClelland with a multi-layered perceptron. The results suggest that it is unlikely that a single-layered perceptron is capable of finding an adequate solution to the problem of mapping stems and past tense forms in input configurations that are sufficiently analogous to English. Second, we explore the input conditions which determine learning in these networks. Several factors that characterize linguistic input are investigated: (a) the nature of the mapping performed by the network (arbitrary, suffixation, identity, and vowel change); (b) the competition effects that arise when the task demands simultaneous learning of distinct mapping types; (c) the role of the type and token frequency of verb stems; and (d) the influence of phonological subregularities in the irregular verbs. Each of these factors is shown to have selective consequences on both successful and erroneous performance in the network. Third, we outline several types of systems which could result in U-shaped acquisition, and discuss the ways in which learning in multi-layered networks can be seen to capture several characteristics of U-shaped learning in children. In general, these models provide information about the role of input in determining the kinds of errors that a network will produce, including the conditions under which rule-like behavior and U-shaped learning will and will not emerge. The results from all simulations are discussed in light of behavioral data on children's acquisition of the past tense and the validity of drawing conclusions about the acquisition of language from models of this sort.     

Children's Productivity in the English Past Tense: The Role of Frequency, Phonology, and Neighborhood Structure

The productive use of English past tense morphology in school-aged children (N = 74; 3 years, 8 months to 13 years, 5 months) is explored using an elicited production task. Errors represented 20% of the responses overall. Virtually all of the children demonstrated productivity with regular (e.g., good) and irregular patterns (zero-marking, e.g., sit → sit; vowel-change, e.g., ride → rid). Overall frequency of errors decreased with age, yet the tendency for certain types of irregularizations increased in the older groups. Analyses across items indicated that all error types were predicted by combinations of item frequency, phonological characteristics of stems and past tense forms, and aspects of phonological past tense "neighborhoods". Contrary to "hybrid" or dual-mechanism models incorporating a phonologically-insensitive default mechanism (e.g., Prasada & Pinker, 1993), these results suggest that children's productivity with regular and irregular patterns is consistent with a phonologically-based constraint satisfaction system similar to that implemented in connectionist models (Daugherty & Seidenberg, 1992; Plunkett & Marchman, 1991, 1993).     

Rules or connections in past-tense inflections: what does the evidence rule out?

Pinker and colleagues propose two mechanisms - a rule system and a lexical memory - to form past tenses and other inflections. They predict that children's acquisition of the regular inflection is sudden; that the regular inflection applies uniformly regardless of phonological, semantic or other factors; and that the rule system is separably vulnerable to disruption. A connectionist account makes the opposite predictions. Pinker has taken existing evidence as support for his theory, but the review of the evidence presented here contradicts this assessment. Instead, it supports all three connectionist predictions: gradual acquisition of the past tense inflection; graded sensitivity to phonological and semantic content; and a single, integrated mechanism for regular and irregular forms, dependent jointly on phonology and semantics.     

On the processing of regular and irregular forms of verbs and nouns: evidence from neuropsychology

Following acquired brain damage, a native English speaking patient (AW) encountered problems accessing phonology in speech production, while her ability to access word meaning appeared to be intact. In a series of tasks, AW was presented either with a verb, and was asked to produce its past tense or past participle (walk --> "walked"), or with a noun, and was asked to produce its plural (glove --> "gloves"). A stark dissociation was found: while AW responded accurately with regular forms of verbs (walked) and nouns (gloves), performance was significantly less accurate with irregular forms (found; children). The appearance of a selective deficit for irregular forms in conditions of impaired lexical access is in line with dual-mechanism accounts, which proposes that irregular forms are specified in the lexicon whereas regular forms are computed via rule-based mechanisms. In contrast, AW's data are problematic for connectionist accounts that do not posit separate mechanisms for processing regular and irregular forms, including the connectionist model recently proposed by Joanisse and Seidenberg (Proceedings of the National Academy of Sciences USA 96 (1999) 7592) which successfully simulated a variety of earlier neuropsychological findings. Analyses of AW's responses shed light on further details of the representation and processing of regular and irregular inflected forms.     

In search of meaning: semantic effects on past-tense inflection

Within single-mechanism connectionist models of inflectional morphology, generating the past-tense form of a verb depends upon the interaction of semantic and phonological representations, with semantic information being particularly important for irregular or exception verbs. We assessed this hypothesis in two experiments requiring normal speakers to produce the past tense from a verb stem that takes a regular or exceptional past tense. Experiment 1 revealed significant latency advantages for high- over low-imageability words for both regular verbs (e.g., "lunged" faster than "loved") and exception items (e.g., "drank" faster than "dealt"); but critically, this effect was significantly larger for exceptions than for regulars. Experiment 2 employed a semantic priming paradigm where participants inflected verb stems (e.g., sit) preceded by related (e.g., chair) or unrelated primes (e.g., jug) and revealed a priming effect in accuracy that was confined to the exception items. Our results are consistent with predictions from single-mechanism connectionist models of inflectional morphology and converge with findings from neurological patients and studies of reading aloud.     

Imaging the past: Neural activation in frontal and temporal regions during regular and irregular past-tense processing

This article presents fMRI evidence bearing on dual-mechanism versus connectionist theories of inflectional morphology. Ten participants were scanned at 4 Tesla as they covertly generated the past tenses of real and nonce (nonword) verbs presented auditorily. Regular past tenses (e.g., walked, wugged) and irregular past tenses (e.g., took, slept) produced similar patterns of activation in the posterior temporal lobe in both hemispheres. In contrast, there was greater activation in left and right inferior frontal gyrus for regular past tenses than for irregular past tenses. Similar previous results have been taken as evidence for the dual-mechanism theory of the past tense (Pinker & Ullman, 2002). However, additional analyses indicated that irregulars that were phonologically similar to regulars (e.g., slept, fled, sold) produced the same level of activation as did regulars, and significantly more activation than did irregulars that were not phonologically similar to regulars (e.g., took, gave). Thus, activation patterns were predicted by phonological characteristics of the past tense rather than by the rule-governed versus exception distinction that is central to the dual-mechanism framework. The results are consistent with a constraint satisfaction model in which phonological, semantic, and other probabilistic constraints jointly determine the past tense, with different degrees of involvement for different verbs.     

Competitive Learning: From Interactive Activation to Adaptive Resonance

Functional and mechanistic comparisons are made between several network models of cognitive processing: competitive learning, interactive activation, adaptive resonance, and back propagation. The starting point of this comparison is the article of Rumelhart and Zipser (1985) on feature discovery through competitive learning. All the models which Rumelhart and Zipser (1985) have described were shown in Grossberg (1976b) to exhibit a type of learning which is temporally unstable. Competitive learning mechanisms can be stabilized in response to an arbitrary input environment by being supplemented with mechanisms for learning top-down expectancies, or templates; for matching bottom-up input patterns with the top-down expectancies; and for releasing orienting reactions in a mismatch situation, thereby updating short-term memory and searching for another internal representation. Network architectures which embody all of these mechanisms were called adaptive resonance models by Grossberg (1976c). Self-stabilizing learning models are candidates for use in real-world applications where unpredictable changes can occur in complex input environments. Competitive learning postulates are inconsistent with the postulates of the interactive activation model of McClelland and Rumelhart (1981), and suggest different levels of processing and interaction rules for the analysis of word recognition. Adaptive resonance models use these alternative levels and interaction rules. The self-organizing learning of an adaptive resonance model is compared and contrasted with the teacher-directed learning of a back propagation model. A number of criteria for evaluating real-time network models of cognitive processing are described and applied.     

A Connectionist Approach to Word Reading and Acquired Dyslexia: Extension to Sequential Processing

A connectionist approach to word reading, based on the principles of distributed representation, graded learning of statistical structure, and interactivity in processing, has led to the development of explicit computational models which account for a wide range of data on normal skilled reading and on patterns of reading impairment due to brain damage. There have, however, been recent empirical challenges to these models, and the approach in general, relating to the influence of orthographic length on the naming latencies of both normal and dyslexic readers. The current work presents a simulation which generates sequential phonological output in response to written input, and which can refixate the input when encountering difficulty. The normal model reads both words and nonwords accurately, and exhibits an effect of orthographic length and a frequency-by-consistency interaction in its naming latencies. When subject to peripheral damage, the model exhibits an increased length effect that interacts with word frequency, characteristic of letter-by-letter reading in pure alexia. Although the model is far from a fully adequate account of all the relevant phenomena, it suggests how connectionist models may be extended to provide deeper insight into sequential processes in reading.     

A Connectionist Model of English Past Tense and Plural Morphology

The acquisition of English noun and verb morphology is modeled using a single-system connectionist network. The network is trained to produce the plurals and past tense forms of a large corpus of monosyllabic English nouns and verbs. The developmental trajectory of network performance is analyzed in detail and is shown to mimic a number of important features of the acquisition of English noun and verb morphology in young children. These include an initial error-free period of performance on both nouns and verbs followed by a period of intermittent over-regularization of irregular nouns and verbs. Errors in the model show evidence of phonological conditioning and frequency effects. Furthermore, the network demonstrates a strong tendency to regularize denominal verbs and deverbal nouns and masters the principles of voicing assimilation. Despite their incorporation into a single-system network, nouns and verbs exhibit some important differences in their profiles of acquisition. Most importantly, noun inflections are acquired earlier than verb inflections. The simulations generate several empirical predictions that can be used to evaluate further the suitability of this type of cognitive architecture in the domain of inflectional morphology.     

Connectionist modelling of the separable processing of consonants and vowels

describe two aphasic patients, with impaired processing of vowels and consonants, respectively. The impairments could not be captured according to the sonority hierarchy or in terms of a feature level analysis. Caramazza et al. claim that this dissociation demonstrates separate representation of the categories of vowels and consonants in speech processing. We present two connectionist models of the management of phonological representations. The models spontaneously develop separable processing of vowels and consonants. The models have two hidden layers and are given as input vowels and consonants represented in terms of their phonological distinctive features. The first model is presented with feature bundles one at a time and the hidden layers have to combine their output to reproduce a unified copy of the feature bundle. In the second model a "fine-coded" layer receives information about feature bundles in isolation, and a "coarse-coded" layer receives information about each feature bundle in the context of the prior and subsequent feature bundle. Coarse-coding facilitated processing of vowels and fine-coding processing of consonants. These models show that separable processing of vowels and consonants is an emergent effect of modular processors operating on feature-based representations. We argue that it is not necessary to postulate an independent level of representation for the consonant/vowel distinction, separate from phonological distinctive features.     

Dynamic self-organization and early lexical development in children

In this study we present a self-organizing connectionist model of early lexical development. We call this model DevLex-II, based on the earlier DevLex model. DevLex-II can simulate a variety of empirical patterns in children's acquisition of words. These include a clear vocabulary spurt, effects of word frequency and length on age of acquisition, and individual differences as a function of phonological short-term memory and associative capacity. Further results from lesioned models indicate developmental plasticity in the network's recovery from damage, in a non-monotonic fashion. We attribute the network's abilities in accounting for lexical development to interactive dynamics in the learning process. In particular, variations displayed by the model in the rate and size of early vocabulary development are modulated by (a) input characteristics, such as word frequency and word length, (b) consolidation of lexical-semantic representation, meaning-form association, and phonological short-term memory, and (c) delayed processes due to interactions among timing, severity, and recoverability of lesion. Together, DevLex and DevLex-II provide an accurate computational account of early lexical development.     

A Behavioral Study of Regularity,Irregularity and Rules in the English Past Tense

Opposing views of storage and processing of morphologically complex words (e.g., past tense) have been suggested: the dual system, whereby regular forms are not in the lexicon but are generated by rule, while irregular forms are explicitly represented; the single system, whereby regular and irregular forms are computed by a single system, using associative connections; and a system whereby phonological rules relate both regular and irregular past to present tense forms. Two reaction time experiments investigated the production of the past tense in English in response to the auditory presentation of the present tense of the verb. The first experiment addressed the methodology of presenting regulars and irregulars in blocked form as in a previous study (Jaeger et al. in Language 72:451–497, 1996). Blocked presentation results showed longer RTs for the elicitation of irregular pasts than for regular pasts; however, there were no differences between regular and irregular elicitation when the presentation was randomized, indicating that it is rules that are being primed. The second experiment tested whether the response time advantage found for blocked regular verbs in the first experiment might also extend to irregular verb forms exhibiting the same sub-regularity (e.g., sing-sang may prime ring-rang). Results showed a trend towards slower RTs when past tense forms from different sub-regularities follow one another, suggesting interference between one sub-regularity and another.     

The role of meaning in inflection: why the past tense does not require a rule

How do we produce the past tenses of verbs? For the last 20 years this question has been the focal domain for conflicting theories of language, knowledge representation, and cognitive processing. On one side of the debate have been similarity-based or single-route approaches that propose that all past tenses are formed simply through phonological analogies to existing past tenses stored in memory. On the other side of the debate are rule-based or dual-route approaches which agree that phonological analogy is important for producing irregular past tenses (e.g., think-->thought), but argue that regular past tenses (e.g., walk-->walked) are generated via a +ed rule and that a principled account of regular inflection can only be given by recourse to explicit rules. This debate has become a crucial battleground for arguments concerning the necessity and importance of abstract mental rules, embracing not only language processing, but also the of nature cognition itself. However, in centering on the roles of phonological similarity and rules, the past tense debate has largely ignored the possible role of semantics in determining inflection. This paper presents five studies that demonstrate a striking and decisive role of semantic similarity in inflection. In fact, semantic factors appear to be more important in inflection than the grammatical considerations put forward by the dual-route account. Further, these new findings provide a new way of discriminating between the claims of single-route (similarity-based) and dual-route (rule-based) approaches. It appears that inflection is carried out through analogical reminding based on semantic and phonological similarity and that a rule-based route is not necessary to account for past tense inflection.     

The past tense inflection project (PTIP): speeded past tense inflections, imageability ratings, and past tense consistency measures for 2,200 verbs

The processes involved in past tense verb generation have been central to models of inflectional morphology. However, the empirical support for such models has often been based on studies of accuracy in past tense verb formation on a relatively small set of items. We present the first large-scale study of past tense inflection (the Past Tense Inflection Project, or PTIP) that affords response time, accuracy, and error analyses in the generation of the past tense form from the present tense form for over 2,000 verbs. In addition to standard lexical variables (such as word frequency, length, and orthographic and phonological neighborhood), we have also developed new measures of past tense neighborhood consistency and verb imageability for these stimuli, and via regression analyses we demonstrate the utility of these new measures in predicting past tense verb generation. The PTIP can be used to further evaluate existing models, to provide well controlled stimuli for new studies, and to uncover novel theoretical principles in past tense morphology.     

Connectionist representation,semantic compositionality,and the instability of concept structure

Summary In the present paper connectionist approaches to the problem of internal representation and the nature of concepts are discussed. In the first part the concept of representation that underlies connectionist modeling is made explicit. It is argued that the connectionist view of representation relies on a correlational theory of semantic content- i.e., the covariation between internal and external states is taken as the basis for ascribing meaning to internal states. The problems and virtues of such a correlational approach to internal representation are addressed. The second part of the paper is concerned with whether connectionism is capable of accounting for the apparent productivity and systematicity of language and thought. There is an evaluation of the recent arguments of Fodor and Pylyshyn, who claim that systematicity can only be explained if one conceives of mental representations as structured symbols composed of context-free constituents. There is a review of empirical evidence that strongly suggests that concepts are not fixed memory structures and that the meaning of constituent symbols varies, depending on the context in which they are embedded. On the basis of this review it is concluded that the meaning of a complex expression is not computed from the context-free meanings of the constituents, and that strong compositionality, as endorsed by Fodor and Pylyshyn (1988), seems implausible as a process theory for the comprehension of complex concepts. Instead, the hypothesis is endorsed that constraint satisfaction in distributed connectionist networks may allow for an alternative account of weak compositionality compatible with the context sensitivity of meaning. In the final section, it is argued that neither mere implementation of a language of thought in connectionist networks nor radical elimination of symbol systems seems to be a fruitful research strategy, but that it might be more useful to discuss how connectionist systems can develop the capacity to use external symbol systems like language or logic without instantiating symbol systems themselves.     

Connectionist models of social learning: a case of learning by observing a simple task]

This article proposes a connectionist model of the social learning theory developed by Bandura (1977). The theory posits that an individual in an interactive situation is capable of learning new behaviours merely by observing them in others. Such learning is acquired through an initial phase in which the individual memorizes what he has observed (observation phase), followed by a second phase where he puts the recorded observations to use as a guide for adjusting his own behaviour (reproduction phase). We shall refer to the two above-mentioned phases to demonstrate that it is conceivable to simulate learning by observation otherwise than through the recording of perceived information using symbolic representation. To this end we shall rely on the formalism of ecological neuron networks (Parisi, Cecconi, & Nolfi, 1990) to implement an agent provided with the major processes identified as essential to learning through observation. The connectionist model so designed shall implement an agent capable of recording perceptive information and producing motor behaviours. The learning situation we selected associates an agent demonstrating goal-achievement behaviour and an observer agent learning the same behaviour by observation. Throughout the acquisition phase, the demonstrator supervises the observer's learning process based on association between spatial information (input) and behavioural information (output). Representation thus constructed then serves as an adjustment guide during the production phase, involving production by the observer of a sequence of actions which he compares to the representation stored in distributed form as constructed through observation. An initial simulation validates model architecture by confirming the requirement for both phases identified in the literature (Bandura, 1977) to simulate learning through observation. The representation constructed over the observation phase evidences acquisition of observed behaviours, although this phase alone is not sufficient to ensure accurate reproduction and must be made functional through the production phase (Deakin & Proteau, 2000). Results obtained through a second simulation replicate those produced by Bandura & Jeffery (1973), who observed that the individual tested following the retention phase recalled recorded information better than he realized in the production phase. The outcome of a third simulation shows that, when performing the transfer task, agents performed the task all the more effectively when they were required to learn a simple path which facilitated knowledge transfer to an adjacent situation. New explanatory assumptions of the mechanics of learning through observation may be produced through OLEANNet. Thus, observed deterioration between memorization and production is caused by successive approximations which occur in the acquisition phase then in the production phase. Further, depending on the type of learning undergone by agents, use of representation as a production guide induces a more or less stringent constraint in the approximation of actual behaviour. This results, during the transfer task, in the ability to effectively generalize acquired knowledge where such knowledge is not specifically related to the task at hand. In conclusion, connectionist model architecture appears valid for modeling learning through observation as defined by Bandura (1977). However, certain limitations appear during implementation, especially in terms of the observed behaviour's availability and the planning of produced behaviours that future developments are liable to counter.     

Overtensing and the effect of regularity

Regularly inflected forms often behave differently in language production than irregular forms. These differences are often used to argue that irregular forms are listed in the lexicon but regular forms are produced by rule. Using an experimental speech production task with adults, it is shown that overtensing errors, where a tensed verb is used in place of an infinitive, predominantly involve irregular forms, but that the differences may be due to phonological confounds, not to regularity per se. Errors involve vowel‐changing irregular forms more than suffixing inflected forms, with at best a small difference between regular ‐ed and irregular ‐en. Frequency effects on overtensing errors require a model in which the past‐tense and base forms of the verb are in competition and in which activation functions are nonlinear, and rule out models with specialized subnetworks for past‐tense forms. Implications for theories of language production are discussed.