Competitive Learning: From Interactive Activation to Adaptive Resonance

Functional and mechanistic comparisons are made between several network models of cognitive processing: competitive learning, interactive activation, adaptive resonance, and back propagation. The starting point of this comparison is the article of Rumelhart and Zipser (1985) on feature discovery through competitive learning. All the models which Rumelhart and Zipser (1985) have described were shown in Grossberg (1976b) to exhibit a type of learning which is temporally unstable. Competitive learning mechanisms can be stabilized in response to an arbitrary input environment by being supplemented with mechanisms for learning top-down expectancies, or templates; for matching bottom-up input patterns with the top-down expectancies; and for releasing orienting reactions in a mismatch situation, thereby updating short-term memory and searching for another internal representation. Network architectures which embody all of these mechanisms were called adaptive resonance models by Grossberg (1976c). Self-stabilizing learning models are candidates for use in real-world applications where unpredictable changes can occur in complex input environments. Competitive learning postulates are inconsistent with the postulates of the interactive activation model of McClelland and Rumelhart (1981), and suggest different levels of processing and interaction rules for the analysis of word recognition. Adaptive resonance models use these alternative levels and interaction rules. The self-organizing learning of an adaptive resonance model is compared and contrasted with the teacher-directed learning of a back propagation model. A number of criteria for evaluating real-time network models of cognitive processing are described and applied.     

No frills: Simple regularities in language can go a long way in the development of word knowledge

Recent years have seen a flourishing of Natural Language Processing models that can mimic many aspects of human language fluency. These models harness a simple, decades-old idea: It is possible to learn a lot about word meanings just from exposure to language, because words similar in meaning are used in language in similar ways. The successes of these models raise the intriguing possibility that exposure to word use in language also shapes the word knowledge that children amass during development. However, this possibility is strongly challenged by the fact that models use language input and learning mechanisms that may be unavailable to children. Across three studies, we found that unrealistically complex input and learning mechanisms are unnecessary. Instead, simple regularities of word use in children's language input that they have the capacity to learn can foster knowledge about word meanings. Thus, exposure to language may play a simple but powerful role in children's growing word knowledge. A video abstract of this article can be viewed at https://youtu.be/dT83dmMffnM .

Research Highlights

• Natural Language Processing (NLP) models can learn that words are similar in meaning from higher-order statistical regularities of word use.
• Unlike NLP models, infants and children may primarily learn only simple co-occurrences between words.
• We show that infants' and children's language input is rich in simple co-occurrence that can support learning similarities in meaning between words.
• We find that simple co-occurrences can explain infants' and children's knowledge that words are similar in meaning.

     

Speech segmentation is facilitated by visual cues

Evidence from infant studies indicates that language learning can be facilitated by multimodal cues. We extended this observation to adult language learning by studying the effects of simultaneous visual cues (nonassociated object images) on speech segmentation performance. Our results indicate that segmentation of new words from a continuous speech stream is facilitated by simultaneous visual input that it is presented at or near syllables that exhibit the low transitional probability indicative of word boundaries. This indicates that temporal audio-visual contiguity helps in directing attention to word boundaries at the earliest stages of language learning. Off-boundary or arrhythmic picture sequences did not affect segmentation performance, suggesting that the language learning system can effectively disregard noninformative visual information. Detection of temporal contiguity between multimodal stimuli may be useful in both infants and second-language learners not only for facilitating speech segmentation, but also for detecting word–object relationships in natural environments.     

Look who's talking: speech style and social context in language input to infants are linked to concurrent and future speech development

Language input is necessary for language learning, yet little is known about whether, in natural environments, the speech style and social context of language input to children impacts language development. In the present study we investigated the relationship between language input and language development, examining both the style of parental speech, comparing ‘parentese’ speech to standard speech, and the social context in which speech is directed to children, comparing one‐on‐one (1:1) to group social interactions. Importantly, the language input variables were assessed at home using digital first‐person perspective recordings of the infants' auditory environment as they went about their daily lives (N =26, 11‐ and 14‐months‐old). We measured language development using (a) concurrent speech utterances, and (b) word production at 24 months. Parentese speech in 1:1 contexts is positively correlated with both concurrent speech and later word production. Mediation analyses further show that the effect of parentese speech‐1:1 on infants' later language is mediated by concurrent speech. Our results suggest that both the social context and the style of speech in language addressed to children are strongly linked to a child's future language development.     

A Connectionist Approach to Word Reading and Acquired Dyslexia: Extension to Sequential Processing

A connectionist approach to word reading, based on the principles of distributed representation, graded learning of statistical structure, and interactivity in processing, has led to the development of explicit computational models which account for a wide range of data on normal skilled reading and on patterns of reading impairment due to brain damage. There have, however, been recent empirical challenges to these models, and the approach in general, relating to the influence of orthographic length on the naming latencies of both normal and dyslexic readers. The current work presents a simulation which generates sequential phonological output in response to written input, and which can refixate the input when encountering difficulty. The normal model reads both words and nonwords accurately, and exhibits an effect of orthographic length and a frequency-by-consistency interaction in its naming latencies. When subject to peripheral damage, the model exhibits an increased length effect that interacts with word frequency, characteristic of letter-by-letter reading in pure alexia. Although the model is far from a fully adequate account of all the relevant phenomena, it suggests how connectionist models may be extended to provide deeper insight into sequential processes in reading.     

Effects of visual information on adults' and infants' auditory statistical learning

Infant and adult learners are able to identify word boundaries in fluent speech using statistical information. Similarly, learners are able to use statistical information to identify word-object associations. Successful language learning requires both feats. In this series of experiments, we presented adults and infants with audio-visual input from which it was possible to identify both word boundaries and word-object relations. Adult learners were able to identify both kinds of statistical relations from the same input. Moreover, their learning was actually facilitated by the presence of two simultaneously present relations. Eight-month-old infants, however, do not appear to benefit from the presence of regular relations between words and objects. Adults, like 8-month-olds, did not benefit from regular audio-visual correspondences when they were tested with tones, rather than linguistic input. These differences in learning outcomes across age and input suggest that both developmental and stimulus-based constraints affect statistical learning.     

How to say "no" to a nonword: a leaky competing accumulator model of lexical decision

We describe a leaky competing accumulator (LCA) model of the lexical decision task that can be used as a response/decision module for any computational model of word recognition. The LCA model uses evidence for a word, operationalized as some measure of lexical activity, as input to the YES decision node. Input to the NO decision node is simply a constant value minus evidence for a word. In this way, evidence for a nonword is a function of time from stimulus onset (as in standard deadline models) modulated by lexical activity via the competitive dynamics of the LCA. We propose a simple mechanism for determining the value of this constant online during the first trials of a lexical decision experiment, such that the model can rapidly optimize speed and accuracy in discriminating words from nonwords. Further optimization is achieved via trial-by-trial adjustments in response criteria as a function of task demands and list context. We show that the LCA model can simulate mean response times and response distributions for correct and incorrect YES and NO decisions for a number of benchmark experiments that have been shown to be fatal for deadline models of lexical decision. Finally, using lexical activity calculated by a computational model of word recognition as input to the LCA decision module, we provide the first item-level simulation of both word and nonword responses in a large-scale database.     

A distributed, developmental model of word recognition and naming

A parallel distributed processing model of visual word recognition and pronunciation is described. The model consists of sets of orthographic and phonological units and an interlevel of hidden units. Weights on connections between units were modified during a training phase using the back-propagation learning algorithm. The model simulates many aspects of human performance, including (a) differences between words in terms of processing difficulty, (b) pronunciation of novel items, (c) differences between readers in terms of word recognition skill, (d) transitions from beginning to skilled reading, and (e) differences in performance on lexical decision and naming tasks. The model's behavior early in the learning phase corresponds to that of children acquiring word recognition skills. Training with a smaller number of hidden units produces output characteristic of many dyslexic readers. Naming is simulated without pronunciation rules, and lexical decisions are simulated without accessing word-level representations. The performance of the model is largely determined by three factors: the nature of the input, a significant fragment of written English; the learning rule, which encodes the implicit structure of the orthography in the weights on connections; and the architecture of the system, which influences the scope of what can be learned.     

The bilingual-dichotic method for learning a foreign-language vocabulary

The bilingual-dichotic method organizes native and foreign words in paired-associate learning for optimum compatibility with study behaviour and asymmetric cerebral hemispheric specialization. Slow-paced auditory presentations alternate dichotic (simultaneous, native word at left ear and foreign word at right ear) and diotic (foreign word at both ears) input while the student views the word list in parallel to the dichotic format (native word-foreign word). More college students attained the criterion of perfect cuedrecall and in significantly less study time using the bilingual-dichotic method to learn English-Russian word pairs compared with those using the conventional (diotic Russian) or a reversed-ear dichotic control method. The new method yielded superior overall performance with significantly more accurate recall of the Russian words, and with shorter latencies of response than that obtained with the conventional method. Analysis suggests the bilingual-dichotic method combines attentional strategies with the right-ear advantage for dichotic verbal material to enhance rehearsal processes of learning. Practical applications of this new method for learning foreign vocabulary may be found in the early stages of foreign-language learning in coordination with other exercises.     

An associative account of the development of word learning

Word learning is a notoriously difficult induction problem because meaning is underdetermined by positive examples. How do children solve this problem? Some have argued that word learning is achieved by means of inference: young word learners rely on a number of assumptions that reduce the overall hypothesis space by favoring some meanings over others. However, these approaches have difficulty explaining how words are learned from conversations or text, without pointing or explicit instruction. In this research, we propose an associative mechanism that can account for such learning. In a series of experiments, 4-year-olds and adults were presented with sets of words that included a single nonsense word (e.g. dax). Some lists were taxonomic (i.,e., all items were members of a given category), some were associative (i.e., all items were associates of a given category, but not members), and some were mixed. Participants were asked to indicate whether the nonsense word was an animal or an artifact. Adults exhibited evidence of learning when lists consisted of either associatively or taxonomically related items. In contrast, children exhibited evidence of word learning only when lists consisted of associatively related items. These results present challenges to several extant models of word learning, and a new model based on the distinction between syntagmatic and paradigmatic associations is proposed.     

儿童早期词汇获得的跨语言/文化研究

婴儿词汇学习是国际语言发展研究的前沿领域, 但大多数研究都是以英语婴儿为研究对象。目前国内关于汉语婴儿早期语言发展研究处于起步阶段。由于汉语在词法、句法等方面具有明显的特殊性, 另外, 成人“言语输入”的语用习惯以及非言语线索都会影响婴儿早期词汇学习, 使得不同语言文化下的婴儿呈现不同的单词学习模式。本项目力图从跨语言视角考察汉语和英语婴儿的早期单词学习(包括单词的理解和产生)以及促进婴儿语言发展的成人言语和非言语输入特征。研究将综合采用实验室实验、半结构化实验室观察和量表测量等研究方法, 利用新的研究技术(如习惯化和IPLP), 从多个角度探索婴儿词汇发展, 以及不同语言文化和个体环境对婴儿早期词汇学习的影响, 揭示词汇获得的跨语言一致性和特异性。研究结果有望对儿童的语言学习提供启示。     

Linguistic versus attentional influences on nonlinguistic categorization in 15-month-old infants

Why linguistic input facilitates nonlinguistic categorization is frequently explained in terms of children's attention to uniquely linguistic forms such as words. However, whether this facilitation is rooted in the children's attention to word forms very early in lexical learning has not been examined directly. A previous experiment (Roberts & Cuff, 1989) provided a set of conditions under which 15-month-olds did not successfully categorize in the absence of linguistic input. The two experiments reported here replicate Roberts and Cuff (1989) exactly, with the exception that either language (Experiment 1) or instrumental music (Experiment 2) was provided as accompanying input. Infants in both experiments successfully categorized and significantly increased attention during habituation. Although directly documenting the influence of language on categorization prior to the “vocabulary explosion,” this influence does not appear attributable to the presence of word forms. Instead, factors common to language and music (e.g., attention-getting properties or factors influencing attention) may facilitate nonlinguistic categorization at the beginnings of word learning.     

Dynamic self-organization and early lexical development in children

In this study we present a self-organizing connectionist model of early lexical development. We call this model DevLex-II, based on the earlier DevLex model. DevLex-II can simulate a variety of empirical patterns in children's acquisition of words. These include a clear vocabulary spurt, effects of word frequency and length on age of acquisition, and individual differences as a function of phonological short-term memory and associative capacity. Further results from lesioned models indicate developmental plasticity in the network's recovery from damage, in a non-monotonic fashion. We attribute the network's abilities in accounting for lexical development to interactive dynamics in the learning process. In particular, variations displayed by the model in the rate and size of early vocabulary development are modulated by (a) input characteristics, such as word frequency and word length, (b) consolidation of lexical-semantic representation, meaning-form association, and phonological short-term memory, and (c) delayed processes due to interactions among timing, severity, and recoverability of lesion. Together, DevLex and DevLex-II provide an accurate computational account of early lexical development.     

A Bayesian Model of Biases in Artificial Language Learning: The Case of a Word‐Order Universal

In this article, we develop a hierarchical Bayesian model of learning in a general type of artificial language‐learning experiment in which learners are exposed to a mixture of grammars representing the variation present in real learners’ input, particularly at times of language change. The modeling goal is to formalize and quantify hypothesized learning biases. The test case is an experiment ( Culbertson, Smolensky, & Legendre, 2012 ) targeting the learning of word‐order patterns in the nominal domain. The model identifies internal biases of the experimental participants, providing evidence that learners impose (possibly arbitrary) properties on the grammars they learn, potentially resulting in the cross‐linguistic regularities known as typological universals. Learners exposed to mixtures of artificial grammars tended to shift those mixtures in certain ways rather than others; the model reveals how learners’ inferences are systematically affected by specific prior biases. These biases are in line with a typological generalization—Greenberg's Universal 18—which bans a particular word‐order pattern relating nouns, adjectives, and numerals.     

Language input and acquisition in a Mayan village: how important is directed speech?

Theories of language acquisition have highlighted the importance of adult speakers as active participants in children’s language learning. However, in many communities children are reported to be directly engaged by their caregivers only rarely ( Lieven, 1994 ). This observation raises the possibility that these children learn language from observing, rather than participating in, communicative exchanges. In this paper, we quantify naturally occurring language input in one community where directed interaction with children has been reported to be rare (Yucatec Mayan). We compare this input to the input heard by children growing up in large families in the United States, and we consider how directed and overheard input relate to Mayan children’s later vocabulary. In Study 1, we demonstrate that 1‐year‐old Mayan children do indeed hear a smaller proportion of total input in directed speech than children from the US. In Study 2, we show that for Mayan (but not US) children, there are great increases in the proportion of directed input that children receive between 13 and 35 months. In Study 3, we explore the validity of using videotaped data in a Mayan village. In Study 4, we demonstrate that word types directed to Mayan children from adults at 24 months (but not word types overheard by children or word types directed from other children) predict later vocabulary. These findings suggest that adult talk directed to children is important for early word learning, even in communities where much of children’s early language input comes from overheard speech. A video abstract of this article can be viewed at http://www.youtube.com/watch?v=ZgA4CxmBTNk&feature=youtu.be     

Using Speakers' Referential Intentions to Model Early Cross-Situational Word Learning

ABSTRACT— Word learning is a "chicken and egg" problem. If a child could understand speakers' utterances, it would be easy to learn the meanings of individual words, and once a child knows what many words mean, it is easy to infer speakers' intended meanings. To the beginning learner, however, both individual word meanings and speakers' intentions are unknown. We describe a computational model of word learning that solves these two inference problems in parallel, rather than relying exclusively on either the inferred meanings of utterances or cross-situational word-meaning associations. We tested our model using annotated corpus data and found that it inferred pairings between words and object concepts with higher precision than comparison models. Moreover, as the result of making probabilistic inferences about speakers' intentions, our model explains a variety of behavioral phenomena described in the word-learning literature. These phenomena include mutual exclusivity, one-trial learning, cross-situational learning, the role of words in object individuation, and the use of inferred intentions to disambiguate reference.     

Effects of Word Processing and Short-term Memory Deficits on Verbal Learning: Evidence from Aphasia

We report two experiments that investigate the abilities of aphasic subjects with lexical and short-term memory impairments to learn supraspan lists of 10 words. We examined effects of stimulus factors (characteristics of words and lists to be learned) and subject factors (verbal and nonverbal STM span, nature of language impairment). Learning ability was influenced by the imageability and frequency of the words to be learned (Experiment 1) and by the linguistic relationship among words in a list (Experiment 2). Additionally, learning ability was affected by the nature of a subject's word processing deficit (whether it involved semantic and/or phonological processes). Phonological ability was positively associated with learning in both experiments, and semantic ability was associated with learning when words in a list were high in both frequency and imageability (Experiment 1) or were contextually related to one another (Experiment 2). Finally, verbal STM span was positively related to learning performance, but the effect was more pronounced for word span than digit span. These relationships among word processing ability, verbal STM span, and learning ability are discussed with reference to learning in aphasia and to models of learning more generally.     

Paired-associate learning and priming effects in amnesia: a neuropsychological study

Despite severe deficits of recall and recognition, amnesic patients can exhibit normal priming effects. Amnesic patients have also been reported to perform well on tests of paired-associate learning that involve related word pairs (e.g., table-chair). The present study investigated the role of priming effects in paired-associate learning. Experiment 1 illustrated the distinction between the memory impairment of amnesic patients and their intact priming ability. Amnesic patients were markedly deficient in learning unrelated word pairs, despite exhibiting normal priming as measured by a word-completion test involving the same words. In Experiment 2A, amnesic patients showed good paired-associate learning for related word pairs, though control subjects still performed significantly better. In addition, the good performance by amnesic patients was short-lived, and performance fell to baseline after a 2-hr delay. Control subjects performed well above baseline at all delay conditions. Experiment 2B showed that the forgetting of related word pairs by amnesic patients followed the same time course as the decay of word priming. Experiment 3 showed that amnesic patients were as good as control subjects at learning related word pairs when incidental learning and test procedures were used (a word-association test). The advantage of control subjects over amnesic patients in Experiments 2A and 2B could therefore be attributed to the explicit learning instructions that are standard in paired-associate tests. Finally, Experiment 4 showed that amnesic patients exhibited normal priming when they were asked to "free associate" to words (e.g., child) that were semantically related to previously presented words (e.g., baby). The results indicate that both priming effects and paired-associate learning of related word pairs depend on activation, a process that is preserved in amnesia. Activation can account for the findings of good performance by amnesic patients on tests of word priming (Experiments 1 and 2B), related paired associates (Experiments 2A and 2B), and word association (Experiments 3 and 4). Activation is a transient phenomenon presumed to operate on and facilitate access to preexisting representations. Control subjects can establish new associations and can strengthen preexisting associations by engaging processes that are impaired in amnesia. As a result, when explicit learning instructions are used to test paired-associate learning of related word pairs, control subjects can learn better and can remember longer than can amnesic patients (Experiments 2A and 2B).(ABSTRACT TRUNCATED AT 400 WORDS)