From exemplar to grammar: a probabilistic analogy-based model of language learning

While rules and exemplars are usually viewed as opposites, this paper argues that they form end points of the same distribution. By representing both rules and exemplars as (partial) trees, we can take into account the fluid middle ground between the two extremes. This insight is the starting point for a new theory of language learning that is based on the following idea: If a language learner does not know which phrase-structure trees should be assigned to initial sentences, s/he allows (implicitly) for all possible trees and lets linguistic experience decide which is the "best" tree for each sentence. The best tree is obtained by maximizing "structural analogy" between a sentence and previous sentences, which is formalized by the most probable shortest combination of subtrees from all trees of previous sentences. Corpus-based experiments with this model on the Penn Treebank and the Childes database indicate that it can learn both exemplar-based and rule-based aspects of language, ranging from phrasal verbs to auxiliary fronting. By having learned the syntactic structures of sentences, we have also learned the grammar implicit in these structures, which can in turn be used to produce new sentences. We show that our model mimicks children's language development from item-based constructions to abstract constructions, and that the model can simulate some of the errors made by children in producing complex questions.     

Categorical invariance and structural complexity in human concept learning

An alternative account of human concept learning based on an invariance measure of the categorical stimulus is proposed. The categorical invariance model (CIM) characterizes the degree of structural complexity of a Boolean category as a function of its inherent degree of invariance and its cardinality or size. To do this we introduce a mathematical framework based on the notion of a Boolean differential operator on Boolean categories that generates the degrees of invariance (i.e., logical manifold) of the category in respect to its dimensions. Using this framework, we propose that the structural complexity of a Boolean category is indirectly proportional to its degree of categorical invariance and directly proportional to its cardinality or size. Consequently, complexity and invariance notions are formally unified to account for concept learning difficulty. Beyond developing the above unifying mathematical framework, the CIM is significant in that: (1) it precisely predicts the key learning difficulty ordering of the SHJ [Shepard, R. N., Hovland, C. L., & Jenkins, H. M. (1961). Learning and memorization of classifications. Psychological Monographs: General and Applied, 75(13), 1-42] Boolean category types consisting of three binary dimensions and four positive examples; (2) it is, in general, a good quantitative predictor of the degree of learning difficulty of a large class of categories (in particular, the 41 category types studied by Feldman [Feldman, J. (2000). Minimization of Boolean complexity in human concept learning. Nature, 407, 630-633]); (3) it is, in general, a good quantitative predictor of parity effects for this large class of categories; (4) it does all of the above without free parameters; and (5) it is cognitively plausible (e.g., cognitively tractable).     

Lexical competition in young children's word learning

In two experiments, 1.5-year-olds were taught novel words whose sound patterns were phonologically similar to familiar words (novel neighbors) or were not (novel nonneighbors). Learning was tested using a picture-fixation task. In both experiments, children learned the novel nonneighbors but not the novel neighbors. In addition, exposure to the novel neighbors impaired recognition performance on familiar neighbors. Finally, children did not spontaneously use phonological differences to infer that a novel word referred to a novel object. Thus, lexical competition--inhibitory interaction among words in speech comprehension--can prevent children from using their full phonological sensitivity in judging words as novel. These results suggest that word learning in young children, as in adults, relies not only on the discrimination and identification of phonetic categories, but also on evaluating the likelihood that an utterance conveys a new word.     

Word learning is 'smart': evidence that conceptual information affects preschoolers' extension of novel words

We examined electrophysiological correlates of conscious change detection versus change blindness for equivalent displays. Observers had to detect any changes, across a visual interruption, between a pair of successive displays. Each display comprised grey circles on a background of alternate black and white stripes. Foreground changes arose when light-grey circles turned dark-grey and vice-versa. Physically stronger background changes arose when all black stripes turned white and vice-versa. Despite their physical strength, background changes were undetected unless attention was directed to them, whereas foreground changes were invariably seen. Event-related potentials revealed that the P300 component was suppressed for unseen background changes, as compared with the same changes when seen. This effect arose first over frontal sites, and then spread to parietal sites. These results extend recent fMRI findings that fronto-parietal activation is associated with conscious visual change detection, to reveal the timing of these neural correlates.     

Theory learning as stochastic search in the language of thought

We present an algorithmic model for the development of children's intuitive theories within a hierarchical Bayesian framework, where theories are described as sets of logical laws generated by a probabilistic context-free grammar. We contrast our approach with connectionist and other emergentist approaches to modeling cognitive development. While their subsymbolic representations provide a smooth error surface that supports efficient gradient-based learning, our symbolic representations are better suited to capturing children's intuitive theories but give rise to a harder learning problem, which can only be solved by exploratory search. Our algorithm attempts to discover the theory that best explains a set of observed data by performing stochastic search at two levels of abstraction: an outer loop in the space of theories and an inner loop in the space of explanations or models generated by each theory given a particular dataset. We show that this stochastic search is capable of learning appropriate theories in several everyday domains and discuss its dynamics in the context of empirical studies of children's learning.     

Discovering syntactic deep structure via Bayesian statistics

In the Bayesian framework, a language learner should seek a grammar that explains observed data well and is also a priori probable. This paper proposes such a measure of prior probability. Indeed it develops a full statistical framework for lexicalized syntax. The learner's job is to discover the system of probabilistic transformations (often called lexical redundancy rules) that underlies the patterns of regular and irregular syntactic constructions listed in the lexicon. Specifically, the learner discovers what transformations apply in the language, how often they apply, and in what contexts. It considers simpler systems of transformations to be more probable a priori. Experiments show that the learned transformations are more effective than previous statistical models at predicting the probabilities of lexical entries, especially those for which the learner had no direct evidence.     

Category learning from equivalence constraints

Information for category learning may be provided as positive or negative equivalence constraints (PEC/NEC)—indicating that some exemplars belong to the same or different categories. To investigate categorization strategies, we studied category learning from each type of constraint separately, using a simple rule-based task. We found that participants use PECs differently than NECs, even when these provide the same amount of information. With informative PECs, categorization was rapid, reasonably accurate and uniform across participants. With informative NECs, performance was rapid and highly accurate for only some participants. When given directions, all participants reached high-performance levels with NECs, but the use of PECs remained unchanged. These results suggest that people may use PECs intuitively, but not perfectly. In contrast, using informative NECs enables a potentially more accurate categorization strategy, but a less natural, one which many participants initially fail to implement—even in this simplified setting.

The paradox of the universal triangle: Concepts,language, and prototypes

For over 300 years, the humble triangle has served as the paradigmatic example of the problem of abstraction. How can we have the idea of a general triangle even though every experience with triangles is with specific ones? Classical cognitive science seemed to provide an answer in symbolic representation. With its easily enumerated necessary and sufficient conditions, the triangle would appear to be an ideal candidate for being represented in a symbolic form. I show that it is not. Across a variety of tasks—drawing, speeded recognition, unspeeded visual judgments, and inference—representations of triangles appear to be graded and context dependent. I show that using the category name “triangle” activates a more prototypical representation than using an arguably coextensive cue, “three-sided polygon”. For example, when asked to draw “triangles” people draw more typical triangles than when asked to draw “three-sided polygons”. Altogether, the results support the view that (even formal) concepts have a graded and flexible structure, which takes on a more prototypical and stable form when activated by category labels.     

Idiomatic syntactic constructions and language learning

This article explores the influence of idiomatic syntactic constructions (i.e., constructions whose phrase structure rules violate the rules that underlie the construction of other kinds of sentences in the language) on the acquisition of phrase structure. In Experiment 1, participants were trained on an artificial language generated from hierarchical phrase structure rules. Some participants were given exposure to an idiomatic construction (IC) during training, whereas others were not. Under some circumstances, the presence of an idiomatic construction in the input aided learners in acquiring the phrase structure of the language. Experiment 2 provides a replication of the first experiment and extends the findings by showing that idiomatic constructions that strongly violate the predictive dependencies that define the phrase structure of the language do not aid learners in acquiring the structure of the language. Together, our data suggest that (a) idiomatic constructions aid learners in acquiring the phrase structure of a language by highlighting relevant structural elements in the language, and (b) such constructions are useful cues to learning to the extent that learners can keep their knowledge of the idiomatic construction separate from their knowledge of the rest of the language.     

Rapid learning of syllable classes from a perceptually continuous speech stream

To learn a language, speakers must learn its words and rules from fluent speech; in particular, they must learn dependencies among linguistic classes. We show that when familiarized with a short artificial, subliminally bracketed stream, participants can learn relations about the structure of its words, which specify the classes of syllables occurring in first and last word positions. By studying the effect of familiarization length, we compared the general predictions of associative theories of learning and those of models postulating separate mechanisms for quickly extracting the word structure and for tracking the syllable distribution in the stream. As predicted by the dual-mechanism model, the preference for structurally correct items was negatively correlated with the familiarization length. This result is difficult to explain by purely associative schemes; an extensive set of neural network simulations confirmed this difficulty. Still, we show that powerful statistical computations operating on the stream are available to our participants, as they are sensitive to co-occurrence statistics among non-adjacent syllables. We suggest that different learning mechanisms analyze speech on-line: A rapid mechanism extracting structural information about the stream, and a slower mechanism detecting statistical regularities among the items occurring in it.     

Language evolution by iterated learning with bayesian agents

Languages are transmitted from person to person and generation to generation via a process of iterated learning: people learn a language from other people who once learned that language themselves. We analyze the consequences of iterated learning for learning algorithms based on the principles of Bayesian inference, assuming that learners compute a posterior distribution over languages by combining a prior (representing their inductive biases) with the evidence provided by linguistic data. We show that when learners sample languages from this posterior distribution, iterated learning converges to a distribution over languages that is determined entirely by the prior. Under these conditions, iterated learning is a form of Gibbs sampling, a widely-used Markov chain Monte Carlo algorithm. The consequences of iterated learning are more complicated when learners choose the language with maximum posterior probability, being affected by both the prior of the learners and the amount of information transmitted between generations. We show that in this case, iterated learning corresponds to another statistical inference algorithm, a variant of the expectation-maximization (EM) algorithm. These results clarify the role of iterated learning in explanations of linguistic universals and provide a formal connection between constraints on language acquisition and the languages that come to be spoken, suggesting that information transmitted via iterated learning will ultimately come to mirror the minds of the learners.     

A rational model of the effects of distributional information on feature learning

Most psychological theories treat the features of objects as being fixed and immediately available to observers. However, novel objects have an infinite array of properties that could potentially be encoded as features, raising the question of how people learn which features to use in representing those objects. We focus on the effects of distributional information on feature learning, considering how a rational agent should use statistical information about the properties of objects in identifying features. Inspired by previous behavioral results on human feature learning, we present an ideal observer model based on nonparametric Bayesian statistics. This model balances the idea that objects have potentially infinitely many features with the goal of using a relatively small number of features to represent any finite set of objects. We then explore the predictions of this ideal observer model. In particular, we investigate whether people are sensitive to how parts co-vary over objects they observe. In a series of four behavioral experiments (three using visual stimuli, one using conceptual stimuli), we demonstrate that people infer different features to represent the same four objects depending on the distribution of parts over the objects they observe. Additionally in all four experiments, the features people infer have consequences for how they generalize properties to novel objects. We also show that simple models that use the raw sensory data as inputs and standard dimensionality reduction techniques (principal component analysis and independent component analysis) are insufficient to explain our results.     

The evolution of frequency distributions: Relating regularization to inductive biases through iterated learning

The regularization of linguistic structures by learners has played a key role in arguments for strong innate constraints on language acquisition, and has important implications for language evolution. However, relating the inductive biases of learners to regularization behavior in laboratory tasks can be challenging without a formal model. In this paper we explore how regular linguistic structures can emerge from language evolution by iterated learning, in which one person’s linguistic output is used to generate the linguistic input provided to the next person. We use a model of iterated learning with Bayesian agents to show that this process can result in regularization when learners have the appropriate inductive biases. We then present three experiments demonstrating that simulating the process of language evolution in the laboratory can reveal biases towards regularization that might not otherwise be obvious, allowing weak biases to have strong effects. The results of these experiments suggest that people tend to regularize inconsistent word-meaning mappings, and that even a weak bias towards regularization can allow regular languages to be produced via language evolution by iterated learning.     

Developmental differences in children's context-dependent word learning

In this study, 2.5-, 3-, and 4-year-olds (N = 108) participated in a novel noun generalization task in which background context was manipulated. During the learning phase of each trial, children were presented with exemplars in one or multiple background contexts. At the test, children were asked to generalize to a novel exemplar in either the same or a different context. The 2.5-year-olds’ performance was supported by matching contexts; otherwise, children in this age group demonstrated context dependent generalization. The 3-year-olds’ performance was also supported by matching contexts; however, children in this age group were aided by training in multiple contexts as well. Finally, the 4-year-olds demonstrated high performance in all conditions. The results are discussed in terms of the relationship between word learning and memory processes; both general memory development and memory developments specific to word learning (e.g., retention of linguistic labels) are likely to support word learning and generalization.     

Learning words’ sounds before learning how words sound: 9-Month-olds use distinct objects as cues to categorize speech information

One of the central themes in the study of language acquisition is the gap between the linguistic knowledge that learners demonstrate, and the apparent inadequacy of linguistic input to support induction of this knowledge. One of the first linguistic abilities in the course of development to exemplify this problem is in speech perception: specifically, learning the sound system of one’s native language. Native-language sound systems are defined by meaningful contrasts among words in a language, yet infants learn these sound patterns before any significant numbers of words are acquired. Previous approaches to this learning problem have suggested that infants can learn phonetic categories from statistical analysis of auditory input, without regard to word referents. Experimental evidence presented here suggests instead that young infants can use visual cues present in word-labeling situations to categorize phonetic information. In Experiment 1, 9-month-old English-learning infants failed to discriminate two non-native phonetic categories, establishing baseline performance in a perceptual discrimination task. In Experiment 2, these infants succeeded at discrimination after watching contrasting visual cues (i.e., videos of two novel objects) paired consistently with the two non-native phonetic categories. In Experiment 3, these infants failed at discrimination after watching the same visual cues, but paired inconsistently with the two phonetic categories. At an age before which memory of word labels is demonstrated in the laboratory, 9-month-old infants use contrastive pairings between objects and sounds to influence their phonetic sensitivity. Phonetic learning may have a more functional basis than previous statistical learning mechanisms assume: infants may use cross-modal associations inherent in social contexts to learn native-language phonetic categories.     

Detailed behavioral analysis as a window into cross-situational word learning

Recent research has demonstrated that word learners can determine word-referent mappings by tracking co-occurrences across multiple ambiguous naming events. The current study addresses the mechanisms underlying this capacity to learn words cross-situationally. This replication and extension of Yu and Smith (2007) investigates the factors influencing both successful cross-situational word learning and mis-mappings. Item analysis and error patterns revealed that the co-occurrence structure of the learning environment as well as the context of the testing environment jointly affected learning across observations. Learners also adopted an exclusion strategy, which contributed conjointly with statistical tracking to performance. Implications for our understanding of the processes underlying cross-situational word learning are discussed.     

A probabilistic model of cross-categorization

Most natural domains can be represented in multiple ways: we can categorize foods in terms of their nutritional content or social role, animals in terms of their taxonomic groupings or their ecological niches, and musical instruments in terms of their taxonomic categories or social uses. Previous approaches to modeling human categorization have largely ignored the problem of cross-categorization, focusing on learning just a single system of categories that explains all of the features. Cross-categorization presents a difficult problem: how can we infer categories without first knowing which features the categories are meant to explain? We present a novel model that suggests that human cross-categorization is a result of joint inference about multiple systems of categories and the features that they explain. We also formalize two commonly proposed alternative explanations for cross-categorization behavior: a features-first and an objects-first approach. The features-first approach suggests that cross-categorization is a consequence of attentional processes, where features are selected by an attentional mechanism first and categories are derived second. The objects-first approach suggests that cross-categorization is a consequence of repeated, sequential attempts to explain features, where categories are derived first, then features that are poorly explained are recategorized. We present two sets of simulations and experiments testing the models’ predictions about human categorization. We find that an approach based on joint inference provides the best fit to human categorization behavior, and we suggest that a full account of human category learning will need to incorporate something akin to these capabilities.     

Procedural learning: A developmental study of motor sequence learning and probabilistic classification learning in school-aged children

In this study, we investigated motor and cognitive procedural learning in typically developing children aged 8–12 years with a serial reaction time (SRT) task and a probabilistic classification learning (PCL) task. The aims were to replicate and extend the results of previous SRT studies, to investigate PCL in school-aged children, to explore the contribution of declarative knowledge to SRT and PCL performance, to explore the strategies used by children in the PCL task via a mathematical model, and to see whether performances obtained in motor and cognitive tasks correlated. The results showed similar learning effects in the three age groups in the SRT and in the first half of the PCL tasks. Participants did not develop explicit knowledge in the SRT task whereas declarative knowledge of the cue–outcome associations correlated with the performances in the second half of the PCL task, suggesting a participation of explicit knowledge after some time of exposure in PCL. An increasing proportion of the optimal strategy use with increasing age was observed in the PCL task. Finally, no correlation appeared between cognitive and motor performance. In conclusion, we extended the hypothesis of age invariance from motor to cognitive procedural learning, which had not been done previously. The ability to adopt more efficient learning strategies with age may rely on the maturation of the fronto-striatal loops. The lack of correlation between performance in the SRT task and the first part of the PCL task suggests dissociable developmental trajectories within the procedural memory system.