Grounding co-occurrence: Identifying features in a lexical co-occurrence model of semantic memory

Lexical co-occurrence models of semantic memory represent word meaning by vectors in a high-dimensional space. These vectors are derived from word usage, as found in a large corpus of written text. Typically, these models are fully automated, an advantage over models that represent semantics that are based on human judgments (e.g., feature-based models). A common criticism of co-occurrence models is that the representations are not grounded: Concepts exist only relative to each other in the space produced by the model. It has been claimed that feature-based models offer an advantage in this regard. In this article, we take a step toward grounding a cooccurrence model. A feed-forward neural network is trained using back propagation to provide a mapping from co-occurrence vectors to feature norms collected from subjects. We show that this network is able to retrieve the features of a concept from its co-occurrence vector with high accuracy and is able to generalize this ability to produce an appropriate list of features from the co-occurrence vector of a novel concept.     

WINDSOR: Windsor improved norms of distance and similarity of representations of semantics

Lexical co-occurrence models of semantic memory form representations of the meaning of a word on the basis of the number of times that pairs of words occur near one another in a large body of text. These models offer a distinct advantage over models that require the collection of a large number of judgments from human subjects, since the construction of the representations can be completely automated. Unfortunately, word frequency, a well-known predictor of reaction time in several cognitive tasks, has a strong effect on the co-occurrence counts in a corpus. Two words with high frequency are more likely to occur together purely by chance than are two words that occur very infrequently. In this article, we examine a modification of a successful method for constructing semantic representations from lexical co-occurrence. We show that our new method eliminates the influence of frequency, while still capturing the semantic characteristics of words.     

Reconstructing maps from text

Previous research has demonstrated that Distributional Semantic Models (DSMs) are capable of reconstructing maps from news corpora (Louwerse & Zwaan, 2009) and novels (Louwerse & Benesh, 2012). The capacity for reproducing maps is surprising since DSMs notoriously lack perceptual grounding. In this paper we investigate the statistical sources required in language to infer maps, and the resulting constraints placed on mechanisms of semantic representation. Study 1 brings word co-occurrence under experimental control to demonstrate that standard DSMs cannot reproduce maps when word co-occurrence is uniform. Specifically, standard DSMs require that direct co-occurrences between city names in a corpus mirror the proximity between the city locations in the map in order to successfully reconstruct the spatial map. Study 2 presents an instance-based DSM that is capable of reconstructing maps independent of the frequency of co-occurrence of city names.     

How lexical is the lexicon? Evidence for integrated auditory memory representations

Previous research has shown that lexical representations must include not only linguistic information (what word was said), but also indexical information (how it was said, and by whom). The present work demonstrates that even this expansion is not sufficient. Seemingly irrelevant information, such as an unattended background sound, is retained in memory and can facilitate subsequent speech perception. We presented participants with spoken words paired with environmental sounds (e.g., a phone ringing), and had them make an “animate/inanimate” decision for each word. Later performance identifying filtered versions of the words was impaired to a similar degree if the voice changed or if the environmental sound changed. Moreover, when quite dissimilar words were used at exposure and test, we observed the same result when we reversed the roles of the words and the environmental sounds. The experiments also demonstrated limits to these effects, with no benefit from repetition. Theoretically, our results support two alternative possibilities: (1) Lexical representations are memory representations, and are not walled off from those for other sounds. Indexical effects reflect simply one type of co-occurrence that is incorporated into such representations. (2) The existing literature on indexical effects does not actually bear on lexical representations – voice changes, like environmental sounds heard with a word, produce implicit memory effects that are not tied to the lexicon. We discuss the evidence and implications of these two theoretical alternatives.     

Embodied attention and word learning by toddlers

Many theories of early word learning begin with the uncertainty inherent to learning a word from its co-occurrence with a visual scene. However, the relevant visual scene for infant word learning is neither from the adult theorist's view nor the mature partner's view, but is rather from the learner's personal view. Here we show that when 18-month old infants interacted with objects in play with their parents, they created moments in which a single object was visually dominant. If parents named the object during these moments of bottom-up selectivity, later forced-choice tests showed that infants learned the name, but did not when naming occurred during a less visually selective moment. The momentary visual input for parents and toddlers was captured via head cameras placed low on each participant's forehead as parents played with and named objects for their infant. Frame-by-frame analyses of the head camera images at and around naming moments were conducted to determine the visual properties at input that were associated with learning. The analyses indicated that learning occurred when bottom-up visual information was clean and uncluttered. The sensory-motor behaviors of infants and parents were also analyzed to determine how their actions on the objects may have created these optimal visual moments for learning. The results are discussed with respect to early word learning, embodied attention, and the social role of parents in early word learning.     

Word chunk frequencies affect the processing of pronominal object-relative clauses

We present experimental support for the view that fine-grained statistical information may play a crucial role in the processing of centre-embedded linguistic structure. Using both offline and online methods, we show that the processing of pronominal object-relative clauses is influenced by the frequency of co-occurrence of the word combinations (chunks) forming the clause. We use materials that are controlled for capacity-based factors that have been previously shown to influence comprehension of relative clauses. The results suggest that, other factors being equal, the frequency of the word chunk forming the clause affects processing difficulty. Analyses of the data indicate that the results cannot be explained by differential access to individual lexical items. Following recent constructivist approaches, we argue that frequency of co-occurrence influences the chunking mechanism by which multiword sequences may become fused into processing units that are easier to access.     

Word chunk frequencies affect the processing of pronominal object-relative clauses

We present experimental support for the view that fine-grained statistical information may play a crucial role in the processing of centre-embedded linguistic structure. Using both offline and online methods, we show that the processing of pronominal object-relative clauses is influenced by the frequency of co-occurrence of the word combinations (chunks) forming the clause. We use materials that are controlled for capacity-based factors that have been previously shown to influence comprehension of relative clauses. The results suggest that, other factors being equal, the frequency of the word chunk forming the clause affects processing difficulty. Analyses of the data indicate that the results cannot be explained by differential access to individual lexical items. Following recent constructivist approaches, we argue that frequency of co-occurrence influences the chunking mechanism by which multiword sequences may become fused into processing units that are easier to access.     

Integrating constraints for learning word-referent mappings

Learning word-referent mappings is complex because the word and its referent tend to co-occur with multiple other words and potential referents. Such complexity has led to proposals for a host of constraints on learning, though how these constraints may interact has not yet been investigated in detail. In this paper, we investigated interactions between word co-occurrence constraints and cross-situational statistics in word learning. Analyses of child-directed speech revealed that when both object-referring and non-referring words occurred in the utterance, referring words were more likely to be preceded by a determiner than when the utterance contained only referring words. In a word learning study containing both referring and non-referring words, learning was facilitated when non-referring words contributed grammatical constraints analogous to determiners. The complexity of multi-word utterances provides an opportunity for co-occurrence constraints to contribute to word-referent mapping, and the learning mechanism is able to integrate these multiple sources of information.     

Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD

In a previous article, we presented a systematic computational study of the extraction of semantic representations from the word-word co-occurrence statistics of large text corpora. The conclusion was that semantic vectors of pointwise mutual information values from very small co-occurrence windows, together with a cosine distance measure, consistently resulted in the best representations across a range of psychologically relevant semantic tasks. This article extends that study by investigating the use of three further factors--namely, the application of stop-lists, word stemming, and dimensionality reduction using singular value decomposition (SVD)--that have been used to provide improved performance elsewhere. It also introduces an additional semantic task and explores the advantages of using a much larger corpus. This leads to the discovery and analysis of improved SVD-based methods for generating semantic representations (that provide new state-of-the-art performance on a standard TOEFL task) and the identification and discussion of problems and misleading results that can arise without a full systematic study.     

Semantic boost on episodic associations: an empirically-based computational model

Words become associated following repeated co-occurrence episodes. This process might be further determined by the semantic characteristics of the words. The present study focused on how semantic and episodic factors interact in incidental formation of word associations. First, we found that human participants associate semantically related words more easily than unrelated words; this advantage increased linearly with repeated co-occurrence. Second, we developed a computational model, SEMANT, suggesting a possible mechanism for this semantic-episodic interaction. In SEMANT, episodic associations are implemented through lateral connections between nodes in a pre-existent self-organized map of word semantics. These connections are strengthened at each instance of concomitant activation, proportionally with the amount of the overlapping activity waves of activated nodes. In computer simulations SEMANT replicated the dynamics of associative learning in humans and led to testable predictions concerning normal associative learning as well as impaired learning in a diffuse semantic system like that characteristic of schizophrenia.     

Identifying influential segments from word co-occurrence networks using AHP

Identifying important segments in textual data seems to be an important area of research for various applications including topic modelling, trend detection, summarization and event detection. In existing research work, different metrics have been studied to analyse the word co-occurrence network. This research work contributes towards non-semantic and an unsupervised topic identification using the word co-occurrence networks. In this research work, keyphrase have been identified by preserving the lexical sequence using a directed and weighted word co-occurrence network. Further AHP (Analytic Hierarchy Process) model based upon four significant attributes of the word co-occurrence networks have been proposed to rank the keyphrases. Most frequently occurring segment is identified as an influential segment. Experimental results proved high effectiveness of the proposed approach. Results for the First Story Detection, 72 Twitter TDT, synthesized Rio Olympics dataset have been discussed to demonstrate its potential in precisely discovering influential segments.     

Gavagai Is as Gavagai Does: Learning Nouns and Verbs From Cross‐Situational Statistics

Learning to map words onto their referents is difficult, because there are multiple possibilities for forming these mappings. Cross‐situational learning studies have shown that word‐object mappings can be learned across multiple situations, as can verbs when presented in a syntactic context. However, these previous studies have presented either nouns or verbs in ambiguous contexts and thus bypass much of the complexity of multiple grammatical categories in speech. We show that noun word learning in adults is robust when objects are moving, and that verbs can also be learned from similar scenes without additional syntactic information. Furthermore, we show that both nouns and verbs can be acquired simultaneously, thus resolving category‐level as well as individual word‐level ambiguity. However, nouns were learned more quickly than verbs, and we discuss this in light of previous studies investigating the noun advantage in word learning.     

Corpus-based estimates of word association predict biases in judgment of word co-occurrence likelihood

This paper draws a connection between statistical word association measures used in linguistics and confirmation measures from epistemology. Having theoretically established the connection, we replicate, in the new context of the judgments of word co-occurrence, an intriguing finding from the psychology of reasoning, namely that confirmation values affect intuitions about likelihood. We show that the effect, despite being based in this case on very subtle statistical insights about thousands of words, is stable across three different experimental settings. Our theoretical and empirical results suggest that factors affecting traditional reasoning tasks are also at play when linguistic knowledge is probed, and they provide further evidence for the importance of confirmation in a new domain.     

Word frequency effects in high-dimensional co-occurrence models: A new approach

The HAL (hyperspace analog to language) model of lexical semantics uses global word co-occurrence from a large corpus of text to calculate the distance between words in co-occurrence space. We have implemented a system called HiDEx (High Dimensional Explorer) that extends HAL in two ways: It removes unwanted influence of orthographic frequency from the measures of distance, and it finds the number of words within a certain distance of the word of interest (NCount, the number of neighbors). These two changes to the HAL model produce     

Exclusion constraints facilitate statistical word learning

The roles of linguistic, cognitive, and social-pragmatic processes in word learning are well established. If statistical mechanisms also contribute to word learning, they must interact with these processes; however, there exists little evidence for such mechanistic synergy. Adults use co-occurrence statistics to encode speech-object pairings with detailed sensitivity in stochastic learning environments (Vouloumanos, 2008). Here, we replicate this statistical work with nonspeech sounds and compare the results with the previous speech studies to examine whether exclusion constraints contribute equally to the statistical learning of speech-object and nonspeech-object associations. In environments in which performance could benefit from exclusion, we find a learning advantage for speech over nonspeech, revealing an interaction between statistical and exclusion processes in associative word learning.     

Prime and prejudice: co-occurrence in the culture as a source of automatic stereotype priming

It has been argued that stereotype priming (response times are faster for stereotypical word pairs, such as black-poor, than for non-stereotypical word pairs, such as black-balmy) is partially a function of biases in the belief system inherent in the culture. In three priming experiments, we provide direct evidence for this position, showing that stereotype priming effects associated with race, gender, and age can be very well explained through objectively measured associative co-occurrence of prime and target in the culture: (a) once objective associative strength between word pairs is taken into account, stereotype priming effects disappear; (b) the relationship between response time and associative strength is identical for social primes and non-social primes. The correlation between associative-value-controlled stereotype priming and self-report measures of racism, sexism, and ageism is near zero. The racist/sexist/ageist in all of us appears to be (at least partially) a reflection of the surrounding culture.     

No frills: Simple regularities in language can go a long way in the development of word knowledge

Recent years have seen a flourishing of Natural Language Processing models that can mimic many aspects of human language fluency. These models harness a simple, decades-old idea: It is possible to learn a lot about word meanings just from exposure to language, because words similar in meaning are used in language in similar ways. The successes of these models raise the intriguing possibility that exposure to word use in language also shapes the word knowledge that children amass during development. However, this possibility is strongly challenged by the fact that models use language input and learning mechanisms that may be unavailable to children. Across three studies, we found that unrealistically complex input and learning mechanisms are unnecessary. Instead, simple regularities of word use in children's language input that they have the capacity to learn can foster knowledge about word meanings. Thus, exposure to language may play a simple but powerful role in children's growing word knowledge. A video abstract of this article can be viewed at https://youtu.be/dT83dmMffnM .

Research Highlights

• Natural Language Processing (NLP) models can learn that words are similar in meaning from higher-order statistical regularities of word use.
• Unlike NLP models, infants and children may primarily learn only simple co-occurrences between words.
• We show that infants' and children's language input is rich in simple co-occurrence that can support learning similarities in meaning between words.
• We find that simple co-occurrences can explain infants' and children's knowledge that words are similar in meaning.

     

Sound specificity effects in spoken word recognition: The effect of integrality between words and sounds

Recent evidence has shown that nonlinguistic sounds co-occurring with spoken words may be retained in memory and affect later retrieval of the words. This sound-specificity effect shares many characteristics with the classic voice-specificity effect. In this study, we argue that the sound-specificity effect is conditional upon the context in which the word and sound coexist. Specifically, we argue that, besides co-occurrence, integrality between words and sounds is a crucial factor in the emergence of the effect. In two recognition-memory experiments, we compared the emergence of voice and sound specificity effects. In Experiment 1 , we examined two conditions where integrality is high. Namely, the classic voice-specificity effect (Exp. 1a) was compared with a condition in which the intensity envelope of a background sound was modulated along the intensity envelope of the accompanying spoken word (Exp. 1b). Results revealed a robust voice-specificity effect and, critically, a comparable sound-specificity effect: A change in the paired sound from exposure to test led to a decrease in word-recognition performance. In the second experiment, we sought to disentangle the contribution of integrality from a mere co-occurrence context effect by removing the intensity modulation. The absence of integrality led to the disappearance of the sound-specificity effect. Taken together, the results suggest that the assimilation of background sounds into memory cannot be reduced to a simple context effect. Rather, it is conditioned by the extent to which words and sounds are perceived as integral as opposed to distinct auditory objects.     

Lexical Association and False Memory for Words in Two Cultures

This study examined the relationship between language experience and false memory produced by the DRM paradigm. The word lists used in Stadler, et al. (Memory & Cognition, 27, 494–500, 1999) were first translated into Chinese. False recall and false recognition for critical non-presented targets were then tested on a group of Chinese users. The average co-occurrence rate of the list word and the critical word was calculated based on two large Chinese corpuses. List–level analyses revealed that the correlation between the American and Taiwanese participants was significant only in false recognition. More importantly, the co-occurrence rate was significantly correlated with false recall and recognition of Taiwanese participants, and not of American participants. In addition, the backward association strength based on Nelson et al. (The University of South Florida word association, rhyme and word fragment norms, 1999) was significantly correlated with false recall of American participants and not of Taiwanese participants. Results are discussed in terms of the relationship between language experiences and lexical association in creating false memory for word lists.     

Co-occurrence frequency evaluated with large language corpora boosts semantic priming effects

In recent decades, many computational techniques have been developed to analyse the contextual usage of words in large language corpora. The present study examined whether the co-occurrence frequency obtained from large language corpora might boost purely semantic priming effects. Two experiments were conducted: one with conscious semantic priming, the other with subliminal semantic priming. Both experiments contrasted three semantic priming contexts: an unrelated priming context and two related priming contexts with word pairs that are semantically related and that co-occur either frequently or infrequently. In the conscious priming presentation (166-ms stimulus-onset asynchrony, SOA), a semantic priming effect was recorded in both related priming contexts, which was greater with higher co-occurrence frequency. In the subliminal priming presentation (66-ms SOA), no significant priming effect was shown, regardless of the related priming context. These results show that co-occurrence frequency boosts pure semantic priming effects and are discussed with reference to models of semantic network.