Word frequency effects in high-dimensional co-occurrence models: A new approach

The HAL (hyperspace analog to language) model of lexical semantics uses global word co-occurrence from a large corpus of text to calculate the distance between words in co-occurrence space. We have implemented a system called HiDEx (High Dimensional Explorer) that extends HAL in two ways: It removes unwanted influence of orthographic frequency from the measures of distance, and it finds the number of words within a certain distance of the word of interest (NCount, the number of neighbors). These two changes to the HAL model produce     

Age effects in a computational model of memory

Previous research demonstrates that semantic priming is relatively unaffected by age (Chiarello, Church, & Hoyer, 1985; Howard, 1988). To determine how age might affect representations in the HAL model of memory, the authors gathered text from older and younger adults and generated a global co-occurrence matrix for each. An analysis demonstrated that, as in humans, there was little difference in measures of semantic priming between memory matrices constructed from the two corpora. However, the authors discovered that semantic word neighborhoods generated by the older adult corpus were denser than corresponding neighborhoods derived from the younger adult corpus. This result has implications for changes in the semantic representation of words due to the aging process.     

From simple associations to the building blocks of language: Modeling meaning in memory with the HAL model

This paper presents a theoretical approach of how simple, episodic associations are transduced into semantic and grammatical categorical knowledge. The approach is implemented in the hyperspace analogue to language (HAL) model of memory, which uses a simple global co-occurrence learning algorithm to encode the context in which words occur. This encoding is the basis for the formation of meaning representations in a high-dimensional context space. Results are presented, and the argument is made that this simple process can ultimately provide the language-comprehension system with semantic and grammatical information required in the comprehension process.     

Three-step priming in lexical decision

In two experiments, we investigated mediated two-step priming (e.g., from LION to STRIPES via TIGER) and three-step priming (e.g., from MANE to STRIPES via LION and TIGER). Experiment 1 showed robust two-step priming in the double lexical decision task. In Experiment 2, we tested for three-step priming and investigated the possibility that it is not association strength based on free association, but frequency of co-occurrence, that causes three-step priming. Co-occurrence has been proposed as a measure of familiarity and semantic relatedness. Significant three-step priming was obtained. Lexical co-occurrence could not account for the effect. However, a more global measure of semantic similarity that includes the similarity of the contexts in which concepts occur revealed that the three-step pairs were semantically related. If this global measure provides a proper estimate of the semantic relatedness of our items, then three-step priming is consistent not only with spreading activation models, but also with distributed memory models and the compound cue model.     

A crossed random effects diffusion model for speeded semantic categorization decisions

Choice reaction times (RTs) are often used as a proxy measure of typicality in semantic categorization studies. However, other item properties have been linked to choice RTs as well. We apply a tailored process model of choice RT to a speeded semantic categorization task in order to deconfound different sources of variability in RT. Our model is based on a diffusion model of choice RT, extended to include crossed random effects (of items and participants). This model retains the interesting process interpretation of the diffusion model’s parameters, but it can be applied to choice RTs even in the case where there are few or no repeated measurements of each participant-item combination. Different aspects of the response process are then linked to different types of item properties. A typicality measure turns out to predict the rate of information uptake, while a lexicographic measure predicts the stimulus encoding time. Accessibility measures cannot reliably predict any component of the decision process.     

A computational approach to modeling population differences

Four experiments were conducted to determine whether the Hyperspace Analogue to Language (HAL) model of semantic memory could differentiate between two different populations. An analysis of the differences in densities (or average distances between word neighbors in semantic space) in HAL matrices—generated from text corpora derived from younger and older adults—confirmed that HAL was able to distinguish between the two age groups. This difference was again detected when structured interview data were used to build the corpora. A third experiment, designed to test the specificity of HAL in detecting differences between groups, did not detect any difference in the densities of the memory representations when older adults generated both the test corpora. The final experiment, conducted on the language of adults with Alzheimer’s and normal adults, again demonstrated that HAL could discriminate between the two populations. These results suggest that HAL is capable of modeling, on the basis of changes in mean density, some of the differences between populations without modifying the model itself but, rather, by changing the text corpus from which the model creates its representations in semantic space.     

Grounding co-occurrence: Identifying features in a lexical co-occurrence model of semantic memory

Lexical co-occurrence models of semantic memory represent word meaning by vectors in a high-dimensional space. These vectors are derived from word usage, as found in a large corpus of written text. Typically, these models are fully automated, an advantage over models that represent semantics that are based on human judgments (e.g., feature-based models). A common criticism of co-occurrence models is that the representations are not grounded: Concepts exist only relative to each other in the space produced by the model. It has been claimed that feature-based models offer an advantage in this regard. In this article, we take a step toward grounding a cooccurrence model. A feed-forward neural network is trained using back propagation to provide a mapping from co-occurrence vectors to feature norms collected from subjects. We show that this network is able to retrieve the features of a concept from its co-occurrence vector with high accuracy and is able to generalize this ability to produce an appropriate list of features from the co-occurrence vector of a novel concept.     

WINDSOR: Windsor improved norms of distance and similarity of representations of semantics

Lexical co-occurrence models of semantic memory form representations of the meaning of a word on the basis of the number of times that pairs of words occur near one another in a large body of text. These models offer a distinct advantage over models that require the collection of a large number of judgments from human subjects, since the construction of the representations can be completely automated. Unfortunately, word frequency, a well-known predictor of reaction time in several cognitive tasks, has a strong effect on the co-occurrence counts in a corpus. Two words with high frequency are more likely to occur together purely by chance than are two words that occur very infrequently. In this article, we examine a modification of a successful method for constructing semantic representations from lexical co-occurrence. We show that our new method eliminates the influence of frequency, while still capturing the semantic characteristics of words.     

The Role of Negative Information in Distributional Semantic Learning

Distributional models of semantics learn word meanings from contextual co‐occurrence patterns across a large sample of natural language. Early models, such as LSA and HAL (Landauer & Dumais, 1997; Lund & Burgess, 1996), counted co‐occurrence events; later models, such as BEAGLE (Jones & Mewhort, 2007), replaced counting co‐occurrences with vector accumulation. All of these models learned from positive information only: Words that occur together within a context become related to each other. A recent class of distributional models, referred to as neural embedding models, are based on a prediction process embedded in the functioning of a neural network: Such models predict words that should surround a target word in a given context (e.g., word2vec; Mikolov, Sutskever, Chen, Corrado, & Dean, 2013). An error signal derived from the prediction is used to update each word's representation via backpropagation. However, another key difference in predictive models is their use of negative information in addition to positive information to develop a semantic representation. The models use negative examples to predict words that should not surround a word in a given context. As before, an error signal derived from the prediction prompts an update of the word's representation, a procedure referred to as negative sampling. Standard uses of word2vec recommend a greater or equal ratio of negative to positive sampling. The use of negative information in developing a representation of semantic information is often thought to be intimately associated with word2vec's prediction process. We assess the role of negative information in developing a semantic representation and show that its power does not reflect the use of a prediction mechanism. Finally, we show how negative information can be efficiently integrated into classic count‐based semantic models using parameter‐free analytical transformations.     

How useful are corpus-based methods for extrapolating psycholinguistic variables?

Subjective ratings for age of acquisition, concreteness, affective valence, and many other variables are an important element of psycholinguistic research. However, even for well-studied languages, ratings usually cover just a small part of the vocabulary. A possible solution involves using corpora to build a semantic similarity space and to apply machine learning techniques to extrapolate existing ratings to previously unrated words. We conduct a systematic comparison of two extrapolation techniques: k-nearest neighbours, and random forest, in combination with semantic spaces built using latent semantic analysis, topic model, a hyperspace analogue to language (HAL)-like model, and a skip-gram model. A variant of the k-nearest neighbours method used with skip-gram word vectors gives the most accurate predictions but the random forest method has an advantage of being able to easily incorporate additional predictors. We evaluate the usefulness of the methods by exploring how much of the human performance in a lexical decision task can be explained by extrapolated ratings for age of acquisition and how precisely we can assign words to discrete categories based on extrapolated ratings. We find that at least some of the extrapolation methods may introduce artefacts to the data and produce results that could lead to different conclusions that would be reached based on the human ratings. From a practical point of view, the usefulness of ratings extrapolated with the described methods may be limited.     

Graph‐Theoretic Properties of Networks Based on Word Association Norms: Implications for Models of Lexical Semantic Memory

We compared the ability of three different contextual models of lexical semantic memory (BEAGLE, Latent Semantic Analysis, and the Topic model) and of a simple associative model (POC) to predict the properties of semantic networks derived from word association norms. None of the semantic models were able to accurately predict all of the network properties. All three contextual models over‐predicted clustering in the norms, whereas the associative model under‐predicted clustering. Only a hybrid model that assumed that some of the responses were based on a contextual model and others on an associative network (POC) successfully predicted all of the network properties and predicted a word's top five associates as well as or better than the better of the two constituent models. The results suggest that participants switch between a contextual representation and an associative network when generating free associations. We discuss the role that each of these representations may play in lexical semantic memory. Concordant with recent multicomponent theories of semantic memory, the associative network may encode coordinate relations between concepts (e.g., the relation between pea and bean, or between sparrow and robin), and contextual representations may be used to process information about more abstract concepts.     

Dynamics of activation of semantically similar concepts during spoken word recognition

Semantic similarity effects provide critical insight into the organization of semantic knowledge and the nature of semantic processing. In the present study, we examined the dynamics of semantic similarity effects by using the visual world eyetracking paradigm. Four objects were shown on a computer monitor, and participants were instructed to click on a named object, during which time their gaze position was recorded. The likelihood of fixating competitor objects was predicted by the degree of semantic similarity to the target concept. We found reliable, graded competition that depended on degree of target-competitor similarity, even for distantly related items for which priming has not been found in previous priming studies. Time course measures revealed a consistently earlier fixation peak for near semantic neighbors relative to targets. Computational investigations with an attractor dynamical model, a spreading activation model, and a decision model revealed that a combination of excitatory and inhibitory mechanisms is required to obtain such peak timing, providing new constraints on models of semantic processing.     

A probabilistic model of semantic plausibility in sentence processing

Experimental research shows that human sentence processing uses information from different levels of linguistic analysis, for example, lexical and syntactic preferences as well as semantic plausibility. Existing computational models of human sentence processing, however, have focused primarily on lexico-syntactic factors. Those models that do account for semantic plausibility effects lack a general model of human plausibility intuitions at the sentence level. Within a probabilistic framework, we propose a wide-coverage model that both assigns thematic roles to verb–argument pairs and determines a preferred interpretation by evaluating the plausibility of the resulting ( verb , role , argument ) triples. The model is trained on a corpus of role-annotated language data. We also present a transparent integration of the semantic model with an incremental probabilistic parser. We demonstrate that both the semantic plausibility model and the combined syntax/semantics model predict judgment and reading time data from the experimental literature.     

Associative priming in faces: Semantic relatedness or simple co-occurrence?

In two experiments, we explored the effects of co-occurrence and semantic relationships in the associative priming of faces. In Experiment 1, pairs of computer-generated human faces were presented simultaneously (i.e., they co-occurred) with no associated semantic information attached to them. A significant facilitation effect in the subsequent recognition of these paired faces (priming) was observed. Thus, repeatedly presenting faces together while keeping semantic information to a minimum appears to be enough to produce associative priming. In Experiment 2, the computer-generated faces were associated with semantic information and again presented ipairs. Priming effects arising from co-occurrence and semantic relatedness were observed. The results from these experiments show that semantic relatedness is not the sole cause of the association between faces; co-occurrence plays a crucial role too. This conclusion has significant implications for the current computational models of face processing.     

Joint effects of semantic priming and repetition in a lexical decision task: Implications for a model of lexical access

An experiment is reported which showed that in a lexical decision task semantic priming by a related preceding word and repetition of target words produce additive effects on decision latency. Previous models of lexical access and modifications of them are discussed, and it is argued that some such models predict an interaction of priming and repetition, while others are insufficiently precise to make a prediction. It is suggested that the generality of effects across tasks requiring lexical access must be established and the components of complex effects must be separated before an adequate model can be devised to account for the data.     

The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics

Notable progress has been made recently on computational models of semantics using vector representations for word meaning (Mikolov, Chen, Corrado, & Dean, 2013; Mikolov, Sutskever, Chen, Corrado, & Dean, 2013). As representations of meaning, recent models presumably hone in on plausible organizational principles for meaning. We performed an analysis on the organization of the skip-gram model’s semantic space. Consistent with human performance (Osgood, Suci, & Tannenbaum, 1957), the skip-gram model primarily relies on affective distinctions to organize meaning. We showed that the skip-gram model accounts for unique variance in behavioral measures of lexical access above and beyond that accounted for by affective and lexical measures. We also raised the possibility that word frequency predicts behavioral measures of lexical access due to the fact that word use is organized by semantics. Deconstruction of the semantic representations in semantic models has the potential to reveal organizing principles of human semantics.     

The structure of semantic person memory: evidence from semantic priming in person recognition

This paper reviews research on the structure of semantic person memory as examined with semantic priming. In this experimental paradigm, a familiarity decision on a target face or written name is usually faster when it is preceded by a related as compared to an unrelated prime. This effect has been shown to be relatively short lived and susceptible to interfering items. Moreover, semantic priming can cross stimulus domains, such that a written name can prime a target face and vice versa. However, it remains controversial whether representations of people are stored in associative networks based on co-occurrence, or in more abstract semantic categories. In line with prominent cognitive models of face recognition, which explain semantic priming by shared semantic information between prime and target, recent research demonstrated that priming could be obtained from purely categorically related, non-associated prime/target pairs. Although strategic processes, such as expectancy and retrospective matching likely contribute, there is also evidence for a non-strategic contribution to priming, presumably related to spreading activation. Finally, a semantic priming effect has been demonstrated in the N400 event-related potential (ERP) component, which may reflect facilitated access to semantic information. It is concluded that categorical relatedness is one organizing principle of semantic person memory.     

The Paradox of Abstraction: Precision Versus Concreteness

We introduce a novel measure of abstractness based on the amount of information of a concept computed from its position in a semantic taxonomy. We refer to this measure as precision. We propose two alternative ways to measure precision, one based on the path length from a concept to the root of the taxonomic tree, and another one based on the number of direct and indirect descendants. Since more information implies greater processing load, we hypothesize that nouns higher in precision will have a processing disadvantage in a lexical decision task. We contrast precision to concreteness, a common measure of abstractness based on the proportion of sensory-based information associated with a concept. Since concreteness facilitates cognitive processing, we predict that while both concreteness and precision are measures of abstractness, they will have opposite effects on performance. In two studies we found empirical support for our hypothesis. Precision and concreteness had opposite effects on latency and accuracy in a lexical decision task, and these opposite effects were observable while controlling for word length, word frequency, affective content and semantic diversity. Our results support the view that concepts organization includes amodal semantic structures which are independent of sensory information. They also suggest that we should distinguish between sensory-based and amount-of-information-based abstractness.     

Semantic distance norms computed from an electronic dictionary (WordNet)

WordNet, an electronic dictionary (or lexical database), is a valuable resource for computational and cognitive scientists. Recent work on the computing of semantic distances among nodes (synsets) in WordNet has made it possible to build a large database of semantic distances for use in selecting word pairs for psychological research. The database now contains nearly 50,000 pairs of words that have values for semantic distance, associative strength, and similarity based on co-occurrence. Semantic distance was found to correlate weakly with these other measures but to correlate more strongly with another measure of semantic relatedness, featural similarity. Hierarchical clustering analysis suggested that the knowledge structure underlying semantic distance is similar in gross form to that underlying featural similarity. In experiments in which semantic similarity ratings were used, human participants were able to discriminate semantic distance. Thus, semantic distance as derived from WordNet appears distinct from other measures of word pair relatedness and is psychologically functional. This database may be downloaded fromwww.psychonomic.org/archive/.