A study of metrics of distance and correlation between ranked lists for compositionality detection

Compositionality in language refers to how much the meaning of some phrase can be decomposed into the meaning of its constituents and the way these constituents are combined. Based on the premise that substitution by synonyms is meaning-preserving, compositionality can be approximated as the semantic similarity between a phrase and a version of that phrase where words have been replaced by their synonyms. Different ways of representing such phrases exist (e.g., vectors (Kiela and Clark, 2013) or language models (Lioma, Simonsen, Larsen, and Hansen, 2015)), and the choice of representation affects the measurement of semantic similarity.We propose a new compositionality detection method that represents phrases as ranked lists of term weights. Our method approximates the semantic similarity between two ranked list representations using a range of well-known distance and correlation metrics. In contrast to most state-of-the-art approaches in compositionality detection, our method is completely unsupervised. Experiments with a publicly available dataset of 1048 human-annotated phrases shows that, compared to strong supervised baselines, our approach provides superior measurement of compositionality using any of the distance and correlation metrics considered.     

A database of orthography-semantics consistency (OSC) estimates for 15,017 English words

Orthography–semantics consistency (OSC) is a measure that quantifies the degree of semantic relatedness between a word and its orthographic relatives. OSC is computed as the frequency-weighted average semantic similarity between the meaning of a given word and the meanings of all the words containing that very same orthographic string, as captured by distributional semantic models. We present a resource including optimized estimates of OSC for 15,017 English words. In a series of analyses, we provide a progressive optimization of the OSC variable. We show that computing OSC from word-embeddings models (in place of traditional count models), limiting preprocessing of the corpus used for inducing semantic vectors (in particular, avoiding part-of-speech tagging and lemmatization), and relying on a wider pool of orthographic relatives provide better performance for the measure in a lexical-processing task. We further show that OSC is an important and significant predictor of reaction times in visual word recognition and word naming, one that correlates only weakly with other psycholinguistic variables (e.g., family size, word frequency), indicating that it captures a novel source of variance in lexical access. Finally, some theoretical and methodological implications are discussed of adopting OSC as one of the predictors of reaction times in studies of visual word recognition.     

Applying an exemplar model to an implicit rule-learning task: Implicit learning of semantic structure

Studies of implicit learning often examine peoples’ sensitivity to sequential structure. Computational accounts have evolved to reflect this bias. An experiment conducted by Neil and Higham [Neil, G. J., & Higham, P. A.(2012). Implicit learning of conjunctive rule sets: An alternative to artificial grammars. Consciousness and Cognition, 21, 1393–1400] points to limitations in the sequential approach. In the experiment, participants studied words selected according to a conjunctive rule. At test, participants discriminated rule-consistent from rule-violating words but could not verbalize the rule. Although the data elude explanation by sequential models, an exemplar model of implicit learning can explain them. To make the case, we simulate the full pattern of results by incorporating vector representations for the words used in the experiment, derived from the large-scale semantic space models LSA and BEAGLE, into an exemplar model of memory, MINERVA 2. We show that basic memory processes in a classic model of memory capture implicit learning of non-sequential rules, provided that stimuli are appropriately represented.     

The University Of Colorado Meaning Norms

There exist surprisingly few normative lists of word meanings even though homographs—words having single spellings but two or more distinct meanings—are useful in studying memory and language. The meaning norms that are available all have one or more weaknesses, including: (1) the collection of free associates rather than meanings as responses to the stimulus words; (2) the collection of single rather than multiple responses to the stimulus words; (3) the inclusion of only the two most frequently occurring meaning categories, rather than all meaning categories, for the stimulus words; (4) omission of the responses typical of each meaning category; (5) inadequate randomization of the presentation order of the stimulus words; and (6) unpaced presentation of the stimulus words. We have compiled meaning norms for 90 common English words of low, medium, and high concreteness using a methodology designed to correct these weaknesses. Analysis showed that words of medium concreteness have significantly more first-response meanings than do words of either low or high concreteness, lending support to the view that concreteness is a categorical, rather than a continuous, semantic attribute.     

Large-scale latent semantic analysis

Latent semantic analysis (LSA) is a statistical technique for representing word meaning that has been widely used for making semantic similarity judgments between words, sentences, and documents. In order to perform an LSA analysis, an LSA space is created in a two-stage procedure, involving the construction of a word frequency matrix and the dimensionality reduction of that matrix through singular value decomposition (SVD). This article presents LANSE, an SVD algorithm specifically designed for LSA, which allows extremely large matrices to be processed using off-the-shelf computer hardware.     

Similarity Judgment Within and Across Categories: A Comprehensive Model Comparison

Similarity is one of the most important relations humans perceive, arguably subserving category learning and categorization, generalization and discrimination, judgment and decision making, and other cognitive functions. Researchers have proposed a wide range of representations and metrics that could be at play in similarity judgment, yet have not comprehensively compared the power of these representations and metrics for predicting similarity within and across different semantic categories. We performed such a comparison by pairing nine prominent vector semantic representations with seven established similarity metrics that could operate on these representations, as well as supervised methods for dimensional weighting in the similarity function. This approach yields a factorial model structure with 126 distinct representation-metric pairs, which we tested on a novel dataset of similarity judgments between pairs of cohyponymic words in eight categories. We found that cosine similarity and Pearson correlation were the overall best performing unweighted similarity functions, and that word vectors derived from free association norms often outperformed word vectors derived from text (including those specialized for similarity). Importantly, models that used human similarity judgments to learn category-specific weights on dimensions yielded substantially better predictions than all unweighted approaches across all types of similarity functions and representations, although dimension weights did not generalize well across semantic categories, suggesting strong category context effects in similarity judgment. We discuss implications of these results for cognitive modeling and natural language processing, as well as for theories of the representations and metrics involved in similarity.     

How useful are corpus-based methods for extrapolating psycholinguistic variables?

Subjective ratings for age of acquisition, concreteness, affective valence, and many other variables are an important element of psycholinguistic research. However, even for well-studied languages, ratings usually cover just a small part of the vocabulary. A possible solution involves using corpora to build a semantic similarity space and to apply machine learning techniques to extrapolate existing ratings to previously unrated words. We conduct a systematic comparison of two extrapolation techniques: k-nearest neighbours, and random forest, in combination with semantic spaces built using latent semantic analysis, topic model, a hyperspace analogue to language (HAL)-like model, and a skip-gram model. A variant of the k-nearest neighbours method used with skip-gram word vectors gives the most accurate predictions but the random forest method has an advantage of being able to easily incorporate additional predictors. We evaluate the usefulness of the methods by exploring how much of the human performance in a lexical decision task can be explained by extrapolated ratings for age of acquisition and how precisely we can assign words to discrete categories based on extrapolated ratings. We find that at least some of the extrapolation methods may introduce artefacts to the data and produce results that could lead to different conclusions that would be reached based on the human ratings. From a practical point of view, the usefulness of ratings extrapolated with the described methods may be limited.     

Effects of achievement contexts on the meaning structure of emotion words

Little is known about the impact of context on the meaning of emotion words. In the present study, we used a semantic profiling instrument (GRID) to investigate features representing five emotion components (appraisal, bodily reaction, expression, action tendencies, and feeling) of 11 emotion words in situational contexts involving success or failure. We compared these to the data from an earlier study in which participants evaluated the typicality of features out of context. Profile analyses identified features for which typicality changed as a function of context for all emotion words, except contentment, with appraisal features being most frequently affected. Those context effects occurred for both hypothesised basic and non-basic emotion words. Moreover, both data sets revealed a four-dimensional structure. The four dimensions were largely similar (valence, power, arousal, and novelty). The results suggest that context may not change the underlying dimensionality but affects facets of the meaning of emotion words.     

Implicit meaning in 18‐month‐old toddlers

A substantial body of evidence demonstrates that infants understand the meaning of spoken words from as early as 6 months. Yet little is known about their ability to do so in the absence of any visual referent, which would offer diagnostic evidence for an adult‐like, symbolic interpretation of words and their use in language mediated thought. We used the head‐turn preference procedure to examine whether infants can generate implicit meanings from word forms alone as early as 18 months of age, and whether they are sensitive to meaningful relationships between words. In one condition, toddlers were presented with lists of words taken from the same taxonomic category (e.g. animals or body parts). In a second condition, words taken from two other categories (e.g. clothes and food items) were interleaved within the same list. Listening times were found to be longer in the related‐category condition than in the mixed‐category condition, suggesting that infants extract the meaning of spoken words and are sensitive to the semantic relatedness between these words. Our results show that infants have begun to construct the rudiments of a semantic system based on taxonomic relations even before they enter a period of accelerated vocabulary growth.     

Semantic similarity, predictability, and models of sentence processing

The effects of word predictability and shared semantic similarity between a target word and other words that could have taken its place in a sentence on language comprehension are investigated using data from a reading time study, a sentence completion study, and linear mixed-effects regression modeling. We find that processing is facilitated if the different possible words that could occur in a given context are semantically similar to each other, meaning that processing is affected not only by the nature of the words that do occur, but also the relationships between the words that do occur and those that could have occurred. We discuss possible causes of the semantic similarity effect and point to possible limitations of using probability as a model of cognitive effort.     

Affective norms for 380 Spanish words belonging to three different semantic categories

Emotional words are increasingly used in the study of word processing. To elucidate whether the experimental effects obtained with these words are due either to their affective content or to other semantic characteristics, it is necessary to conduct experiments with affectively valenced words obtained from different semantic categories. In the present article, we present affective ratings for 380 Spanish words belonging to three semantic categories: animals, people, and objects. The norms are based on the assessments made by 504 participants, who rated about 47 words either in valence and arousal, by using the Self-Assessment Manikin (Bradley & Lang, Journal of Behavioral Therapy and Experimental Psychiatry, 25, 49-59. 1994), or in concreteness and familiarity. These ratings will help researchers select stimuli for experiments in which both the affective properties of words and their membership to a given semantic category have to be taken into account. The database is available as an online supplement for this article.     

Semantic neighborhood effects on the recognition of ambiguous words

The effect of semantic neighborhood on the processing of ambiguous words was examined in two lexical decision experiments. Semantic neighborhood was defined in terms of semantic set size and network connectivity. In Experiment 1, the variables of semantic set size, network connectivity, and ambiguity were crossed. An ambiguity advantage was observed only within small-set low-connectivity words. In Experiment 2, the effect of network connectivity on the processing of words of high and low meaning relatedness was examined. Participants responded more rapidly to words of high meaning relatedness, relative to words of low meaning relatedness, but only within high-connectivity words. These results are interpreted within a framework in which both semantic feedback processes and meaning-level competition can affect the recognition of semantically ambiguous words.     

Composition in distributional models of semantics

Vector-based models of word meaning have become increasingly popular in cognitive science. The appeal of these models lies in their ability to represent meaning simply by using distributional information under the assumption that words occurring within similar contexts are semantically similar. Despite their widespread use, vector-based models are typically directed at representing words in isolation, and methods for constructing representations for phrases or sentences have received little attention in the literature. This is in marked contrast to experimental evidence (e.g., in sentential priming) suggesting that semantic similarity is more complex than simply a relation between isolated words. This article proposes a framework for representing the meaning of word combinations in vector space. Central to our approach is vector composition, which we operationalize in terms of additive and multiplicative functions. Under this framework, we introduce a wide range of composition models that we evaluate empirically on a phrase similarity task.     

Is an ant larger than a lion?

In order to examine the influence exerted by an irrelevant semantic variable in a comparative judgment task, we employed a Stroop-like paradigm. The stimuli were pairs of animal names that were different in their physical and semantic sizes (e.g., ant lion). Participants were asked to judge which of the two words was larger either in physical or in semantic size. Size congruity effect (i.e., faster reaction times with congruent than with incongruent stimuli) appeared in both semantic and physical judgments. The semantic distance effect (i.e., large semantic distances are processed faster than smaller ones), appeared only when the semantic dimension was relevant to the task. The findings indicate that when a word (animal name) is presented, its meaning is accessed automatically. Part of this meaning (at least with our stimuli) relates to the size of the animal in real life. Processing of meaning of the size of the words is carried out in parallel with the extraction of the physical features of the presented stimuli.     

Evaluating the random representation assumption of lexical semantics in cognitive models

A common assumption implicit in cognitive models is that lexical semantics can be approximated by using randomly generated representations to stand in for word meaning. However, the use of random representations contains the hidden assumption that semantic similarity is symmetrically distributed across randomly selected words or between instances within a semantic category. We evaluated this assumption by computing similarity distributions for randomly selected words from a number of well-known semantic measures and comparing them with the distributions from random representations commonly used in cognitive models. The similarity distributions from all semantic measures were positively skewed compared with the symmetric normal distributions assumed by random representations. We discuss potential consequences that this false assumption may have for conclusions drawn from process models that use random representations.     

Exploring What Is Encoded in Distributional Word Vectors: A Neurobiologically Motivated Analysis

The pervasive use of distributional semantic models or word embeddings for both cognitive modeling and practical application is because of their remarkable ability to represent the meanings of words. However, relatively little effort has been made to explore what types of information are encoded in distributional word vectors. Knowing the internal knowledge embedded in word vectors is important for cognitive modeling using distributional semantic models. Therefore, in this paper, we attempt to identify the knowledge encoded in word vectors by conducting a computational experiment using Binder et al.'s (2016) featural conceptual representations based on neurobiologically motivated attributes. In an experiment, these conceptual vectors are predicted from text-based word vectors using a neural network and linear transformation, and prediction performance is compared among various types of information. The analysis demonstrates that abstract information is generally predicted more accurately by word vectors than perceptual and spatiotemporal information, and specifically, the prediction accuracy of cognitive and social information is higher. Emotional information is also found to be successfully predicted for abstract words. These results indicate that language can be a major source of knowledge about abstract attributes, and they support the recent view that emphasizes the importance of language for abstract concepts. Furthermore, we show that word vectors can capture some types of perceptual and spatiotemporal information about concrete concepts and some relevant word categories. This suggests that language statistics can encode more perceptual knowledge than often expected.     

Linguistic semantic agraphia: A dissociation of the lexical spelling system from semantics

Two spelling systems have been described. The phonological system transcodes speech sounds to letters and is thought to be useful for spelling regular words and pronounceable nonwords. Although the second system, the lexical-semantic system, is thought to use visual word images and meaning to spell irregular words, it is not known if this system is dependent on semantic knowledge. We used a homophone spelling test to examine the lexical-semantic system in five patients. The patients were asked to spell individual homophones (doe or dough) using the context of a sentence. Semantically incorrect and correct homophones were spelled equally well, whether they were regular or irregular. These results demonstrate that an irregular word may be spelled without knowledge of the word's meaning. Therefore, the lexical system can be dissociated from semantic influence.     

Representing word meaning and order information in a composite holographic lexicon

The authors present a computational model that builds a holographic lexicon representing both word meaning and word order from unsupervised experience with natural language. The model uses simple convolution and superposition mechanisms (cf. B. B. Murdock, 1982) to learn distributed holographic representations for words. The structure of the resulting lexicon can account for empirical data from classic experiments studying semantic typicality, categorization, priming, and semantic constraint in sentence completions. Furthermore, order information can be retrieved from the holographic representations, allowing the model to account for limited word transitions without the need for built-in transition rules. The model demonstrates that a broad range of psychological data can be accounted for directly from the structure of lexical representations learned in this way, without the need for complexity to be built into either the processing mechanisms or the representations. The holographic representations are an appropriate knowledge representation to be used by higher order models of language comprehension, relieving the complexity required at the higher level.     

Semantic additivity and semantic inhibition: dissociable processes in the cerebral hemispheres?

We report the results of two visual half-field semantic priming experiments using a high proportion of related trials to examine hemisphere asymmetries for semantic processes beyond those attributable to automatic meaning activation. Contrary to previous investigations, we obtained inhibition for unrelated trials in both visual fields. However, priming was additive (being greater for words related via category membership and association than for either single dimension) only when words were presented to the RVF/left hemisphere. A third experiment, using centrally presented stimuli, implied that semantic additivity should be attributed to post-access meaning comparisons and inhibition to the generation of semantic expectancies. These results suggest (1) that inhibition and additivity are potentially dissociable "controlled" semantic processes and (2) that the left hemisphere predominates for meaning integration across successively presented words. The availability of finely tuned meaning integration processes in the left hemisphere may contribute to its superiority in language processing, despite right hemisphere competence for some semantic operations.     

Graph‐Theoretic Properties of Networks Based on Word Association Norms: Implications for Models of Lexical Semantic Memory

We compared the ability of three different contextual models of lexical semantic memory (BEAGLE, Latent Semantic Analysis, and the Topic model) and of a simple associative model (POC) to predict the properties of semantic networks derived from word association norms. None of the semantic models were able to accurately predict all of the network properties. All three contextual models over‐predicted clustering in the norms, whereas the associative model under‐predicted clustering. Only a hybrid model that assumed that some of the responses were based on a contextual model and others on an associative network (POC) successfully predicted all of the network properties and predicted a word's top five associates as well as or better than the better of the two constituent models. The results suggest that participants switch between a contextual representation and an associative network when generating free associations. We discuss the role that each of these representations may play in lexical semantic memory. Concordant with recent multicomponent theories of semantic memory, the associative network may encode coordinate relations between concepts (e.g., the relation between pea and bean, or between sparrow and robin), and contextual representations may be used to process information about more abstract concepts.