Constructing Semantic Models From Words,Images, and Emojis

A number of recent models of semantics combine linguistic information, derived from text corpora, and visual information, derived from image collections, demonstrating that the resulting multimodal models are better than either of their unimodal counterparts, in accounting for behavioral data. Empirical work on semantic processing has shown that emotion also plays an important role especially in abstract concepts; however, models integrating emotion along with linguistic and visual information are lacking. Here, we first improve on visual and affective representations, derived from state-of-the-art existing models, by choosing models that best fit available human semantic data and extending the number of concepts they cover. Crucially then, we assess whether adding affective representations (obtained from a neural network model designed to predict emojis from co-occurring text) improves the model's ability to fit semantic similarity/relatedness judgments from a purely linguistic and linguistic–visual model. We find that, given specific weights assigned to the models, adding both visual and affective representations improves performance, with visual representations providing an improvement especially for more concrete words, and affective representations improving especially the fit for more abstract words.     

Binocular rivalry and semantic processing: out of sight, out of mind

Previous studies of binocular rivalry have shown that some aspects of a phenomenally suppressed stimulus remain available for visual analysis. The question remains, however, whether this analysis extends to the case of semantic information. This experiment examines that question using a semantic-priming paradigm in which prime words were briefly flashed to an eye during either dominance or suppression phases of binocular rivalry. Reaction times on a lexical-decision task were significantly shortened (the semantic-priming effect) only when prime words were presented to an eye during dominance; suppression acted to impair word recognition and to eliminate semantic priming. These results are inconsistent with certain cognitive models of binocular rivalry.     

语义和句法信息在副中央凹加工中的作用

读者能够从副中央凹中提取到什么类型的信息, 是当前阅读眼动研究领域关注的焦点问题.当前普遍认为低水平信息(如正字法信息)可以在副中央凹中得以加工, 但是高水平的信息(如语义和句法信息)能否从副中央凹中进行提取存在争议.本文总结了近年来高水平信息在副中央凹加工的研究进展, 包括拼音文字和非拼音文字(如中文)阅读过程中语义和句法预视效益的研究现状及影响因素, 当前眼动控制模型(如E-Z读者和SWIFT)对预视效益的解释和不足之处, 最后提出未来关于语义和句法预视加工的研究方向.     

句法歧义消解影响因素、认知机制及其神经基础

句法歧义消解主要是指人们在面临多种句法分析可能性的情况下,句法分析器抑制不正确的句法分析而选择正确句法分析的认知过程。研究表明,该过程受到工作记忆、认知控制能力、语言分布频率、语境、韵律等因素的影响。为有效说明人们是如何完成句法歧义消解的,研究者们建构了诸如花园路径模型、约束满意理论、弱交互作用句法分析理论等相关模型。另外,基于脑成像技术研究表明,左侧额下回、左侧颞叶后部、中部和前部在句法消歧任务中都有显著激活。未来研究当围绕着句法消歧中的语义效应、句法和词汇歧义消解机制的异同、句法消歧中抑制能力的效应等方面展开。     

A probabilistic model of semantic plausibility in sentence processing

Experimental research shows that human sentence processing uses information from different levels of linguistic analysis, for example, lexical and syntactic preferences as well as semantic plausibility. Existing computational models of human sentence processing, however, have focused primarily on lexico-syntactic factors. Those models that do account for semantic plausibility effects lack a general model of human plausibility intuitions at the sentence level. Within a probabilistic framework, we propose a wide-coverage model that both assigns thematic roles to verb–argument pairs and determines a preferred interpretation by evaluating the plausibility of the resulting ( verb , role , argument ) triples. The model is trained on a corpus of role-annotated language data. We also present a transparent integration of the semantic model with an incremental probabilistic parser. We demonstrate that both the semantic plausibility model and the combined syntax/semantics model predict judgment and reading time data from the experimental literature.     

Building theories of reading ability: On the relation between individual differences in cognitive skills and reading comprehension

To date most theories of reading ability have emphasized a single factor as the major source of individual differences in performance. However there has been little agreement on what that factor is. However, candidates have included visual discrimination, phonological and semantic recoding, short-term memory, and utilization of linguistic knowledge and context. The single- factor theories are summarized. Literature is then reviewed to show that no single-factor theory is likely to be right, because a very wide range of component skills and abilities has in fact been shown to correlate with reading success. Among them are discrimination of letter location and letter order during perceptual recognition, use of orthographic regularity as an aid to visual code formation, use of spelling-to-sound regularity in phonological recoding, memory for word order, spontaneous identification of syntactic relations, flexibility in prediction from syntactic and semantic context, and context-specificity in semantic encoding. It is concluded that more complex, multifactor models of reading ability are required, and some recent attempts to collect data conducive to such a model are described. In the process, three different approaches to identifying factors relevant to reading success are delineated. These are general abilities assessment, learning potential assessment, and component skills analysis. Two methods of conducting component skills analysis are presented, and it is recommended that they be used as converging operations. Finally, the results of a component skills analysis are used to construct a tentative example of a class of hierarchical models of reading ability that can be pursued developmentally.     

Prototypes,Location, and Associative Networks (PLAN): Towards a Unified Theory of Cognitive Mapping

An integrated representation of large-scale space, or cognitive map, colled PLAN, is presented that attempts to address a broader spectrum of issues than has been previously attempted in a single model. Rather than examining way-finding as a process separate from the rest of cognition, one or the fundamental goals of this work is to examine how the wayfinding process is integrated into general cognition. One result of this approach is that the model is “heads-up,” or scene-based, because it takes advantage of the properties of the human visual system and, particularly, the visual system's split into two pathways. The emphasis on the human location or “where” system is new to cognitive mapping and is port of an attempt to synthesize prototype theory, associative networks and location together in a connectionist system. Not all of PLAN is new, however. Many of its parts have analogues in one or another preexisting theory. What makes PLAN unique is integrating the various components into a coherent whole, and the capacity of this resulting system to speak to a wide range of constraints. Our approach emphasizes adaptiveness; thus, our focus on such issues as ease of use and efficiency of learning. The result is a model that has a stronger relationship both to the environment, and to the ways that humans interact with it, compared with previous models. The resulting model is examined in some detail and compared to other systems.     

A developmental phonological short-term memory deficit: a case study

A developmental case of phonological short-term memory deficit was studied in a highly educated subject. The subject, BS, who had obtained a Ph.D. in molecular biology, demonstrated striking deficits on some short-term memory tasks, particularly for auditorily presented nonword lists. With visual presentation and with meaningful words, he often scored at a normal level. The results indicate a deficit in retaining phonological information but an ability to use visual, lexical, and semantic information to boost recall. Despite this phonological short-term memory deficit, BS scored at a normal level on a syntactic comprehension test and on reading of nonwords. He was impaired, however, on repeated list learning, learning of foreign vocabulary, and transcribing dictated materials. The implications of these results for models of short-term memory and the uses of phonological retention in cognitive processing are discussed.     

Cerebral asymmetry in visual attention

The traditional view of cerebral lateralization of various cognitive functions has been challenged by results from recent experimental and clinical studies. Evidence has been gathered to suggest that a seemingly unitized cognitive function can be further broken down into various processing subcomponents which are distributed across the two hemispheres. For instance, according to such a more complex conceptualization of cerebral lateralization, language is seen not as a unitary ability, but rather as a collection of syntactic, semantic, and prosodic components, with each lateralized in particular manners. In much the same way, the present study attempts to examine the cerebral lateralization patterns of the seemingly unitary visual perception process. In a visual half-field experiment, 20 normal subjects were asked to make same/different judgments to laterally presented arrays of stimuli of the same type as previously studied by Treisman and her colleagues in experiments attempting to separate the preattentive and attentive stages of visual perception. Hemispheric differences were obtained only in tasks requiring attentive processing (i.e., Treisman's glueing). Results indicate a local attentional strategy for the left hemisphere and a global attentional strategy for the right hemisphere.     

Evaluating the random representation assumption of lexical semantics in cognitive models

A common assumption implicit in cognitive models is that lexical semantics can be approximated by using randomly generated representations to stand in for word meaning. However, the use of random representations contains the hidden assumption that semantic similarity is symmetrically distributed across randomly selected words or between instances within a semantic category. We evaluated this assumption by computing similarity distributions for randomly selected words from a number of well-known semantic measures and comparing them with the distributions from random representations commonly used in cognitive models. The similarity distributions from all semantic measures were positively skewed compared with the symmetric normal distributions assumed by random representations. We discuss potential consequences that this false assumption may have for conclusions drawn from process models that use random representations.     

An integrated model of eye movements and visual encoding

Recent computational models of cognition have made good progress in accounting for the visual processes needed to encode external stimuli. However, these models typically incorporate simplified models of visual processing that assume a constant encoding time for all visual objects and do not distinguish between eye movements and shifts of attention. This paper presents a domain-independent computational model, EMMA, that provides a more rigorous account of eye movements and visual encoding and their interaction with a cognitive processor. The visual-encoding component of the model describes the effects of frequency and foveal eccentricity when encoding visual objects as internal representations. The eye-movement component describes the temporal and spatial characteristics of eye movements as they arise from shifts of visual attention. When integrated with a cognitive model, EMMA generates quantitative predictions concerning when and where the eyes move, thus serving to relate higher-level cognitive processes and attention shifts with lower-level eye-movement behavior. The paper evaluates EMMA in three illustrative domains — equation solving, reading, and visual search — and demonstrates how the model accounts for aspects of behavior that simpler models of cognitive and visual processing fail to explain.     

汉语双字词在心理词典中的表征方式:来自fNIRS的证据

以汉语双字构成的真词与假词为实验材料,22名大学生为被试,采用功能性近红外脑成像技术(f NIRS)和事件相关设计,考察被试在完成词汇判断任务时的大脑激活模式,探索汉语双字词在心理词典中的表征方式。结果发现:(1)在完成真假词判断任务时,被试大脑左侧额叶和左侧颞叶均被激活;(2)与判断假词相比,被试在判断真词时显著地激活左额上回和左额中回。这一结果说明汉语双字词在心理词典中是混合表征的。     

Contribution of Callosal Connections to the Interhemispheric Integration of Visuomotor and Cognitive Processes

In recent years, cognitive neuroscience has been concerned with the role of the corpus callosum and interhemispheric communication for lower-level processes and higher-order cognitive functions. There is empirical evidence that not only callosal disconnection but also subtle degradation of the corpus callosum can influence the transfer of information and integration between the hemispheres. The reviewed studies on patients with callosal degradation with and without disconnection indicate a dissociation of callosal functions: while anterior callosal regions were associated with interhemispheric inhibition in situations of semantic (Stroop) and visuospatial (hierarchical letters) competition, posterior callosal areas were associated with interhemispheric facilitation from redundant information at visuomotor and cognitive levels. Together, the reviewed research on selective cognitive functions provides evidence that the corpus callosum contributes to the integration of perception and action within a subcortico-cortical network promoting a unified experience of the way we perceive the visual world and prepare our actions.     

Superadditive effects of multiple lesions in a connectionist architecture: implications for the neuropsychology of optic aphasia

Neuropsychological disorders have traditionally been understood in terms of a focal lesion to a single component of a cognitive architecture. Optic aphasia (OA) defies explanation in this way. In OA, naming of visual stimuli is impaired in the absence of general visual agnosia or anomia. OA has been explained by positing multiple semantic systems or multiple functional pathways to visual naming. M. J. Farah (1990) instead sketched a parsimonious account based on multiple lesions--to pathways mapping visual input to semantic and semantics to naming responses--and the assumption that the effects of the lesions are superadditive. The authors demonstrate superadditive effects of damage in a connectionist architecture and model other phenomena associated with OA. Multiple lesions with superadditivity provide a novel class of explanations for neuropsychological deficits that previously seemed to imply the existence of highly specialized processing components.     

视觉经验缺失对盲人听觉词汇识别的影响

听觉词汇识别包含复杂的认知加工过程。视觉通道受阻的盲人在听觉词汇加工中具有一定听觉补偿优势; 但由于视觉经验的缺失, 盲人对一些视觉相关词(比如颜色词)的语义加工和理解比明眼人弱。未来的研究应对词汇的视觉相关性进行分类讨论; 对音、形、义等多层面及其神经生理机制进行深入探究, 发展符合盲人感知特点的听觉词汇加工模型; 并拓展不同年龄段的发展性研究。最终, 揭示视觉经验缺失对盲人听觉词汇识别影响机制的全貌。     

语义知识神经表征的fMRI研究：通道特异性或类别特异性？

在同一实验中同时考虑类别与通道二个因素,并采用典型的语义提取任务应用功能磁共振成像技术探讨了语义知识的神经基础,结果发现不同类型的语义知识的脑激活模式（含左梭形回的激活）无论在范围还是强度上存在很大的相似性;而且,在严格的统计阈值下,不存在通道或是类别特异性激活的脑区。结果提示语义知识的神经表征分布于整个大脑皮层;在提取物体语义知识时都会同时激活该物体的视觉表象。另外,研究观察到的语义提取过程中BA9区的激活进一步证实并扩展了关于“汉语和表音文字在皮层组织上存在重要差异的观点”。     

视觉词汇加工的动态神经网络及其形成

揭示大脑加工的神经网络机制成为认知神经科学研究的最新取向.本研究以视觉词汇加工脑区(VWFA)的神经功能作为切入点,探讨视觉词汇加工神经网络的动态机制及其形成.研究一考察VWFA在刺激驱动和任务调节下的动态激活,及其与语音、语义脑区所组成神经网络的动态机制.研究二通过跨文化对比以及儿童阅读发展研究,阐明语言经验对视觉词汇加工网络的塑造作用.研究三对比功能网络、静息网络以及白质纤维束联结,探讨视觉词汇加工网络的动态联结及其形成.研究结果有助于建构视觉词汇加工的神经生理模型,为基于脑科学的阅读教学和阅读障碍矫治奠定理论基础,为认知神经科学研究提供了新的思路.     

Dynamics of activation of semantically similar concepts during spoken word recognition

Semantic similarity effects provide critical insight into the organization of semantic knowledge and the nature of semantic processing. In the present study, we examined the dynamics of semantic similarity effects by using the visual world eyetracking paradigm. Four objects were shown on a computer monitor, and participants were instructed to click on a named object, during which time their gaze position was recorded. The likelihood of fixating competitor objects was predicted by the degree of semantic similarity to the target concept. We found reliable, graded competition that depended on degree of target-competitor similarity, even for distantly related items for which priming has not been found in previous priming studies. Time course measures revealed a consistently earlier fixation peak for near semantic neighbors relative to targets. Computational investigations with an attractor dynamical model, a spreading activation model, and a decision model revealed that a combination of excitatory and inhibitory mechanisms is required to obtain such peak timing, providing new constraints on models of semantic processing.     

Developmental changes in word recognition: An information-processing account

Two experiments examining developmental changes in the use of context in single word reading are reported. The first experiment investigated how effectively children can access conceptual knowledge and use this to help their word recognition. The results indicated that young readers can on demand direct their attention to semantic information, and this allows them to reap a relatively greater benefit from context than older more skilful readers. The second experiment attempted to clarify the way such use of contextual information might help in the specific case when a child attempts to decode a new word for the first time. Skilled and unskilled readers pronounced pseudohomophonic nonwords faster when they were primed by a semantic context, and the context effect was greater for unskilled readers. The nonword's graphemic similarity to a lexical item was also important.

In general, the results were consistent with Stanovich's (1980) interactive-compensatory model of reading, and they suggest that in learning to read, several already existing stores of information (e.g. auditory, visual and conceptual) are integrated in order to achieve a solution to the word recognition problem.     

轻度认知损伤的语义记忆研究述评

轻度认知损伤（Mild cognitive impairment,MCI）是介于正常老化与痴呆之间的过渡阶段,表现为与年龄和教育程度不相称的认知功能减退。本文回顾了近期有关MCI语义记忆的研究,分析比较了各研究的实验任务及结果,得出MCI患者存在一定程度的由多种原因造成的语义记忆损伤,语义记忆测验对MCI的早期筛查和转归预测有重要作用。最后指出将来的研究需要将包括语义记忆成分在内的多种认知测验相结合,以便及早发现有可能发展为痴呆的危险个体并开展相应干预。