交流手势认知理论

手势是交流互动中一种重要的非语言媒介, 手势不仅可以辅助语言交流而且具有独立的交流性; 作为和语言共同发生的非语言媒介, 手势交流有助于降低交流认知负荷。文章重点归纳和述评了基于手势和语言表达关系的交流手势理论、交流手势激活理论、交流手势的认知节省理论。未来研究需要进一步考虑交流手势实验研究情境自然性和控制严格性间的平衡, 交流手势和其他非语言因素间的关系, 交流手势认知研究的现实意义。     

交流语言认知理论

交流语言认知是交流认知的典型代表,语言是交流互动中的典型媒介。交流语言具有互动性、合作性、集体性奖赏和个人责任等特征,这决定了交流语言认知过程特征的复杂性、灵活性和现实功用性。文章重点归纳和述评:交流语言加工的同伴特定性理论、时间过程理论、意识性理论。未来研究应进一步考虑:交流语言认知实验研究情境的自然性,交流语言实验设计角度和目的的包容性,交流语言认知和非语言认知间的关系特征等。     

语言伴随性手势是否和语言共享同一交流系统?

语言伴随性手势是人类语言交流的一个普遍的特征, 它可以发挥信息交流的功能。依据产生目的以及适用范围的不同, 手势可以分成表象性手势和非表象性手势。多数研究者认为, 语言和手势动作是“近亲”, 具有“家族相似性”。来自语言发展、认知心理学和认知神经科学的证据均表明, 手势和语言共享同一个交流系统。当手势动作和发音单词意义相同时, 手势动作受到单词发音的干扰, 同时引发发音共振峰2 (F2)的放大。手势和言语之间遵循互相作用理论, 它们作用的基础是语义一致性, 镜像神经系统完成了两者语义一致性的传递。由于语言伴随性手势和语言之间的关系是语言和行为结合良好的特例, 因此, 对语言伴随性手势的研究将有助于对人类心智有一个更深层次的认识和理解。     

手势认知功能研究的新视角：“空间化”手势假设

人类在说话或思考的时候常常伴随着手势。手势是在认知加工或交流过程中自动产生的, 具有表征性, 同时, 手势能够影响人类的认知加工。尽管研究者对手势的概念界定各有侧重, 但普遍认为手势不同于直接行动, 具有认知功能。手势认知功能的代表性理论模型有词汇索引模型、信息打包假设、表象保持理论、语义特殊性假设和嵌入/延展观。根据手势认知功能研究中主要自变量的不同, 可以把手势认知功能分成三种不同的研究范式, 即允许-限制手势的研究范式、侧重手势模式改变的研究范式、侧重情境改变的研究范式。今后值得关注的研究方向除了深入探讨手势认知功能的神经机制、加强对手势认知功能的干预研究外, 提出了建立更具解释力的手势认知功能的理论模型——“空间化”手势假设。     

手势认知功能研究的新视角：“空间化”手势假设

人类在说话或思考的时候常常伴随着手势。手势是在认知加工或交流过程中自动产生的,具有表征性,同时,手势能够影响人类的认知加工。尽管研究者对手势的概念界定各有侧重,但普遍认为手势不同于直接行动,具有认知功能。手势认知功能的代表性理论模型有词汇索引模型、信息打包假设、表象保持理论、语义特殊性假设和嵌入/延展观。根据手势认知功能研究中主要自变量的不同,可以把手势认知功能分成三种不同的研究范式,即允许-限制手势的研究范式、侧重手势模式改变的研究范式、侧重情境改变的研究范式。今后值得关注的研究方向除了深入探讨手势认知功能的神经机制、加强对手势认知功能的干预研究外,提出了建立更具解释力的手势认知功能的理论模型——"空间化"手势假设。     

交流语言认知特征

交流是以语言为核心媒介的一种社会人际互动方式, 与个人语言认知过程不同, 交流互动性决定了交流语言认知过程的复杂性、灵活性。重点归纳和述评：交流语言内容特征、交流语言加工理论、交流语言与非语言因素的关系。未来对于交流语言认知特征的探讨应持一个相对开放的态度; 尝试采用多因素依次叠加的实验设计, 考虑交流实验情境的自然性, 以多种子加工过程为基础全面理解语言认知特征。     

儿童手势及其与学习的关系

手势是在交流或认知过程中产生的、不直接作用于物体的手部运动, 具有具体性和抽象性。其分类主要从手势的来源、手势的内容、手势的意图及手势和言语的匹配性等角度进行划分。不同类型手势在出现时间及发展趋势上存在差异。手势在儿童词汇学习、言语表达、数学问题解决、空间学习及记忆等方面起促进作用, 但对言语理解的影响未得出一致结论。未来可关注不同类型手势与儿童认知发展的关系及对比不同来源手势在各学习领域的优势情况。     

英语多媒体学习中言语关联手势对认知负荷的影响

为了探讨英语多媒体学习中言语关联手势对认知负荷及学习成绩的影响,采用2×2被试间实验设计。结果发现：该手势对认知负荷影响的主效应不显著,但与英语语言技能水平之间存在交互作用,当学生语言技能水平低时,手势增加了认知负荷,反之,则降低认知负荷;该手势对句子转换的成绩没有明显影响,但在学生语言技能高时能提升理解能力的成绩。本研究结果提示,言语关联手势的运用有明显作用,它能提高或降低认知负荷,并对理解能力的成绩产生影响,但其大小和方向依赖于学生英语语言技能水平。     

多元语言文化对个体情境下与社会情境下认知转换功能的差异性影响——来自锡伯族的证据

多民族国家由多个民族和族群组成。个体处于各种民族关系或族群关系中, 需要与其他民族或族群的人协作。“西迁”使我国的锡伯族分居于新疆和东北, 走上了不同的发展道路。新疆锡伯族处于多元语言文化中, 不仅保留了本民族的语言和文字, 也汲取了周边民族的语言文化营养; 而东北锡伯族则与汉族融合, 逐渐失去了本民族的语言和文字, 转用汉语和汉字。采用经典认知转换任务和联合认知转换任务, 对比新疆锡伯族和东北锡伯族在个体情境下和社会情境下的认知转换功能差异, 探讨多元语言文化对个体情境下与社会情境下认知转换功能的影响。结果发现, 在个体情境下, 新疆锡伯族的转换代价更小; 在社会情境下, 新疆锡伯族的联合机制更容易被激活, 他们分配认知资源去表征同伴的任务, 自我-他人的整合程度更高, 转换代价更大。整个研究表明, 多元语言文化对认知转换功能具有不同的影响：促进个体情境下的认知转换功能, 降低社会情境下的认知转换功能。本研究的结果为多民族国家的文化和心理建设提供了重要启示。     

自尊的神经生理基础

自尊是社会心理学领域的经典核心概念之一, 近年来也备受认知神经科学领域的关注。已有的脑成像与神经分泌研究结果显示, 特质性自尊和情境性自尊拥有既各自独立又相互共享的神经机制, 更为重要的是, 特质性自尊会通过影响个体在社会排斥情境中的大脑活动与神经内分泌反应, 调制他们的情境性自尊。为了全面而系统地揭示自尊的内涵与机制, 未来的研究亟需一个整合性的理论框架, 能够将不同类型的自尊, 应激反应模式, 以及大脑特征间的关系加以综合分析。     

Why the spontaneous images created by the hands during talk can help make TV advertisements more effective

The design of effective communications depends upon an adequate model of the communication process. The traditional model is that speech conveys semantic information and bodily movement conveys information about emotion and interpersonal attitudes. But McNeill (2000) argues that this model is fundamentally wrong and that some bodily movements, namely spontaneous hand movements generated during talk (iconic gestures), are integral to semantic communication. But can we increase the effectiveness of communication using this new theory? Focusing on advertising we found that advertisements in which the message was split between speech and iconic gesture (possible on TV) were significantly more effective than advertisements in which meaning resided purely in speech or language (radio/ newspaper). We also found that the significant differences in communicative effectiveness were maintained across five consecutive trials. We compared the communicative power of professionally made TV advertisements in which a spoken message was accompanied either by iconic gestures or by pictorial images, and found the iconic gestures to be more effective. We hypothesized that iconic gestures are so effective because they illustrate and isolate just the core semantic properties of a product. This research suggests that TV advertisements can be made more effective by incorporating iconic gestures with exactly the right temporal and semantic properties.     

Giving cognition a helping hand: The effect of congruent gestures on object name retrieval

The gestures that accompany speech are more than just arbitrary hand movements or communicative devices. They are simulated actions that can both prime and facilitate speech and cognition. This study measured participants’ reaction times for naming degraded images of objects when simultaneously adopting a gesture that was either congruent with the target object, incongruent with it, and when not making any hand gesture. A within‐subjects design was used, with participants (N= 122) naming 10 objects under each condition. Participants named the objects significantly faster when adopting a congruent gesture than when not gesturing at all. Adopting an incongruent gesture resulted in significantly slower naming times. The findings are discussed in the context of the intrapersonal cognitive and facilitatory effects of gestures and underline the relatedness between language, action, and cognition.     

Gestures convey substantive information about a child’s thoughts to ordinary listeners

The gestures that spontaneously occur in communicative contexts have been shown to offer insight into a child’s thoughts. The information gesture conveys about what is on a child’s mind will, of course, only be accessible to a communication partner if that partner can interpret gesture. Adults were asked to observe a series of children who participated ‘live’ in a set of conservation tasks and gestured spontaneously while performing the tasks. Adults were able to glean substantive information from the children’s gestures, information that was not found anywhere in their speech. ‘Gesture-reading’ did, however, have a cost – if gesture conveyed different information from speech, it hindered the listener’s ability to identify the message in speech. Thus, ordinary listeners can and do extract information from a child’s gestures, even gestures that are unedited and fleeting.     

Form and function in early communication: language and pointing gestures

Pointing gestures of verbally advanced 2-year-olds were contrasted with those of less advanced peers, in order to examine the relationships of gesture to language during the acquisition of each. Hypotheses regarding the replacement of gestural functions by speech as verbal skills improve, regarding developmental correspondences between the two communicative domains, and regarding the independence of language acquisition from nonverbal developments were drawn from evolutionary, structuralist, and nativist viewpoints, respectively. Both formal and functional aspects of each communicative skill were measured, and were shown to be largely unrelated, particularly in the gestural domain. No evidence that language replaced gesture for communication in ontogeny was obtained. Correspondences between gesture and language occurred only between functional aspects of each, and the independence of developing language from gestural advances was suggested by the findings.     

Social eye gaze modulates processing of speech and co-speech gesture

In human face-to-face communication, language comprehension is a multi-modal, situated activity. However, little is known about how we combine information from different modalities during comprehension, and how perceived communicative intentions, often signaled through visual signals, influence this process. We explored this question by simulating a multi-party communication context in which a speaker alternated her gaze between two recipients. Participants viewed speech-only or speech + gesture object-related messages when being addressed (direct gaze) or unaddressed (gaze averted to other participant). They were then asked to choose which of two object images matched the speaker’s preceding message. Unaddressed recipients responded significantly more slowly than addressees for speech-only utterances. However, perceiving the same speech accompanied by gestures sped unaddressed recipients up to a level identical to that of addressees. That is, when unaddressed recipients’ speech processing suffers, gestures can enhance the comprehension of a speaker’s message. We discuss our findings with respect to two hypotheses attempting to account for how social eye gaze may modulate multi-modal language comprehension.     

Contrast of hand preferences between communicative gestures and non-communicative actions in baboons: Implications for the origins of hemispheric specialization for language

Gestural communication is a modality considered in the literature as a candidate for determining the ancestral prerequisites of the emergence of human language. As reported in captive chimpanzees and human children, a study in captive baboons revealed that a communicative gesture elicits stronger degree of right-hand bias than non-communicative actions. It remains unclear if it is the communicative nature of this manual behavior which induces such patterns of handedness. In the present study, we have measured hand use for two uninvestigated behaviors in a group of captive olive baboons: (1) a non-communicative self-touching behavior (“muzzle wipe” serving as a control behavior), (2) another communicative gesture (a ritualized “food beg”) different from the one previously studied in the literature (a species-specific threat gesture, namely “hand slap”) in the same population of baboons. The hand preferences for the “food beg” gestures revealed a trend toward right-handedness and significantly correlated with the hand preferences previously reported in the hand slap gesture within the same baboons. By contrast, the hand preferences for the self-touching behaviors did not reveal any trend of manual bias at a group-level nor correlation with the hand preferences of any communicative gestures. These findings provide additional support to the hypothesized existence in baboons of a specific communicative system involved in the production of communicative gestures that may tend to a left-hemispheric dominance and that may differ from the system involved in purely motor functions. The hypothetical implications of these collective results are discussed within the theoretical framework about the origins of hemispheric specialization for human language.     

When do gestures communicate? A meta-analysis

Do the gestures that speakers produce while talking significantly benefit listeners' comprehension of the message? This question has been the topic of many research studies over the previous 35 years, and there has been little consensus. The present meta-analysis examined the effect sizes from 63 samples in which listeners' understanding of a message was compared when speech was presented alone with when speech was presented with gestures. It was found that across samples, gestures do provide a significant, moderate benefit to communication. Furthermore, the magnitude of this effect is moderated by 3 factors. First, effects of gesture differ as a function of gesture topic, such that gestures that depict motor actions are more communicative than those that depict abstract topics. Second, effects of gesture on communication are larger when the gestures are not completely redundant with the accompanying speech; effects are smaller when there is more overlap between the information conveyed in the 2 modalities. Third, the size of the effect of gesture is dependent on the age of the listeners, such that children benefit more from gestures than do adults. Remaining questions for future research are highlighted.     

The relation between gesture and language in aphasic communication

Effective communication in aphasia depends not only on use of preserved linguistic capacities but also (and perhaps primarily) on the capacity to exploit alternative modalities of communication, such as gesture. To ascertain the capacity of aphasic patients to use gesture in their spontaneous communication, informally structured interviews were conducted with two Wernicke's aphasics and two Broca's aphasics, as well as with four normal controls. The performances of the patient groups were compared on the physical parameters of gesture, the points in the communication where gestures occurred, and several facets of the semantics and pragmatics of gesture. Generally speaking, the gestures of the aphasics closely paralleled their speech output: on most indices, the performance of the Wernicke's aphasics more closely resembled that of the normal controls. Wernicke's aphasics differed from normals in the clarity of their language and gestures: While individual linguistic units were often clear, the relation among units was not. In contrast, the Broca's aphasics equaled or surpassed the normal controls in the clarity of their communications. The results offer little support for the view that aphasic patients spontaneously enhance their communicative efficacy through the use of gesture; these findings can, however, be interpreted as evidence in favor of a “central organizer” which controls critical features of communication, irrespective of the modality of expression.     

The repertoire and intentionality of gestural communication in wild chimpanzees

A growing body of evidence suggests that human language may have emerged primarily in the gestural rather than vocal domain, and that studying gestural communication in great apes is crucial to understanding language evolution. Although manual and bodily gestures are considered distinct at a neural level, there has been very limited consideration of potential differences at a behavioural level. In this study, we conducted naturalistic observations of adult wild East African chimpanzees (Pan troglodytes schweinfurthii) in order to establish a repertoire of gestures, and examine intentionality of gesture production, use and comprehension, comparing across manual and bodily gestures. At the population level, 120 distinct gesture types were identified, consisting of 65 manual gestures and 55 bodily gestures. Both bodily and manual gestures were used intentionally and effectively to attain specific goals, by signallers who were sensitive to recipient attention. However, manual gestures differed from bodily gestures in terms of communicative persistence, indicating a qualitatively different form of behavioural flexibility in achieving goals. Both repertoire size and frequency of manual gesturing were more affiliative than bodily gestures, while bodily gestures were more antagonistic. These results indicate that manual gestures may have played a significant role in the emergence of increased flexibility in great ape communication and social bonding.