Speech-associated gestures, Broca’s area, and the human mirror system

Speech-associated gestures are hand and arm movements that not only convey semantic information to listeners but are themselves actions. Broca’s area has been assumed to play an important role both in semantic retrieval or selection (as part of a language comprehension system) and in action recognition (as part of a “mirror” or “observation–execution matching” system). We asked whether the role that Broca’s area plays in processing speech-associated gestures is consistent with the semantic retrieval/selection account (predicting relatively weak interactions between Broca’s area and other cortical areas because the meaningful information that speech-associated gestures convey reduces semantic ambiguity and thus reduces the need for semantic retrieval/selection) or the action recognition account (predicting strong interactions between Broca’s area and other cortical areas because speech-associated gestures are goal-direct actions that are “mirrored”). We compared the functional connectivity of Broca’s area with other cortical areas when participants listened to stories while watching meaningful speech-associated gestures, speech-irrelevant self-grooming hand movements, or no hand movements. A network analysis of neuroimaging data showed that interactions involving Broca’s area and other cortical areas were weakest when spoken language was accompanied by meaningful speech-associated gestures, and strongest when spoken language was accompanied by self-grooming hand movements or by no hand movements at all. Results are discussed with respect to the role that the human mirror system plays in processing speech-associated movements.     

Spoken language and arm gestures are controlled by the same motor control system

Arm movements can influence language comprehension much as semantics can influence arm movement planning. Arm movement itself can be used as a linguistic signal. We reviewed neurophysiological and behavioural evidence that manual gestures and vocal language share the same control system. Studies of primate premotor cortex and, in particular, of the so-called "mirror system", including humans, suggest the existence of a dual hand/mouth motor command system involved in ingestion activities. This may be the platform on which a combined manual and vocal communication system was constructed. In humans, speech is typically accompanied by manual gesture, speech production itself is influenced by executing or observing transitive hand actions, and manual actions play an important role in the development of speech, from the babbling stage onwards. Behavioural data also show reciprocal influence between word and symbolic gestures. Neuroimaging and repetitive transcranial magnetic stimulation (rTMS) data suggest that the system governing both speech and gesture is located in Broca's area. In general, the presented data support the hypothesis that the hand motor-control system is involved in higher order cognition.     

From mouth to hand: gesture, speech, and the evolution of right-handedness

The strong predominance of right-handedness appears to be a uniquely human characteristic, whereas the left-cerebral dominance for vocalization occurs in many species, including frogs, birds, and mammals. Right-handedness may have arisen because of an association between manual gestures and vocalization in the evolution of language. I argue that language evolved from manual gestures, gradually incorporating vocal elements. The transition may be traced through changes in the function of Broca's area. Its homologue in monkeys has nothing to do with vocal control, but contains the so-called "mirror neurons," the code for both the production of manual reaching movements and the perception of the same movements performed by others. This system is bilateral in monkeys, but predominantly left-hemispheric in humans, and in humans is involved with vocalization as well as manual actions. There is evidence that Broca's area is enlarged on the left side in Homo habilis, suggesting that a link between gesture and vocalization may go back at least two million years, although other evidence suggests that speech may not have become fully autonomous until Homo sapiens appeared some 170,000 years ago, or perhaps even later. The removal of manual gesture as a necessary component of language may explain the rapid advance of technology, allowing late migrations of Homo sapiens from Africa to replace all other hominids in other parts of the world, including the Neanderthals in Europe and Homo erectus in Asia. Nevertheless, the long association of vocalization with manual gesture left us a legacy of right-handedness.     

Mirror neurons, the representation of word meaning, and the foot of the third left frontal convolution

Previous neuroimaging research has attempted to demonstrate a preferential involvement of the human mirror neuron system (MNS) in the comprehension of effector-related action word (verb) meanings. These studies have assumed that Broca's area (or Brodmann's area 44) is the homologue of a monkey premotor area (F5) containing mouth and hand mirror neurons, and that action word meanings are shared with the mirror system due to a proposed link between speech and gestural communication. In an fMRI experiment, we investigated whether Broca's area shows mirror activity solely for effectors implicated in the MNS. Next, we examined the responses of empirically determined mirror areas during a language perception task comprising effector-specific action words, unrelated words and nonwords. We found overlapping activity for observation and execution of actions with all effectors studied, i.e., including the foot, despite there being no evidence of foot mirror neurons in the monkey or human brain. These "mirror" areas showed equivalent responses for action words, unrelated words and nonwords, with all of these stimuli showing increased responses relative to visual character strings. Our results support alternative explanations attributing mirror activity in Broca's area to covert verbalisation or hierarchical linearisation, and provide no evidence that the MNS makes a preferential contribution to comprehending action word meanings.     

Gesture in the developing brain

Speakers convey meaning not only through words, but also through gestures. Although children are exposed to co-speech gestures from birth, we do not know how the developing brain comes to connect meaning conveyed in gesture with speech. We used functional magnetic resonance imaging (fMRI) to address this question and scanned 8- to 11-year-old children and adults listening to stories accompanied by hand movements, either meaningful co-speech gestures or meaningless self-adaptors. When listening to stories accompanied by both types of hand movement, both children and adults recruited inferior frontal, inferior parietal, and posterior temporal brain regions known to be involved in processing language not accompanied by hand movements. There were, however, age-related differences in activity in posterior superior temporal sulcus (STSp), inferior frontal gyrus, pars triangularis (IFGTr), and posterior middle temporal gyrus (MTGp) regions previously implicated in processing gesture. Both children and adults showed sensitivity to the meaning of hand movements in IFGTr and MTGp, but in different ways. Finally, we found that hand movement meaning modulates interactions between STSp and other posterior temporal and inferior parietal regions for adults, but not for children. These results shed light on the developing neural substrate for understanding meaning contributed by co-speech gesture.     

Why the spontaneous images created by the hands during talk can help make TV advertisements more effective

The design of effective communications depends upon an adequate model of the communication process. The traditional model is that speech conveys semantic information and bodily movement conveys information about emotion and interpersonal attitudes. But McNeill (2000) argues that this model is fundamentally wrong and that some bodily movements, namely spontaneous hand movements generated during talk (iconic gestures), are integral to semantic communication. But can we increase the effectiveness of communication using this new theory? Focusing on advertising we found that advertisements in which the message was split between speech and iconic gesture (possible on TV) were significantly more effective than advertisements in which meaning resided purely in speech or language (radio/ newspaper). We also found that the significant differences in communicative effectiveness were maintained across five consecutive trials. We compared the communicative power of professionally made TV advertisements in which a spoken message was accompanied either by iconic gestures or by pictorial images, and found the iconic gestures to be more effective. We hypothesized that iconic gestures are so effective because they illustrate and isolate just the core semantic properties of a product. This research suggests that TV advertisements can be made more effective by incorporating iconic gestures with exactly the right temporal and semantic properties.     

From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics

The article analyzes the neural and functional grounding of language skills as well as their emergence in hominid evolution, hypothesizing stages leading from abilities known to exist in monkeys and apes and presumed to exist in our hominid ancestors right through to modern spoken and signed languages. The starting point is the observation that both premotor area F5 in monkeys and Broca's area in humans contain a "mirror system" active for both execution and observation of manual actions, and that F5 and Broca's area are homologous brain regions. This grounded the mirror system hypothesis of Rizzolatti and Arbib (1998) which offers the mirror system for grasping as a key neural "missing link" between the abilities of our nonhuman ancestors of 20 million years ago and modern human language, with manual gestures rather than a system for vocal communication providing the initial seed for this evolutionary process. The present article, however, goes "beyond the mirror" to offer hypotheses on evolutionary changes within and outside the mirror systems which may have occurred to equip Homo sapiens with a language-ready brain. Crucial to the early stages of this progression is the mirror system for grasping and its extension to permit imitation. Imitation is seen as evolving via a so-called simple system such as that found in chimpanzees (which allows imitation of complex "object-oriented" sequences but only as the result of extensive practice) to a so-called complex system found in humans (which allows rapid imitation even of complex sequences, under appropriate conditions) which supports pantomime. This is hypothesized to have provided the substrate for the development of protosign, a combinatorially open repertoire of manual gestures, which then provides the scaffolding for the emergence of protospeech (which thus owes little to nonhuman vocalizations), with protosign and protospeech then developing in an expanding spiral. It is argued that these stages involve biological evolution of both brain and body. By contrast, it is argued that the progression from protosign and protospeech to languages with full-blown syntax and compositional semantics was a historical phenomenon in the development of Homo sapiens, involving few if any further biological changes.     

英语多媒体学习中言语关联手势对认知负荷的影响

为了探讨英语多媒体学习中言语关联手势对认知负荷及学习成绩的影响,采用2×2被试间实验设计。结果发现：该手势对认知负荷影响的主效应不显著,但与英语语言技能水平之间存在交互作用,当学生语言技能水平低时,手势增加了认知负荷,反之,则降低认知负荷;该手势对句子转换的成绩没有明显影响,但在学生语言技能高时能提升理解能力的成绩。本研究结果提示,言语关联手势的运用有明显作用,它能提高或降低认知负荷,并对理解能力的成绩产生影响,但其大小和方向依赖于学生英语语言技能水平。     

Semantic operations in aphasic comprehension: implications for the cortical organization of language.

We provide data on the neurological basis of two semantic operations at the sentence level: aspectual coercion and complement coercion. These operations are characterized by being purely semantic in nature; that is, they lack morphosyntactic reflections. Yet, the operations are mandatory (i.e., they are indispensable for the semantic well formedness of a sentence). Results indicate that, whereas Broca's patients have little or no trouble understanding sentences requiring these operations (performance was above chance for all conditions), Wernicke's patients performed at normal-like levels only for sentences that did not require these operations. These findings suggest that sentence-level semantic operations rely very specifically on the integrity of the cortical area associated with Wernicke's aphasia, but not on the region corresponding to Broca's aphasia. In the context of other findings from lesion and imaging studies, this evidence allows a view of the cortical distribution of language capacity that is drawn along a linguistic line, one which distinguishes syntactic from semantic operations.     

A fMRI study of word retrieval in aphasia

The neural mechanisms underlying recovery of cognitive functions are incompletely understood. Aim of this study was to assess, using functional magnetic resonance (fMRI), the pattern of brain activity during covert word retrieval to letter and semantic cues in five aphasic patients after stroke, in order to assess the modifications of brain function which may be related to recovery. Four out of five patients had undergone language recovery, according to standard testing, after at least 6 months of rehabilitation. The cerebral activation of each patient was evaluated and compared with the activation pattern of normal controls studied with the same fMRI paradigm. In the patients, the pattern of brain activation was influenced by the site and extent of the lesion, by the degree of recovery of language, as reflected by task performance outside the scanner, and by task requirements. In the case of word retrieval to letter cues, a good performance was directly related to the activation in Broca's area, or in the right-sided homologue. On the other hand, in the case of semantic fluency, the relationship between performance level and activation was less clear-cut, because of extensive recruitment of frontal areas in patients with defective performance. These findings suggest that the performance in letter fluency is dependent on the integrity of the left inferior frontal cortex, with the participation of the homologous right hemispheric region when the left inferior frontal cortex is entirely of partially damaged. Semantic fluency, which engages the distributed network of semantic memory, is also associated with more extensive patterns of cerebral activation, which however appear to reflect retrieval effort rather than retrieval success.     

Mirror Neurons and the Evolution of Embodied Language

ABSTRACT— Mirror neurons are a class of neurons first discovered in the monkey premotor cortex that activate both when the monkey executes an action and when it observes the same action made by another individual. These neurons enable individuals to understand actions performed by others. Two subcategories of mirror neurons in monkeys activate when they listen to action sounds and when they observe communicative gestures made by others, respectively. The properties of mirror neurons could constitute a substrate from which more sophisticated forms of communication evolved; this would make sense, given the anatomical and functional homology between part of the monkey premotor cortex and Broca's area (the "speech" area of the brain) in humans. We hypothesize that several components of human language, including some aspects of phonology and syntax, could be embedded in the organizational properties of the motor system and that a deeper knowledge of this system could shed light on how language evolved.     

Motor functions of the Broca's region

Broca's region in the dominant cerebral hemisphere is known to mediate the production of language but also contributes to comprehension. This region evolved only in humans and is constituted of Brodmann's areas 44 and 45 in the inferior frontal gyrus. There is, however, evidence that Broca's region overlaps, at least in part, with the ventral premotor cortex. We summarize the evidence that the motor related part of Broca's area is localized in the opercular portion of the inferior frontal cortex, mainly in area 44 of Brodmann. According to our own data, there seems to be a homology between Brodmann area 44 in humans and the monkey area F5. The non-language related motor functions of Broca's region comprise complex hand movements, associative sensorimotor learning and sensorimotor integration. Brodmann's area 44 is also a part of a specialized parieto-premotor network and interacts significantly with the neighboring premotor areas.     

Making Sense of the Hands and Mouth: The Role of “Secondary” Cues to Meaning in British Sign Language and English

Successful face-to-face communication involves multiple channels, notably hand gestures in addition to speech for spoken language, and mouth patterns in addition to manual signs for sign language. In four experiments, we assess the extent to which comprehenders of British Sign Language (BSL) and English rely, respectively, on cues from the hands and the mouth in accessing meaning. We created congruent and incongruent combinations of BSL manual signs and mouthings and English speech and gesture by video manipulation and asked participants to carry out a picture-matching task. When participants were instructed to pay attention only to the primary channel, incongruent “secondary” cues still affected performance, showing that these are reliably used for comprehension. When both cues were relevant, the languages diverged: Hand gestures continued to be used in English, but mouth movements did not in BSL. Moreover, non-fluent speakers and signers varied in the use of these cues: Gestures were found to be more important for non-native than native speakers; mouth movements were found to be less important for non-fluent signers. We discuss the results in terms of the information provided by different communicative channels, which combine to provide meaningful information.     

Subcortical functions in language: A working model

The current paper explains a model of subcortical language functions that focuses on dynamic interactions between the cortex, the thalamus, and the basal ganglia in the production of spoken language. The model was derived from (a) studies of subcortical lesions and language, (b) studies of subcortical stimulation and language, (c) knowledge regarding neural pathways between various cortical and subcortical structures, and (d) indications that preverbal monitoring of language occurs. In the current model, the thalamus plays roles in cortical arousal and activation and in preverbal semantic monitoring. The basal ganglia function to regulate the degree of excitation conveyed from the thalamus to the cortex and to time the release of formulated language for motor programming. Consistency with classical syndromes of aphasia and potential applications to other areas in the neurosciences are discussed. The current theory, unlike previous formulations, is specific enough that testable hypotheses can be derived.     

Syntactic, prosodic, and semantic processes in the brain: evidence from event-related neuroimaging

The neural network supporting aspects of syntactic, prosodic, and semantic information processing is specified on the basis of two experiments using functional magnetic resonance imaging (fMRI). In these two studies, the presence/absence of lexical-semantic and syntactic information is systematically varied in spoken language stimuli. Inferior frontal and temporal brain areas in the left and the right hemisphere are identified to support different aspects of auditory language processing. Two additional experiments using event-related brain potentials investigate the possible interaction of syntactic and prosodic information, on the one hand, and syntactic and semantic information, on the other. While the first two information types were shown to interact early during processing, the latter two information types do not. Implications for models of auditory language comprehension are discussed.     

The Two-Level Theory of verb meaning: An approach to integrating the semantics of action with the mirror neuron system

Verbs have two separate levels of meaning. One level reflects the uniqueness of every verb and is called the "root". The other level consists of a more austere representation that is shared by all the verbs in a given class and is called the "event structure template". We explore the following hypotheses about how, with specific reference to the motor features of action verbs, these two distinct levels of semantic representation might correspond to two distinct levels of the mirror neuron system. Hypothesis 1: Root-level motor features of verb meaning are partially subserved by somatotopically mapped mirror neurons in the left primary motor and/or premotor cortices. Hypothesis 2: Template-level motor features of verb meaning are partially subserved by representationally more schematic mirror neurons in Brodmann area 44 of the left inferior frontal gyrus. Evidence has been accumulating in support of the general neuroanatomical claims made by these two hypotheses-namely, that each level of verb meaning is associated with the designated cortical areas. However, as yet no studies have satisfied all the criteria necessary to support the more specific neurobiological claims made by the two hypotheses-namely, that each level of verb meaning is associated with mirror neurons in the pertinent brain regions. This would require demonstrating that within those regions the same neuronal populations are engaged during (a) the linguistic processing of particular motor features of verb meaning, (b) the execution of actions with the corresponding motor features, and (c) the observation of actions with the corresponding motor features.     

Semantic processing of mathematical gestures

Objective

To examine whether or not university mathematics students semantically process gestures depicting mathematical functions (mathematical gestures) similarly to the way they process action gestures and sentences. Semantic processing was indexed by the N400 effect.

Results

The N400 effect elicited by words primed with mathematical gestures (e.g. “converging” and “decreasing”) was the same in amplitude, latency and topography as that elicited by words primed with action gestures (e.g. drive and lift), and that for terminal words of sentences.

Significance and conclusion

Findings provide a within-subject demonstration that the topographies of the gesture N400 effect for both action and mathematical words are indistinguishable from that of the standard language N400 effect. This suggests that mathematical function words are processed by the general language semantic system and do not appear to involve areas involved in other mathematical concepts (e.g. numerosity).     

Test-retest reliability in fMRI of language: group and task effects

This paper explores how the test-retest reliability is modulated by different groups of participants and experimental tasks. A group of 12 healthy participants and a group of nine stroke patients performed the same language imaging experiment twice, test and retest, on different days. The experiment consists of four conditions, one audio condition and three audiovisual conditions in which the hands are either resting, gesturing, or performing self-adaptive movements. Imaging data were analyzed using multiple linear regression and the results were further used to generate receiver operating characteristic (ROC) curves for each condition for each individual subject. By using area under the curve as a comparison index, we found that stroke patients have less reliability across time than healthy participants, and that when the participants gesture during speech, their imaging data are more reliable than when they are performing hand movements that are not speech-associated. Furthermore, inter-subject variability is less in the gesture task than in any of the other three conditions for healthy participants, but not for stroke patients.     

The use of hand gestures as self-generated cues for recall of verbally associated targets

This study examined the effects of hand gestures as cues for recall of 40 previously described abstract and concrete words. Participants were either self-cued (SC) with their own self-generated gestures, other-cued (OC) with someone else's gestures, or shown no cues (NC). The SC group had significantly better recall for both word types than either the OC or NC immediately and at a 2-week retrieval interval. Results also showed that when the SC group produced a meaningful gesture, concrete words were cued significantly more often than abstract words, but when total recall (cued and residual) was considered, abstract words were recalled equally well. These results are discussed in the context that hand gestures are a component of subjective organization and are thus distinctive cues for the producer that may facilitate or prime recall.     

“动手不动口”：手部动作与语言进化的关系

语言进化是进化心理学研究领域的重要问题。镜像系统假说、工具制造假说与传授假说从不同角度对手部动作与语言进化间的关系进行了解释, 三种假说都认为人类语言起源于手部动作经验。相关实证研究发现：手语与口语具有一致性特征、语言与手部动作具有共同的神经基础、手势发展可以预测语言发展水平以及手势可以提高工具制造知识的传播效率, 这些研究为三种假说的具体观点提供了实证支持。未来该领域的研究需要关注手势语与口语在进化中的发展关系, 以及人类语言进化与其他认知特征的进化关系。