Intentional and attentional dynamics of speech-hand coordination

Interest is rapidly growing in the hypothesis that natural language emerged from a more primitive set of linguistic acts based primarily on manual activity and hand gestures. Increasingly, researchers are investigating how hemispheric asymmetries are related to attentional and manual asymmetries (i.e., handedness). Both speech perception and production have origins in the dynamical generative movements of the vocal tract known as articulatory gestures. Thus, the notion of a "gesture" can be extended to both hand movements and speech articulation. The generative actions of the hands and vocal tract can therefore provide a basis for the (direct) perception of linguistic acts. Such gestures are best described using the methods of dynamical systems analysis since both perception and production can be described using the same commensurate language. Experiments were conducted using a phase transition paradigm to examine the coordination of speech-hand gestures in both left- and right-handed individuals. Results address coordination (in-phase vs. anti-phase), hand (left vs. right), lateralization (left vs. right hemisphere), focus of attention (speech vs. tapping), and how dynamical constraints provide a foundation for human communicative acts. Predictions from the asymmetric HKB equation confirm the attentional basis of functional asymmetry. Of significance is a new understanding of the role of perceived synchrony (p-centres) during intentional cases of gestural coordination.     

Spoken language and arm gestures are controlled by the same motor control system

Arm movements can influence language comprehension much as semantics can influence arm movement planning. Arm movement itself can be used as a linguistic signal. We reviewed neurophysiological and behavioural evidence that manual gestures and vocal language share the same control system. Studies of primate premotor cortex and, in particular, of the so-called "mirror system", including humans, suggest the existence of a dual hand/mouth motor command system involved in ingestion activities. This may be the platform on which a combined manual and vocal communication system was constructed. In humans, speech is typically accompanied by manual gesture, speech production itself is influenced by executing or observing transitive hand actions, and manual actions play an important role in the development of speech, from the babbling stage onwards. Behavioural data also show reciprocal influence between word and symbolic gestures. Neuroimaging and repetitive transcranial magnetic stimulation (rTMS) data suggest that the system governing both speech and gesture is located in Broca's area. In general, the presented data support the hypothesis that the hand motor-control system is involved in higher order cognition.     

The effects of prohibiting gestures on children's lexical retrieval ability

Two alternative accounts have been proposed to explain the role of gestures in thinking and speaking. The Information Packaging Hypothesis (Kita, 2000) claims that gestures are important for the conceptual packaging of information before it is coded into a linguistic form for speech. The Lexical Retrieval Hypothesis (Rauscher, Krauss & Chen, 1996) sees gestures as functioning more at the level of speech production in helping the speaker to find the right words. The latter hypothesis has not been fully explored with children. In this study children were given a naming task under conditions that allowed and restricted gestures. Children named more words correctly and resolved more 'tip-of-the-tongue' states when allowed to gesture than when not, suggesting that gestures facilitate access to the lexicon in children and are important for speech production as well as conceptualization.     

The role of gestures in spatial working memory and speech

Co-speech gestures traditionally have been considered communicative, but they may also serve other functions. For example, hand-arm movements seem to facilitate both spatial working memory and speech production. It has been proposed that gestures facilitate speech indirectly by sustaining spatial representations in working memory. Alternatively, gestures may affect speech production directly by activating embodied semantic representations involved in lexical search. Consistent with the first hypothesis, we found participants gestured more when describing visual objects from memory and when describing objects that were difficult to remember and encode verbally. However, they also gestured when describing a visually accessible object, and gesture restriction produced dysfluent speech even when spatial memory was untaxed, suggesting that gestures can directly affect both spatial memory and lexical retrieval.     

Neural evidence for the interplay between language, gesture, and action: a review

Co-speech gestures embody a form of manual action that is tightly coupled to the language system. As such, the co-occurrence of speech and co-speech gestures is an excellent example of the interplay between language and action. There are, however, other ways in which language and action can be thought of as closely related. In this paper we will give an overview of studies in cognitive neuroscience that examine the neural underpinnings of links between language and action. Topics include neurocognitive studies of motor representations of speech sounds, action-related language, sign language and co-speech gestures. It will be concluded that there is strong evidence on the interaction between speech and gestures in the brain. This interaction however shares general properties with other domains in which there is interplay between language and action.     

When Do Speakers Use Gestures to Specify Who Does What to Whom? The Role of Language Proficiency and Type of Gestures in Narratives

Previous research has found that iconic gestures (i.e., gestures that depict the actions, motions or shapes of entities) identify referents that are also lexically specified in the co-occurring speech produced by proficient speakers. This study examines whether concrete deictic gestures (i.e., gestures that point to physical entities) bear a different kind of relation to speech, and whether this relation is influenced by the language proficiency of the speakers. Two groups of speakers who had different levels of English proficiency were asked to retell a story in English. Their speech and gestures were transcribed and coded. Our findings showed that proficient speakers produced concrete deictic gestures for referents that were not specified in speech, and iconic gestures for referents that were specified in speech, suggesting that these two types of gestures bear different kinds of semantic relations with speech. In contrast, less proficient speakers produced concrete deictic gestures and iconic gestures whether or not referents were lexically specified in speech. Thus, both type of gesture and proficiency of speaker need to be considered when accounting for how gesture and speech are used in a narrative context.     

Gesture and metaphor comprehension: Electrophysiological evidence of cross-modal coordination by audiovisual stimulation

In recent years, studies have suggested that gestures influence comprehension of linguistic expressions, for example, eliciting an N400 component in response to a speech/gesture mismatch. In this paper, we investigate the role of gestural information in the understanding of metaphors. Event related potentials (ERPs) were recorded while participants viewed video clips of an actor uttering metaphorical expressions and producing bodily gestures that were congruent or incongruent with the metaphorical meaning of such expressions. This modality of stimuli presentation allows a more ecological approach to meaning integration. When ERPs were calculated using gesture stroke as time-lock event, gesture incongruity with metaphorical expression modulated the amplitude of the N400 and of the late positive complex (LPC). This suggests that gestural and speech information are combined online to make sense of the interlocutor’s linguistic production in an early stage of metaphor comprehension. Our data favor the idea that meaning construction is globally integrative and highly context-sensitive.     

Co‐development of child‐mother gestures over the second and the third years

This study looks at whether there is a relationship between mother and infant gesture production. Specifically, it addresses the extent of articulation in the maternal gesture repertoire and how closely it supports the infant production of gestures. Eight Spanish mothers and their 1‐ and 2‐year‐old babies were studied during 1 year of observations. Maternal and child verbal production, gestures and actions were recorded at their homes on five occasions while performing daily routines. Results indicated that mother and child deictic gestures (pointing and instrumental) and representational gestures (symbolic and social) were very similar at each age group and did not decline across groups. Overall, deictic gestures were more frequent than representational gestures. Maternal adaptation to developmental changes is specific for gesturing but not for acting. Maternal and child speech were related positively to mother and child pointing and representational gestures, and negatively to mother and child instrumental gestures. Mother and child instrumental gestures were positively related to action production, after maternal and child speech was partialled out. Thus, language plays an important role for dyadic communicative activities (gesture–gesture relations) but not for dyadic motor activities (gesture–action relations). Finally, a comparison of the growth curves across sessions showed a closer correspondence for mother–child deictic gestures than for representational gestures. Overall, the results point to the existence of an articulated maternal gesture input that closely supports the child gesture production. Copyright © 2006 John Wiley & Sons, Ltd.     

Spoken and gestural production in a naming task by young children with Down syndrome

Lexical production in children with Down syndrome (DS) was investigated by examining spoken naming accuracy and the use of spontaneous gestures in a picture naming task. Fifteen children with DS (range 3.8-8.3 years) were compared to typically developing children (TD), matched for chronological age and developmental age (range 2.6-4.3 years). Relative to TD children, children with DS were less accurate in speech (producing a greater number of unintelligible answers), yet they produced more gestures overall and of these a significantly higher percentage of iconic gestures. Furthermore, the iconic gestures produced by children with DS accompanied by incorrect or no speech often expressed a concept similar to that of the target word, suggesting deeper conceptual knowledge relative to that expressed only in speech.     

Gesturing saves cognitive resources when talking about nonpresent objects

In numerous experimental contexts, gesturing has been shown to lighten a speaker's cognitive load. However, in all of these experimental paradigms, the gestures have been directed to items in the "here-and-now." This study attempts to generalize gesture's ability to lighten cognitive load. We demonstrate here that gesturing continues to confer cognitive benefits when speakers talk about objects that are not present, and therefore cannot be directly indexed by gesture. These findings suggest that gesturing confers its benefits by more than simply tying abstract speech to the objects directly visible in the environment. Moreover, we show that the cognitive benefit conferred by gesturing is greater when novice learners produce gestures that add to the information expressed in speech than when they produce gestures that convey the same information as speech, suggesting that it is gesture's meaningfulness that gives it the ability to affect working memory load.     

The seeds of spatial grammar in the manual modality

Sign languages modulate the production of signs in space and use this spatial modulation to refer back to entities—to maintain coreference. We ask here whether spatial modulation is so fundamental to language in the manual modality that it will be invented by individuals asked to create gestures on the spot. English speakers were asked to describe vignettes under 2 conditions: using gesture without speech, and using speech with spontaneous gestures. When using gesture alone, adults placed gestures for particular entities in non-neutral locations and then used those locations to refer back to the entities. When using gesture plus speech, adults also produced gestures in non-neutral locations but used the locations coreferentially far less often. When gesture is forced to take on the full burden of communication, it exploits space for coreference. Coreference thus appears to be a resilient property of language, likely to emerge in communication systems no matter how simple.     

Voluntary stuttering suppresses true stuttering: a window on the speech perception-production link

In accord with a proposed innate link between speech perception and production (e.g., motor theory), this study provides compelling evidence for the inhibition of stuttering events in people who stutter prior to the initiation of the intended speech act, via both the perception and the production of speech gestures. Stuttering frequency during reading was reduced in 10 adults who stutter by approximately 40% in three of four experimental conditions: (1) following passive audiovisual presentation (i.e., viewing and hearing) of another person producing pseudostuttering (stutter-like syllabic repetitions) and following active shadowing of both (2) pseudostuttered and (3) fluent speech. Stuttering was not inhibited during reading following passive audiovisual presentation of fluent speech. Syllabic repetitions can inhibit stuttering both when produced and when perceived, and we suggest that these elementary stuttering forms may serve as compensatory speech gestures for releasing involuntary stuttering blocks by engaging mirror neuronal systems that are predisposed for fluent gestural imitation.     

Gesture and early bilingual development

The relationship between speech and gestural proficiency was investigated longitudinally (from 2 years to 3 years 6 months, at 6-month intervals) in 5 French-English bilingual boys with varying proficiency in their 2 languages. Because of their different levels of proficiency in the 2 languages at the same age, these children's data were used to examine the relative contribution of language and cognitive development to gestural development. In terms of rate of gesture production, rate of gesture production with speech, and meaning of gesture and speech, the children used gestures much like adults from 2 years on. In contrast, the use of iconic and beat gestures showed differential development in the children's 2 languages as a function of mean length of utterance. These data suggest that the development of these kinds of gestures may be more closely linked to language development than other kinds (such as points). Reasons why this might be so are discussed.     

Metaphor explanation attenuates the right-hand preference for depictive co-speech gestures that imitate actions

Differential activation levels of the two hemispheres due to hemispheric specialization for various linguistic processes might determine hand choice for co-speech gestures. To test this hypothesis, we compared hand choices for gesturing in 20 healthy right-handed participants during explanation of metaphorical vs. non-metaphorical meanings, on the assumption that metaphor explanation enhances the right hemisphere contribution to speech production. Hand choices were analyzed separately for: depictive gestures that imitate action ("character viewpoint gestures," [McNeill, D. (1992). Hand and mind. What gestures reveal about thought. Chicago: University of Chicago Press.]), depictive gestures that express motion, relative locations, and shape ("observer viewpoint gestures"), and "abstract deictic gestures." It was found that the right-hand over left-hand preference was significantly weaker in the metaphor condition than in the non-metaphor conditions for depictive gestures that imitated action. Findings suggest that the activation of the right hemisphere in the metaphor condition reduces the likelihood of left hemisphere generation of gestures that imitate action, thus attenuating the right-hand preference.     

言语与手部运动关系的研究回顾

言语与手部运动之间存在复杂的联系。该文总结了两类手部运动（伴随言语发生的手势运动和抓握运动）与言语之间关系的行为和脑科学研究成果。发现：（1）伴随言语产生的意义手势可促进言语加工,特别是词汇的提取过程;（2）观察手的抓握运动影响言语产生时唇的运动和声音成分;（3）对词语的知觉影响抓握运动的早期计划阶段;（4）言语产生可增加手运动皮层的兴奋性。作者由此认为,言语加工与手势间的联系不仅表现为神经通路的重叠和相互激活,而且可能在外显行为上也相互影响     

Pre-schoolers use head gestures rather than prosodic cues to highlight important information in speech

Previous evidence suggests that children's mastery of prosodic modulations to signal the informational status of discourse referents emerges quite late in development. In the present study, we investigate the children's use of head gestures as it compares to prosodic cues to signal a referent as being contrastive relative to a set of possible alternatives. A group of French-speaking pre-schoolers were audio-visually recorded while playing in a semi-spontaneous but controlled production task, to elicit target words in the context of broad focus, contrastive focus, or corrective focus utterances. We analysed the acoustic features of the target words (syllable duration and word-level pitch range), as well as the head gesture features accompanying these target words (head gesture type, alignment patterns with speech). We found that children's production of head gestures, but not their use of either syllable duration or word-level pitch range, was affected by focus condition. Children mostly aligned head gestures with relevant speech units, especially when the target word was in phrase-final position. Moreover, the presence of a head gesture was linked to greater syllable duration patterns in all focus conditions. Our results show that (a) 4- and 5-year-old French-speaking children use head gestures rather than prosodic cues to mark the informational status of discourse referents, (b) the use of head gestures may gradually entrain the production of adult-like prosodic features, and that (c) head gestures with no referential relation with speech may serve a linguistic structuring function in communication, at least during language development.     

The link between speech perception and production is phonological and abstract: evidence from the shadowing task

This study reports a shadowing experiment, in which one has to repeat a speech stimulus as fast as possible. We tested claims about a direct link between perception and production based on speech gestures, and obtained two types of counterevidence. First, shadowing is not slowed down by a gestural mismatch between stimulus and response. Second, phonetic detail is more likely to be imitated in a shadowing task if it is phonologically relevant. This is consistent with the idea that speech perception and speech production are only loosely coupled, on an abstract phonological level.     

儿童手势及其与学习的关系

手势是在交流或认知过程中产生的、不直接作用于物体的手部运动, 具有具体性和抽象性。其分类主要从手势的来源、手势的内容、手势的意图及手势和言语的匹配性等角度进行划分。不同类型手势在出现时间及发展趋势上存在差异。手势在儿童词汇学习、言语表达、数学问题解决、空间学习及记忆等方面起促进作用, 但对言语理解的影响未得出一致结论。未来可关注不同类型手势与儿童认知发展的关系及对比不同来源手势在各学习领域的优势情况。     

Characteristics of velocity profiles of speech movements

The control of individual speech gestures was investigated by examining laryngeal and tongue movements during vowel and consonant production. A number of linguistic manipulations known to alter the durational characteristics of speech (i.e., speech rate, lexical stress, and phonemic identity) were tested. In all cases a consistent pattern was observed in the kinematics of the laryngeal and tongue gestures. The ratio of maximum instantaneous velocity to movement amplitude, a kinematic index of mass-normalized stiffness, was found to increase systematically as movement duration decreased. Specifically, the ratio of maximum velocity to movement amplitude varied as a function of a parameter, C, times the reciprocal of movement duration. The conformity of the data to this relation indicates that durational change is accomplished by scalar adjustment of a base velocity form. These findings are consistent with the idea that kinematic change is produced by the specification of articulator stiffness.