Read my lips: asymmetries in the visual expression and perception of speech revealed through the McGurk effect

It is well known that the right side of the mouth moves more than the left during speech, but little is known about how this asymmetry affects lipreading. We investigated asymmetries in the visual expression and perception of speech using the McGurk effect-an illusion in which incongruent lip movements cause listeners to misreport sounds. Thirty right-handed participants watched film clips in which the left, the right, or neither side of the mouth was covered. The McGurk effect was attenuated when the right side of the mouth was covered, demonstrating that this side is more important to lipreading than is the left side of the mouth. Mirror-reversed images tested whether the asymmetry was the result of an observer bias toward the left hemispace. The McGurk effect was stronger in the normal than in the mirror orientation when the mouth was fully visible. Thus, observers attend more to what they think is the right side of the speaker's mouth. Asymmetries in mouth movements may reflect the gestural origins of language, which are also right lateralized.     

Cerebral specialization for speech production in persons with Down syndrome.

The study of cerebral specialization in persons with Down syndrome (DS) has revealed an anomalous pattern of organization. Specifically, dichotic listening studies (e.g., Elliott & Weeks, 1993) have suggested a left ear/right hemisphere dominance for speech perception for persons with DS. In the current investigation, the cerebral dominance for speech production was examined using the mouth asymmetry technique. In right-handed, nonhandicapped subjects, mouth asymmetry methodology has shown that during speech, the right side of the mouth opens sooner and to a larger degree then the left side (Graves, Goodglass, & Landis, 1982). The phenomenon of right mouth asymmetry (RMA) is believed to reflect the direct access that the musculature on the right side of the face has to the left hemisphere's speech production systems. This direct access may facilitate the transfer of innervatory patterns to the muscles on the right side of the face. In the present study, the lateralization for speech production was investigated in 10 right-handed participants with DS and 10 nonhandicapped subjects. A RMA at the initiation and end of speech production occurred for subjects in both groups. Surprisingly, the degree of asymmetry between groups did not differ, suggesting that the lateralization of speech production is similar for persons with and persons without DS. These results support the biological dissociation model (Elliott, Weeks, & Elliott, 1987), which holds that persons with DS display a unique dissociation between speech perception (right hemisphere) and speech production (left hemisphere).     

Audiovisual speech from emotionally expressive and lateralized faces

Emotional expression and how it is lateralized across the two sides of the face may influence how we detect audiovisual speech. To investigate how these components interact we conducted experiments comparing the perception of sentences expressed with happy, sad, and neutral emotions. In addition we isolated the facial asymmetries for affective and speech processing by independently testing the two sides of a talker's face. These asymmetrical differences were exaggerated using dynamic facial chimeras in which left- or right-face halves were paired with their mirror image during speech production. Results suggest that there are facial asymmetries in audiovisual speech such that the right side of the face and right-facial chimeras supported better speech perception than their left-face counterparts. Affective information was also found to be critical in that happy expressions tended to improve speech performance on both sides of the face relative to all other emotions, whereas sad emotions generally inhibited visual speech information, particularly from the left side of the face. The results suggest that approach information may facilitate visual and auditory speech detection.     

When half a face is as good as a whole: Effects of simple substantial occlusion on visual and audiovisual speech perception

The visible movement of a talker's face is an influential component of speech perception. However, the ability of this influence to function when large areas of the face (~50%) are covered by simple substantial occlusions, and so are not visible to the observer, has yet to be fully determined. In Experiment 1, both visual speech identification and the influence of visual speech on identifying congruent and incongruent auditory speech were investigated using displays of a whole (unoccluded) talking face and of the same face occluded vertically so that the entire left or right hemiface was covered. Both the identification of visual speech and its influence on auditory speech perception were identical across all three face displays. Experiment 2 replicated and extended these results, showing that visual and audiovisual speech perception also functioned well with other simple substantial occlusions (horizontal and diagonal). Indeed, displays in which entire upper facial areas were occluded produced performance levels equal to those obtained with unoccluded displays. Occluding entire lower facial areas elicited some impairments in performance, but visual speech perception and visual speech influences on auditory speech perception were still apparent. Finally, implications of these findings for understanding the processes supporting visual and audiovisual speech perception are discussed.     

Audiovisual speech from emotionally expressive and lateralized faces

Emotional expression and how it is lateralized across the two sides of the face may influence how we detect audiovisual speech. To investigate how these components interact we conducted experiments comparing the perception of sentences expressed with happy, sad, and neutral emotions. In addition we isolated the facial asymmetries for affective and speech processing by independently testing the two sides of a talker's face. These asymmetrical differences were exaggerated using dynamic facial chimeras in which left- or right-face halves were paired with their mirror image during speech production. Results suggest that there are facial asymmetries in audiovisual speech such that the right side of the face and right-facial chimeras supported better speech perception than their left-face counterparts. Affective information was also found to be critical in that happy expressions tended to improve speech performance on both sides of the face relative to all other emotions, whereas sad emotions generally inhibited visual speech information, particularly from the left side of the face. The results suggest that approach information may facilitate visual and auditory speech detection.     

Face and mouth inversion effects on visual and audiovisual speech perception

Three experiments examined whether image manipulations known to disrupt face perception also disrupt visual speech perception. Research has shown that an upright face with an inverted mouth looks strikingly grotesque whereas an inverted face and an inverted face containing an upright mouth look relatively normal. The current study examined whether a similar sensitivity to upright facial context plays a role in visual speech perception. Visual and audiovisual syllable identification tasks were tested under 4 presentation conditions: upright face-upright mouth, inverted face-inverted mouth, inverted face-upright mouth, and upright face-inverted mouth. Results revealed that for some visual syllables only the upright face-inverted mouth image disrupted identification. These results suggest that upright facial context can play a role in visual speech perception. A follow-up experiment testing isolated mouths supported this conclusion.     

Visuospatial processing and the right-hemisphere interpreter

Popular views of hemispheric asymmetry hold that the left hemisphere is specialized for linguistic and cognitive processes and fine motor control, whereas the right is specialized for visuospatial processing. Although this dichotomy contains more than a grain of truth, it is an oversimplification. Experiments with split-brain patients have demonstrated that the left hemisphere retains relatively sophisticated visuospatial abilities, and that the asymmetries that favor the right hemisphere are subtler than those that favor the left. A consideration of the constructive nature of visual perception, and the organization of the visual system in the two hemispheres suggests that asymmetries are likely to arise relatively late in visual processing in areas that represent both sides of visual space. I present evidence in favor of the view that the right hemisphere can be considered more "visually intelligent" than the left, and postulate the existence of a "right-hemisphere interpreter" dedicated to constructing a representation of the visual world.     

ASYMMETRY IN MOUTH OPENING DURING DIFFERENT SPEECH TASKS

When a normal subject is speaking, the right side of the mouth typically opens more widely or moves over a greater total distance. This asymmetry is most consistent during purely verbal word list generation and verbal recall tasks, less consistent when emotional expression and/or visual imagery is involved, and reversed during smiling. Aphasic patients also show the right bias during word lists, repetition, and conversation, but not during serial speech, singing and smiling. Since observable mouth asymmetry is presumed to result from hemispheric asymmetry in motor control, these observations confirm the major role of a left hemisphere control system for pure verbal expression and provide evidence for involvement of the right hemisphere in mouth motor control during emotional and prosodic expression or visual imagery. Therapeutic possibilities are also suggested.     

Eye movements of monkey observers viewing vocalizing conspecifics

Primates, including humans, communicate using facial expressions, vocalizations and often a combination of the two modalities. For humans, such bimodal integration is best exemplified by speech-reading - humans readily use facial cues to enhance speech comprehension, particularly in noisy environments. Studies of the eye movement patterns of human speech-readers have revealed, unexpectedly, that they predominantly fixate on the eye region of the face as opposed to the mouth. Here, we tested the evolutionary basis for such a behavioral strategy by examining the eye movements of rhesus monkeys observers as they viewed vocalizing conspecifics. Under a variety of listening conditions, we found that rhesus monkeys predominantly focused on the eye region versus the mouth and that fixations on the mouth were tightly correlated with the onset of mouth movements. These eye movement patterns of rhesus monkeys are strikingly similar to those reported for humans observing the visual components of speech. The data therefore suggest that the sensorimotor strategies underlying bimodal speech perception may have a homologous counterpart in a closely related primate ancestor.     

Intentional and attentional dynamics of speech-hand coordination

Interest is rapidly growing in the hypothesis that natural language emerged from a more primitive set of linguistic acts based primarily on manual activity and hand gestures. Increasingly, researchers are investigating how hemispheric asymmetries are related to attentional and manual asymmetries (i.e., handedness). Both speech perception and production have origins in the dynamical generative movements of the vocal tract known as articulatory gestures. Thus, the notion of a "gesture" can be extended to both hand movements and speech articulation. The generative actions of the hands and vocal tract can therefore provide a basis for the (direct) perception of linguistic acts. Such gestures are best described using the methods of dynamical systems analysis since both perception and production can be described using the same commensurate language. Experiments were conducted using a phase transition paradigm to examine the coordination of speech-hand gestures in both left- and right-handed individuals. Results address coordination (in-phase vs. anti-phase), hand (left vs. right), lateralization (left vs. right hemisphere), focus of attention (speech vs. tapping), and how dynamical constraints provide a foundation for human communicative acts. Predictions from the asymmetric HKB equation confirm the attentional basis of functional asymmetry. Of significance is a new understanding of the role of perceived synchrony (p-centres) during intentional cases of gestural coordination.     

Motor control by vision and the evolution of cerebral lateralization

Chicks (4 or 5 days old), which are able to use either eye freely, use the right eye (RE) preferentially in approach to a food dish when a lid, which has to be removed, is visible during approach. They use the left eye (LE) instead when no manipulation is required, but the same dish is similarly visible. The RE is also used preferentially in selecting food grains scattered over the floor; RE use in these two contexts is thus associated with visual control which brings the bill in planned contact with a visible target rather than with approach to a site where it is anticipated that feeding will occur. Zebrafish also use the RE preferentially when preparing to bite a target; during purely visual examination of the same target, this preference disappears. This evidence is used together with evolutionary evidence to support a new hypothesis for the origin of cerebral lateralization: paired anterior eyes evolved in filter-feeding ancestors of the vertebrates as part of the acquisition of prey catching. A key use for early vision was to predict likely contact with prey so as to inhibit reflexes of rejection and avoidance normally elicitated by tactile input to the mouth and so to allow ingestion. Innervation of mouth structures by the left side of the CNS caused control of mouth reflexes to become predominantly a left CNS affair. As visual abilities developed this starting condition meant that control of manipulation (which is by the mouth for most vertebrates) remained predominantly with the left side of the CNS.     

Goal-Directed Aiming: Correcting a Force-Specification Error With the Right and Left Hands

In 2 experiments, the authors examined manual aiming asymmetries as well as the ability of participants to adjust their aiming trajectories following an unexpected change to the inertial resistance to movement. In Experiment 1, participants (N = 11) were able to rapidly adjust their movement trajectories to conform to the new movement requirements. They were faster and more consistent when aiming with their right hand than with their left hand, regardless of whether or not the movement was perturbed. In Experiment 2, participants' (N = 11) vision of the hand was manipulated so that the role of visual feedback in the corrective process could be examined. Vision had an impact not only on performance but also on the characteristics of the movement trajectories. Manual asymmetries in aiming were associated with a right hand superiority during the final corrective stages of the movement.     

Hemispheric lateralization in 47,XXY Klinefelter's syndrome boys

Thirty-two boys with a 47,XXY karyotype were compared with chromosomally normal male controls in their performance on six tasks of hemispheric specialization. The results revealed that the 47,XXY subjects had smaller asymmetries on left hemisphere tasks and larger asymmetries on right hemisphere tasks than controls. Analyses of individual right and left side scores revealed that the atypical lateral asymmetries of the 47,XXYs were due to a shift toward greater right hemisphere involvement on four of the six measures. It was postulated that the slower fetal growth rates of the extra X chromosome group might contribute to their atypical hemispheric specialization and the failure of their left hemisphere to gain dominance over their right in language processing.     

An examination of the right-hemisphere hypothesis of the lateralization of emotion

The Right-Hemisphere Hypothesis posits that emotional stimuli are perceived more efficiently by the right hemisphere than by the left hemisphere. The current research examines this hypothesis by examining hemispheric asymmetries for the conscious and unconscious perception of emotional stimuli. Negative, positive, and neutral words were presented for 17 ms to one visual field or the other. Conscious perception was measured by using a subjective report-of-awareness measure reported by participants on each trial. Unconscious perception was measured using an "exclusion task," a form of word-stem-completion task. Consistent with previous research, there was a right-hemisphere advantage for the conscious perception of negative information. As in previous studies, this advantage for conscious perception occurred at the expense of unconscious perception. Specifically, there was a right-hemisphere inferiority for the unconscious perception of negative information. Contrary to the predictions of the Right-Hemisphere Hypothesis, there were no hemispheric asymmetries for the perception of positive emotional information, thus suggesting that the Right-Hemisphere Hypothesis may not be applicable to all behavioral studies.     

Speech dominance and handedness in the normal human

Spectral analysis was used to measure the coherence or similarity of form between occipital and temporal evoked potentials. In both right- and left-handers, coherence was greater in the left hemisphere for click stimuli and in the right hemisphere for flash stimuli. Similar asymmetries occurred in left but not right speech dominant patients whose speech lateralization has been determined by the intracarotid amytal test. Right-handers and males, however, showed significantly larger amplitudes of auditory responses in the right hemisphere. Left-handers and females reversed this pattern. It was concluded that within a basically left speech dominant organization, males and right-handers would emphasize the verbal, temporal structure of auditory information and the nonverbal, spatial structure of visual information. Females and left-handers would tend to reverse this emphasis.     

Starting with the "right" foot minimizes sprint start time

Although asymmetries in hand and foot performance have been examined using a variety of movement tasks that require the fine control of the timing and amplitude of force generation, foot asymmetries in a functional gross motor movement task, such as the track and field sprint start, have yet to be examined. Twenty individuals (10 experienced, 10 inexperienced) were assessed for pedal asymmetries using the track and field sprint start. Each participant performed 48 starts (24 right foot starts and 24 left foot starts). The pattern of pedal asymmetries was consistent with that of manual asymmetries in that a left foot (i.e., left foot in rear position) reaction time advantage was found while there was a right foot (i.e., right foot in rear position) advantage for movement time and total response time (time from stimulus presentation until the end of the movement). These results are consistent with a right hemisphere specialization for spatio-temporal and attentional processes, and a left hemisphere specialization for movement execution.     

Automatic attention lateral asymmetry in visual discrimination tasks

There is evidence that automatic visual attention favors the right side. This study investigated whether this lateral asymmetry interacts with the right hemisphere dominance for visual location processing and left hemisphere dominance for visual shape processing. Volunteers were tested in a location discrimination task and a shape discrimination task. The target stimuli (S2) could occur in the left or right hemifield. They were preceded by an ipsilateral, contralateral or bilateral prime stimulus (S1). The attentional effect produced by the right S1 was larger than that produced by the left S1. This lateral asymmetry was similar between the two tasks suggesting that the hemispheric asymmetries of visual mechanisms do not contribute to it. The finding that it was basically due to a longer reaction time to the left S2 than to the right S2 for the contralateral S1 condition suggests that the inhibitory component of attention is laterally asymmetric.     

Visual line bisection in sinistrals and dextrals as a function of hemispace, hand, and scan direction

Visual line bisection was investigated in 26 sinistral and 24 dextral subjects as a function of hemispace, hand and scan direction. An ANOVA revealed significant main effects for hand preference, due to the mean bisection errors of dextral subjects being significantly leftward of those of sinistral subjects; for hand, due to the bisection errors of the left hand being significantly to the left of the right hand; and for scan, due to the bisection errors following a left scan being significantly to the left of a right scan. One significant interaction was found, that between hand and direction of scan, due to a significant difference between left and right hands following a scan from the left but not following a scan from the right. For dextral subjects the leftward bisection errors of the left and right hands following a scan from the left, but not for a scan from the right, differed significantly from the midpoint. For sinistral subjects the leftward bisection errors following a scan from the left and rightward bisection errors following a scan from the right differed significantly from the midpoint for the left hand but not for the right hand. No significant main effect or interactions for hemispace were found. This confirms that both sinistral and dextral subjects display pseudoneglect when using their preferred hand and scanning from the left. However, sinistrals, but not dextrals, will display reversed pseudoneglect when using their preferred hand and adopting a scan direction from the right. These results are discussed in terms of the interaction between three factors, whose influence can jointly and severally produce misbisections, hemispheric specialisation for visuospatial function, hemispheric activation for a manual response, and the allocation of visual attention.     

A hemispheric asymmetry for the unconscious perception of emotion

Previous research has demonstrated that hemispheric asymmetries for conscious visual perception do not lead to asymmetries for unconscious visual perception. These studies utilized emotionally neutral items as stimuli. The current research utilized both emotionally negative and neutral stimuli to assess hemispheric differences for conscious and unconscious visual perception. Conscious perception was measured using a subjective measure of awareness reported by participants on each trial. Unconscious perception was measured by an "exclusion task," a form of word-stem-completion task. Consistent with predictions, negative stimuli were consciously perceived most often when presented to the right hemisphere. Negative stimuli presented to the right hemisphere showed no evidence of unconscious perception, suggesting that the hemispheric asymmetry for the conscious perception of negative information occurs at the expense of unconscious perception.