Seeing to hear better: evidence for early audio-visual interactions in speech identification

Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in noise [J. Acoust. Soc. Am. 109 (2001) 2272; J. Acoust. Soc. Am. 108 (2000) 1197]. However, detection is different from comprehension, and it remains to be seen whether improved sensitivity also results in an intelligibility gain in audio-visual speech perception. In this work, we use an original paradigm to show that seeing the speaker's lips enables the listener to hear better and hence to understand better. The audio-visual stimuli used here could not be differentiated by lip reading per se since they contained exactly the same lip gesture matched with different compatible speech sounds. Nevertheless, the noise-masked stimuli were more intelligible in the audio-visual condition than in the audio-only condition due to the contribution of visual information to the extraction of acoustic cues. Replacing the lip gesture by a non-speech visual input with exactly the same time course, providing the same temporal cues for extraction, removed the intelligibility benefit. This early contribution to audio-visual speech identification is discussed in relationships with recent neurophysiological data on audio-visual perception.     

Abortion, Intimacy, and the Duty to Gestate

In this article, I urge that mainstream discussions of abortion are dissatisfying in large part because they proceed in polite abstraction from the distinctive circumstances and meanings of gestation. Such discussions, in fact, apply to abortion conceptual tools that were designed on the premiss that people are physically demarcated, even as gestation is marked by a thorough-going intertwinement. We cannot fully appreciate what is normatively at stake with legally forcing continued gestation, or again how to discuss moral responsibilities to continue gestating, until we appreciate in their own terms the goods and evils distinctive of gestational connection. To underscore the need to explore further the meanings of gestation, I provide two examples of the difference it might make to legal and moral discussions of abortion if we appreciate more fully that gestation is an intimacy.     

The Bursts and Lulls of Multimodal Interaction: Temporal Distributions of Behavior Reveal Differences Between Verbal and Non‐Verbal Communication

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, 2012 ), we focus on measuring the temporal patterns of specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog.     

Performance on auditory and visual temporal information processing is related to psychometric intelligence

The present study investigated the relationship between psychometric intelligence and temporal resolution power (TRP) as simultaneously assessed by auditory and visual psychophysical timing tasks. In addition, three different theoretical models of the functional relationship between TRP and psychometric intelligence as assessed by means of the Adaptive Matrices Test (AMT) were developed. To test the validity of these models, structural equation modeling was applied. Empirical data supported a hierarchical model that assumed auditory and visual modality-specific temporal processing at a first level and amodal temporal processing at a second level. This second-order latent variable was substantially correlated with psychometric intelligence. Therefore, the relationship between psychometric intelligence and psychophysical timing performance can be explained best by a hierarchical model of temporal information processing.     

When two become one: Temporally dynamic integration of the face and voice

In everyday interactions with others, people have to deal with the sight of a face and sound of a voice at the same time. How the perceptual system brings this information together over hundreds of milliseconds to perceive others remains unclear. In 2 studies, we investigated how facial and vocal cues are integrated during real-time social categorization by recording participants' hand movements (via the streaming x, y coordinates of the computer mouse) en route to “male” and “female” responses on the screen. Participants were presented with male and female faces that were accompanied by a same-sex voice morphed to be either sex-typical (e.g., masculinized male voice) or sex-atypical (i.e., feminized male voice). Before settling into ultimate sex categorizations of the face, the simultaneous processing of a sex-atypical voice led the hand to be continuously attracted to the opposite sex-category response across construal. This is evidence that ongoing results from voice perception continuously influence face perception across processing. Thus, social categorization involves dynamic updates of gradual integration of the face and voice.     

Is depression the past tense of anxiety? An empirical study of the temporal distinction

It has long been recognised that depression and anxiety share a common core of negative affect, but research on similarities and differences between these two emotions is growing. The focus of the current study was on whether the timing of a triggering event can determine whether the dominant emotional reaction is depression or anxiety. It was hypothesised that aversive events in the past would elicit more depression than anxiety, whereas the same aversive events in the future would elicit more anxiety than depression. We created temporally varied versions of vignettes describing aversive events occurring at either time, and asked participants to rate the extent to which the events would elicit feelings of depression or anxiety. Results indicated that adverse past events elicited much higher ratings of anticipated depression and adverse future events elicited much higher ratings of anticipated anxiety. Implications for understanding these two emotions and depressive and anxiety disorders are discussed.     

One-year temporal stability and predictive and incremental validity of the Body,Eating, and Exercise Comparison Orientation Measure (BEECOM) among college women

This study examined the one-year temporal stability and the predictive and incremental validity of the Body, Eating, and Exercise Comparison Measure (BEECOM) in a sample of 237 college women who completed study measures at two time points about one year apart. One-year temporal stability was high for the BEECOM total and subscale (i.e., Body, Eating, and Exercise Comparison Orientation) scores. Additionally, the BEECOM exhibited predictive validity in that it accounted for variance in body dissatisfaction and eating disorder symptomatology one year later. These findings held even after controlling for body mass index and existing measures of social comparison orientation. However, results regarding the incremental validity of the BEECOM, or its ability to predict change in these constructs over time, were more mixed. Overall, this study demonstrated additional psychometric properties of the BEECOM among college women, further establishing the usefulness of this measure for more comprehensively assessing eating disorder-related social comparison.     

A dissociation between selective attention and conscious awareness in the representation of temporal order information

Previous electrophysiological studies have shown that attentional selection processes are highly sensitive to the temporal order of task-relevant visual events. When two successively presented colour-defined target stimuli are separated by a stimulus onset asynchrony (SOA) of only 10 ms, the onset latencies of N2pc components to these stimuli (which reflect their attentional selection) precisely match their objective temporal separation. We tested whether such small onset differences are accessible to conscious awareness by instructing participants to report the category (letter or digit) of the first of two target-colour items that were separated by an SOA of 10, 20, or 30 ms. Performance was at chance level for the 10 ms SOA, demonstrating that temporal order information which is available to attentional control processes cannot be utilized for conscious temporal order judgments. These results provide new evidence that selective attention and conscious awareness are functionally separable, and support the hypothesis that attention and awareness operate at different stages of cognitive processing.     

Examining the relationship between skilled music training and attention

While many aspects of cognition have been investigated in relation to skilled music training, surprisingly little work has examined the connection between music training and attentional abilities. The present study investigated the performance of skilled musicians on cognitively demanding sustained attention tasks, measuring both temporal and visual discrimination over a prolonged duration. Participants with extensive formal music training were found to have superior performance on a temporal discrimination task, but not a visual discrimination task, compared to participants with no music training. In addition, no differences were found between groups in vigilance decrement in either type of task. Although no differences were evident in vigilance per se, the results indicate that performance in an attention-demanding temporal discrimination task was superior in individuals with extensive music training. We speculate that this basic cognitive ability may contribute to advantages that musicians show in other cognitive measures.     

Dichotic listening with forced attention in patients with temporal lobe epilepsy: significance of left hemisphere cognitive dysfunction

Fifty right-handed patients with focal temporal lobe epilepsy were administered a dichotic listening test with consonant-vowel syllables under non-forced, forced right and forced left attention conditions, and a neuropsychological test battery. Dichotic listening performance was compared in subgroups with and without left hemisphere cognitive dysfunction, measured by the test battery, and in subgroups with left and right temporal epileptic focus. Left hemisphere cognitive dysfunction led to more correct responses to left ear stimuli in all three attention conditions, and fewer correct responses to right ear stimuli in the non-forced attention condition. This was probably caused by basic left hemisphere perceptual dysfunction. Dichotic listening was less affected by a left-sided epileptic focus than by left hemisphere cognitive dysfunction. General cognitive functioning influenced dichotic listening performance stronger in forced than in non-forced attention conditions. Larger cerebral networks were probably involved in the forced attention conditions due to the emphasis on conscious effort.