Seeing to hear better: evidence for early audio-visual interactions in speech identification

Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in noise [J. Acoust. Soc. Am. 109 (2001) 2272; J. Acoust. Soc. Am. 108 (2000) 1197]. However, detection is different from comprehension, and it remains to be seen whether improved sensitivity also results in an intelligibility gain in audio-visual speech perception. In this work, we use an original paradigm to show that seeing the speaker's lips enables the listener to hear better and hence to understand better. The audio-visual stimuli used here could not be differentiated by lip reading per se since they contained exactly the same lip gesture matched with different compatible speech sounds. Nevertheless, the noise-masked stimuli were more intelligible in the audio-visual condition than in the audio-only condition due to the contribution of visual information to the extraction of acoustic cues. Replacing the lip gesture by a non-speech visual input with exactly the same time course, providing the same temporal cues for extraction, removed the intelligibility benefit. This early contribution to audio-visual speech identification is discussed in relationships with recent neurophysiological data on audio-visual perception.     

Internal versus external attention in social anxiety: an investigation using a novel paradigm

Several cognitive models propose that social anxiety is associated with increased self-focused attention. Indirect evidence for this hypothesis has been provided by questionnaire studies, and by cognitive psychology paradigms that have demonstrated reduced processing of external information during feared social-evaluative situations. However, no studies have simultaneously measured on-line attention to internal and external events. A probe detection task that aimed to measure the balance of attention between internal and external stimuli was developed. High and low socially anxious individuals were instructed to detect two probes. The external probe was superimposed on pictures of faces (happy, neutral, angry) or household objects that were presented on a VDU. The 'internal' probe was a pulse to the finger which participants were led to believe represented significant changes in their physiology. Compared to low speech anxious individuals, high speech anxious individuals showed an internal attentional bias, that was specific to conditions of social-evaluative threat.     

Anxiety and selective attention in obsessive-compulsive disorder

Recently, there has been increasing evidence for information-processing deficits in individuals with obsessive-compulsive disorder (OCD). While impairments in selective attention have been identified to be central to the symptomatology of OCD, the role that situational anxiety plays in attentional processes has not been fully explored. Previous research findings were limited to tasks containing anxiety-relevant materials, only permitting for the evaluation of the impact of anxiety on simultaneous cognitive processing. Furthermore, it has not yet been determined whether the impact of anxiety is limited to selective attention or is indicative of a more general cognitive impairment. This study was designed to examine the role that situational anxiety plays in selective attention impairments. OCD participants and controls were presented with an anxiety producing statement and a neutral statement, followed by the Stroop Task. Results indicated that situational anxiety plays a significant role in the performance of tasks that require selective attention in OCD. A significant deterioration was detected in performance on selective attention tasks for the OCD participants after confronting anxiety-provoking scenarios, as compared to neutral scenarios. Anxiety did not impair performance on simple reading tasks. Possible explanations are discussed.     

Sequential evaluation of behavioral treatments and methylphenidate dosage for children with attention deficit hyperactivity disorder

We used a sequential approach to evaluate the relative and combined effects of different types of behavioral treatments, as well as dosage of methylphenidate (MPH), on the disruptive behavior of 3 students who had been diagnosed with attention deficit hyperactivity disorder. Results showed that individualized behavioral treatments produced decreases in disruptive behavior equivalent to MPH for all 3 participants and demonstrated the need to evaluate behavioral treatments and medication dosage on an individual basis.     

The effects of road traffic noise and meaningful irrelevant speech on different memory systems

To explore why noise has reliable effects on delayed recall in a certain text-reading task, this episodic memory task was employed with other memory tests in a study of road traffic noise and meaningful but irrelevant speech. Context-dependent memory was tested and self-reports of affect were taken. Participants were 96 high school students. The results showed that both road traffic noise and meaningful irrelevant speech impaired recall of the text. Retrieval in noise from semantic memory was also impaired. Attention was impaired by both noise sources, but attention did not mediate the noise effects on episodic memory. Recognition was not affected by noise. Context-dependent memory was not shown. The lack of mediation by attention, and road traffic noise being as harmful as meaningful irrelevant speech, are discussed in relation to where in the input/storing/output sequence noise has its effect and what the distinctive feature of the disturbing noise is.     

Global and local covert visual attention: Evidence from a bayesian hidden markov model

Psychological, psychophysical and physiological research indicates that people switch between two covert attention states, local and global attention, while visually exploring complex scenes. The focus in the local attention state is on specific aspects and details of the scene, and on examining its content with greater visual detail. The focus in the global attention state is on exploring the informative and perceptually salient areas of the scene, and possibly on integrating the information contained therein. The existence of these two visual attention states, their relative prevalence and sequence in time has remained empirically untested to date. To fill this gap, we develop a psychometric model of visual covert attention that extends recent work on hidden Markov models, and we test it using eye-movement data. The model aims to describe the observed time series of saccades typically collected in eye-movement research by assuming a latent Markov process, indicative of the brain switching between global and local covert attention. We allow subjects to be in either state while exploring a stimulus visually, and to switch between them an arbitrary number of times. We relax the no-memory-property of the Markov chain. The model that we develop is estimated with MCMC methodology and calibrated on eye-movement data collected in a study of consumers' attention to print advertisements in magazines.We thank Dominique Claessens of Verify International for making the data available to us.     

Deficit in selective and divided attention associated with cholinergic basal forebrain immunotoxic lesion produced by 192-saporin; motoric/sensory deficit associated with Purkinje cell immunotoxic lesion produced by OX7-saporin

The immunotoxin 192-saporin, infused intracerebroventricularly into rats, destroys cholinergic neurons in the basal forebrain nuclei. Doses required for complete cholinergic loss also kill some Purkinje cells. The immunotoxin OX7-saporin, when infused intraventricularly into rats, destroys Purkinje cells in a pattern similar to that produced by 192-saporin, without affecting cholinergic neurons. Thus, we used OX7-saporin to distinguish behavioral effects of 192-saporin due to cerebellar damage versus those due to cholinergic cell loss. Three doses of 192-saporin (1.6, 2.6, and 3.3 micrograms/rat) were chosen along with a dose of OX7-saporin (2.0 micrograms/rat) that produced Purkinje loss equivalent to the two highest doses of 192-saporin. Groups of Fischer-344 rats were trained in the multiple choice reaction time task and retested with more complex tasks after lesioning. They were also tested in the water maze, passive avoidance, acoustic startle, and open field. The OX7-saporin group exhibited changes in many tests suggesting hypermotility and sensory deficits. The 192-saporin groups differed from the OX7-saporin group when they displayed deficits in multiple choice reaction time tasks in which novel challenges were introduced, including sessions with a noise distractor, shortened and lengthened intertrial intervals, and use of nine instead of five sources of light stimulus. The 192-saporin groups showed no impairment in the other tasks. The cholinergic basal forebrain lesion may mask some of the effects of cerebellar damage up to a threshold after which effects of Purkinje cell loss predominate when 192-saporin is administered intraventricularly.     

Infant responses to adult happy and sad vocal and facial expressions during face-to-face interactions

We examined 5-month-olds’ responses to adult facial versus vocal displays of happy and sad expressions during face-to-face social interactions in three experiments. Infants interacted with adults in either happy-sad-happy or happy-happy-happy sequences. Across experiments, either facial expressions were present while presence/absence of vocal expressions was manipulated or visual access to facial expressions was blocked but vocal expressions were present throughout. Both visual attention and infant affect were recorded. Although infants looked more when vocal expressions were present, they smiled significantly more to happy than to sad facial expressions regardless of presence or absence of the voice. In contrast, infants showed no evidence of differential responding to voices when faces were obscured; their smiling and visual attention simply declined over time. These results extend findings from non-social contexts to social interactions and also indicate that infants may require facial expressions to be present to discriminate among adult vocal expressions of affect.     

The Bursts and Lulls of Multimodal Interaction: Temporal Distributions of Behavior Reveal Differences Between Verbal and Non‐Verbal Communication

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, 2012 ), we focus on measuring the temporal patterns of specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog.     

视觉刺激探测的警觉性注意特征初探

采用"预警提示-目标探测"范式,对视觉刺激探测的警觉性注意特征进行初步探索。实验一表明,随着预备到目标刺激的时间间隔(interstimulus interval,ISI)变化,固定位置上探测目标刺激的速度也随之变化,并且对大概率刺激的探测更快,表现出了警觉性注意的稳定性和选择性特征。实验二进一步考察了目标刺激在不同ISI和不同空间位置的情况,除发现与实验一相似的结果以外,被试探测目标刺激的速度随注意视角增大而下降,表现出了警觉性注意的集中性特征。