The subjective duration of audiovisual looming and receding stimuli

Looming visual stimuli (log-increasing in proximal size over time) and auditory stimuli (of increasing sound intensity over time) have been shown to be perceived as longer than receding visual and auditory stimuli (i.e., looming stimuli reversed in time). Here, we investigated whether such asymmetry in subjective duration also occurs for audiovisual looming and receding stimuli, as well as for stationary stimuli (i.e., stimuli that do not change in size and/or intensity over time). Our results showed a great temporal asymmetry in audition but a null asymmetry in vision. In contrast, the asymmetry in audiovision was moderate, suggesting that multisensory percepts arise from the integration of unimodal percepts in a maximum-likelihood fashion.     

Twice upon a time: multiple concurrent temporal recalibrations of audiovisual speech

Audiovisual timing perception can recalibrate following prolonged exposure to asynchronous auditory and visual inputs. It has been suggested that this might contribute to achieving perceptual synchrony for auditory and visual signals despite differences in physical and neural signal times for sight and sound. However, given that people can be concurrently exposed to multiple audiovisual stimuli with variable neural signal times, a mechanism that recalibrates all audiovisual timing percepts to a single timing relationship could be dysfunctional. In the experiments reported here, we showed that audiovisual temporal recalibration can be specific for particular audiovisual pairings. Participants were shown alternating movies of male and female actors containing positive and negative temporal asynchronies between the auditory and visual streams. We found that audiovisual synchrony estimates for each actor were shifted toward the preceding audiovisual timing relationship for that actor and that such temporal recalibrations occurred in positive and negative directions concurrently. Our results show that humans can form multiple concurrent estimates of appropriate timing for audiovisual synchrony.     

Increased discriminability of authenticity from multimodal laughter is driven by auditory information

We present an investigation of the perception of authenticity in audiovisual laughter, in which we contrast spontaneous and volitional samples and examine the contributions of unimodal affective information to multimodal percepts. In a pilot study, we demonstrate that listeners perceive spontaneous laughs as more authentic than volitional ones, both in unimodal (audio-only, visual-only) and multimodal contexts (audiovisual). In the main experiment, we show that the discriminability of volitional and spontaneous laughter is enhanced for multimodal laughter. Analyses of relationships between affective ratings and the perception of authenticity show that, while both unimodal percepts significantly predict evaluations of audiovisual laughter, it is auditory affective cues that have the greater influence on multimodal percepts. We discuss differences and potential mismatches in emotion signalling through voices and faces, in the context of spontaneous and volitional behaviour, and highlight issues that should be addressed in future studies of dynamic multimodal emotion processing.     

Processing of audiovisually congruent and incongruent speech in school‐age children with a history of specific language impairment: a behavioral and event‐related potentials study

Previous studies indicate that at least some aspects of audiovisual speech perception are impaired in children with specific language impairment (SLI). However, whether audiovisual processing difficulties are also present in older children with a history of this disorder is unknown. By combining electrophysiological and behavioral measures, we examined perception of both audiovisually congruent and audiovisually incongruent speech in school‐age children with a history of SLI (H‐SLI), their typically developing (TD) peers, and adults. In the first experiment, all participants watched videos of a talker articulating syllables ‘ba’, ‘da’, and ‘ga’ under three conditions – audiovisual (AV), auditory only (A), and visual only (V). The amplitude of the N1 (but not of the P2) event‐related component elicited in the AV condition was significantly reduced compared to the N1 amplitude measured from the sum of the A and V conditions in all groups of participants. Because N1 attenuation to AV speech is thought to index the degree to which facial movements predict the onset of the auditory signal, our findings suggest that this aspect of audiovisual speech perception is mature by mid‐childhood and is normal in the H‐SLI children. In the second experiment, participants watched videos of audivisually incongruent syllables created to elicit the so‐called McGurk illusion (with an auditory ‘pa’ dubbed onto a visual articulation of ‘ka’, and the expectant perception being that of ‘ta’ if audiovisual integration took place). As a group, H‐SLI children were significantly more likely than either TD children or adults to hear the McGurk syllable as ‘pa’ (in agreement with its auditory component) than as ‘ka’ (in agreement with its visual component), suggesting that susceptibility to the McGurk illusion is reduced in at least some children with a history of SLI. Taken together, the results of the two experiments argue against global audiovisual integration impairment in children with a history of SLI and suggest that, when present, audiovisual integration difficulties in this population likely stem from a later (non‐sensory) stage of processing.     

Suggestion overrides the Stroop effect in highly hypnotizable individuals

Cognitive scientists distinguish between automatic and controlled mental processes. Automatic processes are either innately involuntary or become automatized through extensive practice. For example, reading words is a purportedly automatic process for proficient readers and the Stroop effect is consequently considered the "gold standard" of automated performance. Although the question of whether it is possible to regain control over an automatic process is mostly unasked, we provide compelling data showing that posthypnotic suggestion reduced and even removed Stroop interference in highly hypnotizable individuals. Drawing on a large sample of highly hypnotizable participants, we examined the effects of suggestion on Stroop performance both with and without a posthypnotic suggestion to perceive the input stream as meaningless symbols. We show that suggestion administered to highly hypnotizable persons significantly reduced Stroop interference and derailed a seemingly automatic process.     

A qualitative analysis of sensory phenomena induced by perceptual deprivation

Previous studies have shown that misperceptions and illusory experiences can occur if sensory stimulation is withdrawn or becomes invariant even for short periods of time. Using a perceptual deprivation paradigm, we created a monotonous audiovisual environment and asked participants to verbally report any auditory, visual or body-related phenomena they experienced. The data (analysed using a variant of interpretative phenomenological analysis) revealed two main themes: (1) reported sensory phenomena have different spatial characteristics ranging from simple percepts to the feeling of immersion in a complex multisensory environment and (2) the active contribution of the perceiver where participants report engaging in exploratory processes even when there is nothing to find. Detailed analysis of the qualitative data further showed that participants who reported more perceptual phenomena were more likely to report internal bodily sensations, move more during the experiment and score higher on the Revised Hallucination Scale than those reporting fewer percepts explicitly linking perceptual deprivation to somatic phenomena. The results demonstrate how the variety of sensory experiences induced by perceptual deprivation can give further insight into the factors mediating conscious awareness and may suggest ways in which the brain imposes meaning on the environment under invariant sensory conditions.     

Distinguishing automatic and controlled components of attitudes from direct and indirect measurement methods

Distinct automatic and controlled processes are presumed to influence social evaluation. Most empirical approaches examine automatic processes using indirect methods, and controlled processes using direct methods. We distinguished processes from measurement methods to test whether a process distinction is more useful than a measurement distinction for taxonomies of attitudes. Results from two studies suggest that automatic components of attitudes can be measured directly. Direct measures of automatic attitudes were reports of gut reactions (Study 1) and behavioral performance in a speeded self-report task (Study 2). Confirmatory factor analyses comparing two-factor models revealed better fits when self-reports of gut reactions and speeded self-reports shared a factor with automatic measures versus sharing a factor with controlled self-report measures. Thus, distinguishing attitudes by the processes they are presumed to measure (automatic versus controlled) is more meaningful than distinguishing based on the directness of measurement.     

McGurk效应的影响因素与神经基础

McGurk效应(麦格克效应)是典型的视听整合现象, 该效应受到刺激的物理特征、注意分配、个体视听信息依赖程度、视听整合能力、语言文化差异的影响。引发McGurk效应的关键视觉信息主要来自说话者的嘴部区域。产生McGurk效应的认知过程包含早期的视听整合(与颞上皮层有关)以及晚期的视听不一致冲突(与额下皮层有关)。未来研究应关注面孔社会信息对McGurk效应的影响, McGurk效应中单通道信息加工与视听整合的关系, 结合计算模型探讨其认知神经机制等。     

Visual contribution to the multistable perception of speech

The multistable perception of speech, or verbal transformation effect, refers to perceptual changes experienced while listening to a speech form that is repeated rapidly and continuously. In order to test whether visual information from the speaker's articulatory gestures may modify the emergence and stability of verbal auditory percepts, subjects were instructed to report any perceptual changes during unimodal, audiovisual, and incongruent audiovisual presentations of distinct repeated syllables. In a first experiment, the perceptual stability of reported auditory percepts was significantly modulated by the modality of presentation. In a second experiment, when audiovisual stimuli consisting of a stable audio track dubbed with a video track that alternated between congruent and incongruent stimuli were presented, a strong correlation between the timing of perceptual transitions and the timing of video switches was found. Finally, a third experiment showed that the vocal tract opening onset event provided by the visual input could play the role of a bootstrap mechanism in the search for transformations. Altogether, these results demonstrate the capacity of visual information to control the multistable perception of speech in its phonetic content and temporal course. The verbal transformation effect thus provides a useful experimental paradigm to explore audiovisual interactions in speech perception.     

A measure for assessing the effects of audiovisual speech integration

We propose a measure of audiovisual speech integration that takes into account accuracy and response times. This measure should prove beneficial for researchers investigating multisensory speech recognition, since it relates to normal-hearing and aging populations. As an example, age-related sensory decline influences both the rate at which one processes information and the ability to utilize cues from different sensory modalities. Our function assesses integration when both auditory and visual information are available, by comparing performance on these audiovisual trials with theoretical predictions for performance under the assumptions of parallel, independent self-terminating processing of single-modality inputs. We provide example data from an audiovisual identification experiment and discuss applications for measuring audiovisual integration skills across the life span.     

情绪和时间辨别任务中视听整合与预先准备效应

本研究分别在时间和情绪认知维度上考察预先准备效应对情绪视听整合的影响。时间辨别任务(实验1)发现视觉引导显著慢于听觉引导,并且整合效应量为负值。情绪辨别任务(实验2)发现整合效应量为正值;在负性情绪整合中,听觉引导显著大于视觉引导;在正性情绪整合中,视觉引导显著大于听觉引导。研究表明,情绪视听整合基于情绪认知加工,而时间辨别会抑制整合;此外,跨通道预先准备效应和情绪预先准备效应都与引导通道有关。     

多感觉整合中的声音诱发闪光错觉效应

声音诱发闪光错觉效应是典型的视听整合错觉现象, 是指当视觉闪光刺激与间隔100 ms内的听觉声音刺激不等数量呈现时, 被试知觉视觉闪光的数量与听觉声音的数量相等。声音诱发闪光错觉的影响因素既包括自下而上和自上而下的被试内差异因素, 也包括视听刺激依赖程度、视听整合的发展和视听刺激知觉敏感性等被试间差异因素。该效应的产生在时程上主要体现在早期加工阶段, 在脑区上主要涉及多处皮层及皮层下相关脑区。未来研究应进一步考察注意、奖赏和视听整合方式等认知加工对声音诱发闪光错觉的影响, 同时也应该关注声音诱发闪光错觉对记忆和学习的影响以及结合计算模型和神经科学的手段进一步探讨其认知神经机制。     

发展性阅读障碍的视听时间敏感性缺陷及其脑神经机制

阅读是一个视听加工过程,阅读障碍的产生可能是过程中视听时间敏感性缺陷的结果。视听时间敏感性指个体对视觉和听觉刺激出现时间的感知能力,可通过同时性判断、时间顺序判断和视听整合考察。研究发现,阅读障碍者在这一能力上表现出行为和脑层面异常。而这些研究多是拼音文字背景,汉语文字下该领域研究相当少。未来需要丰富实验设计,扩大对汉语背景下视听时间敏感性研究,并以此开发干预手段,为阅读障碍的机制和治疗提供借鉴。     

Neural correlates of audiovisual integration of semantic category information

Previous studies have found a late frontal-central audiovisual interaction during the time period about 150-220 ms post-stimulus. However, it is unclear to which process is this audiovisual interaction related: to processing of acoustical features or to classification of stimuli? To investigate this question, event-related potentials were recorded during a words-categorization task with stimuli presented in the auditory-visual modality. In the experiment, congruency of the visual and auditory stimuli was manipulated. Results showed that within the window of about 180-210 ms post-stimulus more positive values were elicited by category-congruent audiovisual stimuli than category-incongruent audiovisual stimuli. This indicates that the late frontal-central audiovisual interaction is related to audiovisual integration of semantic category information.     

When does visual perceptual grouping affect multisensory integration?

Several studies have shown that the direction in which a visual apparent motion stream moves can influence the perceived direction of an auditory apparent motion stream (an effect known as crossmodal dynamic capture). However, little is known about the role that intramodal perceptual grouping processes play in the multisensory integration of motion information. The present study was designed to investigate the time course of any modulation of the cross-modal dynamic capture effect by the nature of the perceptual grouping taking place within vision. Participants were required to judge the direction of an auditory apparent motion stream while trying to ignore visual apparent motion streams presented in a variety of different configurations. Our results demonstrate that the cross-modal dynamic capture effect was influenced more by visual perceptual grouping when the conditions for intramodal perceptual grouping were set up prior to the presentation of the audiovisual apparent motion stimuli. However, no such modulation occurred when the visual perceptual grouping manipulation was established at the same time as or after the presentation of the audiovisual stimuli. These results highlight the importance of the unimodal perceptual organization of sensory information to the manifestation of multisensory integration.     

视听跨通道信息的整合与冲突控制

多通道信息交互是指来自某个感觉通道的信息与另一感觉通道的信息相互作用、相互影响的一系列加工过程。主要包括两个方面：一是不同感觉通道的输入如何整合;二是跨通道信息的冲突控制。本文综述了视听跨通道信息整合与冲突控制的行为心理机制和神经机制,探讨了注意对视听信息整合与冲突控制的影响。未来需探究视听跨通道信息加工的脑网络机制,考察特殊群体的跨通道整合和冲突控制以帮助揭示其认知和社会功能障碍的机制。     

Development of multisensory spatial integration and perception in humans

Previous studies have shown that adults respond faster and more reliably to bimodal compared to unimodal localization cues. The current study investigated for the first time the development of audiovisual (A-V) integration in spatial localization behavior in infants between 1 and 10 months of age. We observed infants' head and eye movements in response to auditory, visual, or both kinds of stimuli presented either 25 degrees or 45 degrees to the right or left of midline. Infants under 8 months of age intermittently showed response latencies significantly faster toward audiovisual targets than toward either auditory or visual targets alone They did so, however, without exhibiting a reliable violation of the Race Model, suggesting that probability summation alone could explain the faster bimodal response. In contrast, infants between 8 and 10 months of age exhibited bimodal response latencies significantly faster than unimodal latencies for both eccentricity conditions and their latencies violated the Race Model at 25 degrees eccentricity. In addition to this main finding, we found age-dependent eccentricity and modality effects on response latencies. Together, these findings suggest that audiovisual integration emerges late in the first year of life and are consistent with neurophysiological findings from multisensory sites in the superior colliculus of infant monkeys showing that multisensory enhancement of responsiveness is not present at birth but emerges later in life.     

Attentional capture in serial audiovisual search tasks

The phenomenon of attentional capture has typically been studied in spatial search tasks. Dalton and Lavie recently demonstrated that auditory attention can also be captured by a singleton item in a rapidly presented tone sequence. In the experiments reported here, we investigated whether these findings extend cross-modally to sequential search tasks using audiovisual stimuli. Participants searched a stream of centrally presented audiovisual stimuli for targets defined on a particular dimension (e.g., duration) in a particular modality. Task performance was compared in the presence versus absence of a unique singleton distractor. Irrelevant auditory singletons captured attention during visual search tasks, leading to interference when they coincided with distractors but to facilitation when they coincided with targets. These results demonstrate attentional capture by auditory singletons during nonspatial visual search.     

跨感觉通道任务中刺激驱动注意捕获的不对称性

通过两个实验探讨了来自不同感觉通道或不同空间方位的奇异刺激在视听搜索范式中的注意捕获。实验一使用定位任务探讨刺激驱动注意捕获中跨通道效应且及其不对称性, 结果发现, 当靶子在听觉通道时, 只有来自听觉同侧空间方位的奇异刺激成功的捕获了注意; 而来自视觉同侧空间和异侧空间的奇异刺激并没有捕获注意。实验二使用检测任务进一步探讨刺激驱动的注意捕获中的跨通道效应且具有不对称性, 结果发现, 来自同侧和异侧的听觉奇异刺激均捕获了注意, 但是视觉奇异刺激却不能捕获注意。研究表明, 视听搜索范式中的奇异刺激的注意捕获具有不对称性, 且存在着一种超感觉通道的注意资源。     

Can expectation enhance response to suggestion? De-automatization illuminates a conundrum

Disparate theoretical viewpoints construe hypnotic suggestibility either as a stable trait, largely determined by underlying cognitive aptitude, or as a flexible skill amenable to attitudinal factors including beliefs and expectations. Circumscribed findings support both views. The present study attempted to consolidate these orthogonal perspectives through the lens of expectancy modification. We surreptitiously controlled light and sound stimuli to convince participants that they were responding strongly to hypnotic suggestions for visual and auditory hallucinations. Extending our previous findings, we indexed hypnotic suggestibility by de-automatizing an involuntary audiovisual phenomenon-the McGurk effect. Here we show that, regardless of expectancy modification, the experimental procedure led to heightened expectations concerning future hypnotic response. We found little effect of expectation, however, on actual response to suggestion. Our findings intimate that, at least in the present experimental context, expectation hardly correlates with--and is unlikely to be a primary determinant of--high hypnotic suggestibility.