通道估计可靠性先验知识在早期的知觉加工阶段影响多感觉信息整合

统计最优化理论认为, 在多感觉信息整合的过程中, 大脑是以加权平均的方式将多个感觉通道的信息整合为统一的感觉信息, 通道信息的利用权重根据各通道信息的可靠性来确定。近期的几个行为研究结果则显示, 有关通道估计可靠性的先验知识同样能够影响通道信息在整合过程中的利用权重。然而, 这些研究结果不能确定先验知识对多感觉信息整合的影响是发生在认知加工的感知觉阶段还是决策阶段。当前研究致力于对此问题进行探讨。实验通过赋予两种颜色字母不同的视、听一致概率(高概率、低概率), 测量并分析被试在各概率水平下对视听一致刺激的反应时。实验数据显示不同的视听一致概率能够调制对视听一致刺激的反应时。该结果揭示通道估计可靠性先验知识在早期的知觉加工阶段影响多感觉信息整合。     

多感觉整合范式中潜在的跨通道转换效应

大脑可以对来自不同感觉通道的信息进行处理与整合。与单一感觉通道相比, 个体对同时呈现在不同感觉通道的目标信号的响应会更快。对于这种现象的一种主要理论解释是共同激活模型, 该模型认为来自不同通道的刺激在特定的脑区汇聚整合, 比如顶叶内沟、颞上沟和前额叶皮层区域。整合后的信号强度更大, 可以更快地触发反应, 但是信号的整合发生在认知加工的哪一阶段目前尚未有明确结论。当个体对出现在不同感觉通道之间的任务转换进行加工时, 产生与感觉通道相关的任务转换的损失小于跨感觉通道转换损失与任务转换损失的总和, 这为与感觉通道相关的转换代价来源于任务设置的惯性和干扰提供了证据。而在单通道和多通道之间发生转换时, 跨通道转换代价会减小甚至消失, 这是由于同时发生的多感觉整合抵消了一部分损失, 这种现象支持了共同激活模型理论。然而, 多感觉信号整合对任务转换的神经加工过程产生怎样的影响并不清楚, 在未来的研究中可以把多感觉整合范式同经典的任务转换范式结合改进, 进而确定跨通道转换的加工机制和多感觉信号整合的发生阶段。     

视听刺激突显性对视觉主导效应的影响

本研究使用空间任务-转换范式,控制视、听刺激的突显性,探讨自下而上注意对视觉主导效应的影响。结果表明视、听刺激突显性显著地影响视觉主导效应,实验一中当听觉刺激为高突显性时,视觉主导效应显著减弱。实验二中当听觉刺激为高突显性并且视觉刺激为低突显性时,视觉主导效应进一步减弱但依然存在。结果支持偏向竞争理论,在跨通道视听交互过程中视觉刺激更具突显性,在多感觉整合过程中更具加工优势。     

集中和分散注意对多感觉整合中听觉主导效应的影响

听觉主导效应是指多感觉通道信息整合过程中,听觉通道中的信息得到优先加工,从而主导其他感觉通道的信息。研究采用经典的声音诱发闪光错觉的范式,通过两个实验操纵了注意资源的分配方式以及实验任务难度,考察了主动注意听觉通道的声音刺激对声音诱发闪光错觉产生的影响,以及任务难度对声音诱发闪光错觉的影响。结果发现:(1)裂变错觉会受到注意资源分配程度的影响,但是融合错觉则不然;(2)任务难度既不会影响裂变错觉,也不会影响融合错觉。说明了分散注意能够影响听觉主导效应中的裂变错觉,并且这种主导效应与任务难度无关。     

视听时、空一致性对Pip-and-Pop效应的影响

多感觉整合遵循空间原则和时间原则, 有研究表明Pip-and-Pop效应产生的原因是多感觉整合, 那么Pip-and-Pop效应是否同样遵循空间原则和时间原则呢?本研究采用动态视觉搜索范式, 通过两个眼动实验考察空间一致性(实验1)和时间一致性(实验2)对Pip-and-Pop效应的影响。结果发现：(1)当视觉目标颜色变化伴随一个同侧的听觉刺激时所产生的Pip-and-Pop效应最大, 对侧条件没有发现Pip-and-Pop效应。(2)视听刺激同时呈现所产生的Pip-and-Pop效应最大, 随视听刺激呈现时间间隔的增加Pip-and-Pop效应逐渐减弱直至消失。结果表明, 视听时、空一致性对Pip-and-Pop效应具有调节作用, 这为多感觉整合在产生Pip-and-Pop效应中的作用提供了证据。     

内外源性空间注意对多感觉整合的影响

本文采用内-外源性空间线索靶子范式, 操控内源性线索有效性(有效线索、无效线索)、外源性线索有效性(有效线索、无效线索)、目标刺激类型(视觉刺激、听觉刺激、视听觉刺激)三个自变量。通过两个不同任务难度的实验(实验1: 简单定位任务; 实验2: 复杂辨别任务)来考察内外源性空间注意对多感觉整合的影响。两个实验结果均发现外源性空间注意显著减弱了多感觉整合效应, 内源性空间注意没有显著增强多感觉整合效应; 实验2中还发现了内源性空间注意会对外源性空间注意减弱多感觉整合效应产生影响。结果表明, 与内源性空间注意不同, 外源性空间注意对多感觉整合的影响不易受任务难度的调控; 当任务较难时内源性空间注意会影响外源性空间注意减弱多感觉整合效应的过程。由此推测, 内外源性空间注意对多感觉整合的调节并非彼此独立、而是相互影响的。     

注意对视听语义整合加工的影响

采用2×3的被试内实验设计,将注意条件和目标刺激类型作为实验变量,考察了指向不同感觉通道的注意对视听语义整合加工的不同影响。结果发现,只有在同时注意视觉和听觉刺激时,被试对语义一致的视听刺激反应最快,即产生冗余信号效应。而在选择性注意一个感觉通道时,语义一致的视听刺激并不具有加工优势。进一步分析发现,在同时注意视觉和听觉时语义一致视听刺激的加工优势源自于其视觉和听觉成分产生了整合。也就是说,只有在同时注意视觉和听觉时,语义一致视听刺激才会产生整合,语义不一致视听刺激不会产生整合。而在选择性注意一个感觉通道时,不论语义是否一致,视听刺激均不会产生整合。     

跨感觉通道任务中刺激驱动注意捕获的不对称性

通过两个实验探讨了来自不同感觉通道或不同空间方位的奇异刺激在视听搜索范式中的注意捕获。实验一使用定位任务探讨刺激驱动注意捕获中跨通道效应且及其不对称性, 结果发现, 当靶子在听觉通道时, 只有来自听觉同侧空间方位的奇异刺激成功的捕获了注意; 而来自视觉同侧空间和异侧空间的奇异刺激并没有捕获注意。实验二使用检测任务进一步探讨刺激驱动的注意捕获中的跨通道效应且具有不对称性, 结果发现, 来自同侧和异侧的听觉奇异刺激均捕获了注意, 但是视觉奇异刺激却不能捕获注意。研究表明, 视听搜索范式中的奇异刺激的注意捕获具有不对称性, 且存在着一种超感觉通道的注意资源。     

外源性注意与多感觉整合的交互关系

外源性注意与多感觉整合的交互关系是一个复杂且具有争议的研究领域, 一直以来备受研究者们关注。为了解释两者间的交互作用机制, 本文基于已有研究成果从两方面综述了外源性注意与多感觉整合的交互关系：(1)外源性注意可以通过自下而上的方式调节多感觉整合, 包括空间不确定性、感知觉敏感度和感觉通道间信号强度差异三种理论假说; (2)多感觉整合可以调节外源性注意。一方面, 来自多感觉通道的刺激能够以自下而上的方式自动整合, 整合后的多感觉通道刺激比单通道刺激具有更大的凸显性从而有效地吸引注意。另一方面, 整合后的多感觉通道刺激能够作为多感觉信号模板存储于大脑之中, 从而在任务中实现自上而下地调节注意捕获。     

内源性空间线索有效性对视听觉整合的影响

采用内源性线索-靶子范式, 操纵线索类型(有效线索、无效线索)和靶刺激通道类型(视觉刺激、听觉刺激、视听觉刺激)两个自变量, 通过两个实验, 分别设置50%和80%两种内源性空间线索有效性来考察不同空间线索有效性条件下内源性空间注意对视听觉整合的影响。结果发现, 当线索有效性为50%时(实验1), 有效线索位置和无效线索位置的视听觉整合效应没有显著差异; 当线索有效性为80%时(实验2), 有效线索位置的视听觉整合效应显著大于无效线索位置的视听觉整合效应。结果表明, 线索有效性不同时, 内源性空间注意对视听觉整合产生了不同的影响, 高线索有效性条件下内源性空间注意能够促进视听觉整合效应。     

视听跨通道信息的整合与冲突控制

多通道信息交互是指来自某个感觉通道的信息与另一感觉通道的信息相互作用、相互影响的一系列加工过程。主要包括两个方面：一是不同感觉通道的输入如何整合;二是跨通道信息的冲突控制。本文综述了视听跨通道信息整合与冲突控制的行为心理机制和神经机制,探讨了注意对视听信息整合与冲突控制的影响。未来需探究视听跨通道信息加工的脑网络机制,考察特殊群体的跨通道整合和冲突控制以帮助揭示其认知和社会功能障碍的机制。     

选择性注意和分配性注意对多感觉整合的不同影响*

以简单图形为视觉刺激,以短纯音作为听觉刺激,通过指导被试注意不同通道（注意视觉、注意听觉、注意视听）以形成不同注意状态（选择性注意和分配性注意）,考察了注意对多感觉整合的影响,发现只有在分配性注意时被试对双通道目标的反应最快最准确。通过竞争模型分析发现,这种对双通道目标的加工优势源自于视听双通道刺激的整合。上述结果表明,只有在分配性注意状态下才会产生多感觉整合。     

Cross-modal dynamic capture: congruency effects in the perception of motion across sensory modalities

This study investigated multisensory interactions in the perception of auditory and visual motion. When auditory and visual apparent motion streams are presented concurrently in opposite directions, participants often fail to discriminate the direction of motion of the auditory stream, whereas perception of the visual stream is unaffected by the direction of auditory motion (Experiment 1). This asymmetry persists even when the perceived quality of apparent motion is equated for the 2 modalities (Experiment 2). Subsequently, it was found that this visual modulation of auditory motion is caused by an illusory reversal in the perceived direction of sounds (Experiment 3). This "dynamic capture" effect occurs over and above ventriloquism among static events (Experiments 4 and 5), and it generalizes to continuous motion displays (Experiment 6). These data are discussed in light of related multisensory phenomena and their support for a "modality appropriateness" interpretation of multisensory integration in motion perception.     

Assessing automaticity in the audiovisual integration of motion

This study examined the multisensory integration of visual and auditory motion information using a methodology designed to single out perceptual integration processes from post-perceptual influences. We assessed the threshold stimulus onset asynchrony (SOA) at which the relative directions (same vs. different) of simultaneously presented visual and auditory apparent motion streams could no longer be discriminated (Experiment 1). This threshold was higher than the upper threshold for direction discrimination (left vs. right) of each individual modality when presented in isolation (Experiment 2). The poorer performance observed in bimodal displays was interpreted as a consequence of automatic multisensory integration of motion information. Experiment 3 supported this interpretation by ruling out task differences as the explanation for the higher threshold in Experiment 1. Together these data provide empirical support for the view that multisensory integration of motion signals can occur at a perceptual level.     

Is neocortex essentially multisensory?

Although sensory perception and neurobiology are traditionally investigated one modality at a time, real world behaviour and perception are driven by the integration of information from multiple sensory sources. Mounting evidence suggests that the neural underpinnings of multisensory integration extend into early sensory processing. This article examines the notion that neocortical operations are essentially multisensory. We first review what is known about multisensory processing in higher-order association cortices and then discuss recent anatomical and physiological findings in presumptive unimodal sensory areas. The pervasiveness of multisensory influences on all levels of cortical processing compels us to reconsider thinking about neural processing in unisensory terms. Indeed, the multisensory nature of most, possibly all, of the neocortex forces us to abandon the notion that the senses ever operate independently during real-world cognition.     

The development of multisensory processes

To understand the development of sensory processes, it is necessary not only to look at the maturation of each of the sensory systems in isolation, but also to study the development of the nervous systems capacity to integrate information across the different senses. It is through such multisensory integration that a coherent perceptual gestalt of the world comes to be generated. In the adult brain, multisensory convergence and integration take place at a number of brainstem and cortical sites, where individual neurons have been found that respond to multisensory stimuli with patterns of activation that depend on the nature of the stimulus complex and the intrinsic properties of the neuron. Parallels between the responses of these neurons and multisensory behavior and perception suggest that they are the substrates that underlie these cognitive processes. In both cat and monkey models, the development of these multisensory neurons and the appearance of their integrative capacity is a gradual postnatal process. For subcortical structures (i.e., the superior colliculus) this maturational process appears to be gated by the appearance of functional projections from regions of association cortex. The slow postnatal maturation of multisensory processes, coupled with its dependency on functional corticotectal connections, suggested that the development of multisensory integration may be tied to sensory experiences acquired during postnatal life. In support of this, eliminating experience in one sensory modality (i.e., vision) during postnatal development severely compromises the integration of multisensory cues. Research is ongoing to better elucidate the critical development antecedents for the emergence of normal multisensory capacity.Edited by Marie-Hélène Giard and Mark WallaceThis revised version was published in May 2004 with corrections to Fig. 1.     

Crossmodal binding: evaluating the "unity assumption" using audiovisual speech stimuli

We investigated whether the "unity assumption," according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1-3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the "unity assumption" in the domain of the multisensory temporal integration of audiovisual speech stimuli.     

The effects of aging on the working memory processes of multimodal information

Normal aging is associated with deficits in working memory processes. However, the majority of research has focused on storage or inhibitory processes using unimodal paradigms, without addressing their relationships using different sensory modalities. Hence, we pursued two objectives. First, was to examine the effects of aging on storage and inhibitory processes. Second, was to evaluate aging effects on multisensory integration of visual and auditory stimuli. To this end, young and older participants performed a multimodal task for visual and auditory pairs of stimuli with increasing memory load at encoding and interference during retention. Our results showed an age-related increased vulnerability to interrupting and distracting interference reflecting inhibitory deficits related to the off-line reactivation and on-line suppression of relevant and irrelevant information, respectively. Storage capacity was impaired with increasing task demands in both age groups. Additionally, older adults showed a deficit in multisensory integration, with poorer performance for new visual compared to new auditory information.     

The co-occurrence of multisensory competition and facilitation

Previous studies of multisensory integration have often stressed the beneficial effects that may arise when information concerning an event arrives via different sensory modalities at the same time, as, for example, exemplified by research on the redundant target effect (RTE). By contrast, studies of the Colavita visual dominance effect (e.g., [Colavita, F. B. (1974). Human sensory dominance. Perception & Psychophysics, 16, 409–412]) highlight the inhibitory consequences of the competition between signals presented simultaneously in different sensory modalities instead. Although both the RTE and the Colavita effect are thought to occur at early sensory levels and the stimulus conditions under which they are typically observed are very similar, the interplay between these two opposing behavioural phenomena (facilitation vs. competition) has yet to be addressed empirically. We hypothesized that the dissociation may reflect two of the fundamentally different ways in which humans can perceive concurrent auditory and visual stimuli. In Experiment 1, we demonstrated both multisensory facilitation (RTE) and the Colavita visual dominance effect using exactly the same audiovisual displays, by simply changing the task from a speeded detection task to a speeded modality discrimination task. Meanwhile, in Experiment 2, the participants exhibited multisensory facilitation when responding to visual targets and multisensory inhibition when responding to auditory targets while keeping the task constant. These results therefore indicate that both multisensory facilitation and inhibition can be demonstrated in reaction to the same bimodal event.