多感觉整合的时间再校准

跨通道刺激的时间同步性是多感觉整合的必要条件, 但是由于刺激物理传导与神经传导的差异, 它们在时间上并非完全匹配。时间再校准指大脑能够适应跨通道刺激间很短的时间延迟的现象, 反映了多感觉整合在时间维度上的可塑性, 表现为适应相继呈现的跨通道刺激后, 主观同时点向时间延迟方向的偏移。本文主要介绍了时间再校准的通道效应与潜在机制, 其初始加工阶段, 它与刺激内容加工的关系及主要的影响因素。今后的研究应当进一步探索时间再校准能否发生于早期加工阶段, 检验其认知过程是否具有双向性, 探讨空间选择性注意的作用, 并结合神经机制的研究, 从综合的视角进行更完善的理论建构。     

注意对视听语义整合加工的影响

采用2×3的被试内实验设计,将注意条件和目标刺激类型作为实验变量,考察了指向不同感觉通道的注意对视听语义整合加工的不同影响。结果发现,只有在同时注意视觉和听觉刺激时,被试对语义一致的视听刺激反应最快,即产生冗余信号效应。而在选择性注意一个感觉通道时,语义一致的视听刺激并不具有加工优势。进一步分析发现,在同时注意视觉和听觉时语义一致视听刺激的加工优势源自于其视觉和听觉成分产生了整合。也就是说,只有在同时注意视觉和听觉时,语义一致视听刺激才会产生整合,语义不一致视听刺激不会产生整合。而在选择性注意一个感觉通道时,不论语义是否一致,视听刺激均不会产生整合。     

情绪效价可预测性对时间捆绑效应的影响

时间捆绑效应指的是主观上主动动作和动作结果的时间点相互靠近的现象。其中, 时间捆绑效应可分为动作捆绑和结果捆绑, 分别对应主动动作和动作结果的时间点变化。本研究通过一个混合设计实验考察了情绪效价可预测性对时间捆绑效应的影响。情绪效价可预测性(可预测、不可预测)为被试内因素, 刺激模态(听觉刺激、视觉刺激)为被试间因素。结果发现, 情绪效价为可预测时, 结果捆绑在主动按键后产生听觉刺激或视觉刺激都出现了增强, 而动作捆绑仅在产生视觉刺激时才出现增强。结果表明, 情绪效价为可预测时会增强时间捆绑效应, 但该作用在动作捆绑和结果捆绑中存在差异。由于时间捆绑效应是反映主动控制感的主要指标, 本研究结果对先进驾驶辅助系统的交互设计具有一定的参考意义。     

多感觉整合中的声音诱发闪光错觉效应

声音诱发闪光错觉效应是典型的视听整合错觉现象, 是指当视觉闪光刺激与间隔100 ms内的听觉声音刺激不等数量呈现时, 被试知觉视觉闪光的数量与听觉声音的数量相等。声音诱发闪光错觉的影响因素既包括自下而上和自上而下的被试内差异因素, 也包括视听刺激依赖程度、视听整合的发展和视听刺激知觉敏感性等被试间差异因素。该效应的产生在时程上主要体现在早期加工阶段, 在脑区上主要涉及多处皮层及皮层下相关脑区。未来研究应进一步考察注意、奖赏和视听整合方式等认知加工对声音诱发闪光错觉的影响, 同时也应该关注声音诱发闪光错觉对记忆和学习的影响以及结合计算模型和神经科学的手段进一步探讨其认知神经机制。     

孤独症谱系障碍者的视听时间整合

视听时间整合是指个体对一定时间间隔内输入的视听刺激进行表征的过程,是视听整合的重要机制。孤独症谱系障碍者的视听时间整合存在缺陷,主要表现为4个方面:视听时间整合窗口较正常个体更宽、更对称;快速视听时间再校准能力不足;听觉时间线索对其视觉搜索促进作用弱;言语刺激的视听时序知觉敏感性低。目前使用的研究任务多样,如声音诱发闪光错觉和"pip-pop"任务从内隐角度探究视听整合中的时间机制,同时性判断、时序判断和优先注视任务主要用于跨通道时序知觉的研究。相关理论从神经加工异常、先验经验不足和视听通道的相互作用等角度解释了其缺陷。未来需要进一步提高研究生态效度,整合理论解释,精确量化诊断指标,同时开发有实效的干预策略。     

节律性时间期待效应不受注意控制的影响

采用双任务范式探讨当听觉节律刺激序列以较慢速度呈现时,其诱导产生的时间期待效应是否受到同时进行的视觉工作记忆任务的影响。结果发现,无论目标刺激是呈现在听觉通道还是视觉通道,双任务和单任务条件下目标刺激出现在规律听觉刺激序列之后被试的反应时均快于目标出现在非规律听觉刺激序列之后,即节律性刺激序列诱导产生的时间期待效应不受工作记忆任务的影响。该结果表明节律性时间期待效应不受注意控制的影响。     

视听刺激突显性对视觉主导效应的影响

本研究使用空间任务-转换范式,控制视、听刺激的突显性,探讨自下而上注意对视觉主导效应的影响。结果表明视、听刺激突显性显著地影响视觉主导效应,实验一中当听觉刺激为高突显性时,视觉主导效应显著减弱。实验二中当听觉刺激为高突显性并且视觉刺激为低突显性时,视觉主导效应进一步减弱但依然存在。结果支持偏向竞争理论,在跨通道视听交互过程中视觉刺激更具突显性,在多感觉整合过程中更具加工优势。     

特征关联的类别间面孔适应

研究通过系列实验探讨了面孔适应不仅仅发生在形状选择性上, 也能发生在任务相关的特征上有内在关联的两个不同类别的物体间。实验1以带有明显性别倾向的物品图片作为适应刺激, 让被试对男女之间morphing程度不同的图片面孔进行性别辨别, 考察了不同适应刺激呈现时间的类别间面孔适应。结果表明适应刺激呈现时间大于50 ms时均存在类别间面孔适应效应。实验2评估了“性别”这一特征以及适应刺激形式在类别间面孔适应中所起的作用, 结果发现带有性别倾向的物品图片、相应的物品名称和性别文字(“男性”、“女性”) 3种适应刺激类型均能产生类别间适应。实验3通过操纵适应刺激上的注意负荷(高负荷、低负荷和无负荷), 探究了注意对类别间面孔适应的影响。结果表明随着注意负荷的增加, 类别间面孔适应效应减小。3个实验报告了一个新异的类别间适应后效, 证明了适应也能发生于在任务相关特征上有内在关联的两个不同类别的物体间。     

听觉刺激对虚拟环境中空间压缩的影响

人们往往会低估虚拟环境中空间的距离(空间压缩)。早期研究发现视听刺激的不一致会改善空间压缩,但听觉刺激在其中的作用及其影响效果需进一步研究。本研究首先探究了听觉刺激的有无对空间压缩的影响,接着操控视听刺激之间的物物距离进一步探究其对空间压缩的影响和改善。研究发现听觉刺激以两种方式影响空间压缩：听觉刺激的存在可以改善空间压缩;空间压缩与视听刺激的物物距离负相关。研究建议为了改善空间压缩,视听刺激应当同时呈现且物物距离至少为1 m。     

跨感觉通道任务中刺激驱动注意捕获的不对称性

通过两个实验探讨了来自不同感觉通道或不同空间方位的奇异刺激在视听搜索范式中的注意捕获。实验一使用定位任务探讨刺激驱动注意捕获中跨通道效应且及其不对称性, 结果发现, 当靶子在听觉通道时, 只有来自听觉同侧空间方位的奇异刺激成功的捕获了注意; 而来自视觉同侧空间和异侧空间的奇异刺激并没有捕获注意。实验二使用检测任务进一步探讨刺激驱动的注意捕获中的跨通道效应且具有不对称性, 结果发现, 来自同侧和异侧的听觉奇异刺激均捕获了注意, 但是视觉奇异刺激却不能捕获注意。研究表明, 视听搜索范式中的奇异刺激的注意捕获具有不对称性, 且存在着一种超感觉通道的注意资源。     

发展性阅读障碍视听时间整合缺陷可能的机制：视听时间再校准能力受损

发展性阅读障碍的本质一直是研究者争论的焦点。大量研究发现, 阅读障碍者具有视听时间整合缺陷。然而, 这些研究仅考察了阅读障碍者视听时间整合加工的整体表现, 也就是平均水平的表现, 却对整合加工的变化过程缺乏探讨。视听时间再校准反映了视听时间整合的动态加工过程, 对内部时间表征与感觉输入之间差异的再校准困难则会导致多感觉整合受损, 而阅读障碍者的再校准相关能力存在缺陷。因此, 视听时间再校准能力受损可能是发展性阅读障碍视听时间整合缺陷的根本原因。未来的研究需要进一步考察发展性阅读障碍者视听时间再校准能力的具体表现, 以及这些表现背后的认知神经机制。     

Spatial grouping resolves ambiguity to drive temporal recalibration

Cross-modal temporal recalibration describes a shift in the point of subjective simultaneity (PSS) between 2 events following repeated exposure to asynchronous cross-modal inputs--the adaptors. Previous research suggested that audiovisual recalibration is insensitive to the spatial relationship between the adaptors. Here we show that audiovisual recalibration can be driven by cross-modal spatial grouping. Twelve participants adapted to alternating trains of lights and tones. Spatial position was manipulated, with alternating sequences of a light then a tone, or a tone then a light, presented on either side of fixation (e.g., left tone--left light--right tone--right light, etc.). As the events were evenly spaced in time, in the absence of spatial-based grouping it would be unclear if tones were leading or lagging lights. However, any grouping of spatially colocalized cross-modal events would result in an unambiguous sense of temporal order. We found that adapting to these stimuli caused the PSS between subsequent lights and tones to shift toward the temporal relationship implied by spatial-based grouping. These data therefore show that temporal recalibration is facilitated by spatial grouping.     

Visual recalibration of auditory speech identification: a McGurk aftereffect

The kinds of aftereffects, indicative of cross-modal recalibration, that are observed after exposure to spatially incongruent inputs from different sensory modalities have not been demonstrated so far for identity incongruence. We show that exposure to incongruent audiovisual speech (producing the well-known McGurk effect) can recalibrate auditory speech identification. In Experiment 1, exposure to an ambiguous sound intermediate between /aba/ and /ada/ dubbed onto a video of a face articulating either /aba/ or /ada/ increased the proportion of /aba/ or /ada/ responses, respectively, during subsequent sound identification trials. Experiment 2 demonstrated the same recalibration effect or the opposite one, fewer /aba/ or /ada/ responses, revealing selective speech adaptation, depending on whether the ambiguous sound or a congruent nonambiguous one was used during exposure. In separate forced-choice identification trials, bimodal stimulus pairs producing these contrasting effects were identically categorized, which makes a role of postperceptual factors in the generation of the effects unlikely.     

Temporal ventriloquism in a purely temporal context

This study examines how audiovisual signals are combined in time for a temporal analogue of the ventriloquist effect in a purely temporal context, that is, no spatial grounding of signals or other spatial facilitation. Observers were presented with two successive intervals, each defined by a 1250-ms tone, and indicated in which interval a brief audiovisual stimulus (visual flash + noise burst) occurred later. In "test" intervals, the audiovisual stimulus was presented with a small asynchrony, while in "probe" intervals it was synchronous and presented at various times guided by an adaptive staircase to find the perceived temporal location of the asynchronous stimulus. As in spatial ventriloquism, and consistent with maximum likelihood estimation (MLE), the asynchronous audiovisual signal was shifted toward the more reliably localized component (audition, for all observers). Moreover, these temporal shifts could be forward or backward in time, depending on the asynchrony order, suggesting perceived timing is not entirely determined by physical timing. However, the critical signature of MLE combination--better bimodal than unimodal precision--was not found. Regardless of the underlying model, these results demonstrate temporal ventriloquism in a paradigm that is defined in a purely temporal context.     

Rapid recalibration to audiovisual asynchrony follows the physical—not the perceived—temporal order

In natural scenes, audiovisual events deriving from the same source are synchronized at their origin. However, from the perspective of the observer, there are likely to be significant multisensory delays due to physical and neural latencies. Fortunately, our brain appears to compensate for the resulting latency differences by rapidly adapting to asynchronous audiovisual events by shifting the point of subjective synchrony (PSS) in the direction of the leading modality of the most recent event. Here we examined whether it is the perceived modality order of this prior lag or its physical order that determines the direction of the subsequent rapid recalibration. On each experimental trial, a brief tone pip and flash were presented across a range of stimulus onset asynchronies (SOAs). The participants’ task alternated over trials: On adaptor trials, audition either led or lagged vision with fixed SOAs, and participants judged the order of the audiovisual event; on test trials, the SOA as well as the modality order varied randomly, and participants judged whether or not the event was synchronized. For test trials, we showed that the PSS shifted in the direction of the physical rather than the perceived (reported) modality order of the preceding adaptor trial. These results suggest that rapid temporal recalibration is determined by the physical timing of the preceding events, not by one’s prior perceptual decisions.     

The effects of voluntary movements on auditory-haptic and haptic-haptic temporal order judgments

In two experiments we investigated the effects of voluntary movements on temporal haptic perception. Measures of sensitivity (JND) and temporal alignment (PSS) were obtained from temporal order judgments made on intermodal auditory-haptic (Experiment 1) or intramodal haptic (Experiment 2) stimulus pairs under three movement conditions. In the baseline, static condition, the arm of the participants remained stationary. In the passive condition, the arm was displaced by a servo-controlled motorized device. In the active condition, the participants moved voluntarily. The auditory stimulus was a short, 500Hz tone presented over headphones and the haptic stimulus was a brief suprathreshold force pulse applied to the tip of the index finger orthogonally to the finger movement. Active movement did not significantly affect discrimination sensitivity on the auditory-haptic stimulus pairs, whereas it significantly improved sensitivity in the case of the haptic stimulus pair, demonstrating a key role for motor command information in temporal sensitivity in the haptic system. Points of subjective simultaneity were by-and-large coincident with physical simultaneity, with one striking exception in the passive condition with the auditory-haptic stimulus pair. In the latter case, the haptic stimulus had to be presented 45ms before the auditory stimulus in order to obtain subjective simultaneity. A model is proposed to explain the discrimination performance.     

Temporal order judgment criteria are affected by synchrony judgment sensitivity

Synchrony perception for audio-visual stimulus pairs is typically studied by using temporal order judgment (TOJ) or synchrony judgment (SJ) tasks. Research has shown that estimates of the point of subjective simultaneity (PSS) obtained using these two methods do not necessarily correspond. Here, we investigate the hypothesis that the PSS estimate obtained in a TOJ task is shifted in the direction of the most sensitive part of the synchrony judgment curve, as obtained in an SJ task. The results confirm that criterion shifts in the TOJ task move the PSS toward regions of the audio-visual temporal interval continuum where discriminations are most sensitive.     

Twice upon a time: multiple concurrent temporal recalibrations of audiovisual speech

Audiovisual timing perception can recalibrate following prolonged exposure to asynchronous auditory and visual inputs. It has been suggested that this might contribute to achieving perceptual synchrony for auditory and visual signals despite differences in physical and neural signal times for sight and sound. However, given that people can be concurrently exposed to multiple audiovisual stimuli with variable neural signal times, a mechanism that recalibrates all audiovisual timing percepts to a single timing relationship could be dysfunctional. In the experiments reported here, we showed that audiovisual temporal recalibration can be specific for particular audiovisual pairings. Participants were shown alternating movies of male and female actors containing positive and negative temporal asynchronies between the auditory and visual streams. We found that audiovisual synchrony estimates for each actor were shifted toward the preceding audiovisual timing relationship for that actor and that such temporal recalibrations occurred in positive and negative directions concurrently. Our results show that humans can form multiple concurrent estimates of appropriate timing for audiovisual synchrony.     

Orientation invariance in visual object priming depends on prime—target asynchrony

Two experiments are reported in which orientation effects on visual object recognition latency were examined. In Experiment 1, we assessed picture-naming performance as a function of image-plane stimulus orientation and found increasing response times with increased misorientation of the stimulus. In Experiment 2, we examined the repetition priming effect on the identification of upright targets as a function of prime orientation. With time delays of 100, 200, or 500 msec between the onset of the prime and that of the target (i.e., stimulus onset asynchrony [SOA]), the magnitude of the priming effect decreased with increasing misorientation of the prime. These results contrast with the orientationinvariant priming effects reported in some previous repetition priming studies. These investigations all used relatively long prime—target SOAs. Confirming the crucial role of the latter variable, Experiment 2 shows that the magnitude of the repetition priming effect is invariant across prime orientations with an SOA of 1,000 msec. The possible implications of the present observations with respect to the issue of orientation invariance versus dependency of the visual object recognition process are discussed.