自我损耗的视听通道效应

采用双任务实验范式,通过两个实验探讨了自我损耗的通道效应。实验1考察了完成视听工作记忆任务对执行手柄任务（非视听任务）的影响,结果发现,完成视觉损耗任务导致手柄任务的成绩显著地变差,完成听觉损耗任务、视听非损耗任务均对手柄任务的成绩没有影响。实验2考察了完成视听工作记忆任务对执行Stroop任务（视觉任务）的影响,结果发现,完成视觉损耗任务导致Stroop任务的成绩显著地变差,完成听觉损耗任务、视听非损耗任务对Stroop任务的成绩没有影响。研究结果表明同一自我控制任务通过视听通道完成对随后另一自我控制任务的影响不同,证实了自我损耗的视听通道效应。     

基于视听双通道的音乐情绪冲突效应及优势通道研究

为探讨基于视听双通道的音乐情绪冲突效应、冲突情境下的优势加工通道和音乐经验对结果的影响,本研究采用音乐表演视频为材料,比较音乐组和非音乐组被试在一致型和不一致型视听双通道下的情绪评定速度、准确性及强度。结果发现:(1)一致型条件下的情绪评定更准确且更强烈;(2)不一致型条件下,被试更多以听觉通道的情绪线索为依据进行情绪类型评定;(3)非音乐组被试比音乐组被试更依赖视觉通道的情绪线索。结果表明:通道间情绪信息的不一致阻碍了音乐情绪加工; 听觉通道是音乐情绪冲突情境下的优势加工通道; 音乐经验降低了情绪冲突效应对音乐组被试的干扰。     

面孔表情和声音情绪信息整合对返回抑制的影响

返回抑制(inhibition of return, IOR)与情绪刺激都具有引导注意偏向、提高搜索效率的特点, 但二者间是否存在一定的交互作用迄今为止尚不明确。研究采用“线索-目标”范式并在视听双通道呈现情绪刺激来考察情绪刺激的加工与IOR的交互作用。实验1中情绪刺激以单通道视觉面孔或一致的视听双通道呈现, 实验2通过在视听通道呈现不一致的情绪刺激进一步考察视听双通道情绪一致刺激对IOR的影响是否是由听觉通道一致的情绪刺激导致的, 即是否对听觉通道的情绪刺激进行了加工。结果发现, 视听双通道情绪一致刺激能够削弱IOR, 但情绪不一致刺激与IOR之间不存在交互作用, 并且单双通道的IOR不存在显著差异。研究结果表明仅在视听双通道呈现情绪一致刺激时, 才会影响同一阶段的IOR, 这进一步支持了IOR的知觉抑制理论。     

预先信息的不确定性与准备时间对任务切换的影响

为揭示任务切换中切换代价的本质, 考察了预先信息与准备时间对任务切换的影响, 结果发现: (1) 预先信息对任务切换有显著的影响效应, 全部信息的反应时和切换代价显著低于无信息和部分信息的反应时和切换代价, 而后二者之间无显著差异; (2) 预先信息的影响效应受到准备时间的调节作用, 在准备时间为100ms时, 预先信息无显著的影响效应, 而在准备时间达到600ms以后, 预先信息的准备效应显著。这说明, 预先信息影响的认知加工是任务设置重构, 并且这种影响效应受到准备时间的调节作用。     

基于视听双通道的音乐情绪加工机制及情绪类型和音乐经验的影响

为了探讨视听双通道下的音乐情绪加工机制及音乐情绪类型和音乐训练背景对加工机制的影响，本研究采用表达开心和悲伤的音乐表演视频为材料，比较音乐组被试和非音乐组被试在单听觉通道、单视觉通道和视听双通道三种情境下的情绪评定速度、正确率和强度。结果发现：1）视听双通道与单视觉通道差异显著，与单听觉通道差异不显著。2）非音乐组被试对悲伤的评定正确率高于音乐组被试，对开心的评定正确率低于音乐组被试。说明音乐情绪加工的视听双通道整合优势仅相对单视觉通道存在；非音乐组被试对视觉通道情绪信息的变化更敏感，音乐组被试更依赖音乐经验；可在音乐表演时加入协调的视觉通道情绪信息帮助没有音乐训练经验的听赏者。     

跨通道视听觉情绪信息对婴儿图式效应的影响

以婴儿和成人不同情绪(积极、中性、消极)的面孔和声音作为实验材料,考察情绪和通道(视觉、听觉、视听觉)对婴儿图式效应的影响。结果发现,在视觉和视听觉通道中,三种情绪均能诱发婴儿图式效应,且中性情绪诱发的婴儿图式效应量最大;在听觉通道中,只有消极情绪诱发了婴儿图式效应,积极和中性情绪未诱发婴儿图式效应。在积极和中性情绪下,视觉和视听觉通道均能诱发婴儿图式效应,且诱发的婴儿图式效应量无显著差异,而听觉通道未诱发婴儿图式效应;在消极情绪下,三种通道均能诱发婴儿图式效应,且诱发的婴儿图式效应量无显著差异。结果表明,婴儿图式效应受到情绪和通道两者交互作用的影响,且情绪对婴儿图式效应的影响具有通道特异性。     

视听刺激突显性对视觉主导效应的影响

本研究使用空间任务-转换范式,控制视、听刺激的突显性,探讨自下而上注意对视觉主导效应的影响。结果表明视、听刺激突显性显著地影响视觉主导效应,实验一中当听觉刺激为高突显性时,视觉主导效应显著减弱。实验二中当听觉刺激为高突显性并且视觉刺激为低突显性时,视觉主导效应进一步减弱但依然存在。结果支持偏向竞争理论,在跨通道视听交互过程中视觉刺激更具突显性,在多感觉整合过程中更具加工优势。     

视听时、空一致性对Pip-and-Pop效应的影响

多感觉整合遵循空间原则和时间原则, 有研究表明Pip-and-Pop效应产生的原因是多感觉整合, 那么Pip-and-Pop效应是否同样遵循空间原则和时间原则呢?本研究采用动态视觉搜索范式, 通过两个眼动实验考察空间一致性(实验1)和时间一致性(实验2)对Pip-and-Pop效应的影响。结果发现：(1)当视觉目标颜色变化伴随一个同侧的听觉刺激时所产生的Pip-and-Pop效应最大, 对侧条件没有发现Pip-and-Pop效应。(2)视听刺激同时呈现所产生的Pip-and-Pop效应最大, 随视听刺激呈现时间间隔的增加Pip-and-Pop效应逐渐减弱直至消失。结果表明, 视听时、空一致性对Pip-and-Pop效应具有调节作用, 这为多感觉整合在产生Pip-and-Pop效应中的作用提供了证据。     

跨感觉通道任务中刺激驱动注意捕获的不对称性

通过两个实验探讨了来自不同感觉通道或不同空间方位的奇异刺激在视听搜索范式中的注意捕获。实验一使用定位任务探讨刺激驱动注意捕获中跨通道效应且及其不对称性, 结果发现, 当靶子在听觉通道时, 只有来自听觉同侧空间方位的奇异刺激成功的捕获了注意; 而来自视觉同侧空间和异侧空间的奇异刺激并没有捕获注意。实验二使用检测任务进一步探讨刺激驱动的注意捕获中的跨通道效应且具有不对称性, 结果发现, 来自同侧和异侧的听觉奇异刺激均捕获了注意, 但是视觉奇异刺激却不能捕获注意。研究表明, 视听搜索范式中的奇异刺激的注意捕获具有不对称性, 且存在着一种超感觉通道的注意资源。     

视觉单通道还是视听双通道?——通道效应的元分析

多媒体学习中的通道效应认为,学习者对"画面+声音解说"组成的视听双通道材料的识记或理解效果要好于"画面+视觉文本"组成的视觉单通道材料,而逆转通道效应则发现了与之相反的结果。针对以往研究缺陷,重新采用元分析技术,以保持测验和迁移测验作为结果变量,考察通道效应的稳定性。经文献检索、纳入与排除后,共获91篇符合元分析的文献。经数据事先合并处理后,保持测验上生成94个独立效应量(8088人),迁移测验上生成83个独立效应量(6664人)。主效应检验发现:视听双通道在保持测验(d=0.24)和迁移测验(d=0.25)上均显著高于视觉单通道;调节效应检验发现:呈现步调、画面动态性和材料时长在保持测验和迁移测验上均对通道效应起到调节作用,通道效应主要在系统步调(d_(保持)=0.43,d_(迁移)=0.44)、动态画面(d_(保持)=0.50,d_(迁移)=0.59)及短时材料(d_(保持)=0.38,d_(迁移)=0.33)条件下出现;所有结果均未发现逆转通道效应。结论认为:相比视觉单通道,当图文信息以视听双通道呈现时更有利于促进学习者对多媒体学习材料的识记和理解,支持了多媒体学习认知理论;呈现步调、画面动态性以及材料时长是通道效应的重要边界条件。     

内外源性空间注意对多感觉整合的影响

本文采用内-外源性空间线索靶子范式, 操控内源性线索有效性(有效线索、无效线索)、外源性线索有效性(有效线索、无效线索)、目标刺激类型(视觉刺激、听觉刺激、视听觉刺激)三个自变量。通过两个不同任务难度的实验(实验1: 简单定位任务; 实验2: 复杂辨别任务)来考察内外源性空间注意对多感觉整合的影响。两个实验结果均发现外源性空间注意显著减弱了多感觉整合效应, 内源性空间注意没有显著增强多感觉整合效应; 实验2中还发现了内源性空间注意会对外源性空间注意减弱多感觉整合效应产生影响。结果表明, 与内源性空间注意不同, 外源性空间注意对多感觉整合的影响不易受任务难度的调控; 当任务较难时内源性空间注意会影响外源性空间注意减弱多感觉整合效应的过程。由此推测, 内外源性空间注意对多感觉整合的调节并非彼此独立、而是相互影响的。     

双通道分配性注意对视听觉返回抑制的影响

本研究基于线索-靶子范式, 操纵目标刺激类型(视觉、听觉、视听觉)与线索有效性(有效线索、中性条件、无效线索)两个自变量, 通过3个实验来考察双通道分配性注意对视听觉返回抑制(inhibition of return, IOR)的影响。实验1 (听觉刺激呈现在左/右侧)结果发现, 在双通道分配性注意条件下, 视觉目标产生显著IOR效应, 而视听觉目标没有产生IOR效应; 实验2 (听觉刺激呈现在左/右侧)与实验3 (听觉刺激呈现在中央)结果发现, 在视觉通道选择性注意条件下, 视觉与视听觉目标均产生显著IOR效应但二者无显著差异。结果表明：双通道分配性注意减弱视听觉IOR效应。     

Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli

We investigated whether the “unity assumption,” according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1–3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the “unity assumption” in the domain of the multisensory temporal integration of audiovisual speech stimuli.     

The co-occurrence of multisensory competition and facilitation

Previous studies of multisensory integration have often stressed the beneficial effects that may arise when information concerning an event arrives via different sensory modalities at the same time, as, for example, exemplified by research on the redundant target effect (RTE). By contrast, studies of the Colavita visual dominance effect (e.g., [Colavita, F. B. (1974). Human sensory dominance. Perception & Psychophysics, 16, 409–412]) highlight the inhibitory consequences of the competition between signals presented simultaneously in different sensory modalities instead. Although both the RTE and the Colavita effect are thought to occur at early sensory levels and the stimulus conditions under which they are typically observed are very similar, the interplay between these two opposing behavioural phenomena (facilitation vs. competition) has yet to be addressed empirically. We hypothesized that the dissociation may reflect two of the fundamentally different ways in which humans can perceive concurrent auditory and visual stimuli. In Experiment 1, we demonstrated both multisensory facilitation (RTE) and the Colavita visual dominance effect using exactly the same audiovisual displays, by simply changing the task from a speeded detection task to a speeded modality discrimination task. Meanwhile, in Experiment 2, the participants exhibited multisensory facilitation when responding to visual targets and multisensory inhibition when responding to auditory targets while keeping the task constant. These results therefore indicate that both multisensory facilitation and inhibition can be demonstrated in reaction to the same bimodal event.     

Audio-visual integration of emotional cues in song

We examined whether facial expressions of performers influence the emotional connotations of sung materials, and whether attention is implicated in audio-visual integration of affective cues. In Experiment 1, participants judged the emotional valence of audio-visual presentations of sung intervals. Performances were edited such that auditory and visual information conveyed congruent or incongruent affective connotations. In the single-task condition, participants judged the emotional connotation of sung intervals. In the dual-task condition, participants judged the emotional connotation of intervals while performing a secondary task. Judgements were influenced by melodic cues and facial expressions and the effects were undiminished by the secondary task. Experiment 2 involved identical conditions but participants were instructed to base judgements on auditory information alone. Again, facial expressions influenced judgements and the effect was undiminished by the secondary task. The results suggest that visual aspects of music performance are automatically and preattentively registered and integrated with auditory cues.     

Crossmodal binding: evaluating the "unity assumption" using audiovisual speech stimuli

We investigated whether the "unity assumption," according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1-3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the "unity assumption" in the domain of the multisensory temporal integration of audiovisual speech stimuli.