Explaining the modality and contiguity effects: New insights from investigating students' viewing behaviour

The study examined viewing behaviour and learning outcome during multimedia learning in order to explore split‐attention processes in modality and spatial contiguity effects. Fourty students viewed a computer instruction depicting the process of lightning. Exploratory text was spoken, written near or written far from accompanying animations. Students who received spoken text outperformed students who received written text in recalling the major steps (retention) and in identifying correct solutions to problems (transfer), replicating a modality effect. Differences between near and far written text presentation in retention (d = 0.51) and transfer (d = 0.68) tests failed statistical significance. Two major characteristics concerning the learners' viewing behaviour were identified: (a) in written text presentation, learning was largely text directed and (b) learning success was related to the time learners' spent looking at animations, indicating that the processing of animations is a crucial factor in explanations of modality and spatial contiguity effects. Copyright © 2009 John Wiley & Sons, Ltd.     

Cognitive load theory,modality of presentation and the transient information effect

The modality effect occurs when audio/visual instructions are superior to visual only instructions. The effect was explored in two experiments conducted within a cognitive load theory framework. In Experiment 1, two groups of primary school students (N = 24) were presented with either audio/visual or visual only instructions on how to read a temperature graph. The group presented with visual text and a diagram rather than audio text and a diagram was superior, reversing most previous data on the modality effect. It was hypothesized that the reason for the reversal was that the transitory auditory text component was too long to be processed easily in working memory compared to more permanent written information. Experiment 2 (N = 64) replicated the experiment with the variation of a reduced length of both auditory and visual text instructions. Results indicated a reinstatement of the modality effect with audio/visual instructions proving superior to visual only instructions. Copyright © 2011 John Wiley & Sons, Ltd.     

Fostering learning from instructional video in a second language

This study investigated the most effective way to present an instructional video that contains words in the students' second language. Korean‐speaking university students received a 16‐min video lesson on Antarctica that included English narration (video + narration group), English text subtitles (video + text group), or English narration with simultaneous text subtitles (video + narration + text group). On a comprehension test, the video + text group scored higher than each of the other two groups, in contrast to the modality effect; and the video + narration + text group outscored the video + narration group, in contrast to the redundancy effect. Each of the lessons that included text was rated as less difficult than the lesson with narration only. The narration + text group reported lower effort than each of the other groups. Results highlight boundary conditions for two principles of multimedia instructional design that apply for college students who are learning in a second language. Theoretical implications are discussed.     

Learner control of the pacing of an online slideshow lesson: Does segmenting help?

How can we improve the instructional effectiveness of an online slideshow lesson? In the present study, college students received a 12‐slide multimedia slideshow lesson on how a geographic information system works. In a 2 × 2 design, the lesson was presented one complete slide at a time (large segment) or added one section of the slide at a time (small segment) when the student pressed the CONTINUE key, and the words were presented in printed form (text) or spoken form (voice). Students performed significantly better on a transfer posttest when the lesson was paced in small segments rather than large segments (d = 0.34); there was no effect or interaction involving modality. The small‐segment version was rated as less difficult than the large‐segment version (d = 0.43). The segmenting principle was supported in the context of online slideshows.     

多媒体学习中线索效应的元分析

作为多媒体学习中一种重要的指导性教学设计形式, 线索能否引导学习者的注意并促进学习效果, 目前结论尚不一致。研究运用元分析技术, 分别以保持测验、迁移测验的成绩作为学习效果的结果变量, 以线索区的注视时间、注视次数作为注意引导的结果变量, 探讨多媒体环境中的线索对学习效果和注意引导的影响, 并通过调节效应检验考察线索效应的边界条件。采用严格标准进行文献检索和筛选后, 共获取43篇符合元分析的文献。在保持测验成绩和迁移测验成绩上, 分别生成了44个(3910人)和41个(3906人)独立效应量; 在对线索区的注视时间和注视次数上, 分别生成了14个(938人)和11个(816人)独立效应量。结果发现, 线索的加入显著提高了保持测验成绩(g = 0.53)和迁移测验成绩(g = 0.36), 同时也增加了学习者对线索区的注视时间(g = 0.50)和注视次数(g = 0.70)。调节效应分析发现：在保持测验成绩上, 静态学习材料、陈述性知识、理工科条件下的线索效应分别高于动态学习材料、程序性知识和文科; 在迁移测验成绩上, 时空线索、静态学习材料、陈述性知识条件下的线索效应分别高于物理线索、动态学习材料和程序性知识; 而在眼动指标上, 线索对注意的引导主要受线索类型的调节, 物理线索条件下学习者对线索区的注视时间和注视次数均高于时空线索。结论认为, 线索的加入确实促进了学习者对学习材料的识记和理解效果, 同时也引导了学习者的注意分配; 但线索效应一定程度上受线索类型、学习材料动态性、知识类型、学科属性等因素的调节。未来研究需进一步考察线索效应的其他边界条件。     

Determinants of structural choice in visually situated sentence production

Three experiments investigated how perceptual, structural, and lexical cues affect structural choices during English transitive sentence production. Participants described transitive events under combinations of visual cueing of attention (toward either agent or patient) and structural priming with and without semantic match between the notional verb in the prime and the target event. Speakers had a stronger preference for passive-voice sentences (1) when their attention was directed to the patient, (2) upon reading a passive-voice prime, and (3) when the verb in the prime matched the target event. The verb-match effect was the by-product of an interaction between visual cueing and verb match: the increase in the proportion of passive-voice responses with matching verbs was limited to the agent-cued condition. Persistence of visual cueing effects in the presence of both structural and lexical cues suggests a strong coupling between referent-directed visual attention and Subject assignment in a spoken sentence.     

线索在多媒体学习中的作用

线索在多媒体学习中作为一种重要的教学设计形式,具有引导注意、组织和整合信息的功能。综述已有研究发现,线索具有引导学习者加工特定位置信息、忽视无关信息干扰的功能;在特定条件下帮助学习者整合图-文信息,并且提高学习者的学习效果;但对于线索信息组织功能尚缺少充分而有力的证据。关于线索对学习效果的影响,大部分研究证实线索能够促进学习。但是由于线索的多样性以及学习材料的复杂性,线索能否有效地提高学习效果仍然存在争议。从认知负荷理论视角综述发现,线索加入并没有明显降低学习者感知到的主观认知负荷;从知觉加工视角而言,线索作为突显的刺激会吸引学习者的注意,从而简化学习者在知觉加工阶段的视觉搜索。未来研究仍然需要关注以下问题:(1)线索是否真的被学习者所注意和加工?(2)重视学习材料多样性比较和不同线索类型差异探讨,(3)关注学习者经验影响,(4)如何在教学设计上保证线索的有效性。     

Working memory interference during processing texts and pictures: Implications for the explanation of the modality effect

Auditory text presentation improves learning with pictures and texts. With sequential text–picture presentation, cognitive models of multimedia learning explain this modality effect in terms of greater visuo‐spatial working memory load with visual as compared to auditory texts. Visual texts are assumed to demand the same working memory subsystem as pictures, while auditory texts make use of an additional cognitive resource. We provide two alternative assumptions that relate to more basic processes: First, acoustic‐sensory information causes a retention advantage for auditory over visual texts which occurs no matter if a picture is presented or not. Second, eye movements during reading hamper visuo‐spatial rehearsal. Two experiments applying elementary procedures provide first evidence for these assumptions. Experiment 1 demonstrates that, regarding text recall, the auditory advantage is independent of visuo‐spatial working memory load. Experiment 2 reveals worse matrix recognition performance after reading text requiring eye movements than after listening or reading without eye movements. Copyright © 2008 John Wiley & Sons, Ltd.     

Procedural Multimedia Presentations: The Effects of Working Memory and Task Complexity on Instruction Time and Assembly Accuracy

Procedural text conveys information of a series of steps to be performed. This study examined the role of verbal and visuo‐spatial WM in comprehension and execution of assembly instructions, as a function of format (text, images, multimedia) and task complexity (three or five steps). One hundred and eight participants read and executed 27 instructions to assemble a LEGO^TM object, in single and dual task conditions. Study times and errors during assembly were measured. Participants processed faster pictorial and multimedia instructions than text instructions, and made fewer errors in the execution of multimedia instructions. Dual task affected more text or picture‐only, than multimedia presentation. A verbal secondary task caused more errors in text or picture‐only presentations, and spatial secondary task also caused interference in text‐only instructions. Overall, these results support the multimedia advantage, and the role of both verbal and visuo‐spatial WM, when understanding instructions. Copyright © 2016 John Wiley & Sons, Ltd.     

多媒体学习中的图文整合:同步化线索和学习者经验的作用

多媒体学习中线索和经验可能在学习过程和效果上具有重要作用。研究控制线索与解说呈现的时间同步性,选取高、低经验大学生被试,采用SMI RED 250记录眼动,探讨同步化线索和经验对多媒体学习过程和效果的影响。结果发现:同步线索条件下测验成绩比非同步线索更高,感知难度更低,对线索区的总注视时间、首次注视时间更长,进入线索区次数和线索区注视点个数更多;高经验者测验成绩比低经验者更好,感知难度更低,但在注视加工上二者没有差异;线索和经验交互作用不显著。结论认为:同步线索更有利于图文整合,降低感知难度,促进注视加工,提高学习成绩;高经验者的经验优势表现在学习效果和主观难度感知上,而非注视加工;同步线索没有弥补低经验者的劣势。     

The perceptual basis of the modality effect in multimedia learning

Various studies have demonstrated an advantage of auditory over visual text modality when learning with texts and pictures. To explain this modality effect, two complementary assumptions are proposed by cognitive theories of multimedia learning: first, the visuospatial load hypothesis, which explains the modality effect in terms of visuospatial working memory overload in the visual text condition; and second, the temporal contiguity assumption, according to which the modality effect occurs because solely auditory texts and pictures can be attended to simultaneously. The latter explanation applies only to simultaneous presentation, the former to both simultaneous and sequential presentation. This paper introduces a third explanation, according to which parts of the modality effect are due to early, sensory processes. This account predicts that-for texts longer than one sentence-the modality effect with sequential presentation is restricted to the information presented most recently. Two multimedia experiments tested the influence of text modality across three different conditions: simultaneous presentation of texts and pictures versus sequential presentation versus presentation of text only. Text comprehension and picture recognition served as dependent variables. An advantage for auditory texts was restricted to the most recent text information and occurred under all presentation conditions. With picture recognition, the modality effect was restricted to the simultaneous condition. These findings clearly support the idea that the modality effect can be attributed to early processes in perception and sensory memory rather than to a working memory bottleneck.     

视觉单通道还是视听双通道?——通道效应的元分析

多媒体学习中的通道效应认为,学习者对"画面+声音解说"组成的视听双通道材料的识记或理解效果要好于"画面+视觉文本"组成的视觉单通道材料,而逆转通道效应则发现了与之相反的结果。针对以往研究缺陷,重新采用元分析技术,以保持测验和迁移测验作为结果变量,考察通道效应的稳定性。经文献检索、纳入与排除后,共获91篇符合元分析的文献。经数据事先合并处理后,保持测验上生成94个独立效应量(8088人),迁移测验上生成83个独立效应量(6664人)。主效应检验发现:视听双通道在保持测验(d=0.24)和迁移测验(d=0.25)上均显著高于视觉单通道;调节效应检验发现:呈现步调、画面动态性和材料时长在保持测验和迁移测验上均对通道效应起到调节作用,通道效应主要在系统步调(d_(保持)=0.43,d_(迁移)=0.44)、动态画面(d_(保持)=0.50,d_(迁移)=0.59)及短时材料(d_(保持)=0.38,d_(迁移)=0.33)条件下出现;所有结果均未发现逆转通道效应。结论认为:相比视觉单通道,当图文信息以视听双通道呈现时更有利于促进学习者对多媒体学习材料的识记和理解,支持了多媒体学习认知理论;呈现步调、画面动态性以及材料时长是通道效应的重要边界条件。     

Interactions Between Exogenous Auditory and Visual Spatial Attention

Six experiments were carried out to investigate the issue of cross-modality between exogenous auditory and visual spatial attention employing Posner's cueing paradigm in detection, localization, and discrimination tasks. Results indicated cueing in detection tasks with visual or auditory cues and visual targets but not with auditory targets (Experiment 1). In the localization tasks, cueing was found with both visual and auditory targets. Inhibition of return was apparent only in the within-modality conditions (Experiment 2). This suggests that it is important whether the attention system is activated directly (within a modality) or indirectly (between modalities). Increasing the cue validity from 50% to 80% influenced performance only in the localization task (Experiment 4). These findings are interpreted as being indicative for modality-specific but interacting attention mechanisms. The results of Experiments 5 and 6 (up/down discrimination tasks) also show cross-modal cueing but not with visual cues and auditory targets. Furthermore, there was no inhibition of return in any condition. This suggests that some cueing effects might be task dependent.     

Effects of cueing and collaboration on the acquisition of complex legal skills

Background. To overcome the teacher bandwidth problem in supporting large groups of students, both automated process support (cueing) and face‐to‐face feedback by peers during small group work (collaboration) can be provided to students. Aim. The purpose of this experimental study was to examine whether a multimedia practical containing cueing could be effectively combined with peer feedback to support the acquisition of the complex skill of preparing a plea in court. Sample. In the context of a regular court practical, 50 junior law students at a Dutch university individually studied a multimedia practical and participated in small group discussions about intermediate learning outcomes of the practical. Method. The study examined the effects of cueing and collaboration on training outcomes and transfer pleas, and on cognitive activity during collaboration, by combining the multimedia practical and small group collaboration to support the complex task of preparing a plea in court. Results. Both cueing and collaboration positively influence training outcomes, with participants without cueing benefiting most from additional collaboration. Transfer plea scores reveal a positive effect of collaboration but a negative effect of cueing. Analysis of discussions during small group collaboration reveals a negative effect of cueing on the level of cognitive activity. Conclusion. Peer feedback during small group work indeed appears to be a feasible option to be combined with or (partially) substitute individualized cueing when training complex learning tasks.     

Combined expectancy effects are modulated by the relation between expectancy cues

Studies of combined expectancies have shown that spatial cueing effects are reduced on trials on which participants have to respond with an unexpected motor response. In the first two experiments the range of reduced expectancy effects is examined. Advance knowledge of the likely response was combined in a trial-by-trial procedure with modality cueing, object cueing, and task cueing. Effects of modality cueing were reduced on trials on which the target requested an unexpected response. However, effects of object cueing as well as effects of task cueing were unaffected by response cueing. Comparing experiments revealed that different types of cues were used in different experiments. To test the effect of type of cue on the interaction of expectancies the third experiment combined spatial cueing with response cueing. When integrated cues were used that cued the likely target location by an arrow and the likely response by an arrow too, spatial cueing effects were reduced on trials with unexpected responses. However, spatial cueing effects remained unaffected by response cueing when separated cues were used consisting in a word cueing the response and an arrow cueing target location. An account for the modulation of combined expectancies by the relation between cues is suggested in terms of the adjusted expectancy model.     

Effects of instruction modality and readback on accuracy in following navigation commands

In 3 experiments, the authors simulated air traffic controllers giving pilots navigation instructions of various lengths. Participants either heard or read the instructions; repeated either all, a reduced form, or none of the instructions; and then followed them by clicking on the specified locations in a space represented by grids on a computer screen. Execution performance for visual presentation was worse than it was for auditory presentation on the longer messages. Repetition of the instructions generally lowered execution performance for longer messages, which required more output, especially with the visual modality, which required phonological receding from visual input to spoken output. An advantage for reduced over full repetition for visual but not for auditory presentation was attributed to an enhanced visual scanning process.     

Spatial attention based on 2D location and relative depth order modulates visual working memory in a 3D environment

The attentional effect on visual working memory (VWM) has been a heated research topic in the past two decades. Studies show that VWM performance for an attended memory item can be improved by cueing its two-dimensional (2D) spatial location during retention. However, few studies have investigated the effect of attentional selection on VWM in a three-dimensional setting, and it remains unknown whether depth information can produce beneficial attentional effects on 2D visual representations similar to 2D spatial information. Here we conducted four experiments, displaying memory items at various stereoscopic depth planes, and examined the retro-cue effects of four types of cues – a cue would either indicate the 2D or depth location of a memory item, and either in the form of physical (directly pointing to a location) or symbolic (numerically mapping onto a location) cues. We found that retro-cue benefits were only observed for cues directly pointing to a 2D location, whereas a null effect was observed for cues directly pointing to a depth location. However, there was a retro-cue effect when cueing the relative depth order, though the effect was weaker than that for cueing the 2D location. The selective effect on VWM based on 2D spatial attention is different from depth-based attention, and the divergence suggests that an object representation is primarily bound with its 2D spatial location, weakly bound with its depth order but not with its metric depth location. This indicates that attentional selection based on memory for depth, particularly metric depth, is ineffective.     

Textures shape the attentional focus: Evidence from exogenous and endogenous cueing

The spatial cueing paradigm (Posner Quarterly Journal of Experimental Psychology 32:3–25, 1980) has often been used to investigate the time course of the deployment of visual attention in space. In a series of eight experiments we investigated whether spatial cues would not only enhance processing of stimuli presented at cued locations, but also enhance processing of the entire texture in which the stimuli were presented. Results showed highest accuracy for responses to stimuli presented at cued locations, a replication of the traditional cueing effect (Posner 1980). Additionally, stimuli presented at uncued locations were responded to with higher accuracy when they were presented inside the same texture as the cued location, as compared with stimuli presented outside the texture with the cued location. To investigate this texture advantage for both automatic and voluntary attention deployment, exogenous and endogenous cues were used. The texture advantage was observed for short interstimulus intervals (ISIs) of 50 and 100 ms for exogenous cues and for a longer ISI of 200 ms for endogenous cues. These findings indicate that the arrangement of task-irrelevant visual stimuli also can have a large impact on the cueing effect. This suggests that visual spatial attention spreads texture-wise across the visual field. Control experiments revealed that the homogeneity within texture elements contributes most to the effect but that the texture advantage is a function of both orientation contrast at the texture border and homogeneity within texture elements.     

The influence of concurrent load on mouthed and vocalized modality effects

A visual search task was coupled with the serial recall of words to assess the extent to which modality effects mediated by vocalizing and silent mouthing reflect an automatically activated preattentive process. Overall, serial position functions systematically changed as concurrent task demands increased, but the magnitude of the modality effect associated with both mouthing and vocalizing was not altered, regardless of whether or not subjects simultaneously searched for digits. These results support the notion that modality effects index a preattentive process that can be activated automatically by either spoken input or gestural cues associated with speech.     

Automatic Mechanisms for Social Attention Are Culturally Penetrable

Are mechanisms for social attention influenced by culture? Evidence that social attention is triggered automatically by bottom‐up gaze cues and is uninfluenced by top‐down verbal instructions may suggest it operates in the same way everywhere. Yet considerations from evolutionary and cultural psychology suggest that specific aspects of one's cultural background may have consequence for the way mechanisms for social attention develop and operate. In more interdependent cultures, the scope of social attention may be broader, focusing on more individuals and relations between those individuals. We administered a multi‐gaze cueing task requiring participants to fixate a foreground face flanked by background faces and measured shifts in attention using eye tracking. For European Americans, gaze cueing did not depend on the direction of background gaze cues, suggesting foreground gaze alone drives automatic attention shifting; for East Asians, cueing patterns differed depending on whether the foreground cue matched or mismatched background cues, suggesting foreground and background gaze information were integrated. These results demonstrate that cultural background influences the social attention system by shifting it into a narrow or broad mode of operation and, importantly, provides evidence challenging the assumption that mechanisms underlying automatic social attention are necessarily rigid and impenetrable to culture.