Working memory interference during processing texts and pictures: Implications for the explanation of the modality effect

Auditory text presentation improves learning with pictures and texts. With sequential text–picture presentation, cognitive models of multimedia learning explain this modality effect in terms of greater visuo‐spatial working memory load with visual as compared to auditory texts. Visual texts are assumed to demand the same working memory subsystem as pictures, while auditory texts make use of an additional cognitive resource. We provide two alternative assumptions that relate to more basic processes: First, acoustic‐sensory information causes a retention advantage for auditory over visual texts which occurs no matter if a picture is presented or not. Second, eye movements during reading hamper visuo‐spatial rehearsal. Two experiments applying elementary procedures provide first evidence for these assumptions. Experiment 1 demonstrates that, regarding text recall, the auditory advantage is independent of visuo‐spatial working memory load. Experiment 2 reveals worse matrix recognition performance after reading text requiring eye movements than after listening or reading without eye movements. Copyright © 2008 John Wiley & Sons, Ltd.     

Comparisons of memory for nonverbal auditory and visual sequential stimuli

Properties of auditory and visual sensory memory were compared by examining subjects' recognition performance of randomly generated binary auditory sequential frequency patterns and binary visual sequential color patterns within a forced-choice paradigm. Experiment 1 demonstrated serial-position effects in auditory and visual modalities consisting of both primacy and recency effects. Experiment 2 found that retention of auditory and visual information was remarkably similar when assessed across a 10 s interval. Experiments 3 and 4, taken together, showed that the recency effect in sensory memory is affected more by the type of response required (recognition vs. reproduction) than by the sensory modality employed. These studies suggest that auditory and visual sensory memory stores for nonverbal stimuli share similar properties with respect to serial-position effects and persistence over time.     

Recognition in a visuospatial memory task: The effect of presentation

The influence of the mode of presentation (simultaneous vs. sequential) on accuracy and latency of visuospatial recognition was explored in three experiments. In Experiment 1, two groups were administered a visuospatial task in which a number of positions were shown either simultaneously or sequentially (in a random order); memory was tested using a recognition procedure of visuospatial patterns, either identical or different (a single cell displaced). The results showed that (1) performance was higher in the simultaneous than in the sequential presentation, and (2) decision time increased with complexity in the sequential presentation but not in the simultaneous presentation. In Experiment 2, the same task was used in three conditions of presentation, simultaneous, random sequential, and ordered sequential; at test, a single location, rather than a pattern, was presented for recognition. The results showed that (1) performance was higher in the simultaneous and in the ordered sequential presentations than in the random sequential one, and (2) decision time increased with complexity. In Experiment 3, the same task was used in the same three conditions of presentation, simultaneous, random sequential, and ordered sequential; at test either an identical or a “displaced” pattern was presented for recognition. The results showed that (1) performance was equivalent in the three types of presentation, and (2) decision time increased with complexity for “hit” items; different patterns of linear relations were observed for “correct rejections” items. The results are interpreted in terms of the organisation of visuospatial working memory, and three types of encoding—extrafigural spatial encoding, visual pattern encoding, and spatial path encoding—were proposed.     

Visuospatial memory and phonological loop in learning from multimedia

The dual‐task paradigm was used to show how visuospatial working memory and the phonological loop are involved in processing scientific texts and illustrations presented via computer. In experiment 1, two presentation formats were compared: text‐only and text‐with‐illustrations. With a concurrent tapping task, the beneficial effect of illustrations disappeared, while a concurrent articulatory task impaired performance similarly in both presentation formats. An analysis of individual differences revealed that this pattern of results was present in high, but not low spatial span subjects. These results support the selective involvement of visuospatial working memory in processing illustrated texts. In Experiment 2, the text‐only presentation format was compared to an illustrations‐only format. The concurrent articulatory task selectively impaired text‐only processing, compared with processing illustrations‐only. In addition, this pattern of results was found for high, but not low digit span subjects. These results suggest that individual differences define the extent to which the two subsystems of working memory are involved in learning from multimedia. These two subsystems would be mainly involved in the maintenance of a visual trace of illustrations and of a verbatim representation of linguistic information respectively, these representations being the basis for higher‐level comprehension processes. Copyright © 2002 John Wiley & Sons, Ltd.     

Stimulus dependence in probabilistic category learning

The current study investigates whether probabilistic categorization on the Weather Prediction task involves a single, modality/domain general learning mechanism or there are modality/domain differences. The same probabilistic categorization task was used in three modalities/domains and two modes of presentation. Cues consisted of visual, auditory-verbal or auditory-nonverbal stimuli, and were presented either sequentially or simultaneously. Results show that while there was no general difference in performance across modalities/domains, the mode of presentation affected them differently. In the visual modality, simultaneous performance had a general advantage over sequential presentation, while in the auditory conditions, there was an initial advantage of simultaneous presentation, which disappeared, and in the non-verbal condition, gave over to a sequential advantage in the later stages of learning. Data suggest that there are strong peripheral modality effects; however, there are no signs of modality/domain of stimuli centrally affecting categorization.     

Modality effect in false recognition: evidence from Chinese characters

Using the Deese/Roediger‐McDermott (DRM) false memory method, Smith and Hunt ( 1998 ) first reported the modality effect on false memory and showed that false recall from DRM lists was lower following visual study than following auditory study, which led to numerous studies on the mechanism of modality effect on false memory and provided many competing explanations. In the present experiment, the authors tested the modality effect in false recognition by using a blocked presentation condition and a random presentation condition. The present experiment found a modality effect different from the results of the previous research; namely, false recognition was shown to be greater following visual study than following auditory study, especially in the blocked presentation condition rather than in the random presentation condition. The authors argued that this reversed modality effect may be due to different encoding and processing characteristics between Chinese characters and English words. Compared with English words, visual graphemes of critical lures in Chinese lists are likely to be activated and encoded in participants' minds, thus it is more difficult for participants to discriminate later inner graphemes from those items presented in visual modality. Hence visual presentation could lead to more false recognition than auditory presentation in Chinese lists. The results in the present experiment demonstrated that semantic activation occurring during the encoding and retrieve phases played an important role in modality effect in false recognition, and our findings might be explained by the activation‐monitoring account.     

跨视听通道的相继错误记忆效应

本研究以12名大学生为被试采用DRM范式,考察了视觉与听觉通道在编码阶段的错误记忆ERP效应,以从更深层面认识错误记忆的内在加工机制。研究结果表明,在编码阶段,在300-500ms及500-700ms的时间窗视觉通道均存在相继错误记忆效应（DIM）,比听觉通道表现出较强的DIM效应,这不仅说明视觉通道与听觉通道在编码阶段具有不同的脑机制,而且表明DIM效应与学习项目的语义加工有着密切的关系。     

Multimedia instructions and cognitive load theory: Effects of modality and cueing

Background: Recent research on the influence of presentation format on the effectiveness of multimedia instructions has yielded some interesting results. According to cognitive load theory (Sweller, Van Merriënboer, & Paas, 1998) and Mayer's theory of multimedia learning (Mayer, 2001), replacing visual text with spoken text (the modality effect) and adding visual cues relating elements of a picture to the text (the cueing effect) both increase the effectiveness of multimedia instructions in terms of better learning results or less mental effort spent. Aims: The aim of this study was to test the generalisability of the modality and cueing effect in a classroom setting. Sample: The participants were 111 second‐year students from the Department of Education at the University of Gent in Belgium (age between 19 and 25 years). Method: The participants studied a web‐based multimedia lesson on instructional design for about one hour. Afterwards they completed a retention and a transfer test. During both the instruction and the tests, self‐report measures of mental effort were administered. Results: Adding visual cues to the pictures resulted in higher retention scores, while replacing visual text with spoken text resulted in lower retention and transfer scores. Conclusions: Only a weak cueing effect and even a reverse modality effect have been found, indicating that both effects do not easily generalise to non‐laboratory settings. A possible explanation for the reversed modality effect is that the multimedia instructions in this study were learner‐paced, as opposed to the system‐paced instructions used in earlier research.     

Testing visual short-term memory: Simultaneous versus sequential presentations

Two experiments testing immediate ordered recall are presented; in these experiments, subjects engaged in repetitive speech (“articulatory suppression”) during a visual presentation in order to prevent auditory recoding of the stimuli. In both experiments, a simultaneous presentation produced results that suggested the use of visual short-term memory, whereas a sequential presentation did not. In Experiment 1, visual confusion errors occurred more often than would be expected by chance for a simultaneous presentation but not for a sequential presentation. In Experiment 2, recall from visual short-term memory was expected to suffer more when subjects wrote a prefix than when they spoke a prefix; this effect occurred for a simultaneous presentation but not for a sequential presentation. These results suggest that existence of a visual short term store that retains a simultaneous presentation but not a sequential presentation.     

Sex differences in the ability to recognise non-verbal displays of emotion: A meta-analysis

The present study aimed to quantify the magnitude of sex differences in humans' ability to accurately recognise non-verbal emotional displays. Studies of relevance were those that required explicit labelling of discrete emotions presented in the visual and/or auditory modality. A final set of 551 effect sizes from 215 samples was included in a multilevel meta-analysis. The results showed a small overall advantage in favour of females on emotion recognition tasks (d = 0.19). However, the magnitude of that sex difference was moderated by several factors, namely specific emotion, emotion type (negative, positive), sex of the actor, sensory modality (visual, audio, audio-visual) and age of the participants. Method of presentation (computer, slides, print, etc.), type of measurement (response time, accuracy) and year of publication did not significantly contribute to variance in effect sizes. These findings are discussed in the context of social and biological explanations of sex differences in emotion recognition.     

Specific conditions for a selective deficit in memory for order in children with dyslexia

Short-term memory (STM) models distinguish between item and order memorization. The present study aims to explore how item and order STM are affected by the nature of the stimuli, the sequential versus simultaneous mode of presentation, the visual versus auditory presentation modality, the possibility of verbal recoding. A total of 20 children with dyslexia were matched one-by-one with 20 typically reading children on sex, age (8–14 years), and grade. Computerized STM tasks were administered while manipulating type (item vs. order), stimuli (letters vs. colors), sequentiality, input and output modality, as well as the presence/absence of articulatory suppression and distractors. General Linear Model analyses were conducted on accuracy scores for item and order STM. Both item and order recall scores were lower for children with dyslexia. Although order STM in the visual input condition turned out to be more impaired than item STM in the dyslexic group, both item and order memory impairments become evident when verbal recoding is prevented through articulatory suppression. Moreover, dyslexic children, unlike typical readers, were not facilitated by the linguistic nature of the stimuli to be remembered. The present findings suggest that the often-reported selective impairment of serial memory in dyslexia is restricted to stimuli that are verbal in nature or can be verbally recoded, whereas both item and order memory impairments become evident when verbal recoding is prevented through articulatory suppression. The presence of distractors is particularly detrimental to STM in the dyslexic group. The sensitivity to distractors, suppression, and stimuli in STM is predictive of reading performance.     

Silent music reading: Auditory imagery and visuotonal modality transfer in singers and non-singers

In daily life, responses are often facilitated by anticipatory imagery of expected targets which are announced by associated stimuli from different sensory modalities. Silent music reading represents an intriguing case of visuotonal modality transfer in working memory as it induces highly defined auditory imagery on the basis of presented visuospatial information (i.e. musical notes). Using functional MRI and a delayed sequence matching-to-sample paradigm, we compared brain activations during retention intervals (10 s) of visual (VV) or tonal (TT) unimodal maintenance versus visuospatial-to-tonal modality transfer (VT) tasks. Visual or tonal sequences were comprised of six elements, white squares or tones, which were low, middle, or high regarding vertical screen position or pitch, respectively (presentation duration: 1.5 s). For the cross-modal condition (VT, session 3), the visuospatial elements from condition VV (session 1) were re-defined as low, middle or high “notes” indicating low, middle or high tones from condition TT (session 2), respectively, and subjects had to match tonal sequences (probe) to previously presented note sequences. Tasks alternately had low or high cognitive load. To evaluate possible effects of music reading expertise, 15 singers and 15 non-musicians were included. Scanner task performance was excellent in both groups. Despite identity of applied visuospatial stimuli, visuotonal modality transfer versus visual maintenance (VT > VV) induced “inhibition” of visual brain areas and activation of primary and higher auditory brain areas which exceeded auditory activation elicited by tonal stimulation (VT > TT). This transfer-related visual-to-auditory activation shift occurred in both groups but was more pronounced in experts. Frontoparietal areas were activated by higher cognitive load but not by modality transfer. The auditory brain showed a potential to anticipate expected auditory target stimuli on the basis of non-auditory information and sensory brain activation rather mirrored expectation than stimulation. Silent music reading probably relies on these basic neurocognitive mechanisms.     

视觉单通道还是视听双通道?——通道效应的元分析

多媒体学习中的通道效应认为,学习者对"画面+声音解说"组成的视听双通道材料的识记或理解效果要好于"画面+视觉文本"组成的视觉单通道材料,而逆转通道效应则发现了与之相反的结果。针对以往研究缺陷,重新采用元分析技术,以保持测验和迁移测验作为结果变量,考察通道效应的稳定性。经文献检索、纳入与排除后,共获91篇符合元分析的文献。经数据事先合并处理后,保持测验上生成94个独立效应量(8088人),迁移测验上生成83个独立效应量(6664人)。主效应检验发现:视听双通道在保持测验(d=0.24)和迁移测验(d=0.25)上均显著高于视觉单通道;调节效应检验发现:呈现步调、画面动态性和材料时长在保持测验和迁移测验上均对通道效应起到调节作用,通道效应主要在系统步调(d_(保持)=0.43,d_(迁移)=0.44)、动态画面(d_(保持)=0.50,d_(迁移)=0.59)及短时材料(d_(保持)=0.38,d_(迁移)=0.33)条件下出现;所有结果均未发现逆转通道效应。结论认为:相比视觉单通道,当图文信息以视听双通道呈现时更有利于促进学习者对多媒体学习材料的识记和理解,支持了多媒体学习认知理论;呈现步调、画面动态性以及材料时长是通道效应的重要边界条件。     

Multi-level mental representations of written,auditory, and audiovisual text in children and adults

We examined whether the comprehension of narrative texts differed between auditory, audiovisual, and written text presentations in a sample of 8- and 10-year-olds and adults. Based on multi-level theories of text comprehension that assume text comprehension to involve at least three levels of mental representation, we applied a sentence recognition task that enabled the separate assessment of the memory of the text surface, text base, and the situation model. Results indicate that 8-year-olds benefit from audiovisual and auditory text presentations in comparison with written text presentations in terms of their memory of situation model information. For 10-year-olds and adults, their text comprehension did not differ between audiovisual, auditory, and written text presentations. Additionally, the mode of text presentation had no effect on the memory of text surface and text base information.     

Reduction and elimination of format effects on recall

Two experiments investigated whether the recall advantage of pictures and spoken words over printed words in working memory (Foos & Goolkasian, 2005; Goolkasian & Foos, 2002) could be reduced by manipulating letter case and sequential versus simultaneous presentation. Participants were required to remember 3 or 6 items presented in varied presentation formats while verifying the accuracy of a sentence. Presenting words in alternating uppercase and lowercase improved recall, and presenting words simultaneously rather than successively removed the effect of presentation format. The findings suggest that when forcing participants to pay attention to printed words you can make them more memorable and thereby diminish or remove any disadvantage in the recall of printed words in comparison with pictures and spoken words.     

Tactile Short-term Memory Revisited

In spite of a large body of empirical research demonstrating the importance of multisensory integration in cognition, there is still little research about multimodal encoding and maintenance effects in working memory. In this study we investigated multimodal encoding in working memory by means of an immediate serial recall task with different modality and format conditions. In a first non-verbal condition participants were presented with sequences of non-verbal inputs representing familiar (concrete) objects, either in visual, auditory or audio-visual formats. In a second verbal condition participants were presented with written, spoken, or bimodally presented words denoting the same objects represented by pictures or sounds in the non-verbal condition. The effects of articulatory suppression were assessed in both conditions. We found a bimodal superiority effect on memory span with non-verbal material, and a larger span with auditory (or bimodal) versus visual presentation with verbal material, with a significant effect of articulatory suppression in the two conditions.     

Auditory and visual differences in time perception? An investigation from a developmental perspective with neuropsychological tests

Adults and children (5- and 8-year-olds) performed a temporal bisection task with either auditory or visual signals and either a short (0.5-1.0s) or long (4.0-8.0s) duration range. Their working memory and attentional capacities were assessed by a series of neuropsychological tests administered in both the auditory and visual modalities. Results showed an age-related improvement in the ability to discriminate time regardless of the sensory modality and duration. However, this improvement was seen to occur more quickly for auditory signals than for visual signals and for short durations rather than for long durations. The younger children exhibited the poorest ability to discriminate time for long durations presented in the visual modality. Statistical analyses of the neuropsychological scores revealed that an increase in working memory and attentional capacities in the visuospatial modality was the best predictor of age-related changes in temporal bisection performance for both visual and auditory stimuli. In addition, the poorer time sensitivity for visual stimuli than for auditory stimuli, especially in the younger children, was explained by the fact that the temporal processing of visual stimuli requires more executive attention than that of auditory stimuli.     

Evidence for different components in children's visuospatial working memory

There are a large number of studies demonstrating that visuospatial working memory (VSWM) involves different subcomponents, but there is no agreement on the identity of these dimensions. The present study attempts to combine different theoretical accounts by measuring VSWM. A battery composed of 13 tests was used to assess working memory and, in particular, the hypothesized mechanisms involved in the tasks, that is, active processing and passive recall of visual versus sequential‐spatial versus simultaneous‐spatial versus verbal tasks. The battery consisted of a number of tests already used in previous studies and new tests developed to examine specific components of working memory. We analysed the psychometric characteristics of the tests, the correlations amongst measured variables and estimated the measured variables with structural equation modelling in children attending third and fourth grades. Results revealed that the best model was composed of a specific verbal factor, three visuospatial passive factors (sequential‐spatial, simultaneous‐spatial, and visual) and one visuospatial active factor.     

The effects of auditory shadowing on recognition of information received visually

When Ss attend to one auditory message, they have no permanent memory for a second auditory message received simultaneously. Generally, it has been argued that a similar effect would occur crossmodally. This hypothesis was tested in the present experiment for messages presented to visual and auditory modalities. All Ss were tested for recognition of information presented either while shadowing or while hearing but not shadowing a passage of prose presented to one ear. One group heard a list of concrete nouns in their other ear. Three other groups received (1) printed words. (2) pictures of objects easily labeled, or (3) pictures of objects difficult to label. The shadowing task produced a decrement m recognition scores for the first three groups but not for the group receiving pictures of objects difficult to label. Further, the shadowing task interfered more with information received auditorily than with any form of visual information. These results suggest that information received visually is stored in a long-term modality-specific memory that may operate independently of the auditory modality.     

New protocol for dissociating visuospatial working memory ability in reaching space and in navigational space

Several studies have demonstrated that the processing of visuospatial memory for locations in reaching space and in navigational space is supported by independent systems, and that the coding of visuospatial information depends on the modality of the presentation (i.e., sequential or simultaneous). However, these lines of evidence and the most common neuropsychological tests used by clinicians to investigate visuospatial memory have several limitations (e.g., they are unable to analyze all the subcomponents of this function and are not directly comparable). Therefore, we developed a new battery of tests that is able to investigate these subcomponents. We recruited 71 healthy subjects who underwent sequential and simultaneous navigational tests by using an innovative sensorized platform, as well as comparable paper tests to evaluate the same components in reaching space (Exp. 1). Consistent with the literature, the principal-component method of analysis used in this study demonstrated the presence of distinct memory for sequences in different portions of space, but no distinction was found for simultaneous presentation, suggesting that different modalities of eye gaze exploration are used when subjects have to perform different types of tasks. For this purpose, an infrared Tobii Eye-Tracking X50 system was used in both spatial conditions (Exp. 2), showing that a clear effect of the presentation modality was due to the specific strategy used by subjects to explore the stimuli in space. Given these findings, the neuropsychological battery established in the present study allows us to show basic differences in the normal coding of stimuli, which can explain the specific visuospatial deficits found in various neurological conditions.