Specifying the boundary conditions of the multimedia effect: The influence of content and its distribution between text and pictures

It was investigated whether the beneficial effect of picture presentation might be influenced by the content conveyed through text and pictures and the way information is distributed between them. Ninety‐nine students learnt in five between‐subjects learning conditions (i.e., text with spatial contents plus pictures, text with visual contents plus pictures, only text with spatial contents, only text with visual contents, only picture) about a tourist centre and a holiday farm. Results showed that pictures (i.e., maps) were beneficial for learning if spatial knowledge had to be acquired, but did not support learning when non‐spatial, visual knowledge had to be acquired. Furthermore, a high overlap of spatial information in text and picture was helpful, which can be explained by the assumption that learning is a text‐guided process. On the other hand, regarding non‐spatial visual information, a high text‐picture overlap did not influence learning, probably because text alone was sufficient for acquiring visual knowledge. The implications of these findings are discussed.     

Working memory interference during processing texts and pictures: Implications for the explanation of the modality effect

Auditory text presentation improves learning with pictures and texts. With sequential text–picture presentation, cognitive models of multimedia learning explain this modality effect in terms of greater visuo‐spatial working memory load with visual as compared to auditory texts. Visual texts are assumed to demand the same working memory subsystem as pictures, while auditory texts make use of an additional cognitive resource. We provide two alternative assumptions that relate to more basic processes: First, acoustic‐sensory information causes a retention advantage for auditory over visual texts which occurs no matter if a picture is presented or not. Second, eye movements during reading hamper visuo‐spatial rehearsal. Two experiments applying elementary procedures provide first evidence for these assumptions. Experiment 1 demonstrates that, regarding text recall, the auditory advantage is independent of visuo‐spatial working memory load. Experiment 2 reveals worse matrix recognition performance after reading text requiring eye movements than after listening or reading without eye movements. Copyright © 2008 John Wiley & Sons, Ltd.     

Visuospatial working memory and mental representation of spatial descriptions

The purpose of the present research is to investigate whether different components of working memory (WM) are involved in processing spatial and nonspatial texts. The interference effects of two concurrent tasks on comprehension and recall of two kinds of text were investigated in two experiments. Each participant listened to a spatial and a nonspatial text, with one of two concurrent tasks: articulatory suppression or spatial tapping. The dependent variables in Experiment 1 were accuracy of recall and verification of information inferred from the texts. In Experiment 2 response times in the verification task were also considered. Results support the hypothesis that verbal and spatial components of working memory are differentially involved in the comprehension and memory of spatial and nonspatial texts, with a selective interference effect of the spatial concurrent task on the spatial text and an interference effect of the verbal concurrent task on both the spatial and nonspatial texts. These effects emerged for recall, sentence verification, and response times. Our findings confirm previous results showing that the verbal component of working memory is involved in the process of text comprehension and memory. In addition, they show that visuospatial working memory is involved, in so far as the text conveys visuospatial information.     

Explicit vs. implicit emotional processing: The interaction between processing type and executive control

The current study examined whether the interaction between emotion and executive control (EC) is modulated by the processing type of the emotional information. Namely, whether the emotional information is explicitly processed, implicitly processed or passively viewed. In each trial, a negative or neutral picture preceded an arrow-flanker stimulus that was congruent or incongruent. Incongruent stimuli are known to recruit EC. Explicit processing of the pictures (Experiment 1a), which required responding to their emotional content, resulted in emotional interference for congruent but not for incongruent stimuli. Similar effects were shown for the passive viewing condition (Experiment 2). In contrast, implicit processing (Experiment 1b), which required responding to non-emotional content, resulted in emotional interference for both congruent and incongruent stimuli. Thus, our findings indicate that implicit emotional processing affects performance independently of EC recruitment. In contrast, explicit emotional processing and passive viewing of emotional pictures lead to reduced emotional interference when EC is recruited.     

Hemispheric processing of anaphoric inferences: the activation of multiple antecedents

This research investigates the hemispheric processing of anaphors when readers activate multiple antecedents. Participants read texts promoting an anaphoric inference and performed a lexical decision task to inference-related target words that were consistent (Experiment 1) or inconsistent (Experiment 2) with the text. These targets were preceded by constrained or less constraining text and were presented to participants' right visual field-left hemisphere or to their left visual field-right hemisphere. In Experiment 1, both hemispheres showed facilitation for consistent antecedents and the right hemisphere showed an advantage over the left hemisphere in processing antecedents when preceded by less constrained text. In Experiment 2, the left hemisphere only showed negative facilitation for inconsistent antecedents. When readers comprehend text with multiple antecedents: both hemispheres process consistent information, the left hemisphere inhibits inconsistent information, and the right hemisphere processes less constrained information.     

Effects of incidental pictorial and verbal adjuncts on text learning

In this study, college students read and studied texts on historical figures in psychology, which were supplemented by drawings and/or brief biographies of these persons. In Experiment 1, a 2 x 2 between-groups design was conducted in which students received one adjunct with each text, both adjuncts, or neither. In Experiment 2, a single group of students received a within-subjects manipulation of the same adjunct conditions. In the between-groups comparison, students receiving biographies learned less of the target text passages, with the group receiving illustrations and biographies performing least accurately. In the within-subject conditions, texts accompanied by an illustration were better learned, with these students doing best on the text with both picture and biography. The results suggest that adjuncts may emphasize some texts, at the expense of learning from the other texts, but that too much adjunct material interferes with the learning of the target passages.     

Semantic priming effects on picture and word processing

The effects of semantic priming on picture and word processing were assessed under conditions in which subjects were required simply to identify stimuli (label pictures or read words) as rapidly as possible. Stimuli were presented in pairs (a prime followed by a target), with half of the pairs containing members of the same semantic category and half containing unrelated concepts. Semantic relatedness was found to facilitate the identification of both pictures (Experiment 1) and words (Experiment 2), and obtained interactions of semantic relatedness and stimulus quality in both experiments suggested that semantic priming affects the initial encoding of both types of stimuli. In Experiment 3, subjects received pairs of pictures, pairs of words, and mixed pairs composed of a picture and a word or of a word and a picture. Significant priming effects were obtained on mixed as well as unmixed pairs, supporting the assumption that pictures and words access semantic information from a common semantic store. Of primary interest was the significantly greater priming obtained in picture-picture pairs than in word-word or mixed pairs. This suggests that, in addition to priming that is mediated by the semantic system, priming may occur in picture-picture pairs that results from the overlap in visual features common to the pictorial representations of objects from the same semantic category.     

Tracing the time course of picture--word processing

A number of independent lines of research have suggested that semantic and articulatory information become available differentially from pictures and words. The first of the experiments reported here sought to clarify the time course by which information about pictures and words becomes available by considering the pattern of interference generated when incongruent pictures and words are presented simultaneously in a Stroop-like situation. Previous investigators report that picture naming is easily disrupted by the presence of a distracting word but that word naming is relatively immune to interference from an incongruent picture. Under the assumption that information available from a completed process may disrupt an ongoing process, these results suggest that words access articulatory information more rapidly than do pictures. Experiment 1 extended this paradigm by requiring subjects to verify the category of the target stimulus. In accordance with the hypothesis that picture access the semantic code more rapidly than words, there was a reversal in the interference pattern: Word categorization suffered considerable disruption, whereas picture categorization was minimally affected by the presence of an incongruent word. Experiment 2 sought to further test the hypothesis that access to semantic and articulatory codes is different for pictures and words by examining memory for those items following naming or categorization. Categorized words were better recognized than named words, whereas the reverse was true for pictures, a result which suggests that picture naming involves more extensive processing than picture categorization. Experiment 3 replicated this result under conditions in which viewing time was held constant. The last experiment extended the investigation of memory differences to a situation in which subjects were required to generate the superordinate category name. Here, memory for categorized pictures was as good as memory for named pictures. Category generation also influenced memory for words, memory performance being superior to that following a yes--no verification of category membership. These experiments suggest a model of information access whereby pictures access semantic information were readily than name information, with the reverse being true for words. Memory for both pictures and words was a function of the amount of processing required to access a particular type of information as well as the extent of response differentiation necessitated by the task.     

The perceptual basis of the modality effect in multimedia learning

Various studies have demonstrated an advantage of auditory over visual text modality when learning with texts and pictures. To explain this modality effect, two complementary assumptions are proposed by cognitive theories of multimedia learning: first, the visuospatial load hypothesis, which explains the modality effect in terms of visuospatial working memory overload in the visual text condition; and second, the temporal contiguity assumption, according to which the modality effect occurs because solely auditory texts and pictures can be attended to simultaneously. The latter explanation applies only to simultaneous presentation, the former to both simultaneous and sequential presentation. This paper introduces a third explanation, according to which parts of the modality effect are due to early, sensory processes. This account predicts that-for texts longer than one sentence-the modality effect with sequential presentation is restricted to the information presented most recently. Two multimedia experiments tested the influence of text modality across three different conditions: simultaneous presentation of texts and pictures versus sequential presentation versus presentation of text only. Text comprehension and picture recognition served as dependent variables. An advantage for auditory texts was restricted to the most recent text information and occurred under all presentation conditions. With picture recognition, the modality effect was restricted to the simultaneous condition. These findings clearly support the idea that the modality effect can be attributed to early processes in perception and sensory memory rather than to a working memory bottleneck.     

情境模型中时间和空间维度更新的相互影响

运用移动窗口技术考察情境模型中时间和空间维度同时或序列转变时二者间的相互关系。结果发现：(1)时间和空间同时转变时, 二者存在易化关系, 但时间对空间的易化作用更大; (2)当二者序列转变时, 以中文为材料的结果只发现时间对空间的易化关系, 而以英语为材料的结果则发现存在双向易化关系, 但时间对空间的易化作用更大。据此, 情境模型中的时间和空间维度更新的捆绑－预期假设得到证明。     

Presentation format effects in a levels-of-processing task

Three experiments were conducted to examine better performance in long-term memory when stimulus items are pictures or spoken words compared to printed words. Hypotheses regarding the allocation of attention to printed words, the semantic link between pictures and processing, and a rich long-term representation for pictures were tested. Using levels-of-processing tasks eliminated format effects when no memory test was expected and processing was deep (El), and when study and test formats did not match (E3). Pictures produced superior performance when a memory test was expected (El & 2) and when study and test formats were the same (E3). Results of all experiments support the attenuation of attention model and that picture superiority is due to a more direct access to semantic processing and a richer visual code. General principles to guide the processing of stimulus information are discussed.     

Processing global information in briefly presented pictures

Summary The effect on object recognition of the processing of the global information in contextually coherent scenes was investigated in two experiments. In Experiment 1 subjects saw 100 ms presentations of line drawings containing objects that were either usual or unusual given the picture context. Following each exposure they were required to select from among four objects the one that had been contained in the scene. The consistency of the three distractor alternatives with the meaning of the picture was varied. Recognition accuracy was poor for unusual objects and when the distractors were consistent with the picture meaning and did not substantially differ from the performance of subjects who selected response alternatives after being provided with a theme of the picture, without actually viewing the pictures. This supports the conclusion that subjects were responding on the basis of the global information but not the local object information in the pictures. When the exposure duration was increased to 2 s (Experiment 2), processing of local information was apparent for both usual and unusual objects but effects of the global information were still evident.     

Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity

We propose a multisensory framework based on Glaser and Glaser's (1989) general reading-naming interference model to account for the semantic priming effect by naturalistic sounds and spoken words on visual picture sensitivity. Four experiments were designed to investigate two key issues: First, can auditory stimuli enhance visual sensitivity when the sound leads the picture as well as when they are presented simultaneously? And, second, do naturalistic sounds (e.g., a dog's "woofing") and spoken words (e.g., /d?g/) elicit similar semantic priming effects? Here, we estimated participants' sensitivity and response criterion using signal detection theory in a picture detection task. The results demonstrate that naturalistic sounds enhanced visual sensitivity when the onset of the sounds led that of the picture by 346 ms (but not when the sounds led the pictures by 173 ms, nor when they were presented simultaneously, Experiments 1-3A). At the same SOA, however, spoken words did not induce semantic priming effects on visual detection sensitivity (Experiments 3B and 4A). When using a dual picture detection/identification task, both kinds of auditory stimulus induced a similar semantic priming effect (Experiment 4B). Therefore, we suggest that there needs to be sufficient processing time for the auditory stimulus to access its associated meaning to modulate visual perception. Besides, the interactions between pictures and the two types of sounds depend not only on their processing route to access semantic representations, but also on the response to be made to fulfill the requirements of the task.     

Spatio-temporal working-memory and short-term object-location tasks use different memory mechanisms

Spatial short-term memory for objects' locations was investigated in a spatial relocation task. During maintenance, dynamic visual noise or spatial tapping were administered as visual or spatial secondary tasks, respectively. Because memory for location should tap the visual component of working memory, a visual but not a spatial secondary task should impair location memory. In fact, neither of the tasks impaired memory (Experiment 1), although the expected dissociation between visual and spatial components was clearly confirmed for a spatio-temporal main task (Corsi test) (Experiment 2). We then contrasted location memory for pictures of objects and of nonsense figures under visual interference. Real objects were relocated much better than nonsense figures, and visual noise was again ineffective (Experiment 3). When spatial tapping was combined with the same material (Experiment 3a), again no influence on memory for locations of objects was observed and only a small influence on remembering nonsense figures. We suggest that the Corsi and the relocation VSWM-tasks use different memory mechanisms. The configuration of objects is reconstructed from perceptual records in an episodic buffer, provided by the same structures that enable visual memory after longer intervals. Rehearsal is not necessary for the persistence of these traces. In contrast, in the Corsi task remembering, a temporal sequence across homogeneous locations needs spatio-temporal marking and therefore active rehearsal of the locations by shifting spatial attention. A spatially demanding secondary task during retention interrupts this rehearsal.     

About the locus of the distractor frequency effect: Evidence from the production of clitic pronouns

In this study, we report results from two experiments in which pictures were shown with superimposed distractors that varied along two dimensions: frequency (high vs. low) and semantic relation with respect to the picture (related vs. unrelated). In one condition of Experiment 1, participants named pictures with a noun utterance; in the other condition of Experiment 1 and in Experiment 2 participants named pictures with a pronominal utterance. Low frequency distractor words produced greater interference with respect to high frequency words in noun production, but not in pronoun production. Critically, a semantic interference effect, greater interference in the semantically related than unrelated condition, was reported in both experiments, suggesting that distractor words were equally processed in both noun and pronoun conditions. These results are discussed in the context of current models of picture–word interference.     

Does text–picture integration also occur with longer text segments?

Text–picture integration is one of the most important cognitive processes when reading illustrated text. There is empirical evidence that text‐picture integration takes place when learning with pictures combined with single sentences. The present experiment investigated whether text–picture integration also takes place when the single sentences are embedded into longer text segments and hence when materials become more complex. In a within‐subjects design, 43 participants read an illustrated story, in which the different combinations of general and specific sentences and pictures, respectively, were embedded. In line with previous findings, participants were more likely to falsely recognize specific versions of the sentences after having studied their general versions combined with specific pictures. Thus, the experiment shows that text–picture integration also occurs when learners have to read longer text passages combined with pictures.     

The drawing effect: Evidence for reliable and robust memory benefits in free recall

In 7 free-recall experiments, the benefit of creating drawings of to-be-remembered information relative to writing was examined as a mnemonic strategy. In Experiments 1 and 2, participants were presented with a list of words and were asked to either draw or write out each. Drawn words were better recalled than written. Experiments 3–5 showed that the memory boost provided by drawing could not be explained by elaborative encoding (deep level of processing, LoP), visual imagery, or picture superiority, respectively. In Experiment 6, we explored potential limitations of the drawing effect, by reducing encoding time and increasing list length. Drawing, relative to writing, still benefited memory despite these constraints. In Experiment 7, the drawing effect was significant even when encoding trial types were compared in pure lists between participants, inconsistent with a distinctiveness account. Together these experiments indicate that drawing enhances memory relative to writing, across settings, instructions, and alternate encoding strategies, both within- and between-participants, and that a deep LoP, visual imagery, or picture superiority, alone or collectively, are not sufficient to explain the observed effect. We propose that drawing improves memory by encouraging a seamless integration of semantic, visual, and motor aspects of a memory trace.     

Metacomprehension judgements reflect the belief that diagrams improve learning from text

In two experiments we systematically explored whether people consider the format of text materials when judging their text learning, and whether doing so might inappropriately bias their judgements. Participants studied either text with diagrams (multimedia) or text alone and made both per-paragraph judgements and global judgements of their text learning. In Experiment 1 they judged their learning to be better for text with diagrams than for text alone. In that study, however, test performance was greater for multimedia, so the judgements may reflect either a belief in the power of multimedia or on-line processing. Experiment 2 replicated this finding and also included a third group that read texts with pictures that did not improve text performance. Judgements made by this group were just as high as those made by participants who received the effective multimedia format. These results confirm the hypothesis that people's metacomprehension judgements can be influenced by their beliefs about text format. Over-reliance on this multimedia heuristic, however, might reduce judgement accuracy in situations where it is invalid.