Short-term memory as a function of input modality

The two experiments reported are concerned with short-term memory for digit lists simultaneously presented both auditorily and visually. Results showed (1) that interpolated written and verbal recall differentially affect retention depending on whether the to-be-recalled list was presented auditorily or visually. (2) That input modality appears to be far more important for recall than was directing subjects' attention to a list during input, when that list might or might not have been subsequently required for recall. The results suggest that short-term storage is modality specific. In this case, Broadbent's P and S mechanisms do not adequately describe what happens during simultaneous visual and auditory presentation. Nor would Sperling's suggestion of a final auditory store appear to be supported.     

Organization by modality,language, and category compared

Summary An experiment was presented to test a functionalistic interpretation of the modality effect. This shows a superior recall performance for auditorily as opposed to visually presented verbal information. A total of 60 subjects were presented with mixed-mode (auditory-visual words), mixed-language (Swedish-English words), or mixed-category (category-unrelated words) lists, and were asked to recall the words of each list in any preferred order. The degree of organization according to modality, language, or category and the recall performance were measured. Organization by modality was significantly higher than organization by language or category as predicted by the functionalistic view proposed. The recall performance obtained for auditory and visual words differed in a way predicted by the functionalistic view.     

Memory storage and retrieval within and across modalities in children

Twenty-four kindergarten and fourth grade children were asked to locate a display card which had been visually or verbally presented. A probe, which identified the card to be located, was presented verbally and visually equally often. The children's ability to recall the location of an item did not differ as a function of the modality to which the material was presented. Nor was recall significantly affected when the presentation modality differed from the probe modality, suggesting that children as young as 5 can cross these sensory modalities to retrieve material with no loss in accuracy. Serial position curves suggest that the verbal and visual material is not stored in a common intersensory store. The primacy effect is found to be stronger with visually presented material and the recency effect strongest with auditorily presented material. Probe modality did not influence the serial position curves.     

The role of visual interference in producing the long-term modality effect

Typically, recall of the last of a list of auditory items greatly exceeds recall of the last of a list of visual items. This modality effect has been found in serial recall, free recall, and recall using the distractor paradigm in which each to-be-remembered item is preceded and followed by distractor activity. One source of the auditory advantage may be visual interference that reduces recall of visual stimuli. In three experiments, sources of visual interference were minimized. Although this manipulation reduced the modality effect, it did not eliminate the effect.     

Immediate serial recall of mixed-modality lists

Seven experiments are reported in which subjects were tested for immediate serial recall of mixed-modality lists. On mixed auditory-visual lists, there was an advantage for auditory items at all serial positions. This was due to both a facilitation of auditory items and an inhibition of visual items on mixed lists, as compared with single-modality lists. When presented on a list containing items read silently, recall of items that were silently mouthed by the subject demonstrated patterns similar to those found with auditory items. When presented on a list containing items read aloud, recall of mouthed items showed patterns similar to those found with silently read items. The auditory advantage on mixed lists was found even when the list items were acoustically similar or identical and was not reduced by midlist auditory suffixes. The results suggest that modality differences in recall of mixed-modality lists are based on information different from that responsible for modality differences in recall of single-modality lists.     

Enumeration produces poor primacy for tactile presentation relative to visual and auditory presentation as the only modality effect

Methodological biases may help explain the modality effect, which is superior recall of auditory recency (end of list) items relative to visual recency items. In 1985 Nairne and McNabb used a counting procedure to reduce methodological biases, and they produced modality-like effects, such that recall of tactile recency items was superior to recall of visual recency items. The present study extended Nairne and McNabb's counting procedure and controlled several variables which may have enhanced recall of tactile end items or disrupted recall of visual end items in their study. Although the results of the present study indicated general serial position effects across tactile, visual, and auditory presentation modalities, the tactile condition showed lower recall for the initial items in the presentation list than the other two conditions. Moreover, recall of the final list item did not differ across the three presentation modalities; modality effects were not found. These results did not replicate the findings of Nairne and McNabb, or much of the past research showing superior recall of auditory recency items. Implications of these findings are discussed.     

Effects of same-modality interference on immediate serial recall of auditory and visual information.

Two studies investigated the effects of same-modality interference on the immediate serial recall of auditorily and visually presented stimuli. Typically, research in which this task is used has been conducted in quiet rooms, excluding auditory information that is extraneous to the auditorily presented stimuli. However, visual information such as background items clearly within the subject's view have not been excluded during visual presentation. Therefore, in both the present studies, the authors used procedures that eliminated extra-list visual interference and introduced extra-list auditory interference. When same-modality interference was eliminated, weak visual recency effects were found, but they were smaller than those that were generated by auditorily presented items. Further, mid-list and end-of-list recall of visually presented stimuli was unaffected by the amount of interfering visual information. On the other hand, the introduction of auditory interference increased mid-list recall of auditory stimuli. The results of Experiment 2 showed that the mid-list effect occurred with a moderate, but not with a minimal or maximal, level of auditory interference, indicating that moderate amounts of auditory interference had an alerting effect that is not present in typical visual interference.     

Phonological short-term store and the nature of the recency effect: Evidence from neuropsychology

The recency effect in immediate recall of lists of unrelated words was investigated in P.V., a left hemisphere-damaged patient, who had a grossly reduced auditory verbal span, attributed to a selective impairment of a phonological short-term store. No recency effect was evident in free recall of auditorily presented material. When the patient was instructed to recall the final items of the list first, the recency performance remained defective, even though P.V. was able to adopt a recall from end order. In the case of visual presentation, P.V.'s free recall performance was within the normal range and a clear recency occurred in the recall from end condition. These results are consistent with the view that the standard recency effect in immediate free recall of auditorily presented material represents the output of a phonological short-term store to which ordinal retrieval strategies, in P.V.'s case unimpaired, are applied. Finally, the contributions of the phonological short-term store and the process of rehearsal to the recency effect and to immediate memory span performance are discussed.     

Repetition learning in the immediate serial recall of visual and auditory materials

In 5 experiments, a Hebb repetition effect, that is, improved immediate serial recall of an (unannounced) repeating list, was demonstrated in the immediate serial recall of visual materials, even when use of phonological short-term memory was blocked by concurrent articulation. The learning of a repeatedly presented letter list in one modality (auditory or visual) did not transfer to give improved performance on the same list in the other modality. This result was not replicated for word lists, however, for which asymmetric transfer was observed. Inferences are made about the structure of short-term memory and about the nature of the Hebb repetition effect.     

The auditory recency advantage in longer term free recall is not enhanced by recalling prerecency items first

Gardiner and Gregg (1979) showed that in a free-recall paradigm in which each list word is embedded in a continuous stream of subject-vocalized distractor activity, recency recall was greater when the words were presented auditorily rather than visually. The experiment described here showed that this auditory advantage persisted even when list and distractor items were both spoken at a controlled pace by the experimenter, and that it was little influenced by instructions to give priority in recall either to the beginning or to the end of the list. These results strengthen the conclusion that this effect cannot be accommodated by any echoic memory theory and, because the effect was not enhanced when prerecency items were recalled first, demonstrate an additional difference between it and the somewhat similar auditory advantage found in immediate recall.     

Accuracy of temporal coding: Auditory-visual comparisons

Three experiments were designed to decide whether temporal information is coded more accurately for intervals defined by auditory events or for those defined by visual events. In the first experiment, the irregular-list technique was used, in which a short list of items was presented, the items all separated by different interstimulus intervals. Following presentation, the subject was given three items from the list, in their correct serial order, and was asked to judge the relative interstimulus intervals. Performance was indistinguishable whether the items were presented auditorily or visually. In the second experiment, two unfilled intervals were defined by three nonverbal signals in either the auditory or the visual modality. After delays of 0, 9, or 18 sec (the latter two filled with distractor activity), the subjects were directed to make a verbal estimate of the length of one of the two intervals, which ranged from 1 to 4 sec and from 10 to 13 sec. Again, performance was not dependent on the modality of the time markers. The results of Experiment 3, which was procedurally similar to Experiment 2 but with filled rather than empty intervals, showed significant modality differences in one measure only. Within the range of intervals employed in the present study, our results provide, at best, only modest support for theories that predict more accurate temporal coding in memory for auditory, rather than visual, stimulus presentation.     

Attention in bisensory simultaneous short-term memory

Subjects were presented with simultaneously visual and auditory material. They were instructed to attend to only one modality but were required to recall materials from both modalities, one before the other. Results supported a previous suggestion that visual recall is minimum under conditions of simultaneous presentation. That is, in situations where S is allowed to divide his attention between both inputs, visual recall was no greater than when his attention was directed away from visual input and to auditory input. It was noted that this difference in recall between the two modalities was limited to verbal material, as other data indicate the opposite effect for nonverbal material.     

Tactile Short-term Memory Revisited

In spite of a large body of empirical research demonstrating the importance of multisensory integration in cognition, there is still little research about multimodal encoding and maintenance effects in working memory. In this study we investigated multimodal encoding in working memory by means of an immediate serial recall task with different modality and format conditions. In a first non-verbal condition participants were presented with sequences of non-verbal inputs representing familiar (concrete) objects, either in visual, auditory or audio-visual formats. In a second verbal condition participants were presented with written, spoken, or bimodally presented words denoting the same objects represented by pictures or sounds in the non-verbal condition. The effects of articulatory suppression were assessed in both conditions. We found a bimodal superiority effect on memory span with non-verbal material, and a larger span with auditory (or bimodal) versus visual presentation with verbal material, with a significant effect of articulatory suppression in the two conditions.     

A developmental study of the Prelinguistic Auditory Store (PAS)

Lists of digits 5 and 7 items in length were presented to second graders, sixth graders, and low-IQ sixth graders in either the visual or auditory modality. Half the auditory lists were followed by the redundant nonrecalled, auditorily presented word “recall” which served as a list suffix. The second graders had the most errors in the ordered recall task followed by the low-IQ sixth- and normal sixth-graders in that order. The size of the modality and suffix effects for the various groups seemed to indicate that, for the younger subjects, a larger proportion of the recall after auditory presentation comes from the Prelinguistic Auditory Store than for the older subjects.     

Processing of words presented simultaneously to eye and ear

The simultaneous processing of auditorily and visually presented messages was examined in three experiments. Subjects searched lists of words for a target word while processing auditorily presented information. Across conditions, subjects searched for (a) target words in a list of words presented auditorily, (b) the same target words in lists presented visually, (c) a member of a taxonomic category in a visually presented list, and (d) a rhyme in a list of words presented visually. The level of processing of a simultaneous auditory message varied across experiments. In experiment 1, subjects shadowed lists of digits. In Experiment 2, subjects reported the antonym of each word in a list. In Experiment 3, subjects named the taxonomic category of each word in a list. In all three experiments, subject had high detection rates for target words presented visually and for category targets but low detection rates for target words presented auditorily and for rhyme targets. These results suggest that processing the semantic properties, but not the acoustic properties, of words presented to the visual modality is independent of simultaneous processing in the auditory modality. Implication for models of selective attention are discussed.     

Does the testing effect depend on presentation modality?

We examined whether the testing effect generalizes to an auditory presentation modality. Five lists of unrelated words (Experiment 1) and related words (Experiment 2) were presented to participants, half of whom studied them visually and half studied them auditorily. Participants in the study-only condition performed a short distractor task following lists 1–4, whereas those in the testing condition completed a short distractor task and then attempted to recall each list. Both groups were subsequently tested on List 5 and on all five lists 30 min later. In both experiments, we found a testing effect for both List 5 and for the final cumulative recall test. However, the effect did not interact with study modality, despite the fact that proactive interference was greater following auditory study. These results have important implications for educational practice, suggesting that initial testing is important for materials presented in auditory as well as visual formats.     

Order of recall,output interference,and the modality effect

Summary In free recall, the order of recall following auditory and visual presentation differs; it tends to be forward for auditory but backward for visual. The first two experiments examined to what extent this difference in output order could account for the modality effect (i.e., a superior retention of auditorily as opposed to visually presented words). Order of recall was manipulated using postcued (Experiment 1) and precued (Experiment 2) procedures. Whereas the modality effect was unaffected with postcueing it was reduced to approximately half its size with precueing. It was concluded from these two studies that although output order cannot explain the whole modality effect, it does seem to play an important role for part of the effect in some situations. Experiments 3 and 4 used a mixed-mode and a probed recall procedure, respectively, to examine the role of output interference in modality experiments. The data suggested that output interference effects were non-monotonic; they were greater for visual than for auditory early in recall, but apparently no different later in recall. The two-store hypotheses (Murdock and Walker, 1969) was elaborated slightly to account for these results.This research was supported by Research Grants APA 146 from the National Research Council of Canada and OMHF 164 from the Ontario Mental Health Foundation. We would like to thank Doris Glavnov and Janet Metcalfe for help with the data analyses.     

Echoic and retrieval accounts of the long-term modality effect tested using the suffix procedure

The long-term modality effect is the advantage in recall of the last of a list of auditory to-be-remembered (TBR) items compared with the last of a list of visual TBR items when the list is followed by a filled retention interval. If the auditory advantage is due to echoic sensory memory mechanisms, then recall of the last auditory TBR item should be substantially reduced when it is followed by a redundant, not-to-be-recalled auditory suffix. Contrary to this prediction, Experiment 1 demonstrated that a redundant auditory suffix does not significantly reduce recall of the last auditory TBR item. In Experiment 2 a nonredundant auditory suffix produced a large reduction in the last auditory item. Redundancy is not the only factor controlling the effectiveness of a suffix, however. Experiment 3 demonstrated that a nonredundant visual suffix does not reduce recall of the last auditory TBR item. These results are discussed in reference to a retrieval account of the long-term modality effect.     

Paradoxical effects of presentation modality on false memory.

Two experiments are reported in which the effects of presentation modality on false memory in recall and recognition are studied. False recall and recognition of critical targets are lower for non-presented items related to a study list when that study list is presented visually than when presented auditorily. This pattern of low levels of false memory for critical targets holds even when participants read the visually presented study items aloud. These results suggest that recollection of visual detail plays a role in the prevention of false memory. However, both the hit rates (true memory) and the false-alarm rates to weakly related distractors (non-critical targets) were higher for visual presentation than for auditory presentation, suggesting that more than one mechanism may underlie false recognition.     

Temporal coding in rhythm tasks revealed by modality effects

Temporal coding has been studied by examining the perception and reproduction of rhythms and by examining memory for the order of events in a list. We attempt to link these research programs both empirically and theoretically. Glenberg and Swanson (1986) proposed that the superior recall of auditory material, compared with visual material, reflects more accurate temporal coding for the auditory material. In this paper, we demonstrate that a similar modality effect can be produced in a rhythm task. Auditory rhythms composed of stimuli of two durations are reproduced more accurately than are visual rhythms. Furthermore, it appears that the auditory superiority reflects enhanced chunking of the auditory material rather than better identification of durations.