The modality effect and articulation

The two experiments reported in this paper investigate the influences of irrelevant articulation on the modality effect in serial recall. Subjects performed a post-list distractor task, which involved generating a well-learned alphabetic sequence for five seconds. The modality effect was impaired when subjects vocalized the letters aloud, but was unaffected by whether the sequence was silently mouthed or silently written. These results show that the modality effect does not arise from the contribution of an articulatory code to recall of the most recent auditory items. They also question the functional equivalence of recency effects for auditory, mouthed and lipread stimuli. On the basis of these and other findings, a multi-component approach to recency effects is proposed.     

The cost of expecting events in the wrong sensory modality

We examined the effects of modality expectancy on human performance. Participants judged azimuth (left vs. right location) for an unpredictable sequence of auditory, visual, and tactile targets. In some blocks, equal numbers of targets were presented in each modality. In others, the majority (75%) of the targets were presented in just one expected modality. Reaction times (RTs) for targets in an unexpected modality were slower than when that modality was expected or when no expectancy applied. RT costs associated with shifting attention from the tactile modality were greater than those for shifts from either audition or vision. Any RT benefits for the most likely modality were due to priming from an event in the same modality on the previous trial, not to the expectancy per se. These results show that stimulus-driven and expectancy-driven effects must be distinguished in studies of attending to different sensory modalities.     

Integrating speech information across talkers,gender, and sensory modality: Female faces and male voices in the McGurk effect

Studies of the McGurk effect have shown that when discrepant phonetic information is delivered to the auditory and visual modalities, the information is combined into a new percept not originally presented to either modality. In typical experiments, the auditory and visual speech signals are generated by the same talker. The present experiment examined whether a discrepancy in the gender of the talker between the auditory and visual signals would influence the magnitude of the McGurk effect. A male talker’s voice was dubbed onto a videotape containing a female talker’s face, and vice versa. The gender-incongruent videotapes were compared with gender-congruent videotapes, in which a male talker’s voice was dubbed onto a male face and a female talker’s voice was dubbed onto a female face. Even though there was a clear incompatibility in talker characteristics between the auditory and visual signals on the incongruent videotapes, the resulting magnitude of the McGurk effectwas not significantly different for the incongruent as opposed to the congruent videotapes. The results indicate that the mechanism for integrating speech information from the auditory and the visual modalities is not disrupted by a gender incompatibility even when it is perceptually apparent. The findings are compatible with the theoretical notion that information about voice characteristics of the talker is extracted and used to normalize the speech signal at an early stage of phonetic processing, prior to the integration of the auditory and the visual information.     

Effects of information load, sensory modality, and age on paced inspection performance

The effects on paced inspection performance of amount of stimulus information, presentation of information visually, as opposed to visually and auditorily, and age were investigated. In comparison to performance with one sequence of visual information, correct identifications of signals declined and false alarms increased when two sequences of visual information were monitored. Increasing the number of classes of signals in the two sequences did not significantly affect performance. However, when one of the two sequences was presented visually and the other auditorily, performance improved. There was a tendency for performance to be lower among older subjects in most experimental conditions.     

The sandwich effect reassessed: Effects of streaming, distraction, and modality

The degree of disruption from interleaving auditory irrelevant items within a sequence of to-be-remembered items--the sandwich effect--was examined in two experiments. Previous demonstrations of the effect have shown that the penalty for interleaving items is small and that changing irrelevant tokens is no more damaging than repeating ones (contrary to the classic changing state effect of irrelevant sound). The results of Experiment 1 suggest that these earlier results were due to the lack of tokens in the irrelevant sequence (in part, the result of using a span method). The results of Experiment 2 also show that the sandwich effect was marked and, further, that it comprised two elements, one due to the partitioning of relevant from irrelevant streams (which may be promoted by similarity of identity within sequence or pitch disparity between sequences), and the other a classic irrelevant sound effect (with effects of changing state). The results are discussed in terms of the role of perceptual organization within and between modalities in short-term memory.     

Integrating speech information across talkers, gender, and sensory modality: female faces and male voices in the McGurk effect.

Studies of the McGurk effect have shown that when discrepant phonetic information is delivered to the auditory and visual modalities, the information is combined into a new percept not originally presented to either modality. In typical experiments, the auditory and visual speech signals are generated by the same talker. The present experiment examined whether a discrepancy in the gender of the talker between the auditory and visual signals would influence the magnitude of the McGurk effect. A male talker's voice was dubbed onto a videotape containing a female talker's face, and vice versa. The gender-incongruent videotapes were compared with gender-congruent videotapes, in which a male talker's voice was dubbed onto a male face and a female talker's voice was dubbed onto a female face. Even though there was a clear incompatibility in talker characteristics between the auditory and visual signals on the incongruent videotapes, the resulting magnitude of the McGurk effect was not significantly different for the incongruent as opposed to the congruent videotapes. The results indicate that the mechanism for integrating speech information from the auditory and the visual modalities is not disrupted by a gender incompatibility even when it is perceptually apparent. The findings are compatible with the theoretical notion that information about voice characteristics of the talker is extracted and used to normalize the speech signal at an early stage of phonetic processing, prior to the integration of the auditory and the visual information.     

Making sense of age-related distractibility: The critical role of sensory modality

Older adults are known to have reduced inhibitory control and therefore to be more distractible than young adults. Recently, we have proposed that sensory modality plays a crucial role in age-related distractibility. In this study, we examined age differences in vulnerability to unimodal and cross-modal visual and auditory distraction. A group of 24 younger (mean age = 21.7 years) and 22 older adults (mean age = 65.4 years) performed visual and auditory n-back tasks while ignoring visual and auditory distraction. Whereas reaction time data indicated that both young and older adults are particularly affected by unimodal distraction, accuracy data revealed that older adults, but not younger adults, are vulnerable to cross-modal visual distraction. These results support the notion that age-related distractibility is modality dependent.     

Order of recall,output interference,and the modality effect

Summary In free recall, the order of recall following auditory and visual presentation differs; it tends to be forward for auditory but backward for visual. The first two experiments examined to what extent this difference in output order could account for the modality effect (i.e., a superior retention of auditorily as opposed to visually presented words). Order of recall was manipulated using postcued (Experiment 1) and precued (Experiment 2) procedures. Whereas the modality effect was unaffected with postcueing it was reduced to approximately half its size with precueing. It was concluded from these two studies that although output order cannot explain the whole modality effect, it does seem to play an important role for part of the effect in some situations. Experiments 3 and 4 used a mixed-mode and a probed recall procedure, respectively, to examine the role of output interference in modality experiments. The data suggested that output interference effects were non-monotonic; they were greater for visual than for auditory early in recall, but apparently no different later in recall. The two-store hypotheses (Murdock and Walker, 1969) was elaborated slightly to account for these results.This research was supported by Research Grants APA 146 from the National Research Council of Canada and OMHF 164 from the Ontario Mental Health Foundation. We would like to thank Doris Glavnov and Janet Metcalfe for help with the data analyses.     

On measuring selective attention to an expected sensory modality

Perceptual judgments can be affected by expectancies regarding the likely target modality. This has been taken as evidence for selective attention to particular modalities, but alternative accounts remain possible in terms of response priming, criterion shifts, stimulus repetition, and spatial confounds. We examined whether attention to a sensory modality would still be apparent when these alternatives were ruled out. Subjects made a speeded detection response (Experiment 1), an intensity or color discrimination (Experiment 2), or a spatial discrimination response (Experiments 3 and 4) for auditory and visual targets presented in a random sequence. On each trial, a symbolic visual cue predicted the likely target modality. Responses were always more rapid and accurate for targets presented in the expected versus unexpected modality, implying that people can indeed selectively attend to the auditory or visual modalities. When subjects were cued to both the probable modality of a target and its likely spatial location (Experiment 4), separable modality-cuing and spatial-cuing effects were observed. These studies introduce appropriate methods for distinguishing attention to a modality from the confounding factors that have plagued previous normal and clinical research.     

The modality effect of ego depletion: Auditory task modality reduces ego depletion

An initial act of self‐control that impairs subsequent acts of self‐control is called ego depletion. The ego depletion phenomenon has been observed consistently. The modality effect refers to the effect of the presentation modality on the processing of stimuli. The modality effect was also robustly found in a large body of research. However, no study to date has examined the modality effects of ego depletion. This issue was addressed in the current study. In Experiment 1, after all participants completed a handgrip task, one group's participants completed a visual attention regulation task and the other group's participants completed an auditory attention regulation task, and then all participants again completed a handgrip task. The ego depletion phenomenon was observed in both the visual and the auditory attention regulation task. Moreover, participants who completed the visual task performed worse on the handgrip task than participants who completed the auditory task, which indicated that there was high ego depletion in the visual task condition. In Experiment 2, participants completed an initial task that either did or did not deplete self‐control resources, and then they completed a second visual or auditory attention control task. The results indicated that depleted participants performed better on the auditory attention control task than the visual attention control task. These findings suggest that altering task modality may reduce ego depletion.     

The effect of study modality on false recognition

A number of previous studies have shown that false recognition of critical items in the Deese/Roediger-McDermott paradigm is reduced when study items are presented visually rather than auditorily; however, this effect has not been uniformly demonstrated. We investigated three potential boundary conditions of the effect of study modality in false recognition. Experiments 1 and 2 showed no reduction in false recognition following visual study presentation when the yes-no recognition test was not preceded by a recall test. Experiment 3 showed that visual study presentation can reduce false recognition without a preceding recall test, if the recognition test uses remember-know instructions. The order of the recognition test items did not influence the effect of visual study presentation on false recognition in Experiment 1. In general, the data imply that distinctive processing at study can reduce false memory in recognition if the test demands draw attention to the dimension of distinctive processing.     

Scaling of apparent accentedness by magnitude estimation and sensory modality matching

In each of three experiments, 24 students judged the accentedness present in the speech of eight Spanish-English bilinguals.Ss gave magnitude estimations and also squeezed a hand dynamometer to indicate the amount of accentedness in the reading of an English passage by each of the speakers. There was significant agreement amongSs regarding the speech samples with each scaling method, and interscale agreement was good. Power functions fitted to the data had exponents falling in the range expected from earlier psychophysical studies. Scale values correlated significantly with the frequency of accented pronunciations by the speakers as judged by two independent judges. The use of these scaling methods for future research on linguistic features of accent and on the relation between accent and language attitudes is discussed.     

Sensory modality and the word-frequency effect

Two experiments were conducted to examine whether the word-frequency effect in recognition memory is primarily a modality-dependent phenomenon. In the first experiment, the presentation modality of a target word was varied orthogonally during the input of the test phases. In the second, the subjects were forced to process each input word at the letter-byo letter level, thus minimizing the orthographical differences between the high- and low-frequency words. The word-frequency effect was found in every experimental condition and should be considered a modality-independent phenomenon. A semantically based interpretation of this effect was proposed.     

The effect of text coherence and modality on metamemory judgements

This study examined the effects of text coherence and modality on the metamemory judgements of Ease of Learning (EOL) and Judgement of Learning (JOL), and on the object-level measure of recall. Previous work found that sentence material set in a coherent, ordered text context was not judged as more memorable than that presented in a context of sentences in a disordered sequence, even though an ordered sequence helped recall (Shaddock & Carroll, 1997). The current study modified Shaddock and Carroll's design by changing the texts used from expository to narrative text. The metamemory judgements and recall were now found to be significantly more sensitive to material learned in an ordered sequence than to material learned in a disordered sequence. Also, JOLs and recall were more sensitive to material that was originally learned in an auditory mode (spoken presentation) than in a visual mode (verbal presentation). The results are discussed in terms of a cue-utilisation approach to metamemory judgements.