The role of sensory-motor modality compatibility in language processing

Language processing requires the combination of compatible (auditory-vocal and visual-manual) or incompatible (auditory-manual and visual-vocal) sensory-motor modalities, and switching between these sensory-motor modality combinations is very common in every-day life. Sensory-motor modality compatibility is defined as the similarity of stimulus modality and the modality of response-related sensory consequences. We investigated the influence of sensory-motor modality compatibility during performing language-related cognitive operations on different linguistic levels. More specifically, we used a variant of the task-switching paradigm, in which participants had to switch between compatible or between incompatible sensory-motor modality combinations during a verbal semantic categorization (Experiment 1) or during a word-form decision (Experiment 2). The data show higher switch costs (i.e., higher reaction times and error rates in switch trials compared to repetition trials) in incompatible sensory-motor modality combinations than in compatible sensory-motor modality combinations. This was true for every language-related cognitive operation, regardless of the individual linguistic level. Taken together, the present study demonstrates that sensory-motor modality compatibility plays an important role in modality switching during language processing.     

Verifying different-modality properties for concepts produces switching costs

According to perceptual symbol systems, sensorimotor simulations underlie the representation of concepts. It follows that sensorimotor phenomena should arise in conceptual processing. Previous studies have shown that switching from one modality to another during perceptual processing incurs a processing cost. If perceptual simulation underlies conceptual processing, then verifying the properties of concepts should exhibit a switching cost as well. For example, verifying a property in the auditory modality (e.g., BLENDER-loud) should be slower after verifying a property in a different modality (e.g., CRANBERRIES-tart) than after verifying a property in the same modality (e.g., LEAVES-rustling). Only words were presented to subjects, and there were no instructions to use imagery. Nevertheless, switching modalities incurred a cost, analogous to the cost of switching modalities in perception. A second experiment showed that this effect was not due to associative priming between properties in the same modality. These results support the hypothesis that perceptual simulation underlies conceptual processing.     

The role of crossmodal competition and dimensional overlap in crossmodal attention switching

Crossmodal selective attention was investigated in a cued task switching paradigm using bimodal visual and auditory stimulation. A cue indicated the imperative modality. Three levels of spatial S–R associations were established following perceptual (location), structural (numerical), and conceptual (verbal) set-level compatibility. In Experiment 1, participants switched attention between the auditory and visual modality either with a spatial-location or spatial-numerical stimulus set. In the spatial-location set, participants performed a localization judgment on left vs. right presented stimuli, whereas the spatial-numerical set required a magnitude judgment about a visually or auditorily presented number word. Single-modality blocks with unimodal stimuli were included as a control condition. In Experiment 2, the spatial-numerical stimulus set was replaced by a spatial-verbal stimulus set using direction words (e.g., “left”). RT data showed modality switch costs, which were asymmetric across modalities in the spatial-numerical and spatial-verbal stimulus set (i.e., larger for auditory than for visual stimuli), and congruency effects, which were asymmetric primarily in the spatial-location stimulus set (i.e., larger for auditory than for visual stimuli). This pattern of effects suggests task-dependent visual dominance.     

Switching between sensory and affective systems incurs processing costs

Recent models of the conceptual system hold that concepts are grounded in simulations of actual experiences with instances of those concepts in sensory-motor systems (e.g., Barsalou, 1999, 2003; Solomon & Barsalou, 2001). Studies supportive of such a viewhave shown that verifying a property of a concept in one modality, and then switching to verify a property of a different concept in a different modality generates temporal processing costs similar to the cost of switching modalities in perception. In addition to non-emotional concepts, the present experiment investigated switching costs in verifying properties of positive and negative (emotional) concepts. Properties of emotional concepts were taken from vision, audition, and the affective system. Parallel to switching costs in neutral concepts, the study showed that for positive and negative concepts, verifying properties from different modalities produced processing costs such that reaction times were longer and error rates were higher. Importantly, this effect was observed when switching from the affective system to sensory modalities, and vice-versa. These results support the embodied cognition view of emotion in humans.     

The role of preparation and cue-modality in crossmodal task switching

The present study examined the effects of cue-based preparation and cue-target modality mapping in crossmodal task switching. In two experiments, we randomly presented lateralized visual and auditory stimuli simultaneously. Subjects were asked to make a left/right judgment for a stimulus in only one of the modalities. Prior to each trial, the relevant stimulus modality was indicated by a visual or auditory cue. The cueing interval was manipulated to examine preparation. In Experiment 1, we used a corresponding mapping of cue-modality and stimulus modality, whereas in Experiment 2 the mapping of cue and stimulus modalities was reversed. We found reduced modality-switch costs with a long cueing interval, showing that attention shifts to stimulus modalities can be prepared, irrespective of cue-target modality mapping. We conclude that perceptual processing in crossmodal switching can be biased in a preparatory way towards task-relevant stimulus modalities.     

非熟练中-英双语者跨语言的错误记忆通道效应

采用DRM范式以112名非熟练中-英双语者为被试进行了跨语言的错误记忆通道效应实验。实验采用三因素混合设计,根据学习通道和测验通道的不同,把被试分为4组,用由汉语与英语单词组成的DRM词表进行学习和测验。结果发现,⑴在非熟练中-英双语被试中存在错误记忆的跨语言现象;⑵非熟练中-英双语者跨语言的错误记忆存在很强的语言特异性,表现为相同语言条件高于不同语言条件、汉语高于英语;⑶在学习与测验相同语言条件下运用校正的再认分数——敏感性指标Pr,进一步探讨错误记忆的通道效应,发现了跟西方不一致的结论：即无论汉语词表还是英语词表,非熟练中-英双语者在视觉通道上的错误记忆明显高于听觉通道     

Cross-modality translations improve recognition by reducing false alarms

Conway and Gathercole [(1990). Writing and long-term memory: Evidence for a “translation” hypothesis. The Quarterly Journal of Experimental Psychology, 42, 513–527] proposed a translation account to explain why certain types of encoding produce benefits in memory: Switching modalities from what is presented to what is encoded enhances item distinctiveness. We investigated this hypothesis in a recognition experiment in which the presentation modality of a study list (visual vs. auditory) and the encoding activity (speaking vs. typing vs. passive encoding) were manipulated between-subjects. Manipulating encoding activity between-subjects ruled out any potential influence of the relationally distinct processing that can occur in a within-subject manipulation (in which all previous translation effects have been demonstrated). We found no overall difference in memory for words presented auditorily vs. visually nor for visual vs. auditory encoding, but critically presentation modality and encoding activity did interact. Translating from one modality to another – particularly from auditory presentation to visual encoding (typing) – led to the best memory discrimination. This was largely because of reduced false alarms, not increased hits, consistent with the distinctiveness heuristic.     

Context effects in auditory implicit memory

The context effect in implicit memory is the finding that presentation of words in meaningful context reduces or eliminates repetition priming compared to words presented in isolation. Virtually all of the research on the context effect has been conducted in the visual modality but preliminary results raise the question of whether context effects are less likely in auditory priming. Context effects in the auditory modality were systematically examined in five experiments using the auditory implicit tests of word-fragment and word-stem completion. The first three experiments revealed the classical context effect in auditory priming: Words heard in isolation produced substantial priming, whereas there was little priming for the words heard in meaningful passages. Experiments 4 and 5 revealed that a meaningful context is not required for the context effect to be obtained: Words presented in an unrelated audio stream produced less priming than words presented individually and no more priming than words presented in meaningful passages. Although context effects are often explained in terms of the transfer-appropriate processing (TAP) framework, the present results are better explained by Masson and MacLeod's (2000) reduced-individuation hypothesis.     

Crossmodal attention switching: Auditory dominance in temporal discrimination tasks

Visual stimuli are often processed more efficiently than accompanying stimuli in another modality. In line with this “visual dominance”, earlier studies on attentional switching showed a clear benefit for visual stimuli in a bimodal visual–auditory modality-switch paradigm that required spatial stimulus localization in the relevant modality. The present study aimed to examine the generality of this visual dominance effect. The modality appropriateness hypothesis proposes that stimuli in different modalities are differentially effectively processed depending on the task dimension, so that processing of visual stimuli is favored in the dimension of space, whereas processing auditory stimuli is favored in the dimension of time. In the present study, we examined this proposition by using a temporal duration judgment in a bimodal visual–auditory switching paradigm. Two experiments demonstrated that crossmodal interference (i.e., temporal stimulus congruence) was larger for visual stimuli than for auditory stimuli, suggesting auditory dominance when performing temporal judgment tasks. However, attention switch costs were larger for the auditory modality than for visual modality, indicating a dissociation of the mechanisms underlying crossmodal competition in stimulus processing and modality-specific biasing of attentional set.     

The Differential Role of Syllabic Structure in Stem Completion for French and English

Four experiments were carried out to examine the role of a word's internal structure (i.e. syllables) in stem completion for French and English speakers. Subjects studied a series of unrelated words, selected so that two words shared their initial consonant-vowel-consonant (CVC) segment (e.g. BALANCE-BALCON). Subjects were then presented with CV or CVC stems (e.g. BA or BAL), half of which corresponded to the studied words' initial segment, and were asked to produce the first word that came to mind. Half the subjects performed the entire task in the auditory modality, half did so in the visual modality (Experiment 1). In both modalities, French subjects completed the stems more often with studied words in which the initial syllable matched the stem structure (e.g. BALCON for BAL) than with studied words that did not match (e.g. BALANCE for BAL). These syllabic effects were dissociable from explicit memory (Experiment 2) and appear to be language-specific, since they were obtained with French speakers but not with English speakers (Experiments 3 and 4). The results are highly consistent with the notion that implicit memory for words reflects the operations of perceptual phonological representations which are organised differently in French and English.     

Auditioning the distinctiveness account: Expanding the production effect to the auditory modality reveals the superiority of writing over vocalising

The production effect (PE) documents the advantage in memory performance for words that are read aloud during study, rather than words that are read silently. Until now, the PE was examined in the visual modality, as the participants read the study words. In the present study, we extended the PE phenomenon and used the auditory modality at study. This novel methodology provides a critical test of the distinctiveness account. Accordingly, the participants heard the study words and learned them by vocal production (saying aloud) or by writing, followed by a free recall test. The use of the auditory modality yielded a memory advantage for words that were written during study over words that were vocally produced. We explain this result in light of the encoding distinctiveness account, suggesting that the PE is determined by the number of different encoding processes involved in learning, emphasising the essential role of active production.     

Modality-specific effects in bilingual language perception

ABSTRACT

Language switching typically leads to language switch costs. Previous studies demonstrated that production-based language switching often results in reduced costs when the languages are assigned to distinct modalities (i.e. bimodal language switching) compared to switching within a modality (i.e. unimodal switching). In Experiment 1–3, we compared unimodal to bimodal switching in language perception. In Experiment 4, unimodal switching was compared to bivocal switching (i.e. languages were assigned to different voices). Experiment 1–3 revealed significantly higher switch costs for bimodal than for unimodal switching, indicating fundamental differences to language production studies. While bimodal switching leads to no (additional) costs but – under specific conditions – even to advantage in production, it increases switch costs in language perception. Moreover, Experiment 4 indicates that this is more a modality-specific rather than a general perception-based effect. Different voices instead of different modalities evoked no similar increase in switch costs.     

The other modality: Auditory stimuli in language switching

Language switching studies typically implement visual stimuli and visual language cues to trigger a concept and a language response, respectively. In the present study we set out to generalise this to another stimulus modality by investigating language switching with auditory stimuli next to visual stimuli. The results showed that switch costs can be obtained with both auditory and visual stimuli. Yet, switch costs were relatively larger with visual stimuli than with auditory stimuli. Both methodological and theoretical implications of these findings are discussed.     

The power of the written word: Comparing word meaning inferencing from auditory and written modalities throughout grade school

Previous research indicates that word learning from auditory contexts may be more effective than written context at least through fourth grade. However, no study has examined contextual differences in word learning in older school-aged children when reading abilities are more developed. Here we examined developmental differences in children’s ability to deduce the meanings of unknown words from the surrounding linguistic context in the auditory and written modalities and sought to identify the most important predictors of success in each modality. A total of 89 children aged 8–15 years were randomly assigned to either read or listen to a narrative that included eight novel words, with five exposures to each novel word. They then completed three posttests to assess word meaning inferencing. Children across all ages performed better in the written modality. Vocabulary was the only significant predictor of success on the word inferencing task. Results indicate support for written stimuli as the most effective modality for novel word meaning deduction. Our findings suggest that the presence of orthographic information facilitates novel word learning even for early, less proficient readers.     

Effects of perceptual modality on verbatim and gist memory

In two experiments, predictions of the fuzzy-trace theory of memory were tested. Perceptual information may play a role in retrieval and recognition processes for verbatim, but not for gist, memory. Perceptual modality effects were assessed in the present study by presenting three-sentence stories (e.g.,The bird is in the cage. The cage is over the table. The bird is yellow) and then testing recognition of probes that varied on three dimensions: (1) semantic accuracy (true vs. false), (2) wording (all original words vs. one novel word included), and (3) sentence type (premise vs. inference). In Experiment 1, study modality (auditory vs. visual) was manipulated, and in Experiment 2, both study and test modalities were manipulated. Despite replicating a number of findings consistent with fuzzy-trace theory (e.g., instruction and probe type effects), the results of both experiments failed to support the idea that perceptual information plays a role in performance on verbatim memory tests.     

Multisensory executive functioning

To better understand the prefrontal circuitry that putatively supports executive functions, such as those involved in switching tasks, we asked whether a current task set is open equally to receiving information from any sensory modality or if it is to some degree modality-specific. Subjects were presented with a sequence of digits to be classified as either odd/even or greater/less than five. The digits were either auditory or visual, with the modality varying randomly. Results demonstrated a reaction time (RT) cost associated with switching between tasks and also an unexpected cost of switching between modalities. When both modality and task switched, the two costs were greater than either of the costs alone, but significantly less than predicted simply by summing the two costs together (i.e., they were underadditive). These data indicate that the frontal mechanisms that allow for a switch in task are only partially modality-specific. Current theories of executive function must be adapted to account for this finding. We also suggest that the present paradigm is amenable to future research aimed at determining precisely how modalities are linked within a task set.     

跨视听通道的相继错误记忆效应

本研究以12名大学生为被试采用DRM范式,考察了视觉与听觉通道在编码阶段的错误记忆ERP效应,以从更深层面认识错误记忆的内在加工机制。研究结果表明,在编码阶段,在300-500ms及500-700ms的时间窗视觉通道均存在相继错误记忆效应（DIM）,比听觉通道表现出较强的DIM效应,这不仅说明视觉通道与听觉通道在编码阶段具有不同的脑机制,而且表明DIM效应与学习项目的语义加工有着密切的关系。     

通道呈现方式与感知学习风格的匹配性对粤-普双言切换代价的影响

选取不同感知学习风格的熟练粤-普双言者被试, 比较在不同通道呈现方式下他们刺激命名任务的表现, 由此考察感知学习风格与通道呈现方式的匹配性对熟练双言者双言切换代价的影响。结果发现, 被试在视觉线索呈现条件下要比在听觉线索条件下的切换代价小; 当感知学习风格与通道呈现方式匹配时, 双言切换代价更低。表明感知学习风格与通道呈现方式的匹配性对于双言切换代价有调节作用。     

Reduction of false recognition through haptic presentation of objects

False recognition of an item that is not presented (the lure) can occur when participants study and are tested on their recognition of items related to the lure. False recognition is reduced when the study and test modalities are congruent (e.g., both visual) rather than different (e.g., visual study and auditory test). The present study examined whether such a congruency effect occurs for haptic and auditory modalities. After studying items presented haptically or auditorily, participants took a haptic or auditory recognition test. False recognition was reduced when both the study and test were haptic, but not when the study was auditory and the test was haptic. These results indicate that cues encoded through the haptic modality can reduce false recognition.     

同音字族内的听觉通道词频效应与同音字表征的激活

对于同音词家族较大的词汇(例如汉语单音节词), 听觉通道的词汇通达和词汇表征的激活会受同音词汇歧义怎样的影响, 仍不很清楚。本研究采用了两个听觉通道的实验。实验一的听写任务发现在孤立音节的同音字选择上存在歧义的同时还存在同音字族内的高频偏向, 并非随机选择; 实验二的音-字同音判断任务对比同音与不同音条件得到同音促进量, 揭示了孤立音节在激活语音表征时还会引起同音高频字表征的自动激活, 而同音低频字则受到抑制。结果说明存在同音字族内的听觉通道词频效应, 同音的高、中、低频字的表征激活具有不同等性, 导致不穷尽通达, 且同音的最高频字得到更多通达机会。这些发现难以被现有的词汇通达和同音词表征激活模型解释, 文章提出一个能够解释这些听觉通道发现的模型。