The other modality: Auditory stimuli in language switching

Language switching studies typically implement visual stimuli and visual language cues to trigger a concept and a language response, respectively. In the present study we set out to generalise this to another stimulus modality by investigating language switching with auditory stimuli next to visual stimuli. The results showed that switch costs can be obtained with both auditory and visual stimuli. Yet, switch costs were relatively larger with visual stimuli than with auditory stimuli. Both methodological and theoretical implications of these findings are discussed.     

Multisensory identification of natural objects in a two-way crossmodal priming paradigm

The question of how vision and audition interact in natural object identification is currently a matter of debate. We developed a large set of auditory and visual stimuli representing natural objects in order to facilitate research in the field of multisensory processing. Normative data was obtained for 270 brief environmental sounds and 320 visual object stimuli. Each stimulus was named, categorized, and rated with regard to familiarity and emotional valence by N=56 participants (Study 1). This multimodal stimulus set was employed in two subsequent crossmodal priming experiments that used semantically congruent and incongruent stimulus pairs in a S1-S2 paradigm. Task-relevant targets were either auditory (Study 2) or visual stimuli (Study 3). The behavioral data of both experiments expressed a crossmodal priming effect with shorter reaction times for congruent as compared to incongruent stimulus pairs. The observed facilitation effect suggests that object identification in one modality is influenced by input from another modality. This result implicates that congruent visual and auditory stimulus pairs were perceived as the same object and demonstrates a first validation of the multimodal stimulus set.     

面孔表情和声音情绪信息整合对返回抑制的影响

返回抑制(inhibition of return, IOR)与情绪刺激都具有引导注意偏向、提高搜索效率的特点, 但二者间是否存在一定的交互作用迄今为止尚不明确。研究采用“线索-目标”范式并在视听双通道呈现情绪刺激来考察情绪刺激的加工与IOR的交互作用。实验1中情绪刺激以单通道视觉面孔或一致的视听双通道呈现, 实验2通过在视听通道呈现不一致的情绪刺激进一步考察视听双通道情绪一致刺激对IOR的影响是否是由听觉通道一致的情绪刺激导致的, 即是否对听觉通道的情绪刺激进行了加工。结果发现, 视听双通道情绪一致刺激能够削弱IOR, 但情绪不一致刺激与IOR之间不存在交互作用, 并且单双通道的IOR不存在显著差异。研究结果表明仅在视听双通道呈现情绪一致刺激时, 才会影响同一阶段的IOR, 这进一步支持了IOR的知觉抑制理论。     

跨感觉通路ERP注意成分的研究

采用提高了非注意纯度的“跨通路延迟反应”实验模式,研究跨通路注意的事件相关电位（ＥＲＰ）成分,以注意ＥＲＰ减去非注意ＥＲＰ的早期差异负波（Ｎｄ１）为主要分析对象。被试者为１２名青年正常人。结果发现：①听觉偏差刺激、视觉标准与偏差刺激皆在其特异性感觉区域的头皮部位,观察到注意引起Ｎ１波幅增大的现象。Ｎｄ１的起始时间与注意Ｎ１相同,而早于非注意Ｎ１,支持注意的早期选择学说;②听觉与视觉偏差刺激产生的Ｎｄ１最大峰分布在它们各自的初级感觉投射区,而标准刺激诱发的Ｎｄ１最大峰却分布于额部,表明跨通路选择性注意对偏差刺激的加工部位主要是通路特异性的,而对标准刺激的加工部位则主要是通路上的。作者据此针对选择性注意中长期争论问题进一步提出,注意的选择性发生时程的早晚可因刺激条件而异,信息过滤器的位置具有可塑性。     

视听刺激突显性对视觉主导效应的影响

本研究使用空间任务-转换范式,控制视、听刺激的突显性,探讨自下而上注意对视觉主导效应的影响。结果表明视、听刺激突显性显著地影响视觉主导效应,实验一中当听觉刺激为高突显性时,视觉主导效应显著减弱。实验二中当听觉刺激为高突显性并且视觉刺激为低突显性时,视觉主导效应进一步减弱但依然存在。结果支持偏向竞争理论,在跨通道视听交互过程中视觉刺激更具突显性,在多感觉整合过程中更具加工优势。     

Knowledge of a second language influences auditory word recognition in the native language

Many studies in bilingual visual word recognition have demonstrated that lexical access is not language selective. However, research on bilingual word recognition in the auditory modality has been scarce, and it has yielded mixed results with regard to the degree of this language nonselectivity. In the present study, we investigated whether listening to a second language (L2) is influenced by knowledge of the native language (L1) and, more important, whether listening to the L1 is also influenced by knowledge of an L2. Additionally, we investigated whether the listener's selectivity of lexical access is influenced by the speaker's L1 (and thus his or her accent). With this aim, Dutch-English bilinguals completed an English (Experiment 1) and a Dutch (Experiment 3) auditory lexical decision task. As a control, the English auditory lexical decision task was also completed by English monolinguals (Experiment 2). Targets were pronounced by a native Dutch speaker with English as the L2 (Experiments 1A, 2A, and 3A) or by a native English speaker with Dutch as the L2 (Experiments 1B, 2B, and 3B). In all experiments, Dutch-English bilinguals recognized interlingual homophones (e.g., lief [sweet]-leaf /li:f/) significantly slower than matched control words, whereas the English monolinguals showed no effect. These results indicate that (a) lexical access in bilingual auditory word recognition is not language selective in L2, nor in L1, and (b) language-specific subphonological cues do not annul cross-lingual interactions.     

Overextinction of the orienting reaction

Sokolov’s 1963 report of return of the orienting reaction (OR) to repeated stimulus presentations following habituation was examined in 18 S’s using the galvanic skin response (GSR) component of the OR and an auditory stimulus of 60 db intensity and 1000 cps. Both group and individual S data analysis indicated a return of the OR which habituated quickly. The number of stimulus presentations to the OR return point showed an inverse relationship to rate of initial OR habituation for individual S’s.     

AUTONOMIC INDICES OF HABITUATION TO COMPLEX AND SIMPLE STIMULI

Skin conductance and heart rate were continuously recorded in 8 subjects repeatedly exposed to a perceptual-conflict test and to auditory signals. (1) Both complex and simple stimuli gave rise to a pronounced increase in skin conductance, the magnitude of response decreasing with number of repetitions. For both types of stimuli a linear relationship was obtained when GSR, measured as the change in log conductance, was plotted against the logarithm of the stimulus number. (2) Each presentation of the conflict test increased heart rate, magnitude of change decreasing approximately linearly with number of trials.     

Signal value and preliminary processes in or elicitation

The evoked cardiac response (ECR) of heart rate deceleration, respiratory pause, the peripheral pulse amplitude response (PPAR) of vascular constriction, and the galvanic skin response (GSR) were recorded from 20 subjects presented with a series of brief target and nontarget stimuli in a random order. Prestimulus vigilance was equivalent for the two types of stimuli since each had to be identified as target or nontarget before being processed, or ignored, respectively. This difference in signal value following identification affected only the GSR, supporting a conceptualization of preliminary processes in orienting response (OR) elicitation rather than Sokolov’s unitary OR concept.     

Effects of phase-of-respiration on GSR detection

Previous studies of visual and auditory signal detection (Flexman & Demaree, 1972; Flexman, 1974; Flexman, Demaree & Simpson, 1974) have found that signals occurring during exhalation are detected with greater sensitivity than are signals occuring during inhalation. The purpose of the present study was to make use of a use of a signal detection paradigm (Green & Swets, 1966) in extending these phase-of-respiration findings to internal event detection, specifically, to the detection of spontaneous galvanic skin responses (GSRs). A secondary purpose of the study was to re-examine Stern's (1972) finding that high magnitude GSRs were better detected than were low magnitude GSRs. It was found that the phase of respiration during which the GRS reached its peak significantly influenced GSR detection, but that unlike studies of external signal detection, detection was greater for GSRs peaking during the inhalation than for GSRs peaking during exhalation. Some possible artifactual sources of this finding are discussed. No significant effect of GSR magnitude on GSR detection was observed although differences were in the expected direction.     

Investigating the modality specificity of response selection using a temporal flanker task

The neurocognitive architecture for response selection is uncertain. Some theorists suggest that it is mediated by an amodal central mechanism, whereas others propose a set of independent control mechanisms. In a functional neuroimaging experiment, we investigated the nature of response selection by examining how its underlying brain mechanisms are affected by stimulus modality. To do this, we used a modified flanker task, in which the target and flanker (distractor) stimuli differed in time rather than space, making it accessible for both visual and auditory stimuli. As in the traditional flanker task, larger reaction times were observed for incongruent than congruent trials (i.e., a congruency effect) for both modalities. Congruency affected brain activation for both modalities in prefrontal cortex, parietal cortex, and the putamen. Modality-dependent activation was found in additional prefrontal and parietal regions for the visual modality and in left inferior prefrontal cortex for the auditory modality. Modality-dependent activity specifically related to response congruency was also found in sensory cortical regions. These data suggest that modality affects the brain regions throughout the cortex mediating response selection even for conceptually identical stimuli and tasks. They are consistent with the hypothesis that (at least partially) independent brain networks mediate response selection and that input modality may be a powerful factor for organizing neural activity to support task performance.     

Microsaccadic responses in a bimodal oddball task

In a visual oddball task the presentation of rare targets induces a prolonged microsaccadic inhibition as compared to standards. Here, we replicated this effect also in the auditory modality. In addition, although auditory standards induced a more limited modulation of microsaccadic frequency as compared to visual standards, auditory oddballs induced a prolonged microsaccadic inhibition. With bimodal standard stimuli the microsaccadic response was determined by the attended modality, resembling that produced by attended unimodal stimuli. The present findings support the idea that the microsaccadic response to oddball and standard stimuli is partly driven by cognitive mechanisms common to both the visual and the auditory modality, and that microsaccades can be used as an implicit behavioral measure of ongoing cognitive processes.     

Perception of auditory motion affects language processing

Previous reports have demonstrated that the comprehension of sentences describing motion in a particular direction (toward, away, up, or down) is affected by concurrently viewing a stimulus that depicts motion in the same or opposite direction. We report 3 experiments that extend our understanding of the relation between perception and language processing in 2 ways. First, whereas most previous studies of the relation between perception and language processing have focused on visual perception, our data show that sentence processing can be affected by the concurrent processing of auditory stimuli. Second, it is shown that the relation between the processing of auditory stimuli and the processing of sentences depends on whether the sentences are presented in the auditory or visual modality.     

The separate and combined effects of GSR biofeedback and delayed auditory feedback on stuttering: A preliminary study

The separate and combined effects of galvanic skin response biofeedback low wave amplitude self regulation (GSR LW) and amplified delayed auditory feedback (DAF) upon stuttering dysfluency are examined. Two college age male stutterers who were seen on an intensive longitudinal basis were the subjects in this study.Results suggest that GSR LW, when it is accomplished, functions to moderately but significantly reduce stuttering dysfluency compared to no GSR biofeedback. This effect does not noticeably dissipate over time. There occurred a marked and sustained reduction in stuttering dysfluency under DAF compared to no DAF conditions. Without separating accomplished and not accomplished GSR LW, there occurred slightly but not statistically significantly fewer blocks under the combined DAF-GSR LW condition than under DAF alone. Nevertheless, it does seem likely, considering these findings overall, that when GSR LW is accomplished, the fluency facilitating effect of its simultaneous combination with DAF is additive i.e., greater than either alone. Possible theoretical and therapeutic implications are discussed.     

Effects of retrieval practice and presentation modality on verbal learning: testing the limits of the testing effect

ABSTRACT

The testing effect refers to improved memory after retrieval practice and has been researched primarily with visual stimuli. In two experiments, we investigated whether the testing effect can be replicated when the to-be-learned information is presented auditorily, or visually?+?auditorily. Participants learned Swahili-English word pairs in one of three presentation modalities – visual, auditory, or visual?+?auditory. This was manipulated between-participants in Experiment 1 and within-participants in Experiment2. All participants studied the word pairs during three study trials. Half of participants practiced recalling the English translations in response to the Swahili cue word twice before the final test whereas the other half simply studied the word pairs twice more. Results indicated an improvement in final test performance in the repeated test condition, but only in the visual presentation modality (Experiments 1 and 2) and in the visual?+?auditory presentation modality (Experiment 2). This suggests that the benefits of practiced retrieval may be limited to information presented in a visual modality.     

Frontal brain electrical activity (EEG) and heart rate in response to affective infant-directed (ID) speech in 9-month-old infants

Many studies have shown that infants prefer infant-directed (ID) speech to adult-directed (AD) speech. ID speech functions to aid language learning, obtain and/or maintain an infant's attention, and create emotional communication between the infant and caregiver. We examined psychophysiological responses to ID speech that varied in affective content (i.e., love/comfort, surprise, fear) in a group of typically developing 9-month-old infants. Regional EEG and heart rate were collected continuously during stimulus presentation. We found the pattern of overall frontal EEG power was linearly related to affective intensity of the ID speech, such that EEG power was greatest in response to fear, than surprise than love/comfort; this linear pattern was specific to the frontal region. We also noted that heart rate decelerated to ID speech independent of affective content. As well, infants who were reported by their mothers as temperamentally distressed tended to exhibit greater relative right frontal EEG activity during baseline and in response to affective ID speech, consistent with previous work with visual stimuli and extending it to the auditory modality. Findings are discussed in terms of how increases in frontal EEG power in response to different affective intensity may reflect the cognitive aspects of emotional processing across sensory domains in infancy.     

不同类型美感体验的自主反应特点

不同美感体验类型有何特异的生理反应？美感的生理反应是否类似于积极情绪？本研究以音乐和图片两类材料为刺激,以心率、皮肤电为生理反应指标,测量了不同审美状态下的自主生理反应情况。结果发现：不论是听觉还是视觉通道,优美、壮美、悲剧、喜剧4种审美风格的刺激都使被试心率和皮肤电下降;不同的美感形态所引起的反应不同,喜剧和壮美风格刺激诱发的自主生理反应变化相对于其他两种风格显得更明显,喜剧风格效果尤为突出;不同美感体验类型特异生理反应具有跨通道的一致性;四种美感体验类型的生理反应都不同于消极情绪,而类似积极情绪,从生理反应视角证明了美感体验递属于积极情绪。     

Crossmodal attention switching: Auditory dominance in temporal discrimination tasks

Visual stimuli are often processed more efficiently than accompanying stimuli in another modality. In line with this “visual dominance”, earlier studies on attentional switching showed a clear benefit for visual stimuli in a bimodal visual–auditory modality-switch paradigm that required spatial stimulus localization in the relevant modality. The present study aimed to examine the generality of this visual dominance effect. The modality appropriateness hypothesis proposes that stimuli in different modalities are differentially effectively processed depending on the task dimension, so that processing of visual stimuli is favored in the dimension of space, whereas processing auditory stimuli is favored in the dimension of time. In the present study, we examined this proposition by using a temporal duration judgment in a bimodal visual–auditory switching paradigm. Two experiments demonstrated that crossmodal interference (i.e., temporal stimulus congruence) was larger for visual stimuli than for auditory stimuli, suggesting auditory dominance when performing temporal judgment tasks. However, attention switch costs were larger for the auditory modality than for visual modality, indicating a dissociation of the mechanisms underlying crossmodal competition in stimulus processing and modality-specific biasing of attentional set.     

A comparison of visual and auditory short-term memory in monkeys (Cebus apella)

Using a visual and an acoustic sample set that appeared to favour the auditory modality of the monkey subjects, in Experiment 1 retention gradients generated in closely comparable visual and auditory matching (go/no-go) tasks revealed a more durable short-term memory (STM) for the visual modality. In Experiment 2, potentially interfering visual and acoustic stimuli were introduced during the retention intervals of the auditory matching task. Unlike the case of visual STM, delay-interval visual stimulation did not affect auditory STM. On the other hand, delay-interval music decreased auditory STM, confirming that the monkeys maintained an auditory trace during the retention intervals. Surprisingly, monkey vocalizations injected during the retention intervals caused much less interference than music. This finding, which was confirmed by the results of Experiments 3 and 4, may be due to differential processing of “arbitrary” (the acoustic samples) and species-specific (monkey vocalizations) sounds by the subjects. Although less robust than visual STM, auditory STM was nevertheless substantial, even with retention intervals as long as 32 sec.     

Is there an advantage for recognizing multi-modal emotional stimuli?

Emotions can be recognized whether conveyed by facial expressions, linguistic cues (semantics), or prosody (voice tone). However, few studies have empirically documented the extent to which multi-modal emotion perception differs from uni-modal emotion perception. Here, we tested whether emotion recognition is more accurate for multi-modal stimuli by presenting stimuli with different combinations of facial, semantic, and prosodic cues. Participants judged the emotion conveyed by short utterances in six channel conditions. Results indicated that emotion recognition is significantly better in response to multi-modal versus uni-modal stimuli. When stimuli contained only one emotional channel, recognition tended to be higher in the visual modality (i.e., facial expressions, semantic information conveyed by text) than in the auditory modality (prosody), although this pattern was not uniform across emotion categories. The advantage for multi-modal recognition may reflect the automatic integration of congruent emotional information across channels which enhances the accessibility of emotion-related knowledge in memory.