Crossmodal transfer of arousal, but not pleasantness, from the musical to the visual domain

Arousal and valence (pleasantness) are considered primary dimensions of emotion. However, the degree to which these dimensions interact in emotional processing across sensory modalities is poorly understood. We addressed this issue by applying a crossmodal priming paradigm in which auditory primes (Romantic piano solo music) varying in arousal and/or pleasantness were sequentially paired with visual targets (IAPS pictures). In Experiment 1, the emotion spaces of 120 primes and 120 targets were explored separately in addition to the effects of musical training and gender. Thirty-two participants rated their felt pleasantness and arousal in response to primes and targets on equivalent rating scales as well as their familiarity with the stimuli. Musical training was associated with elevated familiarity ratings for high-arousing music and a trend for elevated arousal ratings, especially in response to unpleasant musical stimuli. Males reported higher arousal than females for pleasant visual stimuli. In Experiment 2, 40 nonmusicians rated their felt arousal and pleasantness in response to 20 visual targets after listening to 80 musical primes. Arousal associated with the musical primes modulated felt arousal in response to visual targets, yet no such transfer of pleasantness was observed between the two modalities. Experiment 3 sought to rule out the possibility of any order effect of the subjective ratings, and responses of 14 nonmusicians replicated results of Experiment 2. This study demonstrates the effectiveness of the crossmodal priming paradigm in basic research on musical emotions.     

Congruency effects between auditory and tactile motion: Extending the phenomenon of cross-modal dynamic capture

Behavioral studies of multisensory integration in motion perception have focused on the particular case of visual and auditory signals. Here, we addressed a new case: audition and touch. In Experiment 1, we tested the effects of an apparent motion stream presented in an irrelevant modality (audition or touch) on the perception of apparent motion streams in the other modality (touch or audition, respectively). We found significant congruency effects (lower performance when the direction of motion in the irrelevant modality was incongruent with the direction of the target) for the two possible modality combinations. This congruency effect was asymmetrical, with tactile motion distractors having a stronger influence on auditory motion perception than vice versa. In Experiment 2, we used auditory motion targets and tactile motion distractors while participants adopted one of two possible postures: arms uncrossed or arms crossed. The effects of tactile motion on auditory motion judgments were replicated in the arms-uncrossed posture, but they dissipated in the arms-crossed posture. The implications of these results are discussed in light of current findings regarding the representation of tactile and auditory space.     

Cross-modal validation of the Leipzig Affective Norms for German (LANG)

Affective stimuli are increasingly used in emotion research. Typically, stimuli are selected from databases providing affective norms. The validity of these norms is a critical factor with regard to the applicability of the stimuli for emotion research. We therefore probed the validity of the Leipzig Affective Norms for German (LANG) by correlating valence and arousal ratings across different sensory modalities. A sample of 120 words was selected from the LANG database, and auditory recordings of these words were obtained from two professional actors. The auditory stimuli were then rated again for valence and arousal. This cross-modal validation approach yielded very high correlations between auditory and visual ratings (>.95). These data confirm the strong validity of the Leipzig Affective Norms for German and encourage their use in emotion research.     

The role of crossmodal competition and dimensional overlap in crossmodal attention switching

Crossmodal selective attention was investigated in a cued task switching paradigm using bimodal visual and auditory stimulation. A cue indicated the imperative modality. Three levels of spatial S–R associations were established following perceptual (location), structural (numerical), and conceptual (verbal) set-level compatibility. In Experiment 1, participants switched attention between the auditory and visual modality either with a spatial-location or spatial-numerical stimulus set. In the spatial-location set, participants performed a localization judgment on left vs. right presented stimuli, whereas the spatial-numerical set required a magnitude judgment about a visually or auditorily presented number word. Single-modality blocks with unimodal stimuli were included as a control condition. In Experiment 2, the spatial-numerical stimulus set was replaced by a spatial-verbal stimulus set using direction words (e.g., “left”). RT data showed modality switch costs, which were asymmetric across modalities in the spatial-numerical and spatial-verbal stimulus set (i.e., larger for auditory than for visual stimuli), and congruency effects, which were asymmetric primarily in the spatial-location stimulus set (i.e., larger for auditory than for visual stimuli). This pattern of effects suggests task-dependent visual dominance.     

视听刺激突显性对视觉主导效应的影响

本研究使用空间任务-转换范式,控制视、听刺激的突显性,探讨自下而上注意对视觉主导效应的影响。结果表明视、听刺激突显性显著地影响视觉主导效应,实验一中当听觉刺激为高突显性时,视觉主导效应显著减弱。实验二中当听觉刺激为高突显性并且视觉刺激为低突显性时,视觉主导效应进一步减弱但依然存在。结果支持偏向竞争理论,在跨通道视听交互过程中视觉刺激更具突显性,在多感觉整合过程中更具加工优势。     

时间间隔对面部表情唤醒度阈下情绪启动的影响

采用3×4混合实验设计,研究了启动刺激与靶刺激的时间间隔对面部表情唤醒度阈下情绪启动效应的影响,实验因素为启动刺激的唤醒度和时间间隔。结果发现,当时间间隔为200ms、300ms、500ms时,出现了阈下情绪启动效应,时间间隔为700ms时,启动效应消失。上述结果表明阈下情绪启动效应可在唤醒度维度上实现,但此效应只能在启动刺激与靶刺激间隔时间较短的情况下出现。     

Audio-visual integration of emotional cues in song

We examined whether facial expressions of performers influence the emotional connotations of sung materials, and whether attention is implicated in audio-visual integration of affective cues. In Experiment 1, participants judged the emotional valence of audio-visual presentations of sung intervals. Performances were edited such that auditory and visual information conveyed congruent or incongruent affective connotations. In the single-task condition, participants judged the emotional connotation of sung intervals. In the dual-task condition, participants judged the emotional connotation of intervals while performing a secondary task. Judgements were influenced by melodic cues and facial expressions and the effects were undiminished by the secondary task. Experiment 2 involved identical conditions but participants were instructed to base judgements on auditory information alone. Again, facial expressions influenced judgements and the effect was undiminished by the secondary task. The results suggest that visual aspects of music performance are automatically and preattentively registered and integrated with auditory cues.     

Paying attention to emotional images with impact

Emotional stimuli receive high processing priority in attention and memory. This processing "advantage" is generally thought to be predominantly mediated by arousal. However, recent data suggest that ratings of an image's affective "impact" may be a better predictor of recollection than arousal or valence. One interpretation of these findings is that high-impact images may draw an individual's attention, thus facilitating subsequent processing. We investigated the allocation of visual attention to negative emotional images that differed in impact but were matched for valence, arousal, and other characteristics. Participants viewed a central image flanked by 2 neutral indoor or outdoor scenes and made speeded judgments about whether the neutral scenes matched. In Experiment 1, responses were slower on high-impact relative to low-impact or neutral trials. In Experiment 2, responses on high-arousal relative to low-arousal trials did not differ significantly. These data provide evidence for differential allocation of attention to distinct sets of negative, equally arousing images, and argue against the prevailing view that heightened attention to and processing of emotional stimuli relate simply to arousal or valence.     

The interaction between emotion and executive control: Comparison between visual,auditory, and tactile modalities

The reciprocal connections between emotion and attention are vital for adaptive behaviour. Previous results demonstrated that the behavioural effects of emotional stimuli on performance are attenuated when executive control is recruited. The current research studied whether this attenuation is modality dependent. In two experiments, negative and neutral pictures were presented shortly before a visual, tactile, or auditory target in a Simon task. All three modalities demonstrated a Simon effect, a conflict adaptation effect, and an emotional interference effect. However, the interaction between picture valence and Simon congruency was found only in the visual task. Specifically, when the Simon target was visual, emotional interference was reduced during incongruent compared to congruent trials. These findings suggest that although the control-related effects observed in the Simon tasks are not modality dependent, the link between emotion and executive control is modality dependent. Presumably, this link occurs only when the emotional stimulus and the target are presented in the same modality.     

Dutch modality exclusivity norms: Simulating perceptual modality in space

Perceptual information is important for the meaning of nouns. We present modality exclusivity norms for 485 Dutch nouns rated on visual, auditory, haptic, gustatory, and olfactory associations. We found these nouns are highly multimodal. They were rated most dominant in vision, and least in olfaction. A factor analysis identified two main dimensions: one loaded strongly on olfaction and gustation (reflecting joint involvement in flavor), and a second loaded strongly on vision and touch (reflecting joint involvement in manipulable objects). In a second study, we validated the ratings with similarity judgments. As expected, words from the same dominant modality were rated more similar than words from different dominant modalities; but – more importantly – this effect was enhanced when word pairs had high modality strength ratings. We further demonstrated the utility of our ratings by investigating whether perceptual modalities are differentially experienced in space, in a third study. Nouns were categorized into their dominant modality and used in a lexical decision experiment where the spatial position of words was either in proximal or distal space. We found words dominant in olfaction were processed faster in proximal than distal space compared to the other modalities, suggesting olfactory information is mentally simulated as “close” to the body. Finally, we collected ratings of emotion (valence, dominance, and arousal) to assess its role in perceptual space simulation, but the valence did not explain the data. So, words are processed differently depending on their perceptual associations, and strength of association is captured by modality exclusivity ratings.     

Perceptual congruency of audio-visual speech affects ventriloquism with bilateral visual stimuli

Many studies on multisensory processes have focused on performance in simplified experimental situations, with a single stimulus in each sensory modality. However, these results cannot necessarily be applied to explain our perceptual behavior in natural scenes where various signals exist within one sensory modality. We investigated the role of audio-visual syllable congruency on participants' auditory localization bias or the ventriloquism effect using spoken utterances and two videos of a talking face. Salience of facial movements was also manipulated. Results indicated that more salient visual utterances attracted participants' auditory localization. Congruent pairing of audio-visual utterances elicited greater localization bias than incongruent pairing, while previous studies have reported little dependency on the reality of stimuli in ventriloquism. Moreover, audio-visual illusory congruency, owing to the McGurk effect, caused substantial visual interference on auditory localization. Multisensory performance appears more flexible and adaptive in this complex environment than in previous studies.     

A comparison of visual and auditory short-term memory in monkeys (Cebus apella)

Using a visual and an acoustic sample set that appeared to favour the auditory modality of the monkey subjects, in Experiment 1 retention gradients generated in closely comparable visual and auditory matching (go/no-go) tasks revealed a more durable short-term memory (STM) for the visual modality. In Experiment 2, potentially interfering visual and acoustic stimuli were introduced during the retention intervals of the auditory matching task. Unlike the case of visual STM, delay-interval visual stimulation did not affect auditory STM. On the other hand, delay-interval music decreased auditory STM, confirming that the monkeys maintained an auditory trace during the retention intervals. Surprisingly, monkey vocalizations injected during the retention intervals caused much less interference than music. This finding, which was confirmed by the results of Experiments 3 and 4, may be due to differential processing of “arbitrary” (the acoustic samples) and species-specific (monkey vocalizations) sounds by the subjects. Although less robust than visual STM, auditory STM was nevertheless substantial, even with retention intervals as long as 32 sec.     

Norms of valence,arousal, concreteness,familiarity, imageability,and context availability for 1,100 Chinese words

In the present study, we collected valence, arousal, concreteness, familiarity, imageability, and context availability ratings for a total of 1,100 Chinese words. The ratings for all variables were collected with 9-point Likert scales. We tested the reliability of the present database by comparing it to the extant Chinese Affective Word System, and performed split-half correlations for all six variables. We then evaluated the relationships between all variables. Regarding the affective variables, we found a typical quadratic relation between valence and arousal, in line with previous findings. Likewise, significant correlations were found between the semantic variables. Importantly, we explored the relationships between ratings for the affective variables (i.e., valence and arousal) and concreteness ratings, suggesting that valence and arousal ratings can predict concreteness ratings. This database of affective norms will be a valuable source of information for emotion research that makes use of Chinese words, and will enable researchers to use highly controlled Chinese verbal stimuli to more reliably investigate the relation between cognition and emotion.     

Emotional response to musical repetition

Two experiments examined the effects of repetition on listeners' emotional response to music. Listeners heard recordings of orchestral music that contained a large section repeated twice. The music had a symmetric phrase structure (same-length phrases) in Experiment 1 and an asymmetric phrase structure (different-length phrases) in Experiment 2, hypothesized to alter the predictability of sensitivity to musical repetition. Continuous measures of arousal and valence were compared across music that contained identical repetition, variation (related), or contrasting (unrelated) structure. Listeners' emotional arousal ratings differed most for contrasting music, moderately for variations, and least for repeating musical segments. A computational model for the detection of repeated musical segments was applied to the listeners' emotional responses. The model detected the locations of phrase boundaries from the emotional responses better than from performed tempo or physical intensity in both experiments. These findings indicate the importance of repetition in listeners' emotional response to music and in the perceptual segmentation of musical structure.     

Differences in adult and adolescent listeners’ ratings of valence and arousal in emotional prosody

ABSTRACT

Judgments of emotional stimuli’s valence and arousal can differ based on the perceiver’s age. With most of the existing literature on age-related changes in such ratings based on perceptions of visually-presented pictures or words, less is known about how youth and adults perceive and rate the affective information contained in auditory emotional stimuli. The current study examined age-related differences in adolescent (n?=?31; 45% female; aged 12–17, M?=?14.35, SD?=?1.68) and adult listeners’ (n?=?30; 53% female; aged 21–30, M?=?26.20 years, SD?=?2.98) ratings of the valence and arousal of spoken words conveying happiness, anger, and a neutral expression. We also fitted closed curves to the average ratings for each emotional expression to determine their relative position on the valence-arousal plane of an affective circumplex. Compared to adults, adolescents’ ratings of emotional prosody were generally higher in valence, but more constricted in range for both valence and arousal. This pattern of ratings is suggestive of lesser differentiation amongst emotional categories’ holistic properties, which may have implications for the successful recognition and appropriate response to vocal emotional cues in adolescents’ social environments.     

Visual dominance and attention: the Colavita effect revisited

Under many conditions, humans display a robust tendency to rely more on visual information than on other forms of sensory information. Colavita (1974) illustrated this visual dominance effect by showing that naive observers typically fail to respond to clearly suprathreshold tones if these are presented simultaneously with a visual target flash. In the present study, we demonstrate that visual dominance influences performance under more complex stimulation conditions and address the role played by attention in mediating this effect. In Experiment 1, we show the Colavita effect in the simple speeded detection of line drawings and naturalistic sounds, whereas in Experiment 2 we demonstrate visual dominance when the task targets (auditory, visual, or bimodal combinations) are embedded among continuous streams of irrelevant distractors. In Experiments 3-5, we address the consequences of varying the probability of occurrence of targets in each sensory modality. In Experiment 6, we further investigate the role played by attention on visual dominance by manipulating perceptual load in either the visual or the auditory modality. Our results demonstrate that selective attention to a particular sensory modality can modulate--although not completely reverse--visual dominance as illustrated by the Colavita effect.     

Upbeat and happy: Arousal as an important factor in studying attention

The present study examined the effects of music-induced mood changes on different components of visual attention. Affective valence (positive vs. negative) and arousal (high vs. low) were manipulated by having participants listen to one of four versions of a Mozart Sonata that varied in mode (major or minor) and tempo (fast or slow). Attention was measured in three domains—alerting, orienting, and executive control. Affective valence and arousal had an effect on executive control, but not on alerting or orienting. Individuals who experienced positive valence had less efficient control over their responses than those who experienced negative valence, but only when arousal levels were high. Positive and negative valence did not influence executive control measures when arousal levels were low. These findings demonstrate that affective valence and arousal interact with one another to influence the processing of items in visual attention.     

“A sweet smile”: the modulatory role of emotion in how extrinsic factors influence taste evaluation

It has recently been demonstrated that the reported tastes/flavours of food/beverages can be modulated by means of external visual and auditory stimuli such as typeface, shapes, and music. The present study was designed to assess the role of the emotional valence of the product-extrinsic stimuli in such crossmodal modulations of taste. Participants evaluated samples of mixed fruit juice whilst simultaneously being presented with auditory or visual stimuli having either positive or negative valence. The soundtracks had either been harmonised with consonant (positive valence) or dissonant (negative valence) musical intervals. The visual stimuli consisted of images of emotional faces from the International Affective Picture System (IAPS) with valence ratings matched to the soundtracks. Each juice sample was rated on two computer-based scales: One anchored with the words sour and sweet, while the other scale required hedonic ratings. Those participants who tasted the juice sample while presented with the positively-valenced stimuli rated the juice as tasting sweeter compared to negatively-valenced stimuli, regardless of whether the stimuli were visual or auditory. These results suggest that the emotional valence of food-extrinsic stimuli can play a role in shaping food flavour evaluation and liking.     

Upbeat and happy: arousal as an important factor in studying attention

The present study examined the effects of music-induced mood changes on different components of visual attention. Affective valence (positive vs. negative) and arousal (high vs. low) were manipulated by having participants listen to one of four versions of a Mozart Sonata that varied in mode (major or minor) and tempo (fast or slow). Attention was measured in three domains-alerting, orienting, and executive control. Affective valence and arousal had an effect on executive control, but not on alerting or orienting. Individuals who experienced positive valence had less efficient control over their responses than those who experienced negative valence, but only when arousal levels were high. Positive and negative valence did not influence executive control measures when arousal levels were low. These findings demonstrate that affective valence and arousal interact with one another to influence the processing of items in visual attention.     

The insula and evaluative processes

The insula has been implicated as a component of central networks subserving evaluative and affective processes. This study examined evaluative valence and arousal ratings in response to picture stimuli in patients with lesions of the insula and two contrast groups: a control-lesion group (the primary contrast group) and an amygdala-lesion group. Patients rated the positivity and negativity of picture stimuli (from very unpleasant to very pleasant) and how emotionally arousing they found the pictures to be. Compared with patients in the control-lesion group, patients with insular lesions reported reduced arousal in response to both unpleasant and pleasant stimuli, as well as marked attenuation of valence ratings. In contrast, the arousal ratings of patients with amygdala lesions were selectively attenuated for unpleasant stimuli, and these patients' positive and negative valence ratings did not differ from those of the control-lesion group. Results support the view that the insular cortex may play a broad role in integrating affective and cognitive processes, whereas the amygdala may have a more selective role in affective arousal, especially for negative stimuli.