Music to my eyes: cross-modal interactions in the perception of emotions in musical performance

We investigate non-verbal communication through expressive body movement and musical sound, to reveal higher cognitive processes involved in the integration of emotion from multiple sensory modalities. Participants heard, saw, or both heard and saw recordings of a Stravinsky solo clarinet piece, performed with three distinct expressive styles: restrained, standard, and exaggerated intention. Participants used a 5-point Likert scale to rate each performance on 19 different emotional qualities. The data analysis revealed that variations in expressive intention had their greatest impact when the performances could be seen; the ratings from participants who could only hear the performances were the same across the three expressive styles. Evidence was also found for an interaction effect leading to an emergent property, intensity of positive emotion, when participants both heard and saw the musical performances. An exploratory factor analysis revealed orthogonal dimensions for positive and negative emotions, which may account for the subjective experience that many listeners report of having multi-valent or complex reactions to music, such as “bittersweet.”     

Cross-modal interactions in the experience of musical performances: physiological correlates

This experiment was conducted to investigate cross-modal interactions in the emotional experience of music listeners. Previous research showed that visual information present in a musical performance is rich in expressive content, and moderates the subjective emotional experience of a participant listening and/or observing musical stimuli [Vines, B. W., Krumhansl, C. L., Wanderley, M. M., & Levitin, D. J. (2006). Cross-modal interactions in the perception of musical performance. Cognition, 101, 80--113.]. The goal of this follow-up experiment was to replicate this cross-modal interaction by investigating the objective, physiological aspect of emotional response to music measuring electrodermal activity. The scaled average of electrodermal amplitude for visual-auditory presentation was found to be significantly higher than the sum of the reactions when the music was presented in visual only (VO) and auditory only (AO) conditions, suggesting the presence of an emergent property created by bimodal interaction. Functional data analysis revealed that electrodermal activity generally followed the same contour across modalities of presentation, except during rests (silent parts of the performance) when the visual information took on particular salience. Finally, electrodermal activity and subjective tension judgments were found to be most highly correlated in the audio-visual (AV) condition than in the unimodal conditions. The present study provides converging evidence for the importance of seeing musical performances, and preliminary evidence for the utility of electrodermal activity as an objective measure in studies of continuous music-elicited emotions.     

Crossmodal transfer of arousal, but not pleasantness, from the musical to the visual domain

Arousal and valence (pleasantness) are considered primary dimensions of emotion. However, the degree to which these dimensions interact in emotional processing across sensory modalities is poorly understood. We addressed this issue by applying a crossmodal priming paradigm in which auditory primes (Romantic piano solo music) varying in arousal and/or pleasantness were sequentially paired with visual targets (IAPS pictures). In Experiment 1, the emotion spaces of 120 primes and 120 targets were explored separately in addition to the effects of musical training and gender. Thirty-two participants rated their felt pleasantness and arousal in response to primes and targets on equivalent rating scales as well as their familiarity with the stimuli. Musical training was associated with elevated familiarity ratings for high-arousing music and a trend for elevated arousal ratings, especially in response to unpleasant musical stimuli. Males reported higher arousal than females for pleasant visual stimuli. In Experiment 2, 40 nonmusicians rated their felt arousal and pleasantness in response to 20 visual targets after listening to 80 musical primes. Arousal associated with the musical primes modulated felt arousal in response to visual targets, yet no such transfer of pleasantness was observed between the two modalities. Experiment 3 sought to rule out the possibility of any order effect of the subjective ratings, and responses of 14 nonmusicians replicated results of Experiment 2. This study demonstrates the effectiveness of the crossmodal priming paradigm in basic research on musical emotions.     

基于视听双通道的音乐情绪加工机制及情绪类型和音乐经验的影响

为了探讨视听双通道下的音乐情绪加工机制及音乐情绪类型和音乐训练背景对加工机制的影响,本研究采用表达开心和悲伤的音乐表演视频为材料,比较音乐组被试和非音乐组被试在单听觉通道、单视觉通道和视听双通道三种情境下的情绪评定速度、正确率和强度。结果发现：1）视听双通道与单视觉通道差异显著,与单听觉通道差异不显著。2）非音乐组被试对悲伤的评定正确率高于音乐组被试,对开心的评定正确率低于音乐组被试。说明音乐情绪加工的视听双通道整合优势仅相对单视觉通道存在;非音乐组被试对视觉通道情绪信息的变化更敏感,音乐组被试更依赖音乐经验;可在音乐表演时加入协调的视觉通道情绪信息帮助没有音乐训练经验的听赏者。     

基于视听双通道的音乐情绪冲突效应及优势通道研究

为探讨基于视听双通道的音乐情绪冲突效应、冲突情境下的优势加工通道和音乐经验对结果的影响,本研究采用音乐表演视频为材料,比较音乐组和非音乐组被试在一致型和不一致型视听双通道下的情绪评定速度、准确性及强度。结果发现:(1)一致型条件下的情绪评定更准确且更强烈;(2)不一致型条件下,被试更多以听觉通道的情绪线索为依据进行情绪类型评定;(3)非音乐组被试比音乐组被试更依赖视觉通道的情绪线索。结果表明:通道间情绪信息的不一致阻碍了音乐情绪加工; 听觉通道是音乐情绪冲突情境下的优势加工通道; 音乐经验降低了情绪冲突效应对音乐组被试的干扰。     

Emotional response to musical repetition

Two experiments examined the effects of repetition on listeners' emotional response to music. Listeners heard recordings of orchestral music that contained a large section repeated twice. The music had a symmetric phrase structure (same-length phrases) in Experiment 1 and an asymmetric phrase structure (different-length phrases) in Experiment 2, hypothesized to alter the predictability of sensitivity to musical repetition. Continuous measures of arousal and valence were compared across music that contained identical repetition, variation (related), or contrasting (unrelated) structure. Listeners' emotional arousal ratings differed most for contrasting music, moderately for variations, and least for repeating musical segments. A computational model for the detection of repeated musical segments was applied to the listeners' emotional responses. The model detected the locations of phrase boundaries from the emotional responses better than from performed tempo or physical intensity in both experiments. These findings indicate the importance of repetition in listeners' emotional response to music and in the perceptual segmentation of musical structure.     

Probabilistic models of expectation violation predict psychophysiological emotional responses to live concert music

We present the results of a study testing the often-theorized role of musical expectations in inducing listeners’ emotions in a live flute concert experiment with 50 participants. Using an audience response system developed for this purpose, we measured subjective experience and peripheral psychophysiological changes continuously. To confirm the existence of the link between expectation and emotion, we used a threefold approach. (1) On the basis of an information-theoretic cognitive model, melodic pitch expectations were predicted by analyzing the musical stimuli used (six pieces of solo flute music). (2) A continuous rating scale was used by half of the audience to measure their experience of unexpectedness toward the music heard. (3) Emotional reactions were measured using a multicomponent approach: subjective feeling (valence and arousal rated continuously by the other half of the audience members), expressive behavior (facial EMG), and peripheral arousal (the latter two being measured in all 50 participants). Results confirmed the predicted relationship between high-information-content musical events, the violation of musical expectations (in corresponding ratings), and emotional reactions (psychologically and physiologically). Musical structures leading to expectation reactions were manifested in emotional reactions at different emotion component levels (increases in subjective arousal and autonomic nervous system activations). These results emphasize the role of musical structure in emotion induction, leading to a further understanding of the frequently experienced emotional effects of music.     

Judging familiarity and emotion from very brief musical excerpts

In the present study, the gating paradigm was used to measure how much perceptual information that was extracted from musical excerpts needs to be heard to provide judgments of familiarity and of emotionality. Nonmusicians heard segments of increasing duration (250, 500, 1,000 msec, etc.). The stimuli were segments from familiar and unfamiliar musical excerpts in Experiment 1 and from very moving and emotionally neutral musical excerpts in Experiment 2. Participants judged how familiar (Experiment 1) or how moving (Experiment 2) the excerpt was to them. Results show that a feeling of familiarity can be triggered by 500-msec segments, and that the distinction between moving and neutral can be made for 250-msec segments. This finding extends the observation of fast-acting cognitive and emotional processes from face and voice perception to music perception.     

It's not what you play,it's how you play it: Timbre affects perception of emotion in music

Salient sensory experiences often have a strong emotional tone, but the neuropsychological relations between perceptual characteristics of sensory objects and the affective information they convey remain poorly defined. Here we addressed the relationship between sound identity and emotional information using music. In two experiments, we investigated whether perception of emotions is influenced by altering the musical instrument on which the music is played, independently of other musical features. In the first experiment, 40 novel melodies each representing one of four emotions (happiness, sadness, fear, or anger) were each recorded on four different instruments (an electronic synthesizer, a piano, a violin, and a trumpet), controlling for melody, tempo, and loudness between instruments. Healthy participants (23 young adults aged 18–30 years, 24 older adults aged 58–75 years) were asked to select which emotion they thought each musical stimulus represented in a four-alternative forced-choice task. Using a generalized linear mixed model we found a significant interaction between instrument and emotion judgement with a similar pattern in young and older adults (p < .0001 for each age group). The effect was not attributable to musical expertise. In the second experiment using the same melodies and experimental design, the interaction between timbre and perceived emotion was replicated (p < .05) in another group of young adults for novel synthetic timbres designed to incorporate timbral cues to particular emotions. Our findings show that timbre (instrument identity) independently affects the perception of emotions in music after controlling for other acoustic, cognitive, and performance factors.     

How do location and control over the music influence listeners’ responses?

This study uses Mehrabian and Russell's ( 1974 ) Pleasure‐Arousal‐Dominance (PAD) model to consider how responses to both the music heard and overall in‐situ listening experience are influenced by the listener's degree of control over music selected for a particular listening episode and the location in which the listening takes place. Following recruitment via campus advertisements and a university research participation program, 216 individuals completed a background questionnaire and music listening task in a 3 (location) × 2 (experimenter‐ or participant‐selected music) design. After the listening task, participants completed a short questionnaire concerning the music they heard and the overall in‐situ listening experience. Results demonstrated that there was a positive relationship between control and liking for the music and episode, whether the former was considered in terms of: (1) whether the music was self‐selected or experimenter‐selected or (2) overt ratings of perceived control. Furthermore, the location and liking for the music were related to people's judgments of their enjoyment of the overall experience. This research indicates that the PAD model is a useful framework for understanding everyday music listening and supports the contention that, in a musical context, dominance may be operationalized as control over the music.     

The "euphoria" effect of music videos--a study of the reception of music with visual images

An attempt is made to study the impact of visual information on the perception of music by employing (rock) music videos as stimuli. Forty music videos were presented to judges, who either saw the video or only heard the respective pieces of music. They had to judge the emotions conveyed via the piece of music/video on scales and their overall impression on semantic differential scales. Results indicate that visual information as presented in music videos has considerable effects on impressions: When pieces of music are presented as music videos, more positive emotions are attributed, while presentation of the pieces of music alone resulted in more negative emotion attributions. Thus, music videos seem to "euphorize" the recipient. Furthermore, video presentation compared to presentation of music alone evoked more intense "complexity/interest" as well as "activity" judgments, while "evaluation" judgments were not influenced by the medium of presentation. In addition, a number of presentation factors (like the speed of the music or the number of cuts in the videos) do influence impressions. This leads to the conclusion that researchers should pay more attention to such microcharacteristics of stimuli. In general, the effects found to be due to the medium of presentation are independent of the effects on judgments due to content (i.e., presentation factors).     

Specific previous experience affects perception of harmony and meter

Prior knowledge shapes our experiences, but which prior knowledge shapes which experiences? This question is addressed in the domain of music perception. Three experiments were used to determine whether listeners activate specific musical memories during music listening. Each experiment provided listeners with one of two musical contexts that was presented simultaneously with a melody. After a listener was familiarized with melodies embedded in contexts, the listener heard melodies in isolation and judged the fit of a final harmonic or metrical probe event. The probe event matched either the familiar (but absent) context or an unfamiliar context. For both harmonic (Experiments 1 and 3) and metrical (Experiment 2) information, exposure to context shifted listeners' preferences toward a probe matching the context that they had been familiarized with. This suggests that listeners rapidly form specific musical memories without explicit instruction, which are then activated during music listening. These data pose an interesting challenge for models of music perception which implicitly assume that the listener's knowledge base is predominantly schematic or abstract.     

Music skills and the expressive interpretation of music in children with Williams-Beuren syndrome: pitch, rhythm, melodic imagery, phrasing, and musical affect.

This paper studied music in 14 children and adolescents with Williams-Beuren syndrome (WBS), a multi-system neurodevelopmental disorder, and 14 age-matched controls. Five aspects of music were tested. There were two tests of core music domains, pitch discrimination and rhythm discrimination. There were two tests of musical expressiveness, melodic imagery and phrasing. There was one test of musical interpretation, the ability to identify the emotional resonance of a musical excerpt. Music scores were analyzed by means of logistic regressions that modeled outcome (higher or lower music scores) as a function of group membership (WBS or Control) and cognitive age. Compared to age peers, children with WBS had similar levels of musical expressiveness, but were less able to discriminate pitch and rhythm, or to attach a semantic interpretation to emotion in music. Music skill did not vary with cognitive age. Musical strength in individuals with WBS involves not so much formal analytic skill in pitch and rhythm discrimination as a strong engagement with music as a means of expression, play, and, perhaps, improvisation.     

Music Skills and the Expressive Interpretation of Music in Children with Williams-Beuren Syndrome: Pitch,Rhythm, Melodic Imagery,Phrasing, and Musical Affect

This paper studied music in 14 children and adolescents with Williams-Beuren syndrome (WBS), a multi-system neurodevelopmental disorder, and 14 age-matched controls. Five aspects of music were tested. There were two tests of core music domains, pitch discrimination and rhythm discrimination. There were two tests of musical expressiveness, melodic imagery and phrasing. There was one test of musical interpretation, the ability to identify the emotional resonance of a musical excerpt. Music scores were analyzed by means of logistic regressions that modeled outcome (higher or lower music scores) as a function of group membership (WBS or Control) and cognitive age. Compared to age peers, children with WBS had similar levels of musical expressiveness, but were less able to discriminate pitch and rhythm, or to attach a semantic interpretation to emotion in music. Music skill did not vary with cognitive age. Musical strength in individuals with WBS involves not so much formal analytic skill in pitch and rhythm discrimination as a strong engagement with music as a means of expression, play, and, perhaps, improvisation.     

不同目标标记方式对多目标追踪表现的影响

有关多目标追踪的研究多采用目标闪烁数次或目标周围出现方框等视觉方式对目标进行标记。在已有研究基础上,通过2个实验比较了不同目标数量下视觉平行标记、视觉序列标记、听觉序列标记(视听跨通道标记)和视听双通道序列标记4种目标标记方式对多目标追踪表现的影响。研究结果表明在多目标追踪任务中,线索阶段目标的视觉标记比视听跨通道标记更有效,且不受目标的序列标记或平行标记的影响,而视觉和听觉编码结合（视听双通道标记）可以促进任务难度较高时的追踪表现。     

The ethical potential of sound in public space: Migrant pan flute music and its potential to create moments of conviviality in a ‘failed’ public square

In this paper we examine the notion that music in public space could be understood in terms of ethical potential, where new sensibilities for thinking, feeling, seeing and being with others might be imagined and practiced. We do this by considering how musical performances by migrants impact on inclusive forms of place (re-)making, affective enactments of public space and emotional accounts of belonging and ‘the other’. The paper draws on an ethnographic exploration of South American pan flute musicians, performing music at Sergels torg, a central square in Stockholm, Sweden. Through fieldwork with a combination of qualitative techniques, including observation, interviews and sensory methods such as photography, video and recorded ‘sound walks’ we trace the affective aspects of encounters with busking and the impact of music on place. We highlight the ethical potential of music in the experience of urban moments and its capacity to reconfigure space. We find that encounters with sound can produce new spaces of conviviality and inclusion; it can soothe, animate and soften urban spaces. However, a positive encounter with difference through sound depends on a favourable social, physical and temporal context, and because busking serves to make marginalised voices heard (both literally and metaphorically), it can be experienced as troubling for precisely this reason. Thus, we need to take into account the full complexity of the dynamics between sound and place, in considering this relationship as a novel window to the ethical potential of the urban encounter.     

Distinct effects of positive and negative music on older adults' auditory target identification performances

Older adults, compared to younger adults, are more likely to attend to pleasant situations and avoid unpleasant ones. Yet, it is unclear whether such a phenomenon may be generalized to musical emotions. In this study, we investigated whether there is an age-related difference in how musical emotions are experienced and how positive and negative music influences attention performances in a target identification task. Thirty-one young and twenty-eight older adults were presented with 40 musical excerpts conveying happiness, peacefulness, sadness, and threat. While listening to music, participants were asked to rate their feelings and monitor each excerpt for the occurrence of an auditory target. Compared to younger adults, older adults reported experiencing weaker emotional activation when listening to threatening music and showed higher level of liking for happy music. Correct reaction times (RTs) for target identification were longer for threatening than for happy music in older adults but not in younger adults. This suggests that older adults benefit from a positive musical context and can regulate emotion elicited by negative music by decreasing attention towards it (and therefore towards the auditory target).     

Individual differences in music performance

Music cognition depends on the existence and deployment of processes for detecting, storing and organizing musical materials according to underlying structural features. Common cultural experiences develop these processes to a certain degree, but specifically designed and supported learning environments are required to achieve the levels of expertise required to perform western art music. Certain motivational and social factors are therefore implicated in the maintenance of activities that promote skill-acquisition, such as practice. Expert musical performance is not just a matter of technical motor skill, it also requires the ability to generate expressively different performances of the same piece of music according to the nature of intended structural and emotional communication. This review examines these abilities and describes how some of them have been shown to have lawful relationships to objective musical and extra-musical parameters. Psychological research is thus engaged in a process of demystifying musical expertise, a process that helps to improve upon culturally prevalent, but ultimately non-explanatory, notions of inborn 'talent'.     

Memory for facial expression is influenced by the background music playing during study

The effect of the emotional quality of study-phase background music on subsequent recall for happy and sad facial expressions was investigated. Undergraduates (N = 48) viewed a series of line drawings depicting a happy or sad child in a variety of environments that were each accompanied by happy or sad music. Although memory for faces was very accurate, emotionally incongruent background music biased subsequent memory for facial expressions, increasing the likelihood that happy faces were recalled as sad when sad music was previously heard, and that sad faces were recalled as happy when happy music was previously heard. Overall, the results indicated that when recalling a scene, the emotional tone is set by an integration of stimulus features from several modalities.