Building a controllable expressive speech synthesis system with multiple emotion strengths

Emotion is considered to be an essential element in the performance of human-computer interactions. In expressive synthesis speech, it is important to generate emotional speech that reflects subtle and complex emotional states. However, there has been limited research on how to effectively synthesize emotional speech using different levels of emotion strength with intuitive control, which is difficult to be modeled effectively. In this paper, we explore an expressive speech synthesis model that can be used to produce speech with multiple emotion strengths. Unlike previous studies that encoded emotions into discrete codes, we propose an embedding vector to continuously control the emotion strength, which is a data-driven method to synthesize speech with a fine control over the emotions. Compared with the models using the retraining technique or a one-hot vector, our proposed model using an embedding vector can explicitly learn the high-level emotion strength from the low-level acoustic features. As a result, we can control the emotion strength of synthetic speech in a relatively predictable and globally consistent way. The objective and subjective evaluation tests show that our proposed model achieves state-of-the-art performance in terms of model flexibility and controllability.     

Cognition and Emotion Lecture at the 2010 SPSP Emotion Preconference

One of the most fundamental distinctions in the field of emotion is the distinction between emotion generation and emotion regulation. This distinction fits comfortably with folk theories, which view emotions as passions that arise unbidden and then must be controlled. But is it really helpful to distinguish between emotion generation and emotion regulation? In this article, we begin by offering working definitions of emotion generation and emotion regulation. We argue that in some circumstances, the distinction between emotion generation and emotion regulation is indeed useful. We point both to citation patterns, which indicate that researchers from across a number of sub-areas within psychology are making this distinction, and to empirical studies, which indicate the utility of this distinction in many different research contexts. We then consider five ways in which the distinction between emotion generation and emotion regulation can be problematic. We suggest that it is time to move beyond debates about whether this distinction is useful to a more specific consideration of when and in what ways this distinction is useful, and in this spirit, we offer recommendations for future research.     

Humour as emotion regulation: The differential consequences of negative versus positive humour

Humour is often seen as an adaptive coping strategy; however, the empirical literature is inconclusive. One possible explanation is that different types of humour have different adaptive consequences. In the present research, we predicted that positive (good-natured) humour would be more effective at regulating negative emotions than negative (mean-spirited) humour. In Study 1, participants were shown negative pictures two times. First, they simply viewed the pictures and rated their levels of positive and negative emotions. Second, they were instructed to: (a) view; (b) use positive humour; or (c) use negative humour, and then rate their reactions. Compared to negative humour, positive humour was more successful at down-regulating negative and up-regulating positive emotion. In Study 2, we replicated these findings and showed that these effects cannot be explained by differences in difficulty between the two humour conditions, participants’ expectations, or social desirability. Taken together, these findings suggest that positive (but not negative) humour may be an effective form of emotion regulation.     

Knowing me,knowing you: emotion differentiation in oneself is associated with recognition of others’ emotions

ABSTRACT

Previous research has found that individuals vary greatly in emotion differentiation, that is, the extent to which they distinguish between different emotions when reporting on their own feelings. Building on previous work that has shown that emotion differentiation is associated with individual differences in intrapersonal functions, the current study asks whether emotion differentiation is also related to interpersonal skills. Specifically, we examined whether individuals who are high in emotion differentiation would be more accurate in recognising others’ emotional expressions. We report two studies in which we used an established paradigm tapping negative emotion differentiation and several emotion recognition tasks. In Study 1 (N?=?363), we found that individuals high in emotion differentiation were more accurate in recognising others’ emotional facial expressions. Study 2 (N?=?217), replicated this finding using emotion recognition tasks with varying amounts of emotional information. These findings suggest that the knowledge we use to understand our own emotional experience also helps us understand the emotions of others.     

Emotions associated with different textures during touch

Haptics plays an important role in emotion perception. However, most studies of the affective aspects of haptics have investigated emotional valence rather than emotional categories. In the present study, we explored the associations of different textures with six basic emotions: fear, anger, happiness, disgust, sadness and surprise. Participants touched twenty-one different textures and evaluated them using six emotional scales. Additionally, we explored whether individual differences in participants’ levels of alexithymia are related to the intensity of emotions associated with touching the textures. Alexithymia is a trait related to difficulties in identifying, describing and communicating emotions to others. The findings show that people associated touching different textures with distinct emotions. Textures associated with each of the basic emotions were identified. The study also revealed that a higher alexithymia level corresponds to a higher intensity of associations between textures and the emotions of disgust, anger and sadness.     

Classifying emotion: a developmental account

The aim of this paper is to propose a systematic classification of emotions which can also characterize their nature. The first challenge we address is the submission of clear criteria for a theory of emotions that determine which mental phenomena are emotions and which are not. We suggest that emotions as a subclass of mental states are determined by their functional roles. The second and main challenge is the presentation of a classification and theory of emotions that can account for all existing varieties. We argue that we must classify emotions according to four developmental stages: 1. pre-emotions as unfocussed expressive emotion states, 2. basic emotions, 3. primary cognitive emotions, and 4. secondary cognitive emotions. We suggest four types of basic emotions (fear, anger, joy and sadness) which are systematically differentiated into a diversity of more complex emotions during emotional development. The classification distinguishes between basic and non-basic emotions and our multi-factorial account considers cognitive, experiential, physiological and behavioral parameters as relevant for constituting an emotion. However, each emotion type is constituted by a typical pattern according to which some features may be more significant than others. Emotions differ strongly where these patterns of features are concerned, while their essential functional roles are the same. We argue that emotions form a unified ontological category that is coherent and can be well defined by their characteristic functional roles. Our account of emotions is supported by data from developmental psychology, neurobiology, evolutionary biology and sociology.     

Spatializing Emotion: No Evidence for a Domain‐General Magnitude System

People implicitly associate different emotions with different locations in left‐right space. Which aspects of emotion do they spatialize, and why? Across many studies people spatialize emotional valence, mapping positive emotions onto their dominant side of space and negative emotions onto their non‐dominant side, consistent with theories of metaphorical mental representation. Yet other results suggest a conflicting mapping of emotional intensity (a.k.a., emotional magnitude), according to which people associate more intense emotions with the right and less intense emotions with the left — regardless of their valence; this pattern has been interpreted as support for a domain‐general system for representing magnitudes. To resolve the apparent contradiction between these mappings, we first tested whether people implicitly map either valence or intensity onto left‐right space, depending on which dimension of emotion they attend to (Experiments 1a, b). When asked to judge emotional valence, participants showed the predicted valence mapping. However, when asked to judge emotional intensity, participants showed no systematic intensity mapping. We then tested an alternative explanation of findings previously interpreted as evidence for an intensity mapping (Experiments 2a, b). These results suggest that previous findings may reflect a left‐right mapping of spatial magnitude (i.e., the size of a salient feature of the stimuli) rather than emotion. People implicitly spatialize emotional valence, but, at present, there is no clear evidence for an implicit lateral mapping of emotional intensity. These findings support metaphor theory and challenge the proposal that mental magnitudes are represented by a domain‐general metric that extends to the domain of emotion.     

‘Taking emotions seriously’: Feeling female and becoming-surfer through UK Surf Space

In this paper, I consider the importance of “taking emotions seriously” in relation to my research into women and surfing in the UK. I draw on feminist work in the areas of emotion, space and subjectivity, and relevant literature on surfing, in order to explore the significance of affectual and emotional experiences for women who surf. I contend that by exploring how women emotionally experience surfing spaces, we might better understand the dynamic intricacies of surfing as an embodied, gendered space. In particular, I explore feelings of fear, freedom, belonging and anger in the context of female subjectivities caught up in a spatial becoming-surfer. My findings suggest that although surfing remains dominated by men, the fluidity of the space offers many possibilities for women to experience their gendered embodiment differently. I also contend that an association of surfing with masculinity continues to shape gendered relations and spatial becomings in the surf.     

Rumination,emotional intensity and emotional clarity

It has been suggested that a high tendency to ruminate presents a deficient emotion regulation. Past research found that people with high tendency to ruminate show sustained attention for negative stimuli and increased negative thinking, which may result in intensified experiences of negative emotions. Moreover, high level of rumination was associated with low emotional understanding. Accordingly, we hypothesized (1) high ruminators (HR) experience more intense emotional reactions than low ruminators (LR) for negative but not positive emotions, (2) LR have higher emotional clarity than HR, and (3) there would be the same pattern of results for brooding but not for reflective pondering. Participants completed a demographic questionnaire, a rumination response style questionnaire, and the Beck Depression Inventory-II. They also rated emotional intensity and identified emotion type for scene pictures from the CAP-D (Categorized Affective Pictures Database). The highest (HR) and lowest (LR) quarters of ruminators were compared on levels of emotional intensity and emotional clarity. We found HR experienced negative emotions more intensely than LR, with no difference for positive emotions. In contrast to our hypothesis, the two groups did not differ in their emotion understanding. This pattern of results was found for brooding but not for reflective pondering. Our research sheds light on the mechanism underlying rumination and emotion regulation.     

Remembering the silver lining: Reappraisal and positive bias in memory for emotion

Reappraisal and distraction, unlike suppression, are known to decrease the intensity of negative emotion in the short term. Little is known about long-term characteristics associated with emotion regulation strategies, however. In a longitudinal study, we examined the relation between the strategies people reported using to regulate emotions during a stressful situation and their later memory for their emotions. Students in Italy rated the intensity of positive and negative emotions they were experiencing as they prepared for their high school exit exam. They also rated the extent to which they were regulating emotion using reappraisal, distraction, and suppression. Six weeks later, students recalled their pre-exam emotions. The more students reported engaging in reappraisal before the exam, the more they overestimated positive emotion and underestimated negative emotion when recalling their experience. The association between reported reappraisal and memory bias was partially mediated by positive changes over time in students’ appraisals of the exam preparation experience. Reports of engaging in distraction and suppression were not associated with memory bias. Because remembered emotion guides future choices, these findings suggest that reappraisal is a highly adaptive strategy for coping with stressful situations, not only in the short run, but also in the long run.     

The effect of self- and interpersonal emotion regulation on athletes’ anxiety and goal achievement in competition

ObjectivesThere exists a wealth of evidence that athletes must regulate their emotions for optimal performance and wellbeing. In addition to athletes’ attempts to regulate their own emotions, they may also attempt to regulate each other’s emotions (interpersonal emotion regulation). Though self- and interpersonal emotion regulation likely co-occur, previous research has not explored how these strategies concurrently impact athletes’ emotions and performance outcomes. In the current study, we examined whether athletes’ emotional self-regulation and the receipt of interpersonal emotion regulation from their teammates were related to their anxiety and goal achievement during competition.DesignQuantitative, cross-sectional retrospective survey design.MethodData were gathered following sport competitions from 509 participants from 50 interdependent sport teams from Canada and the UK (M_age = 19.0, SD = 3.1).ResultsAnalysis of the data using structural equation modeling revealed that after accounting for pre-competition anxiety, received interpersonal emotion regulation was not associated with anxiety during competition, though affect-worsening self-regulation was positively associated with anxiety during competition. Received interpersonal emotion regulation was also not associated with goal achievement, yet affect-improving and affect-worsening self-regulation were associated with goal achievement. Nevertheless, when the influence of emotional self-regulation on anxiety and goal achievement was set to zero, affect-improving and affect-worsening interpersonal emotion regulation were associated with anxiety during competition and affect-improving interpersonal emotion regulation was associated with goal achievement.ConclusionsThese data can be interpreted as evidence that emotion regulation actions between teammates are important for anxiety and performance outcomes, albeit this effect is attenuated in the presence of athletes’ own emotional self-regulation. These results extend the extant research on self- and interpersonal emotion regulation in sport, and in line with these observations, we highlight a number of future research opportunities for researchers examining emotion regulation in performance contexts.     

Cross‐cultural differences in emotion suppression in everyday interactions

Previous research suggests that in collectivistic cultures, people tend to suppress their emotions more than in individualistic cultures. Little research, however, has explored cross‐cultural differences in emotion regulation in everyday interactions. Using a daily social interaction method, we examined whether people from collectivistic backgrounds (Chinese exchange students and immigrants from the Moluccas, Indonesia) living in the Netherlands differed from those from individualistic backgrounds (Dutch natives) in emotion suppression during everyday interactions. We also examined whether this depended on their relationship with the interaction partner(s). We found that Chinese participants suppressed positive and negative emotions more than Dutch and Moluccan participants and that this was related to differences in interdependent and independent self‐construal across the samples. We also found that Chinese participants suppressed positive emotions less in interactions with close others, whereas Dutch participants suppressed negative emotions more with non‐close others. No such differences were found for Moluccans. Our findings support the idea that people from collectivistic cultures suppress emotions more than those from individualistic cultures, but they also suggest that this depends on who the interaction partner is. Furthermore, they suggest that emotion suppression may change when people with collectivistic backgrounds have been raised in individualistic cultures.     

Emotion perception and recognition: An exploration of cultural differences and similarities

The electroencephalogram (EEG) is a powerful method for investigation of different cognitive processes. Recently, EEG analysis became very popular and important, with classification of these signals standing out as one of the mostly used methodologies. Emotion recognition is one of the most challenging tasks in EEG analysis since not much is known about the representation of different emotions in EEG signals. In addition, inducing of desired emotion is by itself difficult, since various individuals react differently to external stimuli (audio, video, etc.). In this article, we explore the task of emotion recognition from EEG signals using distance-based time-series classification techniques, involving different individuals exposed to audio stimuli. Furthermore, since some of the participants in the experiment do not understand the language of the stimuli, we also investigate the impact of language understanding on emotion perception. Using time-series distances as features for the construction of new data representations, applied here for the first time to emotion recognition and related tasks, lead to excellent classification performance, indicating that differences between EEG signals can be used to build successful models for recognition of emotions, individuals, and other related tasks. In the process, we observed that cultural differences between the subjects did not have a significant impact on the recognition tasks and models.     

Happy people thrive on adversity: Pre-existing mood moderates the effect of emotion inductions on creative thinking

Past research on the relationship between affect and creativity has yielded contradictory results. Most of the evidence has tended to show that brief positive emotions as well as more enduring positive moods enhance creativity. No study to date, however, has attempted to determine whether the influence of momentary emotions on creativity depends on pre-existing moods. In the present study, 96 undergraduates completed one of two creative tasks (generating or evaluating captions for photographs) on three occasions, after watching videos designed to induce positive, neutral, or negative emotions. Participants also completed a questionnaire assessing depressed mood. Results confirmed that the effect of emotion inductions on creativity depended on pre-existing mood. Participants low in depression wrote more creative captions and rated captions more accurately with induced negative emotion than with induced positive emotion. In contrast, participants high in depression appeared impervious to the effect of emotion inductions.     

The Role of Basic, Self-Conscious and Self-Conscious Evaluative Emotions in Children’s Memory and Understanding of Emotion

In past research, emotion has been classified as basic, self-conscious or self-conscious evaluative, with each type of emotion being progressively more difficult for children to understand (Lewis, M. (2000a). In M. Lewis & J. M. Haviland-Jones (Eds.), Handbook of emotions (2nd ed., pp. 265–280), New York, NY: The Guilford Press). Although researchers have examined children’s understanding of individual emotions (e.g., guilt), researchers have not assessed children’s understanding and memory for emotions based on this classification. In the present research, 6-, 8- and 10-year-old children’s memory and understanding of basic, self-conscious and self-conscious evaluative emotions were examined. Although a memory advantage was found for emotions, this effect was less so for the younger children and less so for non-basic emotions. In fact, 6-year-old children and, to a lesser extent 8-year-old children, were more likely than older children to recall self-conscious and self-conscious evaluative emotions with basic emotion labels, and were more likely to explain them using basic emotion labels. Overall, negative emotions (e.g., mad, guilt) were better recalled than positive emotions (e.g., happy, pride), regardless of type of emotion. Gender differences were found as girls were more likely to remember emotion than boys, especially when the emotion action was specifically labeled and a female character experienced it. Proportions of this research were presented at the biennial meeting of the Society for Research in Child Development (2005), Atlanta, Georgia and the meeting of the Cognitive Development Society (2005), San Diego, CA.     

Less is more? Effects of exhaustive vs. minimal emotion labelling on emotion regulation strategy planning

Previous research suggests that labelling emotions, or describing affective states using emotion words, facilitates emotion regulation. But how much labelling promotes emotion regulation? And which emotion regulation strategies does emotion labelling promote? Drawing on cognitive theories of emotion, we predicted that labelling emotions using fewer words would be less confusing and would facilitate forms of emotion regulation requiring more cognitively demanding processing of context. Participants (N?=?82) mentally immersed themselves in an emotional vignette, were randomly assigned to an exhaustive or minimal emotion labelling manipulation, and then completed an emotion regulation strategy planning task. Minimal (vs. exhaustive) emotion labelling promoted higher subjective emotional clarity. Furthermore, in terms of specific emotion regulation strategies, minimal emotion labelling prompted more plans for problem solving and marginally more plans for reappraisal, but did not affect plans for behavioural activation or social support seeking. We discuss implications for the cognitive mechanisms supporting the generation of emotion regulation strategies.     

Emotional cartography as a window into children's well-being: Visualizing the felt geographies of place

More often than not, Geographic Information Systems (GIS) excludes emotion and qualitative analysis from studies of people-place relationships in favor of quantitative approaches. We employ emotional cartography as a form of qualitative GIS (qualGIS) to elevate emotions from the periphery to the center of dialogue about children's well-being. We highlight the ontological parallels between qualGIS, emotional cartography and children in society, and advance emotion maps as a way to visualize different spatial and emotional realities. In reflecting upon the felt geography of our own childhood places, we affirm the importance of children's emotional attachments to places as well as the centrality of ‘messy’ human experiences in GIS. To conclude, we discuss the implications of emotional cartography for researchers, planners and GIS, paying special attention to children's well-being amidst the current COVID-19 pandemic. In particular, this includes a call to ‘witness’ and to foster spatial empathy among those advocating for children.     

Space for emotion in the spaces of activism

This paper explores the role of emotions in activism. Although, increasingly, researchers have examined what emotions inspire or deter different forms of political and social movement activism, this paper takes a new direction by considering what spaces, practices and emotional stances are necessary to sustain individual and collective resistance in the long-term. We argue that we need to sustain activism through emotional reflexivity, building sustaining spaces to create space for emotion in activism. Using empirical examples from different forms of autonomous (anti-capitalist) activism in Britain, the role and importance of emotions to the sustainability of activism is explored. In particular, we consider the role of different spaces in sustaining activists through the cycles of protest, what spaces of activism can be opened up by a closer attention to emotions, and how the spaces in which protest and other activist practices take place shape the emotional and affective engagements of participants. As autonomous forms of activism attempt to prefiguratively enact new post-capitalist social relations in the here and now, we suggest there is still some way to go in changing affective relationships within many of these groups.     

Yes I can: Expected success promotes actual success in emotion regulation

People who expect to be successful in regulating their emotions tend to experience less frequent negative emotions and are less likely to suffer from depression. It is not clear, however, whether beliefs about the likelihood of success in emotion regulation can shape actual emotion regulation success. To test this possibility, we manipulated participants' beliefs about the likelihood of success in emotion regulation and assessed their subsequent ability to regulate their emotions during a negative emotion induction. We found that participants who were led to expect emotion regulation to be more successful were subsequently more successful in regulating their emotional responses, compared to participants in the control condition. Our findings demonstrate that expected success can contribute to actual success in emotion regulation.     

Spontaneous emotion regulation to positive and negative stimuli

The ability to regulate one’s emotions is an integral part of human social behavior. One antecedent emotion regulation strategy, known as reappraisal, is characterized by cognitively evaluating an emotional stimulus to alter its emotional impact and one response-focused strategy, suppression, is aimed at reducing behavioral output. People are capable of using these specific emotion regulation strategies when instructed to do so; however, it is equally important to investigate natural and self-selected strategy use. This study was designed to determine to what extent people spontaneously regulate their emotions and the emotion regulation strategies they choose to achieve their regulatory goals. Participants were given no instructions to regulate their emotions before they were shown a negative and a positive film clip, but were instead asked afterwards about the specific strategies that they had used. Participants reported regulating their emotions more to the negative film than to the positive film. Reappraisal was more frequently selected as an emotion regulation strategy than suppression. As expected, participants with high baseline respiratory sinus arrhythmia (RSA) adopted reappraisal strategies more than those with low RSA but, surprisingly, RSA was not associated with facial expressivity. Suggestions for future research in this relatively young field of spontaneous emotion regulation are offered.