The wisdom of the body: Listeners' autonomic arousal distinguishes between spontaneous and posed vocal emotions

It has been the matter of much debate whether perceivers are able to distinguish spontaneous vocal expressions of emotion from posed vocal expressions (e.g., emotion portrayals). In this experiment, we show that such discrimination can be manifested in the autonomic arousal of listeners during implicit processing of vocal emotions. Participants (N = 21, age: 20–55 years) listened to two consecutive blocks of brief voice clips and judged the gender of the speaker in each clip, while we recorded three measures of sympathetic arousal of the autonomic nervous system (skin conductance level, mean arterial blood pressure, pulse rate). Unbeknownst to the listeners, the blocks consisted of two types of emotional speech: spontaneous and posed clips. As predicted, spontaneous clips yielded higher arousal levels than posed clips, suggesting that listeners implicitly distinguished between the two kinds of expression, even in the absence of any requirement to retrieve emotional information from the voice. We discuss the results with regard to theories of emotional contagion and the use of posed stimuli in studies of emotions.     

Facial and vocal cues of deception and honesty

Facial expressions and vocal cues (filtered speech) of honest and deceptive messages were examined in posed and spontaneous situations. The question of interest was the degree to which nonverbal cues transmit information about deception. Results indicated that (a) for both the facial and vocal channels, posing (as compared to spontaneous behavior) produced a higher level of communication accuracy; (b) facial expressions of deceptive (as compared to honest) messages were rated as less pleasant, while vocal expressions of deception were rated as less honest, less assertive, and less dominant, particularly in the posed condition; (c) the sender's ability to convey honesty was negatively correlated with his/her ability to convey deception, suggesting the existence of a demeanor bias—individual senders tend to appear and sound consistently honest (or dishonest) regardless of whether they deliver an honest or a deceptive message; (d) in the posing condition, the sender's abilities to convey honesty/deception via facial and vocal cues were positively and significantly correlated, whereas in the spontaneous condition they were not; and (e) senders whose full (unfiltered) speech indicated more involvement with their responses were judged as more honest from both their vocal (filtered speech) and facial cues, in both the honest and deceptive conditions.     

Task and sex modulate the brain response to emotional incongruity in Asian listeners

In order to recognize banter or sarcasm in social interactions, listeners must integrate verbal and vocal emotional expressions. Here, we investigated event-related potential correlates of this integration in Asian listeners. We presented emotional words spoken with congruous or incongruous emotional prosody. When listeners classified word meaning as positive or negative and ignored prosody, incongruous trials elicited a larger late positivity than congruous trials in women but not in men. Sex differences were absent when listeners evaluated the congruence between word meaning and emotional prosody. The similarity of these results to those obtained in Western listeners suggests that sex differences in emotional speech processing depend on attentional focus and may reflect culturally independent mechanisms.     

Facial expression and sarcasm.

This study examined facial expression in the presentation of sarcasm. 60 responses (sarcastic responses = 30, nonsarcastic responses = 30) from 40 different speakers were coded by two trained coders. Expressions in three facial areas--eyebrow, eyes, and mouth--were evaluated. Only movement in the mouth area significantly differentiated ratings of sarcasm from nonsarcasm.     

Posed and spontaneous communication of emotion via facial and vocal cues1

Subjects' facial expressions were videotaped without their knowledge while they watched two pleasant and two unpleasant videotaped scenes (spontaneous facial encoding). Later, subjects' voices were audiotaped while describing their reactions to the scenes (vocal encoding). Finally, subjects were videotaped with their knowledge while they posed appropriate facial expressions to the scenes (posed facial encoding). The videotaped expressions were presented for decoding to the same subjects. The vocal material, both the original version and an electronically filtered version, was rated by judges other than the original senders. Results were as follows: (a) accuracy of vocal encoding (measured by ratings of both the filtered and unfiltered versions) was positively related to accuracy of facial encoding; (b) posing increased the accuracy of facial communication, particularly for more pleasant affects and less intense affects; (c) encoding of posed cues was correlated with encoding of spontaneous cues and decoding of posed cues was correlated with decoding of spontaneous cues; (d) correlations, within encoding and decoding, of similar scenes were positive while those among dissimilar scenes were low or negative; (e) while correlations between total encoding and total decoding were positive and low, correlations between encoding and decoding of the same scene were negative; (f) there were sex differences in decoding ability and in the relationships of personality variables with encoding and decoding of facial cues.     

On the role of the anterior cingulate cortex in phonation: A case report

A 41-year-old male patient is presented with a lesion in the anterior cingulate cortex, medial orbital cortex, and rostral striatum bilaterally and supplementary motor area on the left side. The patient first exhibited a state of akinetic mutism which lasted about 6 weeks. During this state, no volitional vocal utterances were made; there were, however, occasional groans of pain. During recovery, the mute phase was replaced by a state in which the patient could whisper but not phonate verbal utterances. About 10 weeks after the accident, phonation was restored. The speech was characterized, however, by monotonous intonation and a very low frequency of spontaneous utterances. While the frequency of spontaneous speech improved noticeably during the following months, emotional intonation remained permanently defective. A comparison of the present case with other cases from the literature as well as experimental monkey data suggest that the anterior cingulate cortex is involved in the volitional control of emotional vocal utterances.     

Vocal Features of Conversational Sarcasm: A Comparison of Methods

This study investigated vocal cues that differentiate sarcastic utterances from non-sarcastic utterances. Utterances were drawn from videotapes of participant interviews and arranged on a master tape for analysis. Utterances that were identified as sarcastic by speakers and recognized as sarcastic by listeners were randomly arranged with utterances identified and recognized as non-sarcastic by the same participants. Both sarcastic and non-sarcastic utterances were analyzed by two methods–acoustic analysis and perceptual coding. The acoustic analysis proved slightly more successful than the perceptual coding in discriminating between sarcastic and non-sarcastic utterances. The acoustic analysis indicated that fundamental frequency, frequency range, length of utterance, and total amount of sound significantly discriminated sarcastic from non-sarcastic utterances. The perceptual coding method revealed that pitch range, length of utterance, and total amount of sound significantly discriminated sarcastic from non-sarcastic utterances. Moderate correlations were found between the acoustic and perceptual variables.     

Learning to recognize talkers from natural,sinewave, and reversed speech samples

In 5 experiments, the authors investigated how listeners learn to recognize unfamiliar talkers and how experience with specific utterances generalizes to novel instances. Listeners were trained over several days to identify 10 talkers from natural, sinewave, or reversed speech sentences. The sinewave signals preserved phonetic and some suprasegmental properties while eliminating natural vocal quality. In contrast, the reversed speech signals preserved vocal quality while distorting temporally based phonetic properties. The training results indicate that listeners learned to identify talkers even from acoustic signals lacking natural vocal quality. Generalization performance varied across the different signals and depended on the salience of phonetic information. The results suggest similarities in the phonetic attributes underlying talker recognition and phonetic perception.     

Recognizing intentions in infant-directed speech: evidence for universals

In all languages studied to date, distinct prosodic contours characterize different intention categories of infant-directed (ID) speech. This vocal behavior likely exists universally as a species-typical trait, but little research has examined whether listeners can accurately recognize intentions in ID speech using only vocal cues, without access to semantic information. We recorded native-English-speaking mothers producing four intention categories of utterances (prohibition, approval, comfort, and attention) as both ID and adult-directed (AD) speech, and we then presented the utterances to Shuar adults (South American hunter-horticulturalists). Shuar subjects were able to reliably distinguish ID from AD speech and were able to reliably recognize the intention categories in both types of speech, although performance was significantly better with ID speech. This is the first demonstration that adult listeners in an indigenous, nonindustrialized, and nonliterate culture can accurately infer intentions from both ID speech and AD speech in a language they do not speak.     

Perceiving verbal and vocal emotions in a second language

Emotional inferences from speech require the integration of verbal and vocal emotional expressions. We asked whether this integration is comparable when listeners are exposed to their native language and when they listen to a language learned later in life. To this end, we presented native and non-native listeners with positive, neutral and negative words that were spoken with a happy, neutral or sad tone of voice. In two separate tasks, participants judged word valence and ignored tone of voice or judged emotional tone of voice and ignored word valence. While native listeners outperformed non-native listeners in the word valence task, performance was comparable in the voice task. More importantly, both native and non-native listeners responded faster and more accurately when verbal and vocal emotional expressions were congruent as compared to when they were incongruent. Given that the size of the latter effect did not differ as a function of language proficiency, one can conclude that the integration of verbal and vocal emotional expressions occurs as readily in one's second language as it does in one's native language.     

Perceiving verbal and vocal emotions in a second language

Emotional inferences from speech require the integration of verbal and vocal emotional expressions. We asked whether this integration is comparable when listeners are exposed to their native language and when they listen to a language learned later in life. To this end, we presented native and non-native listeners with positive, neutral and negative words that were spoken with a happy, neutral or sad tone of voice. In two separate tasks, participants judged word valence and ignored tone of voice or judged emotional tone of voice and ignored word valence. While native listeners outperformed non-native listeners in the word valence task, performance was comparable in the voice task. More importantly, both native and non-native listeners responded faster and more accurately when verbal and vocal emotional expressions were congruent as compared to when they were incongruent. Given that the size of the latter effect did not differ as a function of language proficiency, one can conclude that the integration of verbal and vocal emotional expressions occurs as readily in one's second language as it does in one's native language.     

Teaching 'Yes' and 'No' responses to children with multiple disabilities through a program including microswitches linked to a vocal output device

The present study introduced two children (a boy and a girl of 9.2 and 12.3 yr. of age, respectively, with multiple disabilities) to an intervention program for establishing Yes and No responses in relation to preferred and nonpreferred stimuli and assessed the acquisition and generalization of those responses. The program involved the use of microswitches, which monitored the nonverbal Yes and No responses performed by the children and ensured that these responses produced the matching verbal utterances via a vocal output device. Analysis showed children acquired Yes and No responses in relation to the stimuli used for the intervention and, to a comparable degree, to the stimuli used for generalization checks as well. An interview of 30 educational staff of a center for persons with multiple disabilities indicated that they considered the presence of Yes and No verbal utterances preferable to the absence of such utterances, and thought that the utterances could be helpful for the children's response acquisition and would be useful and pleasing for staff and parents. General implications of the findings were discussed.     

Teaching spontaneous responses to young children with autism

Using a multiple probe design across responses, we demonstrated the effectiveness of intensive intervention in establishing spontaneous verbal responses to 2 3-year-old children with autism with generalization to novel settings involving novel persons. Intervention involved discrete-trial instruction (i.e., repeated instructional opportunities presented in close proximity to high rates of reinforcement), specific prompts, and error correction. Spontaneous responses were defined as specific verbal utterances (e.g., the child says "bless you") following discriminative stimuli that did not involve explicit vocal directives (e.g., adult sneeze). The development of effective interventions to address the social-communicative needs of very young children with autism is discussed.     

Exploring the Process of Inference Generation in Sarcasm: A Review of Normal and Clinical Studies

This article evaluates two theoretical accounts of how sarcasm is understood; the traditional model, which asserts that listeners derive a counterfactual inference from the sarcastic comment, and relevance theory, which asserts that listeners recognize sarcasm as a scornful echo of a previous assertion. Evidence from normal speakers provides only partial support for both theories. Evidence from brain-injured populations suggests that aspects of the pragmatic process can be arrested in ways not predicted by either theory. It is concluded that sarcasm is more effortful to process than nonsarcastic comments and that inferences about the facts of the situation and the mental state of the speaker (e.g., attitudes, knowledge, and intentions) are important to comprehending sarcasm. It is questioned whether inferences about mental state are relatively more difficult for brain-injured subjects and, if so, whether this is a continuum of difficulty or reflects reliance upon different cognitive processes.     

Does node stability underlie the verbal transformation effect? A test of node structure theory

Continuous repetition of a word causes listeners to hear the word transform into other utterances, an illusion known as the verbal transformation effect. Node structure theory (MacKay, 1987) provides a useful framework for understanding the illusion, positing that the transformations listeners report are a function of the stability of the node that represents the repeating stimulus. In Experiment 1, the accuracy of this account was investigated, using stimuli that varied from words to phonotactically illegal pseudowords. Experiments 2 and 3 replicated and generalized the findings of Experiment 1, which support a conceptualization of node stability slightly different from that embodied in node structure theory. A new method of measuring lexical influences in the verbal transformation effect is also introduced.     

THE CONTRIBUTION OF KINESIC ILLUSTRATORS TOWARD THE COMPREHENSION OF VERBAL BEHAVIOR WITHIN UTTERANCES

The study assessed the extent to which a speaker's visible body movements can improve verbal comprehension for listeners. Subjects responded to multiple-choice items designed to test their comprehension of 12 videotaped spoken utterances which had been obtained by asking speakers to describe either objects in motion (e.g., a tennis ball, a car, spraying water) or abstract concepts. The 60 subjects each responded to stimuli in one of three presentation conditions (audiovisual, audiovisual without lip and facial cues, and audio-alone) over four signal-to-noise ratios. The results indicated that: (1) visual cues can at times significantly improve comprehension scores, even with lip and facial cues not present; (2) visual cues are increasingly useful as noise is introduced; (3) visual cues assist the comprehension of certain grammatical types of verbal segments regardless of semantic content expressed in those type segments.     

Understanding speaker attitudes from prosody by adults with Parkinson's disease

The ability to interpret vocal (prosodic) cues during social interactions can be disrupted by Parkinson's disease, with notable effects on how emotions are understood from speech. This study investigated whether PD patients who have emotional prosody deficits exhibit further difficulties decoding the attitude of a speaker from prosody. Vocally inflected but semantically nonsensical ‘pseudo‐utterances’ were presented to listener groups with and without PD in two separate rating tasks. Task 1 required participants to rate how confident a speaker sounded from their voice and Task 2 required listeners to rate how polite the speaker sounded for a comparable set of pseudo‐utterances. The results showed that PD patients were significantly less able than HC participants to use prosodic cues to differentiate intended levels of speaker confidence in speech, although the patients could accurately detect the polite/impolite attitude of the speaker from prosody in most cases. Our data suggest that many PD patients fail to use vocal cues to effectively infer a speaker's emotions as well as certain attitudes in speech such as confidence, consistent with the idea that the basal ganglia play a role in the meaningful processing of prosodic sequences in spoken language ( Pell & Leonard, 2003 ).     

Processes and products of imitation: Additional evidence that imitation is progressive

The spontaneous verbal interactions of two children and their babysitter were recorded over the period of half a year. The children were males, from middle-class backgrounds, and 2 years old at the beginning of the observation period. Imitated utterances were found to surpass quantitatively spontaneous ones on the dimensions of grammatical and semantic complexity. With few exceptions, new structures appeared first in imitative utterances and only later in spontaneous ones. Both the higher complexity and the prior appearance of new constructions in imitative utterances suggest that imitation fulfilled a progressive function for the observed children.     

Control of voice gender in pre‐pubertal children

Adult listeners are capable of identifying the gender of speakers as young as 4 years old from their voice. In the absence of a clear anatomical dimorphism in the dimensions of pre‐pubertal boys' and girls' vocal apparatus, the observed gender differences may reflect children's regulation of their vocal behaviour. A detailed acoustic analysis was conducted of the utterances of 34 6‐ to 9‐year‐old children, in their normal voices and also when asked explicitly to speak like a boy or a girl. Results showed statistically significant shifts in fundamental and formant frequency values towards those expected from the sex dimorphism in adult voices. Directions for future research on the role of vocal behaviours in pre‐pubertal children's expression of gender are considered.     

Talker-specific learning in speech perception

The effects of perceptual learning of talker identity on the recognition of spoken words and sentences were investigated in three experiments. In each experiment, listeners were trained to learn a set of 10 talkers’ voices and were then given an intelligibility test to assess the influence of learning the voices on the processing of the linguistic content of speech. In the first experiment, listeners learned voices from isolated words and were then tested with novel isolated words mixed in noise. The results showed that listeners who were given words produced by familiar talkers at test showed better identification performance than did listeners who were given words produced by unfamiliar talkers. In the second experiment, listeners learned novel voices from sentence-length utterances and were then presented with isolated words. The results showed that learning a talker’s voice from sentences did not generalize well to identification of novel isolated words. In the third experiment, listeners learned voices from sentence-length utterances and were then given sentence-length utterances produced by familiar and unfamiliar talkers at test. We found that perceptual learning of novel voices from sentence-length utterances improved speech intelligibility for words in sentences. Generalization and transfer from voice learning to linguistic processing was found to be sensitive to the talker-specific information available during learning and test. These findings demonstrate that increased sensitivity to talker-specific information affects the perception of the linguistic properties of speech in isolated words and sentences.