Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli

We investigated whether the “unity assumption,” according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1–3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the “unity assumption” in the domain of the multisensory temporal integration of audiovisual speech stimuli.     

When the face fits: Recognition of celebrities from matching and mismatching faces and voices

The results of two experiments are presented in which participants engaged in a face-recognition or a voice-recognition task. The stimuli were face–voice pairs in which the face and voice were co-presented and were either “matched” (same person), “related” (two highly associated people), or “mismatched” (two unrelated people). Analysis in both experiments confirmed that accuracy and confidence in face recognition was consistently high regardless of the identity of the accompanying voice. However accuracy of voice recognition was increasingly affected as the relationship between voice and accompanying face declined. Moreover, when considering self-reported confidence in voice recognition, confidence remained high for correct responses despite the proportion of these responses declining across conditions. These results converged with existing evidence indicating the vulnerability of voice recognition as a relatively weak signaller of identity, and results are discussed in the context of a person-recognition framework.     

Influence of vowel context on the audio‐visual speech perception of voiced stop consonants

The work reported here investigated whether the extent of McGurk effect differs according to the vowel context, and differs when cross‐modal vowels are matched or mismatched in Japanese. Two audio‐visual experiments were conducted to examine the process of audio‐visual phonetic‐feature extraction and integration. The first experiment was designed to compare the extent of the McGurk effect in Japanese in three different vowel contexts. The results indicated that the effect was largest in the /i/ context, moderate in the /a/ context, and almost nonexistent in the /u/ context. This suggests that the occurrence of McGurk effect depends on the characteristics of vowels and the visual cues from their articulation. The second experiment measured the McGurk effect in Japanese with cross‐modal matched and mismatched vowels, and showed that, except with the /u/ sound, the effect was larger when the vowels were matched than when they were mismatched. These results showed, again, that the extent of McGurk effect depends on vowel context and that auditory information processing before phonetic judgment plays an important role in cross‐modal feature integration.     

Crossmodal binding: evaluating the "unity assumption" using audiovisual speech stimuli

We investigated whether the "unity assumption," according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1-3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the "unity assumption" in the domain of the multisensory temporal integration of audiovisual speech stimuli.     

Psychophysics of the McGurk and other audiovisual speech integration effects

When the auditory and visual components of spoken audiovisual nonsense syllables are mismatched, perceivers produce four different types of perceptual responses, auditory correct, visual correct, fusion (the so-called McGurk effect), and combination (i.e., two consonants are reported). Here, quantitative measures were developed to account for the distribution of the four types of perceptual responses to 384 different stimuli from four talkers. The measures included mutual information, correlations, and acoustic measures, all representing audiovisual stimulus relationships. In Experiment 1, open-set perceptual responses were obtained for acoustic /bɑ/ or /lɑ/ dubbed to video /bɑ, dɑ, gɑ, vɑ, zɑ, lɑ, wɑ, eɑ/. The talker, the video syllable, and the acoustic syllable significantly influenced the type of response. In Experiment 2, the best predictors of response category proportions were a subset of the physical stimulus measures, with the variance accounted for in the perceptual response category proportions between 17% and 52%. That audiovisual stimulus relationships can account for perceptual response distributions supports the possibility that internal representations are based on modality-specific stimulus relationships.     

Lexical influences in audiovisual speech perception

Phoneme identification with audiovisually discrepant stimuli is influenced hy information in the visual signal (the McGurk effect). Additionally, lexical status affects identification of auditorily presented phonemes. The present study tested for lexical influences on the McGurk effect. Participants identified phonemes in audiovisually discrepant stimuli in which lexical status of the auditory component and of a visually influenced percept was independently varied. Visually influenced (McGurk) responses were more frequent when they formed a word and when the auditory signal was a nonword (Experiment 1). Lexical effects were larger for slow than for fast responses (Experiment 2), as with auditory speech, and were replicated with stimuli matched on physical properties (Experiment 3). These results are consistent with models in which lexical processing of speech is modality independent.     

Perception of intersensory synchrony in audiovisual speech: not that special

Perception of intersensory temporal order is particularly difficult for (continuous) audiovisual speech, as perceivers may find it difficult to notice substantial timing differences between speech sounds and lip movements. Here we tested whether this occurs because audiovisual speech is strongly paired (“unity assumption”). Participants made temporal order judgments (TOJ) and simultaneity judgments (SJ) about sine-wave speech (SWS) replicas of pseudowords and the corresponding video of the face. Listeners in speech and non-speech mode were equally sensitive judging audiovisual temporal order. Yet, using the McGurk effect, we could demonstrate that the sound was more likely integrated with lipread speech if heard as speech than non-speech. Judging temporal order in audiovisual speech is thus unaffected by whether the auditory and visual streams are paired. Conceivably, previously found differences between speech and non-speech stimuli are not due to the putative “special” nature of speech, but rather reflect low-level stimulus differences.     

Biased lineup instructions and face identification from video images

Previous eyewitness memory research has shown that biased lineup instructions reduce identification accuracy, primarily by increasing false-positive identifications in target-absent lineups. Because some attempts at identification do not rely on a witness's memory of the perpetrator but instead involve matching photos to images on surveillance video, the authors investigated the effects of biased instructions on identification accuracy in a matching task. In Experiment 1, biased instructions did not affect the overall accuracy of participants who used video images as an identification aid, but nearly all correct decisions occurred with target-present photo spreads. Both biased and unbiased instructions resulted in high false-positive rates. In Experiment 2, which focused on video-photo matching accuracy with target-absent photo spreads, unbiased instructions led to more correct responses (i.e., fewer false positives). These findings suggest that investigators should not relax precautions against biased instructions when people attempt to match photos to an unfamiliar person recorded on video.     

The Effect of Predicate Matching on Perceived Understanding and Factual Recall

Grinder and Bandler (1976) asserted that if therapists communicate with clients using verbal predicates that match the modality of the clients' primary representational system (PRS), it will be easier for the clients to understand the therapist and to believe that they are understood. In this study we investigated the relationship between predicate matching and understanding. Participants (N=99) listened to a recorded narrative under either a “matched” or “mismatched” experimental condition. Participants' objective understanding (factual recall) and subjective understanding (feelings of having understood) were assessed. The results weakly supported a claim of enhanced accuracy of understanding on one measure of objective understanding, and they suggested greater understanding and recall for visual PRS participants.     

Infant face interest is associated with voice information and maternal psychological health

Early infant interest in their mother's face is driven by an experience based face processing system, and is associated with maternal psychological health, even within a non clinical community sample. The present study examined the role of the voice in eliciting infants’ interest in mother and stranger faces and in the association between infant face interest and maternal psychological health.Infants aged 3.5-months were shown photographs of their mother's and a stranger's face paired with an audio recording of their mother's and a stranger's voice that was either matched (e.g., mother's face and voice) or mismatched (e.g., mother's face and stranger's voice). Infants spent more time attending to the stranger's matched face and voice than the mother's matched face and voice and the mismatched faces and voices. Thus, infants demonstrated an earlier preference for a stranger's face when given voice information than when the face is presented alone. In the present sample, maternal psychological health varied with 56.7% of mothers reporting mild mood symptoms (depression, anxiety or stress response to childbirth). Infants of mothers with significant mild maternal mood symptoms looked longer at the faces and voices compared to infants of mothers who did not report mild maternal mood symptoms. In sum, infants’ experience based face processing system is sensitive to their mothers’ maternal psychological health and the multimodal nature of faces.     

Predictive validity of the CognitiveVs. Somatic Anxiety Distinction

The purpose of this study was to test the predictive validity of an instrument, The Cognitive-Somatic Anxiety Questionnaire (CSAQ), which purports to measure separately cognitive and somatic anxiety. Twelve subjects with either predominantly cognitive or somatic anxiety received a treatment that matched their predominant anxiety mode (“matched”), and twelve received a treatment which addressed the secondary anxiety mode (“mismatched”). After five sessions, the “matched” subjects reported significantly fewer anxiety symptoms than the “mismatched” subjects. The results support the predictive validity of the CSAQ, and suggest that anxiety may not be a unidimensional phenomenon. The implications of these results for clinical practice are discussed.     

迫选规则体验法: 检验规范性和描述性风险决策理论的新尝试

为寻求检验规范性和描述性风险决策理论的通用标准, 本研究以期望价值理论和齐当别抉择模型为例, 探讨了“迫选规则体验法”的适用性。被试为120名大学生, 实验任务为要求被试分别完成自主决策(采用未知规则: 真规则)和规则迫选决策(遵循给定规则: 假规则)任务, 并对决策后的情感和认可程度进行评定。研究发现: (1) 被试在自主决策条件下比在规则迫选条件下体验到的正性情感程度更强, 负性情感的程度更弱; (2) 被试在自主决策与规则迫选决策两种条件下做出的相同决策越多, 该被试对迫选规则更加认可并体验到的正性情感程度越强, 负性情感的程度越弱; (3) 与期望价值理论相比, 齐当别抉择模型可能符合更多决策者的实际决策规则。这些结果表明, 作为检验规范性和描述性风险决策理论的新尝试, 迫选规则体验法可能更有助于回答“决策者实际采用的决策规则是什么”的问题。     

非言语声音影响汉语听者言语声音的知觉

采用启动范式, 以汉语听者为被试, 考察了非言语声音是否影响言语声音的知觉。实验1考察了纯音对辅音范畴连续体知觉的影响, 结果发现纯音影响到辅音范畴连续体的知觉, 表现出频谱对比效应。实验2考察了纯音和复合音对元音知觉的影响, 结果发现与元音共振峰频率一致的纯音或复合音加快了元音的识别, 表现出启动效应。两个实验一致发现非言语声音能够影响言语声音的知觉, 表明言语声音知觉也需要一个前言语的频谱特征分析阶段, 这与言语知觉听觉理论的观点一致。     

Priming of non-speech vocalizations in male adults: the influence of the speaker's gender

Previous research reported a priming effect for voices. However, the type of information primed is still largely unknown. In this study, we examined the influence of speaker's gender and emotional category of the stimulus on priming of non-speech vocalizations in 10 male participants, who performed a gender identification task. We found a significant priming effect, which was greater for female than male vocalizations. In contrast, the emotional category of the stimuli did not appear to modulate the magnitude of priming. These data suggest that different aspects of the information conveyed in non-speech vocalizations may contribute differently to behavioral priming, possibly influenced by task relevance.     

The Role of Parenthood and College Education in the Self-Concept of College Students: Explicit and Implicit Assessments of Gendered Aspirations

This research examined the extent to which parenthood and college education are incorporated into the self-concept of college students. A US sample of undergraduates (90 men, 87 women) attending a large and ethnically diverse university completed explicit and implicit measures of identification with “parenthood” and “college education,” associations between these concepts and gender categories, and gender identification. Explicitly, men and women identified strongly with college education. Implicitly, women identified equally with parenthood and college education, whereas men identified more strongly with college education. In addition, implicit measures revealed that traditional gender roles accounted for a stronger identification with parenthood for participants who displayed a female identity and a stronger identification with college education for participants who displayed a male identity.     

The McGurk effect in infants

In the McGurk effect, perceptual identification of auditory speech syllables is influenced by simultaneous presentation of discrepant visible speech syllables. This effect has been found in subjects of different ages and with various native language backgrounds. But no McGurk tests have been conducted with prelinguistic infants. In the present series of experiments, 5-month-old English-exposed infants were tested for the McGurk effect. Infants were first gaze-habituated to an audiovisual /va/. Two different dishabituation stimuli were then presented: audio /ba/-visual /va/ (perceived by adults as /va/), and audio /da/-visual /va/ (perceived by adults as /da/). The infants showed generalization from the audiovisual /va/ to the audio /ba/-visual /va/ stimulus but not to the audio /da/-visual /va/ stimulus. Follow-up experiments revealed that these generalization differences were not due to a general preference for the audio /da/-visual /va/ stimulus or to the auditory similarity of /ba/ to /va/ relative to /da/. These results suggest that the infants were visually influenced in the same way as Englishspeaking adults are visually influenced.     

Integrating speech information across talkers,gender, and sensory modality: Female faces and male voices in the McGurk effect

Studies of the McGurk effect have shown that when discrepant phonetic information is delivered to the auditory and visual modalities, the information is combined into a new percept not originally presented to either modality. In typical experiments, the auditory and visual speech signals are generated by the same talker. The present experiment examined whether a discrepancy in the gender of the talker between the auditory and visual signals would influence the magnitude of the McGurk effect. A male talker’s voice was dubbed onto a videotape containing a female talker’s face, and vice versa. The gender-incongruent videotapes were compared with gender-congruent videotapes, in which a male talker’s voice was dubbed onto a male face and a female talker’s voice was dubbed onto a female face. Even though there was a clear incompatibility in talker characteristics between the auditory and visual signals on the incongruent videotapes, the resulting magnitude of the McGurk effectwas not significantly different for the incongruent as opposed to the congruent videotapes. The results indicate that the mechanism for integrating speech information from the auditory and the visual modalities is not disrupted by a gender incompatibility even when it is perceptually apparent. The findings are compatible with the theoretical notion that information about voice characteristics of the talker is extracted and used to normalize the speech signal at an early stage of phonetic processing, prior to the integration of the auditory and the visual information.     

Integrating speech information across talkers, gender, and sensory modality: female faces and male voices in the McGurk effect.

Studies of the McGurk effect have shown that when discrepant phonetic information is delivered to the auditory and visual modalities, the information is combined into a new percept not originally presented to either modality. In typical experiments, the auditory and visual speech signals are generated by the same talker. The present experiment examined whether a discrepancy in the gender of the talker between the auditory and visual signals would influence the magnitude of the McGurk effect. A male talker's voice was dubbed onto a videotape containing a female talker's face, and vice versa. The gender-incongruent videotapes were compared with gender-congruent videotapes, in which a male talker's voice was dubbed onto a male face and a female talker's voice was dubbed onto a female face. Even though there was a clear incompatibility in talker characteristics between the auditory and visual signals on the incongruent videotapes, the resulting magnitude of the McGurk effect was not significantly different for the incongruent as opposed to the congruent videotapes. The results indicate that the mechanism for integrating speech information from the auditory and the visual modalities is not disrupted by a gender incompatibility even when it is perceptually apparent. The findings are compatible with the theoretical notion that information about voice characteristics of the talker is extracted and used to normalize the speech signal at an early stage of phonetic processing, prior to the integration of the auditory and the visual information.     

Psychophysiological effects of emotional dissonance in a face-to-face service interaction

Abstract

Current research demonstrates that requirements to express emotions which are not genuinely felt in the particular situation (emotional dissonance) are associated with negative long and short-term effects. The aim of the present study was to investigate: (a) the psychophysiological short-term effects of emotional dissonance in a face-to-face service interaction and (b) the moderating role of gender and neuroticism. In total, 32 women and 27 men were instructed to play the role of a service employee, who had to interact with an angry and rude customer. Half of the sample was given information that the organization expected them to be friendly (emotional dissonance), the other half was told they were expected to act naturally and show their genuine feelings (no emotional dissonance). Subjective and behavioral responses revealed that participants in the “friendly” condition modulated their emotional expressions to a greater extent than those in the “naturally” condition. Participants in the “friendly” condition showed stronger systolic and diastolic blood pressure responses. Furthermore, hierarchical regression analyses yielded that gender and neuroticism partly moderated the association between emotional dissonance and participants' psychophysiological response.