Evaluating the influence of the 'unity assumption' on the temporal perception of realistic audiovisual stimuli

Vatakis, A. and Spence, C. (in press) [Crossmodal binding: Evaluating the 'unity assumption' using audiovisual speech stimuli. Perception &Psychophysics] recently demonstrated that when two briefly presented speech signals (one auditory and the other visual) refer to the same audiovisual speech event, people find it harder to judge their temporal order than when they refer to different speech events. Vatakis and Spence argued that the 'unity assumption' facilitated crossmodal binding on the former (matching) trials by means of a process of temporal ventriloquism. In the present study, we investigated whether the 'unity assumption' would also affect the binding of non-speech stimuli (video clips of object action or musical notes). The auditory and visual stimuli were presented at a range of stimulus onset asynchronies (SOAs) using the method of constant stimuli. Participants made unspeeded temporal order judgments (TOJs) regarding which modality stream had been presented first. The auditory and visual musical and object action stimuli were either matched (e.g., the sight of a note being played on a piano together with the corresponding sound) or else mismatched (e.g., the sight of a note being played on a piano together with the sound of a guitar string being plucked). However, in contrast to the results of Vatakis and Spence's recent speech study, no significant difference in the accuracy of temporal discrimination performance for the matched versus mismatched video clips was observed. Reasons for this discrepancy are discussed.     

Investigating the effects of inversion on configural processing with an audiovisual temporal-order judgment task

Research has shown that inversion is more detrimental to the perception of faces than to the perception of other types of visual stimuli. Inverting a face results in an impairment of configural information processing that leads to slowed early face processing and reduced accuracy when performance is tested in face recognition tasks. We investigated the effects of inverting speech and non-speech stimuli on audiovisual temporal perception. Upright and inverted audiovisual video clips of a person uttering syllables (experiments 1 and 2), playing musical notes on a piano (experiment 3), or a rhesus monkey producing vocalisations (experiment 4) were presented. Participants made unspeeded temporal-order judgments regarding which modality stream (auditory or visual) appeared to have been presented first. Inverting the visual stream did not have any effect on the sensitivity of temporal discrimination responses in any of the four experiments, thus implying that audiovisual temporal integration is resilient to the effects of orientation in the picture plane. By contrast, the point of subjective simultaneity differed significantly as a function of orientation only for the audiovisual speech stimuli but not for the non-speech stimuli or monkey calls. That is, smaller auditory leads were required for the inverted than for the upright-visual speech stimuli. These results are consistent with the longer processing latencies reported previously when human faces are inverted and demonstrates that the temporal perception of dynamic audiovisual speech can be modulated by changes in the physical properties of the visual speech (ie by changes in orientation).     

Phonetic recalibration only occurs in speech mode

Upon hearing an ambiguous speech sound dubbed onto lipread speech, listeners adjust their phonetic categories in accordance with the lipread information (recalibration) that tells what the phoneme should be. Here we used sine wave speech (SWS) to show that this tuning effect occurs if the SWS sounds are perceived as speech, but not if the sounds are perceived as non-speech. In contrast, selective speech adaptation occurred irrespective of whether listeners were in speech or non-speech mode. These results provide new evidence for the distinction between a speech and non-speech processing mode, and they demonstrate that different mechanisms underlie recalibration and selective speech adaptation.     

Temporal processing capabilities in repetition conduction aphasia

This study investigates the temporal resolution capacities of the central-auditory system in a subject (NP) suffering from repetition conduction aphasia. More specifically, the patient was asked to detect brief gaps between two stretches of broadband noise (gap detection task) and to evaluate the duration of two biphasic (WN-3) continuous noise elements, starting with white noise (WN) followed by 3 kHz bandpass-filtered noise (duration discrimination task). During the gap detection task, the two portions of each stimulus were either identical (“intra-channel condition”) or differed (“inter-channel condition”) in the spectral characteristics of the leading and trailing acoustic segments. NP did not exhibit any deficits in the intra-channel condition of the gap detection task, indicating intact auditory temporal resolution across intervals of 1–3 ms. By contrast, the inter-channel condition yielded increased threshold values. Based upon the “multiple-looks” model of central-auditory processing, this profile points at a defective integration window operating across a few tens of milliseconds – a temporal range associated with critical features of the acoustic speech signal such as voice onset time and formant transitions. Additionally, NP was found impaired during a duration discrimination task addressing longer integration windows (ca. 150 ms). Concerning speech, this latter time domain approximately corresponds to the duration of stationary segmental units such as fricatives and long vowels. On the basis of our results we suggest, that the patient’s auditory timing deficits in non-speech tasks may account, at least partially, for his impairments in speech processing.     

Temporal order perception of auditory stimuli is selectively modified by tonal and non-tonal language environments

The close relationship between temporal perception and speech processing is well established. The present study focused on the specific question whether the speech environment could influence temporal order perception in subjects whose language backgrounds are distinctively different, i.e., Chinese (tonal language) vs. Polish (non-tonal language). Temporal order thresholds were measured for both monaurally presented clicks and binaurally presented tone pairs. Whereas the click experiment showed similar order thresholds for the two language groups, the experiment with tone pairs resulted in different observations: while Chinese demonstrated better performance in discriminating the temporal order of two “close frequency” tone pairs (600 Hz and 1200 Hz), Polish subjects showed a reversed pattern, i.e., better performance for “distant frequency” tone pairs (400 Hz and 3000 Hz). These results indicate on the one hand a common temporal mechanism for perceiving the order of two monaurally presented stimuli, and on the other hand neuronal plasticity for perceiving the order of frequency-related auditory stimuli. We conclude that the auditory brain is modified with respect to temporal processing by long-term exposure to a tonal or a non-tonal language. As a consequence of such an exposure different cognitive modes of operation (analytic vs. holistic) are selected: the analytic mode is adopted for “distant frequency” tone pairs in Chinese and for “close frequency” tone pairs in Polish subjects, whereas the holistic mode is selected for “close frequency” tone pairs in Chinese and for “distant frequency” tone pairs in Polish subjects, reflecting a double dissociation of function.     

Time perception,attention, and memory: A selective review

This article provides a selective review of time perception research, mainly focusing on the authors' research. Aspects of psychological time include simultaneity, successiveness, temporal order, and duration judgments. In contrast to findings at interstimulus intervals or durations less than 3.0–5.0 s, there is little evidence for an “across-senses” effect of perceptual modality (visual vs. auditory) at longer intervals or durations. In addition, the flow of time (events) is a pervasive perceptual illusion, and we review evidence on that. Some temporal information is encoded All rights reserved. relatively automatically into memory: People can judge time-related attributes such as recency, frequency, temporal order, and duration of events. Duration judgments in prospective and retrospective paradigms reveal differences between them, as well as variables that moderate the processes involved. An attentional-gate model is needed to account for prospective judgments, and a contextual-change model is needed to account for retrospective judgments.     

Predictability affects the perception of audiovisual synchrony in complex sequences

The ability to make accurate audiovisual synchrony judgments is affected by the "complexity" of the stimuli: We are much better at making judgments when matching single beeps or flashes as opposed to video recordings of speech or music. In the present study, we investigated whether the predictability of sequences affects whether participants report that auditory and visual sequences appear to be temporally coincident. When we reduced their ability to predict both the next pitch in the sequence and the temporal pattern, we found that participants were increasingly likely to report that the audiovisual sequences were synchronous. However, when we manipulated pitch and temporal predictability independently, the same effect did not occur. By altering the temporal density (items per second) of the sequences, we further determined that the predictability effect occurred only in temporally dense sequences: If the sequences were slow, participants' responses did not change as a function of predictability. We propose that reduced predictability affects synchrony judgments by reducing the effective pitch and temporal acuity in perception of the sequences.     

An acquired deficit of audiovisual speech processing

We report a 53-year-old patient (AWF) who has an acquired deficit of audiovisual speech integration, characterized by a perceived temporal mismatch between speech sounds and the sight of moving lips. AWF was less accurate on an auditory digit span task with vision of a speaker's face as compared to a condition in which no visual information from the lower face was available. He was slower in matching words to pictures when he saw congruent lip movements compared to no lip movements or non-speech lip movements. Unlike normal controls, he showed no McGurk effect. We propose that multisensory binding of audiovisual language cues can be selectively disrupted.     

Lapsed attention to elapsed time? Individual differences in working memory capacity and temporal reproduction

Working memory capacity (WMC) predicts individual differences in a wide range of mental abilities. In three experiments we examined whether WMC would predict temporal judgment. Low-WMC temporal reproductions were consistently too long for the shortest duration and too short for the longest, but were accurate (unbiased) for the intermediate. In contrast, high-WMC temporal reproductions were more accurate (unbiased) across the range. Thus low-WMC showed a classic “migration effect” (Vierordt's Law) to a greater extent than high-WMC. Furthermore reproduction errors depended more on temporal context than the absolute durations of “shortest,” “longest,” and “intermediate.” Low-WMC reproductions were overall more variable than high-WMC. General fluid intelligence (gF) was also related to temporal bias and variability. However, WMC-related timing differences were only attenuated and not eliminated with gF as covariate. Results are discussed in terms of attention, memory, and other psychological constructs.     

Degrading phonetic information affects matching of audiovisual speech in adults,but not in infants

Infants and adults are well able to match auditory and visual speech, but the cues on which they rely (viz. temporal, phonetic and energetic correspondence in the auditory and visual speech streams) may differ. Here we assessed the relative contribution of the different cues using sine-wave speech (SWS). Adults (N = 52) and infants (N = 34, age ranged in between 5 and 15 months) matched 2 trisyllabic speech sounds (‘kalisu’ and ‘mufapi’), either natural or SWS, with visual speech information. On each trial, adults saw two articulating faces and matched a sound to one of these, while infants were presented the same stimuli in a preferential looking paradigm. Adults’ performance was almost flawless with natural speech, but was significantly less accurate with SWS. In contrast, infants matched the sound to the articulating face equally well for natural speech and SWS. These results suggest that infants rely to a lesser extent on phonetic cues than adults do to match audio to visual speech. This is in line with the notion that the ability to extract phonetic information from the visual signal increases during development, and suggests that phonetic knowledge might not be the basis for early audiovisual correspondence detection in speech.     

The relationship between feature binding and consciousness: evidence from asynchronous multi-modal stimuli

Processing the various features from different feature maps and modalities in coherent ways requires a dedicated integration mechanism (“the binding problem”). Many authors have related feature binding to conscious awareness but little is known about how tight this relationship really is. We presented subjects with asynchronous audiovisual stimuli and tested whether the two features were integrated. The results show that binding took place up to 350 ms feature-onset asynchronies, suggesting that integration covers a relatively wide temporal window. We also asked subjects to explicitly judge whether the two features would belong to the same or to the different events. Unsurprisingly, synchrony judgments decreased with increasing asynchrony. Most importantly, feature binding was entirely unaffected by conscious experience: features were bound whether they were experienced as occurring together or as belonging to a separate events, suggesting that the conscious experience of unity is not a prerequisite for, or a direct consequence of binding.     

Audiovisual semantic interference and attention: Evidence from the attentional blink paradigm

In the present study we investigate the role of attention in audiovisual semantic interference, by using an attentional blink paradigm. Participants were asked to make an unspeeded response to the identity of a visual target letter. This target letter was preceded at various SOAs by a synchronized audiovisual letter-pair, which was either congruent (e.g. hearing an “F” and viewing an “F”) or incongruent (e.g. hearing an “F” and viewing a “Z”). In Experiment 1, participants were asked to match the members of the audiovisual letter-pair. In Experiment 2, participants were asked to ignore the synchronized audiovisual letter-pairs altogether and only report the visual target. In Experiment 3, participants were asked to identify only one of the audiovisual letters (identify the auditory letter, and ignore the synchronized visual letter, or vice versa). An attentional blink was found in all three experiments indicating that the audiovisual letter-pairs were processed. However, a congruency effect on subsequent target detection was observed in Experiments 1 and 3, but not in Experiment 2. The results indicate that attention to the semantic contents of at least one modality is necessary to establish audiovisual semantic interference.     

Resources versus respect: Social judgments based on targets' power and status positions

In two experiments, we investigate how individuals' levels of power and status interact to determine how they are perceived by others. We find that power and status have similar, positive, effects on judged dominance. We also find that power has a negative effect on perceived warmth, but status moderates this “power penalty”: high power without status is associated with low warmth, but power with status is judged warm. Consequently, we find high status individuals, regardless of power level, are perceived positively - dominant and warm - whereas high power-low status individuals are judged most negatively — dominant and cold (Experiments 1 and 2). As a result, perceivers expect positive interactions with high status individuals, but negative interactions with high power, low status individuals (Experiment 2). These findings provide insight into power and status social judgments, and further our understanding of similarities and differences between these distinct, yet related, constructs.     

Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli

We investigated whether the “unity assumption,” according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1–3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the “unity assumption” in the domain of the multisensory temporal integration of audiovisual speech stimuli.     

Crossmodal binding: evaluating the "unity assumption" using audiovisual speech stimuli

We investigated whether the "unity assumption," according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1-3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the "unity assumption" in the domain of the multisensory temporal integration of audiovisual speech stimuli.     

Strength of Preference and Effects of Valence in the Domains of Gains and Losses

In comparative judgments that follow binary choices, judgments of “How much better is a preferred option?” and “How much worse is a less preferred option?” may differ in their magnitudes (“valence effects”). This paper analyzed cognitive processes that underlie positive valence effects (“Better” exceeding “Worse”) and negative valence effects (Worse exceeding Better). My analyses used a “focus shift model.” The focus shift model postulates that choice options are represented as sets of desirable and undesirable features. Difference judgments are reached by assigning subjective weights onto such features and integrating weighted feature contributions. Positive and negative valence effects reflect the differences in subjective weighting depending on valence of judgments. Experiments 1 through 3 showed that systematic positive valence effects were observed in the domain of gains, whereas negative valence effects were observed in the domain of losses. Estimates of subjective weights showed that valence effects occurred when participants heavily weighted desirable features in preferred options and undesirable features in less preferred options. These patterns of subjective weighting for positive and negative valence effects were consistent with the focus shift model. Data were more consistent with the focus shift model over alternative explanations of valence effects. Relationship with relevant economic and psychological phenomena are discussed.     

非言语声音影响汉语听者言语声音的知觉

采用启动范式, 以汉语听者为被试, 考察了非言语声音是否影响言语声音的知觉。实验1考察了纯音对辅音范畴连续体知觉的影响, 结果发现纯音影响到辅音范畴连续体的知觉, 表现出频谱对比效应。实验2考察了纯音和复合音对元音知觉的影响, 结果发现与元音共振峰频率一致的纯音或复合音加快了元音的识别, 表现出启动效应。两个实验一致发现非言语声音能够影响言语声音的知觉, 表明言语声音知觉也需要一个前言语的频谱特征分析阶段, 这与言语知觉听觉理论的观点一致。     

The contextual and systematic nature of life satisfaction judgments

Five studies were conducted to examine the nature of life satisfaction judgments. When the category of “excitement” was made accessible experimentally, individuals based their life satisfaction judgments more heavily on the frequency of excitement, in comparison to a “peaceful” condition in Study 1 and to both “neutral priming” and “no-priming” conditions in Study 2. A 7-day diary study (Study 3) showed that as “excitement” became naturally more accessible on weekends, the correlations between excitement and daily satisfaction also increased significantly. Study 3 thus illustrated a systematic contextual shift in the bases of life satisfaction judgments. Study 4 showed that high sensation seekers, for whom “excitement” should be chronically accessible, based their life satisfaction judgments more heavily on the frequency of excitement than did low sensation seekers. Finally, Study 5 demonstrated that the chronic accessibility of “excitement” measured at Time 1 predicted the degree to which individuals based their life satisfaction judgments on the frequency of excitement at Time 2. Altogether, these five studies highlight the contextually sensitive, yet systematic nature of life satisfaction judgments.     

A foundation for support theory based on a non-Boolean event space

A new foundation is presented for the theory of subjective judgments of probability known in the psychological literature as “Support Theory”. It is based on new complementation operations that, unlike those of classical probability theory (set-theoretic complementation) and classical logic (negation), need not satisfy the principles of the Law of The Excluded Middle and the Law of Double Complementation. Interrelationships between the new complementation operations and the Kahneman and Tversky judgmental heuristic of availability are described.