Use of visual information in speech perception: evidence for a visual rate effect both with and without a McGurk effect

The McGurk effect, where an incongruent visual syllable influences identification of an auditory syllable, does not always occur, suggesting that perceivers sometimes fail to use relevant visual phonetic information. We tested whether another visual phonetic effect, which involves the influence of visual speaking rate on perceived voicing (Green & Miller, 1985), would occur in instances when the McGurk effect does not. In Experiment 1, we established this visual rate effect using auditory and visual stimuli matching in place of articulation, finding a shift in the voicing boundary along an auditory voice-onset-time continuum with fast versus slow visual speech tokens. In Experiment 2, we used auditory and visual stimuli differing in place of articulation and found a shift in the voicing boundary due to visual rate when the McGurk effect occurred and, more critically, when it did not. The latter finding indicates that phonetically relevant visual information is used in speech perception even when the McGurk effect does not occur, suggesting that the incidence of the McGurk effect underestimates the extent of audio-visual integration.     

Garner interference is not solely driven by stimulus uncertainty

The speeded classification tasks popularized by Garner (1974), with their accompanying labels of separable and integral dimensions, have become the dominant paradigm for characterizing perceptual interactions between stimulus dimensions. Separable dimensions like color and shape can be selectively attended to at will, but integral dimensions like hue and brightness cannot. When classifying stimuli with respect to a single dimension, integral stimuli lead to faster performance in the baseline block, where only that single dimension varies, than in the filtering block, in which a second dimension is allowed to vary irrelevantly. No such difference is predicted for separable dimensions lead to no difference. The comparison between baseline and filtering confounds a change in the number of stimuli with a change in the number of variant dimensions. A new experimental condition is proposed utilizing three stimulus dimensions to hold the number of stimuli constant while introducing variance along a second irrelevant dimension. Data indicate that interference increases, suggesting that changes in the number of potential stimuli within a block are not necessary for Garner interference, in contrast to the accounts provided by extant quantitative models.     

Processing two dimensions of nonspeech stimuli: the auditory-phonetic distinction reconsidered.

Nonspeech stimuli were varied along two dimensions--intensity and rise time. In a series of speeded classification tasks, subjected were asked to identify the stimuli in terms of one of these dimensions. Identification time for the dimension of rise time increased when there was irrelevant variation in intensity; however, identification of intensity was unaffected by irrelevant variation in rise time. When the two dimensions varied redundantly, identification time decreased. This pattern of results is virtually identical to that obtained previously for stimuli that vary along a linguistic and a nonlinguistic dimension. The present data, taken together with those from other studies using the same stimuli, suggest that the mechanisms underlying the auditory-phonetic distinction should be reconsidered. The results are also discussed in terms of general models of multidimensional information processing.     

Dimensional interactions in vibrotactile information processing

The effect of irrelevant dimensional variation on the processing of vibrotactile stimuli was measured. Six observers performed a speeded classification task with stimuli varying along the dimensions of pitch and loudness. Choice reaction times were obtained for stimuli differing on one dimension alone, on two correlated dimensions, or on two orthogonally varied dimensions. Compared to one-dimension performance, reaction times were faster in the correlated condition and slower in the orthogonal condition. In general, these findings agreed with similar experiments in other modalities, with the exception that the effects in this study tended to be stronger for cases in which loudness, rather than pitch, was the relevant dimension. The results are explained in terms of the integrality of pitch and loudness and of the relative discriminability of dimensions.     

The role of visual information in the processing of

Visual information provided by a talker’s mouth movements can influence the perception of certain speech features. Thus, the “McGurk effect” shows that when the syllable /bi/ is presented audibly, in synchrony with the syllable /gi/, as it is presented visually, a person perceives the talker as saying /di/. Moreover, studies have shown that interactions occur between place and voicing features in phonetic perception, when information is presented audibly. In our first experiment, we asked whether feature interactions occur when place information is specified by a combination of auditory and visual information. Members of an auditory continuum ranging from /ibi/ to /ipi/ were paired with a video display of a talker saying /igi/. The auditory tokens were heard as ranging from /ibi/ to /ipi/, but the auditory-visual tokens were perceived as ranging from /idi/ to /iti/. The results demonstrated that the voicing boundary for the auditory-visual tokens was located at a significantly longer VOT value than the voicing boundary for the auditory continuum presented without the visual information. These results demonstrate that place-voice interactions are not limited to situations in which place information is specified audibly. In three follow-up experiments, we show that (1) the voicing boundary is not shifted in the absence of a change in the global percept, even when discrepant auditory-visual information is presented; (2) the number of response alternatives provided for the subjects does not affect the categorization or the VOT boundary of the auditory-visual stimuli; and (3) the original effect of a VOT boundary shift is not replicated when subjects are forced by instruction to \ldrelabel\rd the /b-p/auditory stimuli as/d/or/t/. The subjects successfully relabeled the stimuli, but no shift in the VOT boundary was observed.     

Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task

The McGurk effect is usually presented as an example of fast, automatic, multisensory integration. We report a series of experiments designed to directly assess these claims. We used a syllabic version of the speeded classification paradigm, whereby response latencies to the first (target) syllable of spoken word-like stimuli are slowed down when the second (irrelevant) syllable varies from trial to trial. This interference effect is interpreted as a failure of selective attention to filter out the irrelevant syllable. In Experiment 1 we reproduced the syllabic interference effect with bimodal stimuli containing auditory as well as visual lip movement information, thus confirming the generalizability of the phenomenon. In subsequent experiments we were able to produce (Experiment 2) and to eliminate (Experiment 3) syllabic interference by introducing 'illusory' (McGurk) audiovisual stimuli in the irrelevant syllable, suggesting that audiovisual integration occurs prior to attentional selection in this paradigm.     

Classification errors and response times over multiple distributed sessions as a function of category structure

Learning difficulty orderings for categorical stimuli have long provided an empirical foundation for concept learning and categorization research. The conventional approach seeks to determine learning difficulty orderings in terms of mean classification accuracy. However, it is relatively rare that the stability of such orderings is tested over a period of extended learning. Further, research rarely explores dependent variables beyond classification accuracy that may also indicate relative learning difficulty, such as classification response times (RTs). Using a family of category structures defined over three binary dimensions and four positive examples that is well-known for its robust learning difficulty ordering, we report the results of two experiments that test the stability of the ordering (in terms of both errors and RTs) over multiple category learning sessions. The experimental stimuli consisted of instantiations of each of the six category structures in the family. These take the form of categories consisting of four “flasks” that vary along the binary features of size (large or small), shape (circular or triangular), and color (black or white). Experiment 1 shows that when participants are randomly presented instances of all six types, the difficulty ordering remains stable across all three sessions. This stability is present in terms of mean accuracy (errors) as well as mean RTs. In Experiment 2, participants were repeatedly exposed to category instances of a single type. In terms of errors, the ordering is revealed in the first session and disappears in later sessions. The opposite trend is observed for classification RTs: The ordering is not present in the first session but is revealed in later sessions. This suggests that even when individuals reach a relative degree of expertise in terms of reduced errors, the original degree of difficulty continues to influence processing. We interpret these results in the context of the concept learning and perceptual expertise literatures.     

The role of visual information in the processing of place and manner features in speech perception

Visual information provided by a talker's mouth movements can influence the perception of certain speech features. Thus, the "McGurk effect" shows that when the syllable (bi) is presented audibly, in synchrony with the syllable (gi), as it is presented visually, a person perceives the talker as saying (di). Moreover, studies have shown that interactions occur between place and voicing features in phonetic perception, when information is presented audibly. In our first experiment, we asked whether feature interactions occur when place information is specificed by a combination of auditory and visual information. Members of an auditory continuum ranging from (ibi) to (ipi) were paired with a video display of a talker saying (igi). The auditory tokens were heard as ranging from (ibi) to (ipi), but the auditory-visual tokens were perceived as ranging from (idi) to (iti). The results demonstrated that the voicing boundary for the auditory-visual tokens was located at a significantly longer VOT value than the voicing boundary for the auditory continuum presented without the visual information. These results demonstrate that place-voice interactions are not limited to situations in which place information is specified audibly.(ABSTRACT TRUNCATED AT 250 WORDS)     

Garner interference and temporal information processing

The aim of the study was to explore whether temporal information processing can interfere with performance of a non-temporal task. A new methodology based on the Garner paradigm was employed. Participants were asked to classify two-dimensional stimuli according to either length or duration, with and without variation in the other (task-irrelevant) dimension. Garner interference was detected only with respect to classification by length when irrelevant variation in duration was present. Stroop interference was detected only in classification by length across compatible and non-compatible values of length and duration. Classification by length took more time when done with variation in duration than when duration was constant. Classification by length also took more time when length and duration were not compatible than when they were compatible. The findings indicate that the processing of duration is similar to the processing of other perceptual dimensions. The processing of duration consumes attentional resources and can interfere with the processing of other perceptual dimensions. The findings support attentional models of prospective duration judgment.     

认知负载对基于动机的情绪刺激加工的影响

结合数数任务和情绪flanker任务,探究不同认知负载条件下,高低趋近动机对情绪刺激加工的影响。结果发现：高负载组中,高趋近动机条件下,对积极目标刺激的反应显著快于消极目标刺激;低趋近动机条件下,对消极目标刺激的反应显著快于积极目标刺激。且高负载组中,各条件下的反应大多都显著慢于低负载组。低负载组中没有发现各条件之间的差异。结果表明,高负载条件下,高趋近动机促进了积极情绪刺激的加工,低趋近动机促进了消极情绪刺激的加工,高负载在一定程度上阻碍了情绪刺激的加工。     

认知负载对基于动机的情绪刺激加工的影响

结合数数任务和情绪flanker任务,探究不同认知负载条件下,高低趋近动机对情绪刺激加工的影响。结果发现：高负载组中,高趋近动机条件下,对积极目标刺激的反应显著快于消极目标刺激;低趋近动机条件下,对消极目标刺激的反应显著快于积极目标刺激。且高负载组中,各条件下的反应大多都显著慢于低负载组。低负载组中没有发现各条件之间的差异。结果表明,高负载条件下,高趋近动机促进了积极情绪刺激的加工,低趋近动机促进了消极情绪刺激的加工,高负载在一定程度上阻碍了情绪刺激的加工。     

Hard and soft interacting dimensions: differential effects of dual context on classification

In this article, we extend Garner's speeded classification procedure to investigate processes underlying the interaction of the auditory dimensions pitch, loudness, and timbre. In the experiments reported here, subjects classified attributes on these three auditory dimensions. Our extended procedure, called multiclass, is based conceptually on our model of how such dimensions interact; the model explains the perception of attributes from an attended dimension through the action of contextual constraints created by variation along an unattended dimension. Two forms of context are present simultaneously in each multiclass task: intraclass context, variation along the unattended dimension that interferes with the classification of attributes, and redundant context, variation along the unattended dimension that enhances classification. We find that such dual-context situations reliably distinguish two kinds of interacting dimensions. Subjects classifying hard dimensions, here pitch and timbre, resist the ill effects of intraclass context and reap gains from redundant context. Subjects classifying soft dimensions, here loudness, show interference because the attributes are veiled perceptually in dual context. These findings, we argue, demonstrate the power of the multiclass procedure and fit well our view that dimensional interaction entails processing both at the level of the stimulus whole and at the level of stimulus attributes.     

Hard andsoft interacting dimensions: Differential effects of dual context on classification

In this article, we extend Garner’s speeded classification procedure to investigate processes underlying the interaction of the auditory dimensions pitch, loudness, and timbre. In the experiments reported here, subjects classified attributes on these three auditory dimensions. Our extended procedure, calledmulticlass, is based conceptually on our model of how such dimensions interact; the model explains the perception of attributes from an attended dimension through the action of contextual constraints created by variation along an unattended dimension. Two forms of context are present simultaneously in each multiclass task:intraclass context, variation along the unattended dimension that interferes with the classification of attributes, andredundant context, variation along the unattended dimension that enhances classification. We find that such dual-context situations reliably distinguish two kinds of interacting dimensions. Subjects classifying HARD dimensions, here pitch and timbre, resist the ill effects of intraclass.context and reap gains from redundant context. Subjects classifying SOFT dimensions, here loudness, show interference because the attributes are veiled perceptually in dual context, These findings, we argue, demonstrate the power of the multiclass procedure and fit well our view-that dimensional interaction entails processing both at the level of the stimulus whale and at the level of stimulus attributes.     

The separability of space and time: Dimensional interaction in the memory trace

The speeded-classification paradigm, adapted from the study of perceptual interactions, was used to examine interactions among features of the short-term memory trace. In each of four experiments, a trial began with the presentation of a pair of stimuli, each member of which occurred either first or second in time and above or below a fixation point. One of the two stimuli was then presented again, and the task was to classify its prior temporal (Experiments 1 and 3) or spatial (Experiments 2 and 4) position. The main question of interest asked whether subjects could selectively attend to one of these occurrence dimensions while ignoring irrelevant variation along the other. The results suggested that whereas subjects can selectively ignore temporal or spatial variation when no recall of the irrelevant dimension’s value is required, they suffer interference when values on both dimensions must be remembered. The obtained patterns of interference are consistent withpostperceptual interactions of the-spatial and temporal components of the memory trace.     

Influence of auditory and audiovisual stimuli on the right–left prevalence effect

When auditory stimuli are used in two-dimensional spatial compatibility tasks, where the stimulus and response configurations vary along the horizontal and vertical dimensions simultaneously, a right–left prevalence effect occurs in which horizontal compatibility dominates over vertical compatibility. The right–left prevalence effects obtained with auditory stimuli are typically larger than that obtained with visual stimuli even though less attention should be demanded from the horizontal dimension in auditory processing. In the present study, we examined whether auditory or visual dominance occurs when the two-dimensional stimuli are audiovisual, as well as whether there will be cross-modal facilitation of response selection for the horizontal and vertical dimensions. We also examined whether there is an additional benefit of adding a pitch dimension to the auditory stimulus to facilitate vertical coding through use of the spatial-musical association of response codes (SMARC) effect, where pitch is coded in terms of height in space. In Experiment 1, we found a larger right–left prevalence effect for unimodal auditory than visual stimuli. Neutral, non-pitch coded, audiovisual stimuli did not result in cross-modal facilitation, but did show evidence of visual dominance. The right–left prevalence effect was eliminated in the presence of SMARC audiovisual stimuli, but the effect influenced horizontal rather than vertical coding. Experiment 2 showed that the influence of the pitch dimension was not in terms of influencing response selection on a trial-to-trial basis, but in terms of altering the salience of the task environment. Taken together, these findings indicate that in the absence of salient vertical cues, auditory and audiovisual stimuli tend to be coded along the horizontal dimension and vision tends to dominate audition in this two-dimensional spatial stimulus–response task.     

影响视觉复合刺激中整体和局部性质加工的几种因素

文章研究了视知觉组织,空间位置不确定性和视野位置等因素对复合刺激中整体和局部性质加工的影响。实验发现,（１）当复合刺激呈现在白背景上时,反应时和干扰作用都表明复复合刺激中大图的加工优于小图,刺激位置的不确定削弱大图对小图的干扰作用;而当复合刺激呈现在由“＋”组成的背影或时,小图的加工优于大图,刺激位置的不确定延长分辩小图的ＲＴ;（２）与周国刺激相比,分辨视野中央的复合刺激的ＲＴ较短,这种影响对分辨     

Processes underlying dimensional interactions: Correspondences between linguistic and nonlinguistic dimensions

In six experiments, we examined speeded classification when one dimension was linguistic and the other was nonlinguistic. In five of these, attributes on the dimensions corresponded meaningfully, having in common the concepts "high" and "low." For example, in Experiment 1, the visually presented words HI and LO were paired with high- or low-pitched tones; in Experiment 2, the dimensions were visual words and vertical position, in Experiment 3, they were spoken words and position, and in Experiments 4 and 5, spoken words and pitch. For each dimension in each pair, subjects suffered Garner interference when dimensions were varied orthogonally. Garner interference remained constant across 15 blocks of trials (Experiment 5). Subjects also showed significant congruity effects in all experiments, with attributes from congruent stimuli (e.g., HI/high pitch) classified faster than attributes from incongruent stimuli (e.g., HI/low pitch). These results differ from those obtained previously with noncorresponding pairs of linguistic-nonlinguistic dimensions. The results also differ from those obtained with traditional Stroop dimensions (colors and color words; Experiment 6), which showed minimal Garner interference and diminishing congruity effects across blocks of trials. We conclude that the interactions found here represent cross-talk between channels within a semantic level of processing. We contrast our view with current models of dimensional interaction.     

The effect of selective adaptation on the identification of speech sounds

An experiment was performed to determine the effect of selective adaptation on the identification of synthetic speech sounds which varied along the phonetic dimensionplace of articulation. Adaptation with a stimulus of a particular place value led to a reduction in the number of test stimuli identified as having that place value. An identification shift was obtained even when the acoustic information specifying place value for the adapting stimulus had virtually nothing in common with the information specifying place value for any of the test stimuli. Removing the vowel portion of an adapting stimulus eliminated identification shift only when the resulting stimulus was no longer perceived as speech-like. The results indicate that at least part of the adaptation effect occurs at a site of phonetic, not merely acoustic, feature analysis.

     

Matching patterns vs matching digits: The effect of memory dependence and complexity on “same”-“different” reaction times

Two experiments are reported in which “same”-“different” reaction times (RTs) were collected to pairs of stimuli. In Experiment 1 stimuli were matrix patterns, and in Experiment 2 stimuli were digits. In both experiments, the pairs were presented simultaneously (discrimination task) and successively (memory task) for a set of nine simple and a set of nine complex stimuli. The following results were obtained: discrimination RTs were longer than memory RTs; RTs to complex stimuli were longer than RTs to simple stimuli; “same” RTs were faster than “different” RTs across all conditions except simple pattern discrimination, for which “different” RTs were faster than “same” RTs; and discrimination RTs for complex patterns were longer than would be predicted from the other conditions. Some evidence was obtained that the form of encoding for both patterns and digits in the memory task was visual. These results are discussed in terms of encoding and comparison strategies.