Detection and resolution of audio-visual incompatibility in the perception of vowels

If audio and video recordings of a talker speaking consonant-vowel syllables containing different consontants are approximately synchronised, observers may fail to detect conflict between the modalities and perceive consonants presented in neither individual modality. The present experiments demonstrate an analogous effect in the perception of vowels. Vision can bias the identity of an acoustical vowel to be more like the vowel presented visually, even when observers detect conflict and are instructed to report only what they hear. The size of the effect is positively related to the size of the physical difference between the visible configuration of the lips and the configuration that would naturally accompany the acoustical vowel. In demonstrating these and other phenomena in audio-visual speech perception, observers behave as if they compute a continuous estimate of the filter function of the vocal tract from both visual and acoustical evidence. If the visual evidence is potent, observers may appear to interpret the acoustical evidence in novel ways. However, these compromises can be predicted from known patterns of acoustical similarity and visual distinctiveness and do not require ad hoc explanations involving categorical levels of perceptual process.