Auditory color constancy: Calibration to reliable spectral properties across nonspeech context and targets

Brief experience with reliable spectral characteristics of a listening context can markedly alter perception of subsequent speech sounds, and parallels have been drawn between auditory compensation for listening context and visual color constancy. In order to better evaluate such an analogy, the generality of acoustic context effects for sounds with spectral-temporal compositions distinct from speech was investigated. Listeners identified nonspeech sounds—extensively edited samples produced by a French horn and a tenor saxophone—following either resynthesized speech or a short passage of music. Preceding contexts were “colored” by spectral envelope difference filters, which were created to emphasize differences between French horn and saxophone spectra. Listeners were more likely to report hearing a saxophone when the stimulus followed a context filtered to emphasize spectral characteristics of the French horn, and vice versa. Despite clear changes in apparent acoustic source, the auditory system calibrated to relatively predictable spectral characteristics of filtered context, differentially affecting perception of subsequent target nonspeech sounds. This calibration to listening context and relative indifference to acoustic sources operates much like visual color constancy, for which reliable properties of the spectrum of illumination are factored out of perception of color.     

Temporally nonadjacent nonlinguistic sounds affect speech categorization

Speech perception is an ecologically important example of the highly context-dependent nature of perception; adjacent speech, and even nonspeech, sounds influence how listeners categorize speech. Some theories emphasize linguistic or articulation-based processes in speech-elicited context effects and peripheral (cochlear) auditory perceptual interactions in non-speech-elicited context effects. The present studies challenge this division. Results of three experiments indicate that acoustic histories composed of sine-wave tones drawn from spectral distributions with different mean frequencies robustly affect speech categorization. These context effects were observed even when the acoustic context temporally adjacent to the speech stimulus was held constant and when more than a second of silence or multiple intervening sounds separated the nonlinguistic acoustic context and speech targets. These experiments indicate that speech categorization is sensitive to statistical distributions of spectral information, even if the distributions are composed of nonlinguistic elements. Acoustic context need be neither linguistic nor local to influence speech perception.     

Crossmodal duration perception involves perceptual grouping, temporal ventriloquism, and variable internal clock rates

Here, we investigate how audiovisual context affects perceived event duration with experiments in which observers reported which of two stimuli they perceived as longer. Target events were visual and/or auditory and could be accompanied by nontargets in the other modality. Our results demonstrate that the temporal information conveyed by irrelevant sounds is automatically used when the brain estimates visual durations but that irrelevant visual information does not affect perceived auditory duration (Experiment 1). We further show that auditory influences on subjective visual durations occur only when the temporal characteristics of the stimuli promote perceptual grouping (Experiments 1 and 2). Placed in the context of scalar expectancy theory of time perception, our third and fourth experiments have the implication that audiovisual context can lead both to changes in the rate of an internal clock and to temporal ventriloquism-like effects on perceived on- and offsets. Finally, intramodal grouping of auditory stimuli diminished any crossmodal effects, suggesting a strong preference for intramodal over crossmodal perceptual grouping (Experiment 5).     

Assessment of infant cry: acoustic cry analysis and parental perception

Infant crying signals distress to potential caretakers who can alleviate the aversive conditions that gave rise to the cry. The cry signal results from coordination among several brain regions that control respiration and vocal cord vibration from which the cry sounds are produced. Previous work has shown a relationship between acoustic characteristics of the cry and diagnoses related to neurological damage, SIDS, prematurity, medical conditions, and substance exposure during pregnancy. Thus, assessment of infant cry provides a window into the neurological and medical status of the infant. Assessment of infant cry is brief and noninvasive and requires recording equipment and a standardized stimulus to elicit a pain cry. The typical protocol involves 30 seconds of crying from a single application of the stimulus. The recorded cry is submitted to an automated computer analysis system that digitizes the cry and either presents a digital spectrogram of the cry or calculates measures of cry characteristics. The most common interpretation of cry measures is based on deviations from typical cry characteristics. Another approach evaluates the pattern across cry characteristics suggesting arousal or under-arousal or difficult temperament. Infants with abnormal cries should be referred for a full neurological evaluation. The second function of crying--to elicit caretaking--involves parent perception of the infant's needs. Typically, parents are sensitive to deviations in cry characteristics, but their perception can be altered by factors in themselves (e.g., depression) or in the context (e.g., culture). The potential for cry assessment is largely untapped. Infant crying and parental response is the first language of the new dyadic relationship. Deviations in the signal and/or misunderstanding the message can compromise infant care, parental effectiveness, and undermine the budding relationship. (c) 2005 Wiley-Liss, Inc. MRDD Research Reviews 2005;11:83-93.     

When sound affects vision: effects of auditory grouping on visual motion perception

Two identical visual targets moving across each other can be perceived either to bounce off or to stream through each other. A brief sound at the moment the targets coincide biases perception toward bouncing. We found that this bounce-inducing effect was attenuated when other identical sounds (auditory flankers) were presented 300 ms before and after the simultaneous sound. The attenuation occurred only when the simultaneous sound and auditory flankers had similar acoustic characteristics and the simultaneous sound was not salient. These results suggest that there is an aspect of auditory-grouping (saliency-assigning) processes that is context-sensitive and can be utilized by the visual system for solving ambiguity. Furthermore, control experiments revealed that such auditory context did not affect the perceptual qualities of the simultaneous sound. Because the attenuation effect is not manifest in the perception of acoustic characteristics of individual sound elements, we conclude that it is a genuine cross-modal effect.     

Visual motion influences the contingent auditory motion aftereffect

In this study, we show that the contingent auditory motion aftereffect is strongly influenced by visual motion information. During an induction phase, participants listened to rightward-moving sounds with falling pitch alternated with leftward-moving sounds with rising pitch (or vice versa). Auditory aftereffects (i.e., a shift in the psychometric function for unimodal auditory motion perception) were bigger when a visual stimulus moved in the same direction as the sound than when no visual stimulus was presented. When the visual stimulus moved in the opposite direction, aftereffects were reversed and thus became contingent upon visual motion. When visual motion was combined with a stationary sound, no aftereffect was observed. These findings indicate that there are strong perceptual links between the visual and auditory motion-processing systems.     

Changes in cry acoustics and distress ratings while the infant is crying

Acoustics and distress ratings were examined during four minutes of naturally occurring crying from 20 healthy, 1‐month‐old infants. Two listeners made continuous judgements of infant distress during every 10‐s segment of each cry sample. Dysphonation, number of wails, and pause duration were related to distress ratings across the 4 min of crying. Means and variances of ratings and acoustics, and the predictive value of each acoustic variable to ratings, differed in the first and second halves of the samples. Differences in the pattern of results occurred for individual infants. The results highlight the importance of studying the cry as a dynamic acoustic signal that may provide unique information about an individual infant's level of distress. Copyright © 2009 John Wiley & Sons, Ltd.     

Infant crying,risk status and social support in families of preterm and term infants

Infant crying influences the caregiver and the broader caregiving environment. In this study, cry acoustics were recorded and acoustically analyzed from a sample of fullterm and preterm infants at 40 weeks gestational age, along with the medical risk and socioeconomic status (SES) of the family. Following factor analysis of the cry acoustics, cry factors, along with medical risk and SES were used to predict patterns of social support in the informal (family, friends) and formal (health care providers) social support networks at 44 weeks gestational age. One cry factor, temporal patterning, indicative of the influence of respiratory factors on the infant's cry, predicted a significant amount of variance in the amount of support from the informal network, beyond that predicted from medical risk and SES. Medical risk alone predicted the amount of contact with the formal network, and SES predicted satisfaction with help from the formal network. There were different patterns of relationship between cry acoustics and social support for families with term and preterm infants, indicating that caregivers may interpret and respond to different information in the acoustics of their infants' cries. These findings have implications for understanding how infant crying and behavior influence the caregiving environment and for the clinical management of early cry problems in families with infants differing in risk status. © 1998 John Wiley & Sons, Ltd.     

From sensory to long-term memory: evidence from auditory memory reactivation studies

Everyday experience tells us that some types of auditory sensory information are retained for long periods of time. For example, we are able to recognize friends by their voice alone or identify the source of familiar noises even years after we last heard the sounds. It is thus somewhat surprising that the results of most studies of auditory sensory memory show that acoustic details, such as the pitch of a tone, fade from memory in ca. 10-15 s. One should, therefore, ask (1) what types of acoustic information can be retained for a longer term, (2) what circumstances allow or help the formation of durable memory records for acoustic details, and (3) how such memory records can be accessed. The present review discusses the results of experiments that used a model of auditory recognition, the auditory memory reactivation paradigm. Results obtained with this paradigm suggest that the brain stores features of individual sounds embedded within representations of acoustic regularities that have been detected for the sound patterns and sequences in which the sounds appeared. Thus, sounds closely linked with their auditory context are more likely to be remembered. The representations of acoustic regularities are automatically activated by matching sounds, enabling object recognition.     

Sounds exaggerate visual shape

While perceiving speech, people see mouth shapes that are systematically associated with sounds. In particular, a vertically stretched mouth produces a /woo/ sound, whereas a horizontally stretched mouth produces a /wee/ sound. We demonstrate that hearing these speech sounds alters how we see aspect ratio, a basic visual feature that contributes to perception of 3D space, objects and faces. Hearing a /woo/ sound increases the apparent vertical elongation of a shape, whereas hearing a /wee/ sound increases the apparent horizontal elongation. We further demonstrate that these sounds influence aspect ratio coding. Viewing and adapting to a tall (or flat) shape makes a subsequently presented symmetric shape appear flat (or tall). These aspect ratio aftereffects are enhanced when associated speech sounds are presented during the adaptation period, suggesting that the sounds influence visual population coding of aspect ratio. Taken together, these results extend previous demonstrations that visual information constrains auditory perception by showing the converse - speech sounds influence visual perception of a basic geometric feature.     

Perceptual dimensions of infant cry signals: a semantic differential analysis

In the present study the Semantic Differential technique was used to examine perceived characteristics of infant cry sounds. 24 cries (6 each of pain, hunger, birth, and "pleasure") were rated by 39 mothers on 50 scales. A factor analysis of the scale ratings uncovered three main factors, labelled Affect, Potency, and Evaluation, respectively. 5 "factorially-pure" scales were selected to represent each factor. A hierarchical cluster analysis of the mean factor-scale ratings uncovered 3 major cry clusters which essentially represented 3 cry-type groups, hunger, pleasure, and pain/birth. A close correspondence was found between this set of clusters and another set generated independently from cry-recognition data, indicating that the semantic differential factor-scales effectively discriminated perceptually distinct cries. This procedure adequately separates the physical (acoustic) properties of signals from their listener-perceived (auditory) ones. This distinction is not possible with studies which depend upon multivariate techniques and casts doubt on their findings.     

Affective reactivity to cry sounds predicts young women’s reactivity and behavior in a simulated caregiving task

Different populations of adults (experienced vs. inexperienced caregivers, men vs. women, abusive vs. nonabusive parents, etc.) have been reported to differ in their affective reactions to the sounds of infant crying. These differences are thought to impact caregiving behavior and, in some instances, to affect long-term outcomes for infants. There can be great intra-group variation, however, even when group differences are significant; modeling developmental process will require a finer grained approach. We have undertaken a pair of studies intended to validate the Negative Affect Scale (NA) from the PANAS as a measure of individuals’ affective reactivity to cry sounds. In Study 1, 306 young women who were not yet mothers listened either to infant crying or to birdsong. The results supported the NA as a measure of reactivity to crying. In Study 2, a new sample of 301 young women listened to crying in a screening task; a group of “high reactors” (n = 21) and a group of “low reactors” (n = 22) then participated in a simulated caregiving situation. Individuals’ affective reactivity to the caregiving simulation mirrored their affective reactivity in the screening task, and rates and overall organization of caregiving behavior differed between the groups. Changes in negative affect, then, appear to be both a result of infant crying and a determinant of some aspects of caregiving behavior. Further studies will extend these laboratory results to real infants and their caregivers, and further validate the NA as a measure of individual differences in reactivity to cry sounds.     

Spectral information in nonspeech contexts influences children’s categorization of ambiguous speech sounds

For both adults and children, acoustic context plays an important role in speech perception. For adults, both speech and nonspeech acoustic contexts influence perception of subsequent speech items, consistent with the argument that effects of context are due to domain-general auditory processes. However, prior research examining the effects of context on children’s speech perception have focused on speech contexts; nonspeech contexts have not been explored previously. To better understand the developmental progression of children’s use of contexts in speech perception and the mechanisms underlying that development, we created a novel experimental paradigm testing 5-year-old children’s speech perception in several acoustic contexts. The results demonstrated that nonspeech context influences children’s speech perception, consistent with claims that context effects arise from general auditory system properties rather than speech-specific mechanisms. This supports theoretical accounts of language development suggesting that domain-general processes play a role across the lifespan.     

The influence of selective attention to auditory and visual speech on the integration of audiovisual speech information

Conflicting visual speech information can influence the perception of acoustic speech, causing an illusory percept of a sound not present in the actual acoustic speech (the McGurk effect). We examined whether participants can voluntarily selectively attend to either the auditory or visual modality by instructing participants to pay attention to the information in one modality and to ignore competing information from the other modality. We also examined how performance under these instructions was affected by weakening the influence of the visual information by manipulating the temporal offset between the audio and video channels (experiment 1), and the spatial frequency information present in the video (experiment 2). Gaze behaviour was also monitored to examine whether attentional instructions influenced the gathering of visual information. While task instructions did have an influence on the observed integration of auditory and visual speech information, participants were unable to completely ignore conflicting information, particularly information from the visual stream. Manipulating temporal offset had a more pronounced interaction with task instructions than manipulating the amount of visual information. Participants' gaze behaviour suggests that the attended modality influences the gathering of visual information in audiovisual speech perception.     

基于通道的内源性注意对声音诱发闪光错觉的影响

声音诱发闪光错觉是指当视觉闪光伴随不相等数量的听觉声音在100 ms内相继或同时呈现时, 个体会错觉性地知觉到视觉闪光的个数与听觉声音的数量相等。研究采用经典的声音诱发闪光错觉范式, 将注意以内源性的方式指向视觉通道或者听觉通道, 考察了基于通道的内源性注意对声音诱发闪光错觉的影响。结果发现, 当注意内源性地指向视觉通道时, 相对于基线条件, 裂变错觉的量显著减小; 当注意内源性地指向听觉通道时, 相对于基线条件, 裂变错觉的量存在增大的趋势。说明了基于通道的内源性注意可以影响声音诱发闪光错觉中的裂变错觉, 但不影响融合错觉。     

Left-Hemisphere Advantage for Click Consonants is Determined by Linguistic Significance and Experience

Abstract—Left-hemisphere (LH) superiority for speech perception is a fundamental neurocognitive aspect of language, and is particularly strong for consonant perception. Two key theoretical aspects of the LH advantage for consonants remain controversial, however: the processing mode (auditory vs. linguistic) and the developmental basis of the specialization (innate vs. experience dependent). Click consonants offer a unique opportunity to evaluate these theoretical issues. Brief and spectrally complex, oral clicks exemplify the acoustic properties that have been proposed for an auditorily based LH specialization, yet they retain linguistic significance only for listeners whose languages employ them as consonants (e.g., Zulu). Speakers of other languages (e.g., English) perceive these clicks as nonspeech sounds. We assessed Zulu versus English listeners' hemispheric asymmetries for clicks, in and out of syllable context, in a dichotic-listening task. Performance was good for both groups, but only Zulus showed an LH advantage. Thus, linguistic processing and experience both appear to be crucial.     

Examination of perceptual reorganization for nonnative speech contrasts: Zulu click discrimination by English-speaking adults and infants

The language environment modifies the speech perception abilities found in early development. In particular, adults have difficulty perceiving many nonnative contrasts that young infants discriminate. The underlying perceptual reorganization apparently occurs by 10-12 months. According to one view, it depends on experiential effects on psychoacoustic mechanisms. Alternatively, phonological development has been held responsible, with perception influenced by whether the nonnative sounds occur allophonically in the native language. We hypothesized that a phonemic process appears around 10-12 months that assimilates speech sounds to native categories whenever possible; otherwise, they are perceived in auditory or phonetic (articulatory) terms. We tested this with English-speaking listeners by using Zulu click contrasts. Adults discriminated the click contrasts; performance on the most difficult (80% correct) was not diminished even when the most obvious acoustic difference was eliminated. Infants showed good discrimination of the acoustically modified contrast even by 12-14 months. Together with earlier reports of developmental change in perception of nonnative contrasts, these findings support a phonological explanation of language-specific reorganization in speech perception.     

What the voice reveals: within- and between-category stereotyping on the basis of voice

The authors report research that attempts to shift the traditional focus of visual cues to auditory cues as a basis for stereotyping. Moreover, their approach examines whether gender-signaling vocal cues lead not only to between-category but also to within-category gender stereotyping. Study 1 showed that both men and women vary within category in how feminine their voices sound and that perceptions of vocal femininity are highly consensual. Furthermore, the measured acoustic characteristics that differed between gender were also related to perceptions of within-gender femininity. Subsequent studies demonstrated that variability in vocal femininity affects gender stereotyping when the targets are all of the same gender (Study 2) and when the targets are of different genders (Study 3). In the latter case, evidence of both category-based and feature-based stereotyping was found. Mediation analyses showed that the relationship between acoustics and stereotyping was in part due to femininity.     

What Are You Waiting For? Real‐Time Integration of Cues for Fricatives Suggests Encapsulated Auditory Memory

Speech unfolds over time, and the cues for even a single phoneme are rarely available simultaneously. Consequently, to recognize a single phoneme, listeners must integrate material over several hundred milliseconds. Prior work contrasts two accounts: (a) a memory buffer account in which listeners accumulate auditory information in memory and only access higher level representations (i.e., lexical representations) when sufficient information has arrived; and (b) an immediate integration scheme in which lexical representations can be partially activated on the basis of early cues and then updated when more information arises. These studies have uniformly shown evidence for immediate integration for a variety of phonetic distinctions. We attempted to extend this to fricatives, a class of speech sounds which requires not only temporal integration of asynchronous cues (the frication, followed by the formant transitions 150–350 ms later), but also integration across different frequency bands and compensation for contextual factors like coarticulation. Eye movements in the visual world paradigm showed clear evidence for a memory buffer. Results were replicated in five experiments, ruling out methodological factors and tying the release of the buffer to the onset of the vowel. These findings support a general auditory account for speech by suggesting that the acoustic nature of particular speech sounds may have large effects on how they are processed. It also has major implications for theories of auditory and speech perception by raising the possibility of an encapsulated memory buffer in early auditory processing.     

Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity

We propose a multisensory framework based on Glaser and Glaser's (1989) general reading-naming interference model to account for the semantic priming effect by naturalistic sounds and spoken words on visual picture sensitivity. Four experiments were designed to investigate two key issues: First, can auditory stimuli enhance visual sensitivity when the sound leads the picture as well as when they are presented simultaneously? And, second, do naturalistic sounds (e.g., a dog's "woofing") and spoken words (e.g., /d?g/) elicit similar semantic priming effects? Here, we estimated participants' sensitivity and response criterion using signal detection theory in a picture detection task. The results demonstrate that naturalistic sounds enhanced visual sensitivity when the onset of the sounds led that of the picture by 346 ms (but not when the sounds led the pictures by 173 ms, nor when they were presented simultaneously, Experiments 1-3A). At the same SOA, however, spoken words did not induce semantic priming effects on visual detection sensitivity (Experiments 3B and 4A). When using a dual picture detection/identification task, both kinds of auditory stimulus induced a similar semantic priming effect (Experiment 4B). Therefore, we suggest that there needs to be sufficient processing time for the auditory stimulus to access its associated meaning to modulate visual perception. Besides, the interactions between pictures and the two types of sounds depend not only on their processing route to access semantic representations, but also on the response to be made to fulfill the requirements of the task.