Seeing to hear better: evidence for early audio-visual interactions in speech identification

Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in noise [J. Acoust. Soc. Am. 109 (2001) 2272; J. Acoust. Soc. Am. 108 (2000) 1197]. However, detection is different from comprehension, and it remains to be seen whether improved sensitivity also results in an intelligibility gain in audio-visual speech perception. In this work, we use an original paradigm to show that seeing the speaker's lips enables the listener to hear better and hence to understand better. The audio-visual stimuli used here could not be differentiated by lip reading per se since they contained exactly the same lip gesture matched with different compatible speech sounds. Nevertheless, the noise-masked stimuli were more intelligible in the audio-visual condition than in the audio-only condition due to the contribution of visual information to the extraction of acoustic cues. Replacing the lip gesture by a non-speech visual input with exactly the same time course, providing the same temporal cues for extraction, removed the intelligibility benefit. This early contribution to audio-visual speech identification is discussed in relationships with recent neurophysiological data on audio-visual perception.     

The Bursts and Lulls of Multimodal Interaction: Temporal Distributions of Behavior Reveal Differences Between Verbal and Non‐Verbal Communication

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, 2012 ), we focus on measuring the temporal patterns of specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog.     

Distress tolerance but not impulsivity predicts outcome in concurrent treatment for posttraumatic stress disorder and substance use disorder

Posttraumatic stress disorder (PTSD) and substance use disorder (SUD) commonly co-occur. Emerging research suggests that both distress tolerance and impulsivity may underlie this comorbidity. However, to our knowledge no studies have examined whether these 2 constructs predict outcome in PTSD-SUD treatment. The current study investigated whether pretreatment distress tolerance and impulsivity predicted posttreatment PTSD and cravings severity in a sample of 70 Veterans receiving concurrent treatment for PTSD and SUD in a residential day treatment program. Veterans completed measures of symptom severity before and after treatment. Results demonstrated that pretreatment distress tolerance predicted posttreatment PTSD severity while controlling for pretreatment PTSD. By contrast, pretreatment impulsivity was not predictive of posttreatment PTSD while controlling for pretreatment values. Neither distress tolerance nor impulsivity predicted posttreatment cravings severity. The findings support the notion that distress tolerance may help to explain the co-occurrence of PTSD and SUD and suggest that targeting this construct in PTSD-SUD treatment may be important for successful outcomes.     

A COMPARISON OF THE EFFECTS OF BRIEF RULES, A TIMER, AND PREFERRED TOYS ON SELF-CONTROL

Some children make impulsive choices (i.e., choose a small but immediate reinforcer over a large but delayed reinforcer). Previous research has shown that delay fading, providing an alternative activity during the delay, teaching participants to repeat a rule during the delay, combining delay fading with an alternative activity, and combining delay fading with a countdown timer are effective for increasing self‐control (i.e., choosing the large but delayed reinforcer over the small but immediate reinforcer). The purpose of the current study was to compare the effects of various interventions in the absence of delay fading (i.e., providing brief rules, providing a countdown timer during the delay, or providing preferred toys during the delay) on self‐control. Results suggested that providing brief rules or a countdown timer during the delay was ineffective for enhancing self‐control. However, providing preferred toys during the delay effectively enhanced self‐control.     

Links between self-reported and laboratory behavioral impulsivity

Havik, M., Jakobson, A., Tamm, M., Paaver, M., Konstabel, K., Uusberg, A., Allik, J., Ööpik, V. & Kreegipuu, K. (2012). Links between self‐reported and laboratory behavioral impulsivity. Scandinavian Journal of Psychology 53, 216–223. A major problem in the research considering impulsivity is the lack of mutual understanding on how to measure and define impulsivity. Our study examined the relationship between self‐reported impulsivity, behavioral excitatory and inhibitory processes and time perception. Impulsivity – fast, premature, thoughtless or disinhibited behavior – was assessed in 58 normal, healthy participants (30 men, mean age 21.9 years). Self‐reported impulsivity as measured by Adaptive and Maladaptive Impulsivity Scale (AMIS) and behavioral excitatory and inhibitory processes as measured by Stop Signal Task were not directly related. Time perception, measured by the retrospective Time Estimation Task, was related to both. The length of the perceived time interval was positively correlated to AMIS Disinhibition subscale and negatively to several Stop Signal Task parameters. The longer subjects perceived the duration to last, the higher was their score on Disinhibition scale and the faster were their reactive responses in the Stop Signal Task. In summary our findings support the idea of cognitive tempo as a possible mechanism underlying impulsive behavior.     

Performance on auditory and visual temporal information processing is related to psychometric intelligence

The present study investigated the relationship between psychometric intelligence and temporal resolution power (TRP) as simultaneously assessed by auditory and visual psychophysical timing tasks. In addition, three different theoretical models of the functional relationship between TRP and psychometric intelligence as assessed by means of the Adaptive Matrices Test (AMT) were developed. To test the validity of these models, structural equation modeling was applied. Empirical data supported a hierarchical model that assumed auditory and visual modality-specific temporal processing at a first level and amodal temporal processing at a second level. This second-order latent variable was substantially correlated with psychometric intelligence. Therefore, the relationship between psychometric intelligence and psychophysical timing performance can be explained best by a hierarchical model of temporal information processing.     

When two become one: Temporally dynamic integration of the face and voice

In everyday interactions with others, people have to deal with the sight of a face and sound of a voice at the same time. How the perceptual system brings this information together over hundreds of milliseconds to perceive others remains unclear. In 2 studies, we investigated how facial and vocal cues are integrated during real-time social categorization by recording participants' hand movements (via the streaming x, y coordinates of the computer mouse) en route to “male” and “female” responses on the screen. Participants were presented with male and female faces that were accompanied by a same-sex voice morphed to be either sex-typical (e.g., masculinized male voice) or sex-atypical (i.e., feminized male voice). Before settling into ultimate sex categorizations of the face, the simultaneous processing of a sex-atypical voice led the hand to be continuously attracted to the opposite sex-category response across construal. This is evidence that ongoing results from voice perception continuously influence face perception across processing. Thus, social categorization involves dynamic updates of gradual integration of the face and voice.     

Delay discounting: I'm a k, you're a k

Delay discounting is the decline in the present value of a reward with delay to its receipt. Across a variety of species, populations, and reward types, value declines hyperbolically with delay. Value declines steeply with shorter delays, but more shallowly with longer delays. Quantitative modeling provides precise measures to characterize the form of the discount function. These measures may be regarded as higher-order dependent variables, intervening variables, or hypothetical constructs. I suggest the degree of delay discounting may be a personality trait. In the end, the ontological status of measures of delay discounting is irrelevant. Whatever delay discounting may be, its study has provided the field of behavior analysis and other areas measures with robust generality and predictive validity for a variety of significant human problems. Research on moderating the degree of delay discounting has the potential to produce substantial societal benefits.     

Is depression the past tense of anxiety? An empirical study of the temporal distinction

It has long been recognised that depression and anxiety share a common core of negative affect, but research on similarities and differences between these two emotions is growing. The focus of the current study was on whether the timing of a triggering event can determine whether the dominant emotional reaction is depression or anxiety. It was hypothesised that aversive events in the past would elicit more depression than anxiety, whereas the same aversive events in the future would elicit more anxiety than depression. We created temporally varied versions of vignettes describing aversive events occurring at either time, and asked participants to rate the extent to which the events would elicit feelings of depression or anxiety. Results indicated that adverse past events elicited much higher ratings of anticipated depression and adverse future events elicited much higher ratings of anticipated anxiety. Implications for understanding these two emotions and depressive and anxiety disorders are discussed.     

Patterns of Responding on a Balloon Analogue Task Reveal Individual Differences in Overall Risk-Taking: Choice Between Guaranteed and Uncertain Cash

ABSTRACT. We explored the utility of analyzing within- and between-balloon response patterns on a balloon analogue task (BAT) in relation to overall risk scores, and to a choice between a small guaranteed cash reward and an uncertain reward of the same expected value. Young adults (n = 61) played a BAT, and then were offered a choice between $5 in cash and betting to win $0 to $15. Between groups, pumping was differentially influenced by explosions and by the number of successive unexploded balloons, with risk takers responding increasingly on successive balloons after an explosion. Within-balloons, risk takers showed a characteristic pattern of constant high rate, while non-risk takers showed a characteristic variable lower rate. Overall, results show that the higher number of pumps and explosions that characterize risk takers at a molar level, result from particular forms of adaptation to the positive and negative outcomes of choices seen at a molecular level.

Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/vgen.