Performance on auditory and visual temporal information processing is related to psychometric intelligence

The present study investigated the relationship between psychometric intelligence and temporal resolution power (TRP) as simultaneously assessed by auditory and visual psychophysical timing tasks. In addition, three different theoretical models of the functional relationship between TRP and psychometric intelligence as assessed by means of the Adaptive Matrices Test (AMT) were developed. To test the validity of these models, structural equation modeling was applied. Empirical data supported a hierarchical model that assumed auditory and visual modality-specific temporal processing at a first level and amodal temporal processing at a second level. This second-order latent variable was substantially correlated with psychometric intelligence. Therefore, the relationship between psychometric intelligence and psychophysical timing performance can be explained best by a hierarchical model of temporal information processing.     

Crossmodal attention switching: Auditory dominance in temporal discrimination tasks

Visual stimuli are often processed more efficiently than accompanying stimuli in another modality. In line with this “visual dominance”, earlier studies on attentional switching showed a clear benefit for visual stimuli in a bimodal visual–auditory modality-switch paradigm that required spatial stimulus localization in the relevant modality. The present study aimed to examine the generality of this visual dominance effect. The modality appropriateness hypothesis proposes that stimuli in different modalities are differentially effectively processed depending on the task dimension, so that processing of visual stimuli is favored in the dimension of space, whereas processing auditory stimuli is favored in the dimension of time. In the present study, we examined this proposition by using a temporal duration judgment in a bimodal visual–auditory switching paradigm. Two experiments demonstrated that crossmodal interference (i.e., temporal stimulus congruence) was larger for visual stimuli than for auditory stimuli, suggesting auditory dominance when performing temporal judgment tasks. However, attention switch costs were larger for the auditory modality than for visual modality, indicating a dissociation of the mechanisms underlying crossmodal competition in stimulus processing and modality-specific biasing of attentional set.     

Timing is everything: Changes in presentation rate have opposite effects on auditory and visual implicit statistical learning

Implicit statistical learning (ISL) is exclusive to neither a particular sensory modality nor a single domain of processing. Even so, differences in perceptual processing may substantially affect learning across modalities. In three experiments, statistically equivalent auditory and visual familiarizations were presented under different timing conditions that either facilitated or disrupted temporal processing (fast or slow presentation rates). We find an interaction of rate and modality of presentation: At fast rates, auditory ISL was superior to visual. However, at slow presentation rates, the opposite pattern of results was found: Visual ISL was superior to auditory. Thus, we find that changes to presentation rate differentially affect ISL across sensory modalities. Additional experiments confirmed that this modality-specific effect was not due to cross-modal interference or attentional manipulations. These findings suggest that ISL is rooted in modality-specific, perceptually based processes.     

Timing is everything: changes in presentation rate have opposite effects on auditory and visual implicit statistical learning

Implicit statistical learning (ISL) is exclusive to neither a particular sensory modality nor a single domain of processing. Even so, differences in perceptual processing may substantially affect learning across modalities. In three experiments, statistically equivalent auditory and visual familiarizations were presented under different timing conditions that either facilitated or disrupted temporal processing (fast or slow presentation rates). We find an interaction of rate and modality of presentation: At fast rates, auditory ISL was superior to visual. However, at slow presentation rates, the opposite pattern of results was found: Visual ISL was superior to auditory. Thus, we find that changes to presentation rate differentially affect ISL across sensory modalities. Additional experiments confirmed that this modality-specific effect was not due to cross-modal interference or attentional manipulations. These findings suggest that ISL is rooted in modality-specific, perceptually based processes.     

Shared and distinct factors driving attention and temporal processing across modalities

In addition to the classic finding that “sounds are judged longer than lights,” the timing of auditory stimuli is often more precise and accurate than is the timing of visual stimuli. In cognitive models of temporal processing, these modality differences are explained by positing that auditory stimuli more automatically capture and hold attention, more efficiently closing an attentional switch that allows the accumulation of pulses marking the passage of time (Penney, Gibbon, & Meck, 2000). However, attention is a multifaceted construct, and there has been little attempt to determine which aspects of attention may be related to modality effects. We used visual and auditory versions of the Continuous Temporal Expectancy Task (CTET; O'Connell et al., 2009) a timing task previously linked to behavioral and electrophysiological measures of mind-wandering and attention lapses, and tested participants with or without the presence of a video distractor. Performance in the auditory condition was generally superior to that in the visual condition, replicating standard results in the timing literature. The auditory modality was also less affected by declines in sustained attention indexed by declines in performance over time. In contrast, distraction had an equivalent impact on performance in the two modalities. Analysis of individual differences in performance revealed further differences between the two modalities: Poor performance in the auditory condition was primarily related to boredom whereas poor performance in the visual condition was primarily related to distractibility. These results suggest that: 1) challenges to different aspects of attention reveal both modality-specific and nonspecific effects on temporal processing, and 2) different factors drive individual differences when testing across modalities.     

Modality-specific auditory and visual temporal processing deficits

We studied the attentional blink (AB) and the repetition blindness (RB) effects using an audio-visual presentation procedure designed to overcome several potential methodological confounds in previous cross-modal research. In Experiment 1, two target digits were embedded amongst letter distractors in two concurrent streams (one visual and the other auditory) presented from the same spatial location. Targets appeared in either modality unpredictably at different temporal lags, and the participants' task was to recall the digits at the end of the trial. We evaluated both AB and RB for pairs of targets presented in either the same or different modalities. Under these conditions both AB and RB were observed in vision, AB but not RB was observed in audition, and there was no evidence of AB or RB cross-modally from audition to vision or vice versa. In Experiment 2, we further investigated the AB by including Lag 1 items and observed Lag 1 sparing, thus ruling out the possibility that the observed effects were due to perceptual and/or conceptual masking. Our results support a distinction between a modality-specific interference at the attentional selection stage and a modality-independent interference at later processing stages. They also provide a new dissociation between the AB and RB.     

Involuntary listening aids seeing: evidence from human electrophysiology

It is well known that sensory events of one modality can influence judgments of sensory events in other modalities. For example, people respond more quickly to a target appearing at the location of a previous cue than to a target appearing at another location, even when the two stimuli are from different modalities. Such cross-modal interactions suggest that involuntary spatial attention mechanisms are not entirely modality-specific. In the present study, event-related brain potentials (ERPs) were recorded to elucidate the neural basis and timing of involuntary, cross-modal spatial attention effects. We found that orienting spatial attention to an irrelevant sound modulates the ERP to a subsequent visual target over modality-specific, extrastriate visual cortex, but only after the initial stages of sensory processing are completed. These findings are consistent with the proposal that involuntary spatial attention orienting to auditory and visual stimuli involves shared, or at least linked, brain mechanisms.     

The contribution of rapid visual and auditory processing to the reading of irregular words and pseudowords presented singly and in contiguity

This study examined the relative involvement of rapid auditory and visual temporal resolution mechanisms in the reading of phonologically regular pseudowords and English irregular words presented both in isolation and in contiguity as a series of six words. Seventy-nine undergraduates participated in a range of reading, visual temporal, and auditory temporal tasks. The correlation analyses suggested a general timing mechanism across modalities. There were more significant correlations between the visual temporal measures and irregular word reading and between the auditory measures and pseudoword reading. Auditory gap detection predicted pseudoword reading accuracies. The low temporal frequency flicker contrast sensitivity measure predicted the accuracies of isolated irregular words and pseudowords presented in contiguity. However, when a combined speed-accuracy score was used, visible persistence at both low and high spatial frequencies and auditory gap detection were active in the reading of pseudowords presented in contiguity. Sensory processing skills in both visual and auditory modalities accounted for some of the variance in the reading performance of normal undergraduates, not just reading-impaired students.     

Twice upon a time: multiple concurrent temporal recalibrations of audiovisual speech

Audiovisual timing perception can recalibrate following prolonged exposure to asynchronous auditory and visual inputs. It has been suggested that this might contribute to achieving perceptual synchrony for auditory and visual signals despite differences in physical and neural signal times for sight and sound. However, given that people can be concurrently exposed to multiple audiovisual stimuli with variable neural signal times, a mechanism that recalibrates all audiovisual timing percepts to a single timing relationship could be dysfunctional. In the experiments reported here, we showed that audiovisual temporal recalibration can be specific for particular audiovisual pairings. Participants were shown alternating movies of male and female actors containing positive and negative temporal asynchronies between the auditory and visual streams. We found that audiovisual synchrony estimates for each actor were shifted toward the preceding audiovisual timing relationship for that actor and that such temporal recalibrations occurred in positive and negative directions concurrently. Our results show that humans can form multiple concurrent estimates of appropriate timing for audiovisual synchrony.     

Variable foreperiods and temporal discrimination

Temporal judgements are often accounted for by a single-clock hypothesis. The output of such a clock is reported to depend on the allocation of attention. In the present series of experiments, the influence of attention on temporal information processing is investigated by systematic variations of the period preceding brief empty intervals to be judged. Two indicators of timing performance, temporal sensitivity, reflecting discrimination performance, and perceived duration served as dependent variables. Foreperiods ranged from 0.3 to 0.6 s in Experiments 1 to 4. When the foreperiod varied randomly from trial to trial, perceived duration was longer with increasing length of foreperiod (Experiments 1 and 3 with brief auditory markers and Experiment 4 with brief visual markers), an effect that disappeared with no trial-to-trial variations (Experiment 2). Longer foreperiods also enhanced performance on temporal discrimination of auditory empty intervals with a base duration of 100 ms (Experiments 1 and 5), whereas discrimination performance was unaffected for auditory intervals with a base duration of 500 ms (Experiment 3). The variable-foreperiod effect on perceived duration also held when foreperiods ranged from 0.6 to 1.5 s (Experiments 5-7). Findings suggest that foreperiods appear to effectively modulate attention mechanisms necessary for temporal information processing. However, alternative explanations such as assimilation or compatibility effects cannot be totally discarded.     

It's about time: revisiting temporal processing deficits in dyslexia

Temporal processing in French children with dyslexia was evaluated in three tasks: a word identification task requiring implicit temporal processing, and two explicit temporal bisection tasks, one in the auditory and one in the visual modality. Normally developing children matched on chronological age and reading level served as a control group. Children with dyslexia exhibited robust deficits in temporal tasks whether they were explicit or implicit and whether they involved the auditory or the visual modality. First, they presented larger perceptual variability when performing temporal tasks, whereas they showed no such difficulties when performing the same task on a non‐temporal dimension (intensity). This dissociation suggests that their difficulties were specific to temporal processing and could not be attributed to lapses of attention, reduced alertness, faulty anchoring, or overall noisy processing. In the framework of cognitive models of time perception, these data point to a dysfunction of the ‘internal clock’ of dyslexic children. These results are broadly compatible with the recent temporal sampling theory of dyslexia.     

Action enhances auditory but not visual temporal sensitivity

People naturally dance to music, and research has shown that rhythmic auditory stimuli facilitate production of precisely timed body movements. If motor mechanisms are closely linked to auditory temporal processing, just as auditory temporal processing facilitates movement production, producing action might reciprocally enhance auditory temporal sensitivity. We tested this novel hypothesis with a standard temporal-bisection paradigm, in which the slope of the temporal-bisection function provides a measure of temporal sensitivity. The bisection slope for auditory time perception was steeper when participants initiated each auditory stimulus sequence via a keypress than when they passively heard each sequence, demonstrating that initiating action enhances auditory temporal sensitivity. This enhancement is specific to the auditory modality, because voluntarily initiating each sequence did not enhance visual temporal sensitivity. A control experiment ruled out the possibility that tactile sensation associated with a keypress increased auditory temporal sensitivity. Taken together, these results demonstrate a unique reciprocal relationship between auditory time perception and motor mechanisms. As auditory perception facilitates precisely timed movements, generating action enhances auditory temporal sensitivity.     

The reading ability of good and poor temporal processors among a group of college students

In this study, we examined whether good auditory and good visual temporal processors were better than their poor counterparts on certain reading measures. Various visual and auditory temporal tasks were administered to 105 undergraduates. They read some phonologically regular pseudowords and irregular words that were presented sequentially in the same ("word" condition) and in different ("line" condition) locations. Results indicated that auditory temporal acuity was more relevant to reading, whereas visual temporal acuity was more relevant to spelling. Good auditory temporal processors did not have the advantage in processing pseudowords, even though pseudoword reading correlated significantly with auditory temporal processing. These results suggested that some higher cognitive or phonological processes mediated the relationship between auditory temporal processing and pseudoword reading. Good visual temporal processors did not have the advantage in processing irregular words. They also did not process the line condition more accurately than the word condition. The discrepancy might be attributed to the use of normal adults and the unnatural reading situation that did not fully capture the function of the visual temporal processes. The distributions of auditory and visual temporal processing abilities were co-occurring to some degree, but they maintained considerable independence. There was also a lack of a relationship between the type and severity of reading deficits and the type and number of temporal deficits.     

Hearing what the eyes see: auditory encoding of visual temporal sequences

When the senses deliver conflicting information, vision dominates spatial processing, and audition dominates temporal processing. We asked whether this sensory specialization results in cross-modal encoding of unisensory input into the task-appropriate modality. Specifically, we investigated whether visually portrayed temporal structure receives automatic, obligatory encoding in the auditory domain. In three experiments, observers judged whether the changes in two successive visual sequences followed the same or different rhythms. We assessed temporal representations by measuring the extent to which both task-irrelevant auditory information and task-irrelevant visual information interfered with rhythm discrimination. Incongruent auditory information significantly disrupted task performance, particularly when presented during encoding; by contrast, varying the nature of the rhythm-depicting visual changes had minimal impact on performance. Evidently, the perceptual system automatically and obligatorily abstracts temporal structure from its visual form and represents this structure using an auditory code, resulting in the experience of "hearing visual rhythms."     

发展性阅读障碍的视听时间敏感性缺陷及其脑神经机制

阅读是一个视听加工过程,阅读障碍的产生可能是过程中视听时间敏感性缺陷的结果。视听时间敏感性指个体对视觉和听觉刺激出现时间的感知能力,可通过同时性判断、时间顺序判断和视听整合考察。研究发现,阅读障碍者在这一能力上表现出行为和脑层面异常。而这些研究多是拼音文字背景,汉语文字下该领域研究相当少。未来需要丰富实验设计,扩大对汉语背景下视听时间敏感性研究,并以此开发干预手段,为阅读障碍的机制和治疗提供借鉴。     

On the disruption of visual memory: Interference produced by visual report cues

Report of single letters from centrally-fixated, seven-letter, target rows was probed by either auditory or visual cues. The target rows were presented for 100 ms, and the report cues were single digits which indicated the spatial location of a letter. In three separate experiments, report was always better with the auditory cues. The advantage for the auditory cues was maintained both when target rows were masked by a patterned stimulus and when the auditory cues were presented 500 ms later than comparable visual cues. The results indicate that visual cues produce modality-specific interference which operates at a level of processing beyond iconic representation.     

发展性阅读障碍的听觉时间加工缺陷

发展性阅读障碍是指个体在智力正常并且不缺乏学校教育的情况下, 仍无法获得与年龄相匹配的阅读技能的一种学习障碍, 其缺陷的本质一直是研究者争论的焦点。大量研究显示, 阅读障碍者常表现出听觉时间加工损伤。在行为层面, 阅读障碍者难以辨别快速、连续呈现刺激的顺序以及刺激本身的动态时间特征。在神经层面, 阅读障碍者诱发的失匹配负波更弱且具有异常的神经同步加工。这些损伤同时存在于对言语和非言语刺激的加工中, 表明听觉时间加工缺陷非言语加工所特有。未来的研究还需阐明以下几个问题：1)阅读障碍的听觉时间加工缺陷发生在哪些时间窗口, 随年龄增长如何变化; 2)阅读障碍听觉时间加工缺陷在神经层面的时间进程是怎样的; 3)听觉时间加工缺陷是否为阅读障碍的核心缺陷。     

The role of visual and auditory temporal processing in reading irregular and nonsense words.

In the present study, the role of rapid visual and auditory temporal processing in reading irregular and nonsense words was investigated with a group of normal readers. One hundred and five undergraduates participated in various visual and auditory temporal-processing tasks. Readers who primarily adopted the phonological route in reading (nonsense-word readers) showed a trend for better auditory temporal resolution but readers who primarily adopted sight word skills (irregular-word readers) did not exhibit better visual temporal resolution. Both the correlation and stepwise multiple-regression analyses, however, revealed a relationship between visual temporal processing and irregular-word reading as well as a relationship between auditory temporal processing and nonsense-word reading. The results support the involvement of visual and auditory processing in reading irregular and nonsense words respectively, and were discussed with respect to recent findings that only dyslexics with phonological impairment will display temporal deficits. Further, the temporal measures were not effective discriminants for the reading groups, suggesting a lack of association between reading ability and the choice of reading strategy.     

Auditory memory can be object based

Identifying how memories are organized remains a fundamental issue in psychology. Previous work has shown that visual short-term memory is organized according to the object of origin, with participants being better at retrieving multiple pieces of information from the same object than from different objects. However, it is not yet clear whether similar memory structures are employed for other modalities, such as audition. Under analogous conditions in the auditory domain, we found that short-term memories for sound can also be organized according to object, with a same-object advantage being demonstrated for the retrieval of information in an auditory scene defined by two complex sounds overlapping in both space and time. Our results provide support for the notion of an auditory object, in addition to the continued identification of similar processing constraints across visual and auditory domains. The identification of modality-independent organizational principles of memory, such as object-based coding, suggests possible mechanisms by which the human processing system remembers multimodal experiences.     

What limits working memory capacity? Evidence for modality-specific sources to the simultaneous storage of visual and auditory arrays

There is considerable debate on whether working memory (WM) storage is mediated by distinct subsystems for auditory and visual stimuli (Baddeley, 1986) or whether it is constrained by a single, central capacity-limited system (Cowan, 2006). Recent studies have addressed this issue by measuring the dual-task cost during the concurrent storage of auditory and visual arrays (e.g., Cocchini, Logie, Della Sala, MacPherson, & Baddeley, 2002; Fougnie & Marois, 2006; Saults & Cowan, 2007). However, studies have yielded widely different dual-task costs, which have been taken to support both modality-specific and central capacity-limit accounts of WM storage. Here, we demonstrate that the controversies regarding such costs mostly stem from how these costs are measured. Measures that compare combined dual-task capacity with the higher single-task capacity support a single, central WM store when there is a large disparity between the single-task capacities (Experiment 1) but not when the single-task capacities are well equated (Experiment 2). In contrast, measures of the dual-task cost that normalize for differences in single-task capacity reveal evidence for modality-specific stores, regardless of single-task performance. Moreover, these normalized measures indicate that dual-task cost is much smaller if the tasks do not involve maintaining bound feature representations in WM (Experiment 3). Taken together, these experiments not only resolve a discrepancy in the field and clarify how to assess the dual-task cost but also indicate that WM capacity can be constrained both by modality-specific and modality-independent sources of information processing.