Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type

When an audio-visual event is perceived in the natural environment, a physical delay will always occur between the arrival of the leading visual component and that of the trailing auditory component. This natural timing relationship suggests that the point of subjective simultaneity (PSS) should occur at an auditory delay greater than or equal to 0 msec. A review of the literature suggests that PSS estimates derived from a temporal order judgment (TOJ) task differ from those derived from a synchrony judgment (SJ) task, with (unnatural) auditory-leading PSS values reported mainly for the TOJ task. We report data from two stimulus types that differed in terms of complexity--namely, (1) a flash and a click and (2) a bouncing ball and an impact sound. The same participants judged the temporal order and synchrony of both stimulus types, using three experimental methods: (1) a TOJ task with two response categories ("audio first" or "video first"), (2) an SJ task with two response categories ("synchronous" or "asynchronous"; SJ2), and (3) an SJ task with three response categories ("audio first," "synchronous," or "video first"; SJ3). Both stimulus types produced correlated PSS estimates with the SJ tasks, but the estimates from the TOJ procedure were uncorrelated with those obtained from the SJ tasks. These results suggest that the SJ task should be preferred over the TOJ task when the primary interest is i n perceived audio-visualsynchrony.     

On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model

Research on the perception of temporal order uses either temporal-order judgment (TOJ) tasks or synchrony judgment (SJ) tasks, in both of which two stimuli are presented with some temporal delay and observers must judge the order of presentation. Results generally differ across tasks, raising concerns about whether they measure the same processes. We present a model including sensory and decisional parameters that places these tasks in a common framework that allows studying their implications on observed performance. TOJ tasks imply specific decisional components that explain the discrepancy of results obtained with TOJ and SJ tasks. The model is also tested against published data on audiovisual temporal-order judgments, and the fit is satisfactory, although model parameters are more accurately estimated with SJ tasks. Measures of latent point of subjective simultaneity and latent sensitivity are defined that are invariant across tasks by isolating the sensory parameters governing observed performance, whereas decisional parameters vary across tasks and account for observed differences across them. Our analyses concur with other evidence advising against the use of TOJ tasks in research on perception of temporal order.     

Somatosensory prior entry assessed with temporal order judgments and simultaneity judgments

Attention is central to perception, yet a clear understanding of how attention influences the latency of perception has proven surprisingly elusive. Recent research has indicated that spatially attended stimuli are perceived earlier than unattended stimuli across a range of sensory modalities-an effect termed prior entry. However, the method commonly used to measure this, the temporal order judgment (TOJ) task, has been criticized as susceptible to response bias, despite deliberate attempts to minimize such bias. A preferred alternative is the simultaneity judgment (SJ) task. We tested the prior-entry hypothesis for somatosensory stimuli using both a TOJ task (replicating an earlier experiment) and an SJ task. Prior-entry effects were found for both, though the effect was reduced in the SJ task. Additional experiments (TOJ and SJ) using visual cues established that the earlier perception of cued tactile targets does not result from intramodal sensory interactions between tactile cues and targets.     

Fitting model-based psychometric functions to simultaneity and temporal-order judgment data: MATLAB and R routines

Research on temporal-order perception uses temporal-order judgment (TOJ) tasks or synchrony judgment (SJ) tasks in their binary SJ2 or ternary SJ3 variants. In all cases, two stimuli are presented with some temporal delay, and observers judge the order of presentation. Arbitrary psychometric functions are typically fitted to obtain performance measures such as sensitivity or the point of subjective simultaneity, but the parameters of these functions are uninterpretable. We describe routines in MATLAB and R that fit model-based functions whose parameters are interpretable in terms of the processes underlying temporal-order and simultaneity judgments and responses. These functions arise from an independent-channels model assuming arrival latencies with exponential distributions and a trichotomous decision space. Different routines fit data separately for SJ2, SJ3, and TOJ tasks, jointly for any two tasks, or also jointly for the three tasks (for common cases in which two or even the three tasks were used with the same stimuli and participants). Additional routines provide bootstrap p-values and confidence intervals for estimated parameters. A further routine is included that obtains performance measures from the fitted functions. An R package for Windows and source code of the MATLAB and R routines are available as Supplementary Files.     

Sound improves diminished visual temporal sensitivity in schizophrenia

Visual temporal processing and multisensory integration (MSI) of sound and vision were examined in individuals with schizophrenia using a visual temporal order judgment (TOJ) task. Compared to a non-psychiatric control group, persons with schizophrenia were less sensitive judging the temporal order of two successively presented visual stimuli. However, their sensitivity to visual temporal order improved as in the control group when two accessory sounds were added (temporal ventriloquism). These findings indicate that individuals with schizophrenia have diminished sensitivity to visual temporal order, but no deficits in the integration of low-level auditory and visual information.     

Increased visual bias in children with developmental coordination disorder: Evidence from a visual-tactile temporal order judgment task

BackgroundPrevious studies have suggested that children with developmental coordination disorder (DCD) rely heavily on vision to perform movements, which may contribute to their clumsy movements. However, few studies have objectively and quantitatively investigated the perceptual biases of children with DCD.MethodsA visual-tactile temporal order judgment (TOJ) task was used to measure and compare the perceptual biases of 19 children with DCD and 19 age- and sex-matched typically developing children. The point of subjective equality, which demonstrates when “visual first” and “tactile first” judgment probabilities are equal (50%), obtained by analyzing the results of the visual-tactile TOJ task, was used as an indicator of perceptual biases. Further, variables (age and manual dexterity in all participants; motor function, autism spectrum disorder and attention-deficit hyperactivity disorder traits, and depressive symptoms in children with DCD) associated with perceptual biases were examined with correlation analysis.ResultsChildren with DCD had significantly stronger visual bias than typically developing children. Overall correlation analysis showed that increased visual bias was significantly correlated with poor manual dexterity.ConclusionChildren with DCD had a strong visual bias, which was associated with poor manual dexterity.     

Inhibition and disinhibition of return: Evidence from temporal order judgments

In the present study the temporal order judgment (TOJ) task was used to investigate whether or not inhibition of return (IOR) affects perceptual processing. Previous failures to obtain IOR in the TOJ task have been taken to suggest that IOR does not affect perceptual processing (e.g., Maylor, 1985). The present study showed that IOR is modulated by the temporal disparity between successive targets as well as the relative order in which they appear at cued and uncued locations. Consequently, IOR affects TOJs in some conditions but not in others. The selective occurrence of IOR in the TOJ task provides converging support for the notion that IOR does affect perceptual processing, and also accounts for the previous failures to observe IOR in the TOJ task. Moreover, these and other results suggest that inhibitory processing at the cued location can be dis-inhibited when stimulation occurs at other locations.     

The role of tone duration in dichotic temporal order judgment

Over the past two decades, a number of studies have been based on the dichotic temporal order judgment (TOJ) paradigm, to compare auditory temporal processing in various subpopulations to that of young, normal controls. The reported estimates of dichotic TOJ thresholds, expressed as the interstimulus intervals (ISIs) for 75 % accuracy among the controls, have varied. In the present study, we examined the influence of tone duration, within the 10- to 40-msec range, on dichotic TOJ accuracy and threshold. The results indicated that increases in either ISI or tone duration increased dichotic TOJ accuracy similarly, implying that changes in tone duration may affect dichotic TOJs simply by adding to the delay between onset of the tone at the lead ear and onset of the tone at the lag ear. The dichotic TOJ thresholds from three published studies that had used tones as stimuli and the dichotic thresholds from the present study all fell within 0.5 standard deviations from the theoretical line of a 10-msec reduction in threshold for every 10-msec increase in tone duration. When the dichotic TOJ threshold data from the present study and from the published studies were converted to stimulus onset asynchronies (SOAs) and plotted as a function of tone duration, they fell very close to or on a zero-slope line, indicating that when ISI and duration are summed to yield an SOA, dichotic TOJ thresholds are invariant to tone duration within the range of 10–40 msec.     

Contextual relative temporal duration judgment: an investigation of sequence interruptions

How do listeners judge relative duration? There currently are three primary classes of proposed timing mechanisms: interval based (judgments of discrete events), beat based (judgments of beat synchrony), and oscillator based (judgments of relative phase or synchrony). In the present research, these mechanisms were examined in terms of predictions both about how an induction sequence of preceding intervals affects relative temporal duration comparison in a simple, two-interval task and about how a pause (e.g., an interstimulus interval) within the presented sequence affects relative temporal duration judgment. Results indicated that a relative temporal judgment was best when all intervals preceding the to-be-judged interval were equal in duration to the to-be-judged interval. Results also indicated that whereas short pauses significantly impair a relative temporal duration judgment, long pauses generally do not reduce the effectiveness of the induction sequence. The results are not entirely consistent with current conceptualizations of any of the proposed mechanisms but can be fully accommodated with simple modifications to the oscillator-based mechanism.     

Better late than never: how onsets and offsets influence prior entry and exit

The three experiments presented in the paper examine visual prior entry (determining which of two stimuli appeared first) and prior exit (determining which of two stimuli disappeared first) effects with a temporal order judgment (TOJ) task. In addition to using onset and offset targets, the preceding cues also consisted of either onset or offset stimuli. Typical, and equivalent, prior entry effects were found when either onset or offset cues preceded the onset targets. Unexpectedly large prior exit effects where found with the offset targets, with offset cues producing greater capture effects than onset cues. These findings are consistent with the notion that more attention is allocated to searching the visual field when targets are more difficult to find. In addition, the results indicate that attentional control settings may be more likely to occur with more difficult searches. In addition, these findings demonstrate that TOJ tasks provide extremely precise measures of the allocation of attention and are very sensitive to a range of task manipulations.     

Disinhibition of return: Unnecessary and unlikely

Recently, from data obtained with a temporal order judgment (TOJ) task, Gibson and Egeth (1994) concluded that inhibition of return (IOR; a response time effect that reveals slower responding to targets at previously cued versus uncued locations) reflects impaired perceptual processing. By replotting their data, we demonstrate that the perception of temporal order is influenced only by the facilitatory effect of a cue at short stimulus onset asynchronies (SOAs) and is unaffected by IOR at long SO As. The target paper proposed that, when extra stimuli are presented at task-relevant locations (i.e., in the TOJ task), IOR is prevented by a hypothetical process that is known as disinhibition of return (DOR). We argue that the assumptions that IOR affects perceptual processing and that DOR exists are unnecessary, as a more parsimonious response-based interpretation of IOR is consistent with their data. Further, we summarize recent results and present new data that demonstrate that DOR is unlikely.     

Dissociations between reaction times and temporal order judgments: a diffusion model approach

A diffusion model for simple reaction time (RT) and temporal order judgment (TOJ) tasks was developed to account for a commonly observed dissociation between these 2 tasks: Most stimulus manipulations (e.g., intensity) have larger effects in RT tasks than in TOJ tasks. The model assumes that a detection criterion determines the level of sensory evidence needed to conclude that a stimulus has been presented. Analysis of the performance that would be achieved with different possible criterion settings revealed that performance was optimal with a lower criterion setting for the TOJ task than for the RT task. In addition, the model predicts that effects of stimulus manipulations should increase with the size of the detection criterion. Thus, the model suggests that commonly observed dissociations between RT and TOJ tasks may simply be due to performance optimization in the face of conflicting task demands.     

The "temporal processing deficit" hypothesis in dyslexia: new experimental evidence

The notion that developmental dyslexia may result from a general, nonspecific, defect in perceiving rapidly changing auditory signals is a current subject of debate (so-called "temporal processing deficit" hypothesis). Thirteen phonological dyslexics (age 10-13 years) and 10 controls matched for chronological and reading age were compared on a temporal order judgment (TOJ) task using the succession of two consonants (/p/-/s/) within a cluster. In order to test the relevance of the temporal deficit hypothesis, the task also included two additional conditions where either the two stimuli were artificially slowed or the interstimulus interval was expanded. As expected, the TOJ performance was significantly poorer in dyslexics than in controls. Moreover, in the "slowed speech" condition dyslexics' performance improved to reach the normal controls' level, whereas no significant improvement occurred when increasing the interstimulus interval. Finally dyslexics' performances, especially on the slowed condition, were found correlated with several tests of phonological processing (phoneme deletion, rhyme judgment, and nonword spelling tasks). These results lend support to the general temporal deficit theory of dyslexia.     

Temporal, but not spatial, context modulates a masked prime's effect on temporal order judgment, but not on response latency

Three experiments investigated a dissociation originally described by Neumann, Esselmann, and Klotz. Stimuli were geometric shapes, preceded by similar shapes that were masked by metacontrast. Each experiment consisted of three parts. In the reaction time (RT) part, participants saw an array of geometric shapes, one of which was marked by bars, and had to respond to the marked shape's position by pressing an appropriate button. A prime (a similar, but smaller stimulus) preceded either the marked or an unmarked stimulus. In the temporal order judgment (TOJ) part, the task was to judge the temporal order of the marked and the unmarked stimulus. In the detection part, detectability of the prime was tested. Although its detectability was zero or close to zero, the prime affected both RT and the apparent onset as measured by TOJ. The effect on RT was significantly larger than the effect on TOJ (Exp. 1). Increasing the spatial context (number of non-target stimuli in the display) did not affect this pattern (Exp. 2). By contrast, reducing the temporal context (range of stimulus onset asynchronies) abolished the prime's effect in the TOJ task, although the prime affected RT under identical conditions. It is concluded that partially different mechanisms mediate the prime's effect in the two tasks and that the effect of stimulus context on TOJ found in the Neumann et al. study was due to temporal, not spatial context. Received: 17 December 1997 / Accepted: 15 December 1998     

Differential effects of visual-spatial attention on response latency and temporal-order judgment

Summary Theorists from both classical structuralism and modern attention research have claimed that attention to a sensory stimulus enhances processing speed. However, they have used different operations to measure this effect, viz., temporal-order judgment (TOJ) and reaction-time (RT) measurement. We report two experiments that compared the effect of a spatial cue on RT and TOJ. Experiment 1 demonstrated that a nonmasked, peripheral cue (the brief brightening of a box) affected both RT and TOJ. However, the former effect was significantly larger than the latter. A masked cue had a smaller, but reliable, effect on TOJ. In Experiment 2, the effects of a masked cue on RT and TOJ were compared under identical stimulus conditions. While the cue had a strong effect on RT, it left TOJ unaffected. These results suggest that a spatial cue may have dissociable effects on response processes and the processes that lead to a conscious percept. Implications for the concept of direct parameter specification and for theories of visual attention are discussed.     

Effet de l’action motrice sur les asynchronies perceptives

In agreement with the hypothesis of differential treatment of the intrinsic (color, shape, category…) and extrinsic (position, orientation…) visual characteristics along the ventral and dorsal pathways of the visual system (Milner & Goodale, 1995), the study of temporal order judgment (TOJ) of the variation of two visual attributes highlighted the perceptual asynchrony even though these changes occur synchronously. In this context, we investigated the role of action in the perception of perceptual asynchrony, especially the effect of a reaching movement on the TOJ of position and color changes of a target occurring at different times of movement execution. In the absence of voluntary action, the point of subjective simultaneity (PSS) shows that the color changes must occurr 46.6 ms before the change of position to give rise to a synchronous perception of these two changes. Performing a reaching movement reduced significantly the PSS (12.4 ms), but only if the changes occur near movement end-point. If changes occur during movement execution, the PSS (40.2 ms) was not different from that obtained in the perceptual condition. These results suggest that endogenous signals associated with voluntary motor action contribute to the reduction of perceptual asynchrony in relation to the goal of the action. We discuss the possibility that, in the context of the action, the motor system contributes to the binding of objects sensory attributes as well as to the sense of agency.     

Perception of intersensory synchrony in audiovisual speech: not that special

Perception of intersensory temporal order is particularly difficult for (continuous) audiovisual speech, as perceivers may find it difficult to notice substantial timing differences between speech sounds and lip movements. Here we tested whether this occurs because audiovisual speech is strongly paired (“unity assumption”). Participants made temporal order judgments (TOJ) and simultaneity judgments (SJ) about sine-wave speech (SWS) replicas of pseudowords and the corresponding video of the face. Listeners in speech and non-speech mode were equally sensitive judging audiovisual temporal order. Yet, using the McGurk effect, we could demonstrate that the sound was more likely integrated with lipread speech if heard as speech than non-speech. Judging temporal order in audiovisual speech is thus unaffected by whether the auditory and visual streams are paired. Conceivably, previously found differences between speech and non-speech stimuli are not due to the putative “special” nature of speech, but rather reflect low-level stimulus differences.     

Temporal processing and phonological impairment in dyslexia: effect of phoneme lengthening on order judgment of two consonants

The evidence of supporting phonological deficit as a cause of developmental dyslexia has been accumulating rapidly over the past 2 decades, yet the exact mechanisms underlying this deficit remain controversial. Some authors assume that a temporal processing deficit is the source of the phonological disorder observed in dyslexic children. Others maintain that the phonological deficit in dyslexia is basically linguistic, not acoustic, in nature. Three experiments were conducted and tested the impact of the temporal alteration and the impact of complex syllabic structure on consonant order judgments. Thirteen phonological dyslexics (age 10-13) and 10 controls matched for chronologial age were compared on a Temporal Order Judgment (TOJ) task using the succession of two consonants (/p/ /s/) within a cluster. In order to test the possible relevance of the temporal deficit hypothesis, the task also included two additional conditions where either the two stimuli were artificially slowed or two phonological structures were opposed (CCV and CVCV). As expected, the TOJ performance was significantly poorer in dyslexics than in controls. Moreover, in the "slowed speech" condition dyslexics' performance improved to reach the normal controls' level, whereas manipulating the phonological structure complexity provided no significant improvement. Finally dyslexics' performances, especially on the slowed condition, were found correlated with several tests of phonological processing. These results lend support to the general temporal deficit theory of dyslexia.     

Do working memory-driven attention shifts speed up visual awareness?

Previous research has shown that content representations in working memory (WM) can bias attention in favor of matching stimuli in the scene. Using a visual prior-entry procedure, we here investigate whether such WM-driven attention shifts can speed up the conscious awareness of memory-matching relative to memory-mismatching stimuli. Participants were asked to hold a color cue in WM and to subsequently perform a temporal order judgment (TOJ) task by reporting either of two different-colored circles (presented to the left and right of fixation with a variable temporal interval) as having the first onset. One of the two TOJ circles could match the memory cue in color. We found that awareness of the temporal order of the circle onsets was not affected by the contents of WM, even when participants were explicitly informed that one of the TOJ circles would always match the WM contents. The null effect of WM on TOJs was not due to an inability of the memory-matching item to capture attention, since response times to the target in a follow-up experiment were improved when it appeared at the location of the memory-matching item. The present findings suggest that WM-driven attention shifts cannot accelerate phenomenal awareness of matching stimuli in the visual field.     

触觉时序知觉手臂交叉效应的性别差异

本研究探讨触觉时序知觉的手臂交叉效应是否存在性别差异。通过两个实验在较短和较长的SOA条件下考察男性和女性被试在基于体觉和基于外部空间的触觉时序判断任务中的表现。结果表明,男性与女性被试在基于体觉和基于外部空间的触觉时序判断任务中均存在手臂交叉效应,SOA较短时男性被试的手臂交叉效应显著小于女性被试,但在SOA较长的条件下手臂交叉效应没有明显的性别差异。触觉时序知觉手臂交叉效应的性别差异可能与空间知觉能力和生理解剖学因素有关。