Processing Pitch and Temporal Structures in Music Reading: Independent or Interactive Processing Mechanisms?

Music provides a useful domain in which to study how the different attributes of complex multidimensional stimuli are processed both separately and in combination. Much research has been devoted to addressing how the dimension of pitch and time are co-processed in music listening tasks. Neuropsychological studies have provided evidence for a certain degree of independence between pitch and temporal processing, although there are also many experimental reports favouring interactive models of pitch and temporal processing. Here we extended these investigations by examining the processing of pitch and temporal structures when music is presented in the visual modality (i.e. in the form of music notation). In two experiments, musician subjects were presented with visual musical stimuli containing both pitch and temporal information for a brief amount of time, and they were subsequently required to recall both the pitch and temporal information. In Experiment 1, we documented that concurrent, unattended, pitch and rhythmic auditory interference stimuli disrupted the recall of pitch, but not time. In Experiment 2, we showed that manipulating the tonal structure of the visual presentation stimuli affected the recall of pitch, but not time. On the other hand, manipulating the metrical properties of the visual stimuli affected recall of time, and pitch to a certain extent. Taken together, these results suggest that the processing of pitch is constrained by the processing of time, but the processing of time is not affected by the processing of pitch. These results do not support either strong independence or interactive models of pitch and temporal processing, but they suggest that the processing of time can occur independently from the processing of pitch when performing a written recall task.     

Binding time: Evidence for integration of temporal stimulus features

Several lines of evidence suggest that during processing of events, the features of these events become connected via episodic bindings. Such bindings have been demonstrated for a large number of visual and auditory stimulus features, like color and orientation, or pitch and loudness. Importantly, most visual and auditory events typically also involve temporal features, like onset time or duration. So far, however, whether temporal stimulus features are also bound into event representations has never been tested directly. The aim of the present study was to investigate possible binding between stimulus duration and other features of auditory events. In Experiment 1, participants had to respond with two keys to a low or high pitch sinus tone. Critically, the tones were presented with two different presentation durations. Sequential analysis of RT data indicated binding of stimulus duration into the event representation: at pitch repetitions, performance was better when both pitch and duration repeated, relative to when only pitch repeated and duration switched. This finding was replicated with loudness as relevant stimulus feature in Experiment 2. In sum, the results demonstrate that temporal features are bound into auditory event representations. This finding is an important advancement for binding theory in general, and raises several new questions for future research.     

Multi-Encoding for Pitch Information of Tone Sequences

This study was designed to investigate the possibilities that subjects would visualize an auditory contour as a visual contour (visual imagery) to encode pitch information of tone sequences (Experiment 1), and that subjects would be motivated to attempt to engage in covert rehearsal with multi-code (Experiment 2). The findings from these experiments suggest that: (a) Whereas the highly musically trained subjects were able to encode pitches as accurate notes on a staff, the less well musically trained subjects encoded the pitch sequence as a contour. It is quite evident that there is an intermodal analogy between the perception of pitch relationships and that of relationships in visual space. (b) Pitch rehearsal of auditory information along with note names (dual-code) and staff notation accompanied by pitch rehearsal with note names (triple-code) were the most effective strategies for highly trained subjects with pitches of tonal sequences; melodic contour accompanied by pitch rehearsal (dual-code) was used by highly trained subjects with atonal sequences and by less well trained subjects with both types of sequences.     

A transient processing deficit following selection of an auditory target

Two experiments were conducted to examine the possibility that a transient processing deficit might result following selection of an auditory target. In both experiments, listeners were presented with a sequence of tones and required to make separate detection judgments, one regarding a target and the second regarding a subsequent probe. The target was presented following a variable number of tones, and the probe was presented in one of the seven or eight possible temporal positions following the target. The experiments differed only in that in Experiment 1 the first post-target item could be either a distractor tone or a probe, whereas in Experiment 2 the first post-target item was replaced by a silent period. In both experiments, a deficit in processing of the probe was apparent when it was presented in one of the first few positions following the target.     

Effects of temporal trial-by-trial cuing on early and late stages of auditory processing: Evidence from event-related potentials

Temporal-cuing studies show faster responding to stimuli at an attended versus unattended time point. Whether the mechanisms involved in this temporal orienting of attention are located early or late in the processing stream has not been answered unequivocally. To address this question, we measured event-related potentials in two versions of an auditory temporal cuing task: Stimuli at the uncued time point either required a response (Experiment 1) or did not (Experiment 2). In both tasks, attention was oriented to the cued time point, but attention could be selectively focused on the cued time point only in Experiment 2. In both experiments, temporal orienting was associated with a late positivity in the timerange of the P3. An early enhancement in the timerange of the auditory N1 was observed only in Experiment 2. Thus, temporal attention improves auditory processing at early sensory levels only when it can be focused selectively.     

Harmonic and rhythmic influences on musical expectancy

The effects of harmony and rhythm on expectancy formation were studied in two experiments. In both studies, we generated musical passages consisting of a melodic line accompanied by four harmonic (chord) events. These sequences varied in their harmonic content, the rhythmic periodicity of the three context chords prior to the final chord, and the ending time of the final chord itself. In Experiment 1, listeners provided ratings for how well the final chord in a chord sequence fit their expectations for what was to come next; analyses revealed subtle changes in ratings as a function of both harmonic and rhythmic variation. Experiment 2 extended these results; listeners made a speeded reaction time judgment on whether the final chord of a sequence belonged with its set of context chords. Analysis of the reaction time data suggested that harmonic and rhythmic variation also influenced the speed of musical processing. These results are interpreted with reference to current models of music cognition, and they highlight the need for rhythmical weighting factors within the psychological representation of tonal/pitch information.     

Acuity of auditory images in pitch and time

We examined the pitch and temporal acuity of auditory expectations/images formed under attentional-cuing and imagery task conditions, in order to address whether auditory expectations and auditory images are functionally equivalent. Across three experiments, we observed that pitch acuity was comparable between the two task conditions, whereas temporal acuity deteriorated in the imagery task. A fourth experiment indicated that the observed pitch acuity could not be attributed to implicit influences of the primed context alone. Across the experiments, image acuity in both pitch and time was better in listeners with more musical training. The results support a view that auditory images are multifaceted and that their acuity along any given dimension depends partially on the context in which they are formed.     

The representation of pitch in auditory imagery: Evidence from S-R compatibility and distance effects

In three experiments we explored the nature of representations constructed during the perception and imagination of pitch. We employed a same–different task to eliminate the influence of nonauditory information and to minimise use of cognitive strategies on auditory imagery. A reference tone of frequency 1000, 1500, or 2000 Hz, or an imagined tone of a pitch indicated by a visual cue, was followed by a comparison tone (1000, 1500, or 2000 Hz) to which either a speeded same or different response was required. In separate experiments, same–different judgements were mapped to vertically (Experiments 1 and 2) and horizontally arranged responses (Experiment 3). Judgements of tones closer in pitch yielded longer reaction times and higher error rates than more distant tones, indicating a pitch distance effect for perceptual and imagery tasks alike. In addition, in the imagery task, same–different responses were faster when low-pitched tones demanded a bottom or left key response and high-pitched tones a top or right response than vice versa, suggesting that pitch is coded spatially. Together, these behavioural effects support the assumption that both perceived and imagined pitch are translated into an analogical representation in the spatial domain.     

Evaluation of an imputed pitch velocity model of the auditory tau effect

This article extends an imputed pitch velocity model of the auditory kappa effect proposed by Henry and McAuley (2009a) to the auditory tau effect. Two experiments were conducted using an AXB design in which listeners judged the relative pitch of a middle target tone (X) in ascending and descending three-tone sequences. In Experiment 1, sequences were isochronous, establishing constant fast, medium, and slow velocity conditions. No systematic distortions in perceived target pitch were observed, and thresholds were similar across velocity conditions. Experiment 2 introduced to-be-ignored variations in target timing. Variations in target timing that deviated from constant velocity conditions introduced systematic distortions in perceived target pitch, indicative of a robust auditory tau effect. Consistent with an auditory motion hypothesis, the magnitude of the tau effect was larger at faster velocities. In addition, the tau effect was generally stronger for descending sequences than for ascending sequences. Combined with previous work on the auditory kappa effect, the imputed velocity model and associated auditory motion hypothesis provide a unified quantitative account of both auditory tau and kappa effects. In broader terms, these findings add support to the view that pitch and time relations in auditory patterns are fundamentally interdependent.     

Auditory attention to frequency and time: an analogy to visual local-global stimuli

Two priming experiments demonstrated exogenous attentional persistence to the fundamental auditory dimensions of frequency (Experiment 1) and time (Experiment 2). In a divided-attention task, participants responded to an independent dimension, the identification of three-tone sequence patterns, for both prime and probe stimuli. The stimuli were specifically designed to parallel the local-global hierarchical letter stimuli of [Navon D. (1977). Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9, 353-383] and the task was designed to parallel subsequent work in visual attention using Navon stimuli [Robertson, L. C. (1996). Attentional persistence for features of hierarchical patterns. Journal of Experimental Psychology: General, 125, 227-249; Ward, L. M. (1982). Determinants of attention to local and global features of visual forms. Journal of Experimental Psychology: Human Perception and Performance, 8, 562-581]. The results are discussed in terms of previous work in auditory attention and previous approaches to auditory local-global processing.     

Resetting the pitch-analysis system: 1. Effects of rise times of tones in noise backgrounds or of harmonics in a complex tone

The question of whether sudden increases in the amplitude of pure-tone components would perceptually isolate them from a more complex spectrum was investigated in two experiments. In Experiment 1, a 3.5-sec noise was played as a masker. During the noise, two pure-tone components of different frequencies appeared in succession. Subjects were asked to judge whether the pitch sequence went up or down. The rise time of these components had only a small and inconsistent effect on discrimination. In Experiment 2, the 3.5-sec background signal was a complex tone. The amplitudes of two of its components were incremented in succession. Again, subjects judged whether the pitch pattern went up or down. This time there was a sizable, monotonic effect of the rise time of the increments, with more rapid increments leading to better discrimination. The difference between the two results is interpreted in terms of the auditory system’s response to changing and unchanging signals and the role of its “sudden-change” responses in attracting perceptual processing to certain spectral regions.     

Responsiveness of Western adults to pitch-distributional information in melodic sequences

Responsiveness of musically trained and untrained adults to pitch-distributional information in melodic contexts was assessed. In Experiment 1, melodic contexts were pure-tone sequences, generated from either a diatonic or one of four nondiatonic tonesets, in which pitch-distributional information was manipulated by variation of the relative frequency of occurrence of tones from the toneset. Both the assignment of relative frequency of occurrence to tones and the construction of the (fixed) temporal order of tones within the sequences contravened the conventions of western tonal music. A probe-tone technique was employed. Each presentation of a sequence was followed by a probe tone, one of the 12 chromatic notes within the octave. Listeners rated the goodness of musical fit of the probe tone to the sequence. Probe-tone ratings were significantly related to frequency of occurrence of the probe tone in the sequence for both trained and untrained listeners. In addition, probe-tone ratings decreased as the pitch distance between the probe tone and the final tone of the sequence increased. For musically trained listeners, probe-tone ratings for diatonic sequences tended also to reflect the influence of an internalized tonal schema. Experiment 2 demonstrated that the temporal location of tones in the sequences could not alone account for the effect of frequency of occurrence in Experiment 1. Experiment 3 tested musically untrained listeners under the conditions of Experiment 1, with the exception that the temporal order of tones in each sequence was randomized across trials. The effect of frequency of occurrence found in Experiment 1 was replicated and strengthened.     

Higher order pattern structure influences auditory representational momentum

Representational momentum refers to the phenomenon that observers tend to incorrectly remember an event undergoing real or implied motion as shifted beyond its actual final position. This has been demonstrated in both visual and auditory domains. In 5 pitch discrimination experiments, listeners heard tone sequences that implied either linear, periodic, or null motions in pitch space. Their task was to judge whether the pitch of a probe tone following each sequence was the same or different from the final sequence tone. Results suggested that listeners made errors consistent with extrapolation of coherent pitch patterns (linear, periodic) but not with incoherent (null) ones. Hypotheses associated with internalized physical principles and pattern-based expectations are discussed.     

Absolute pitch in infancy and adulthood: the role of tonal structure

Despite many similarities in infant and adult auditory processing, the literature suggests that two aspects of music perception, pitch processing and knowledge of tonal structure, change over development. The current experiments assess the use of absolute and relative pitch cues in a tone sequence statistical learning task containing tonal structure. The results suggest that infants preferentially process absolute pitch patterns in continuous tone sequences, supporting the hypothesis that absolute pitch is present in infancy, whereas adults tracked both absolute and relative pitch patterns. Infants and adults detected the tonal structure in the input, suggesting that humans are attuned to basic aspects of tonality early in life.     

The effect of task and pitch structure on pitch-time interactions in music

Musical pitch-time relations were explored by investigating the effect of temporal variation on pitch perception. In Experiment 1, trained musicians heard a standard tone followed by a tonal context and then a comparison tone. They then performed one of two tasks. In the cognitive task, they indicated whether the comparison tone was in the key of the context. In the perceptual task, they judged whether the comparison tone was higher or lower than the standard tone. For both tasks, the comparison tone occurred early, on time, or late with respect to temporal expectancies established by the context. Temporal variation did not affect accuracy in either task. Experiment 2 used the perceptual task and varied the pitch structure by employing either a tonal or an atonal context. Temporal variation did not affect accuracy for tonal contexts, but did for atonal contexts. Experiment 3 replicated these results and controlled potential confounds. We argue that tonal contexts bias attention toward pitch and eliminate effects of temporal variation, whereas atonal contexts do not, thus fostering pitch-time interactions. Psychonomic Society, Inc.     

Tonal expectations influence pitch perception

In this study, we investigated the influence of tonal relatedness on pitch perception in melodies. Tonal expectations for target tones were manipulated in melodic contexts while controlling sensory expectations, thus allowing us to assess specifically the influence oftonal expectations on pitch perception. Three experimentsprovided converging evidence that tonal relatedness modulates pitch perception in nonmusician listeners. Experiment 1 showed, with a rating task, the influence of the tonal relatedness of a target tone on listeners' judgments of tuning/mistuning. Experiment 2 showed, with a priming task, that pitch processing of in-tune tones was faster for tonally related targets than for less related targets. Experiment 3 showed, with a comparison task, that discrimination performance for small mistunings was better when the to-be-compared tones were tonally related to the melodic context. Findings are discussed in relation to psychoacoustic research on contextual pitch perception and to studies showing facilitation of early processing steps via knowledge- and attention-related processes.     

Learning of timing patterns and the development of temporal expectations

The present study investigated the learning of a culturally unfamiliar musical rhythm, leading to the development of temporal expectations, and it explored the potential for generalization across tempi and tasks. With that aim, we adapted the serial reaction time task to examine the learning of temporal structures by an indirect method. The temporal pattern employed was based on a complex interval ratio (2:3) and compared to one based on a simple interval ratio (1:2). In the exposure phase, non-musician participants performed a two-choice speeded discrimination task that required responding by key press to each event of the simple or complex auditory pattern. Participants were not informed about the temporal regularities; their task solely concerned the discrimination task. During exposure (Experiments 1–3), response times decreased over time for both temporal patterns, but particularly for the events following the longer interval of the more complex 2:3 pattern. Exposure further influenced performance in subsequent testing phases, notably the precision of tap timing in a production task (Experiment 2) and temporal expectations in a perception task (Experiment 3). Our findings promote the new paradigm introduced here as a method to investigate the learning of temporal structures.     

Effects of visual attentional load on low-level auditory scene analysis

The sharing of processing resources between the senses was investigated by examining the effects of visual task load on auditory event-related brain potentials (ERPs). In Experiment 1, participants completed both a zero-back and a one-back visual task while a tone pattern or a harmonic series was presented. N1 and P2 waves were modulated by visual task difficulty, but neither mismatch negativity (MMN) elicited by deviant stimuli from the tone pattern nor object-related negativity (ORN) elicited by mistuning from the harmonic series was affected. In Experiment 2, participants responded to identity (what) or location (where) in vision, while ignoring sounds alternating in either pitch (what) or location (where). Auditory ERP modulations were consistent with task difficulty, rather than with task specificity. In Experiment 3, we investigated auditory ERP generation under conditions of no visual task. The results are discussed with respect to a distinction between process-general (N1 and P2) and processspecific (MMN and ORN) auditory ERPs.     

The effect of exogenous spatial attention on auditory information processing

This study investigated the effect of exogenous spatial attention on auditory information processing. In Experiments 1, 2 and 3, temporal order judgment tasks were performed to examine the effect. In Experiment 1 and 2, a cue tone was presented to either the left or right ear, followed by sequential presentation of two target tones. The subjects judged the order of presentation of the target tones. The results showed that subjects heard both tones simultaneously when the target tone, which was presented on the same side as the cue tone, was presented after the target tone on the opposite side. This indicates that spatial exogenous attention was aroused by the cue tone, and facilitated subsequent auditory information processing. Experiment 3 examined whether both cue position and frequency influence the resulting information processing. The same effect of spatial attention was observed, but the effect of attention to a certain frequency was only partially observed. In Experiment 4, a tone fusion judgment task was performed to examine whether the effect of spatial attention occurred in the initial stages of hearing. The result suggests that the effect occurred in the later stages of hearing.     

Knowing when to hear aids what to hear

Temporal preparation often has been assumed to influence motor stages of information processing. Recent studies, however, challenge this notion and provide evidence for a facilitation of visual processing. The present study was designed to investigate whether perceptual processing in the auditory domain also benefits from temporal preparation. To this end, we employed a pitch discrimination task. In Experiment 1, discrimination performance was clearly improved when participants were temporally prepared. This finding was confirmed in Experiment 2, which ruled out possible influences of short-term memory. The results support the notion that temporal preparation enhances perceptual processing not only in the visual, but also in the auditory, modality.