Auditory feedback in music performance: the role of transition-based similarity

Past research has suggested that the disruptive effect of altered auditory feedback depends on how structurally similar the sequence of feedback events is to the planned sequence of actions. Three experiments pursued one basis for similarity in musical keyboard performance: matches between sequential transitions in spatial targets for movements and the melodic contour of auditory feedback. Trained pianists and musically untrained persons produced simple tonal melodies on a keyboard while hearing feedback sequences that either matched the planned melody or were contour-preserving variations of that melody. Sequence production was disrupted among pianists when feedback events were serially shifted by one event, similarly for shifts of planned melodies and tonal variations but less so for shifts of atonal variations. Nonpianists were less likely to be disrupted by serial shifts of variations but showed similar disruption to pianists for shifts of the planned melody. Thus, transitional properties and tonal schemata may jointly determine perception-action similarity during musical sequence production, and the tendency to generalize from a planned sequence to variations of it may develop with the acquisition of skill.     

“Deafness” effects in detecting alterations to auditory feedback during sequence production

Past research has shown that when discrete responses are associated with a perceptual goal, performers may have difficulty detecting stimuli that are commensurate with that goal. Three experiments are reported here that test whether such effects extend to sequence production. In Experiment 1, participants performed 8-note melodies repeatedly, and on each trial a single tone could be altered with respect to its pitch and/or synchrony with actions. Results suggested a selective deficit of detection when feedback pitch was unchanged and the event was slightly delayed. Experiment 2 showed that this “deafness” to feedback is limited to rhythmic motor tasks that require sequencing, in that similar effects did not emerge when participants produced pitch sequences by tapping a single key repeatedly. A third experiment demonstrated similar results to Experiment 1 when the mapping of keys to pitches on the keyboard was reversed. Taken together, results suggest a selective deafness to response-congruent delayed feedback, consistent with the idea that performers suppress previously planned events during production.     

Effects of hearing the past, present, or future during music performance

Three experiments were performed to explore the effects of mismatches between actions (key-presses) and the contents of auditory feedback (pitch events) during music performance. Pianists performed melodies from memory during altered auditory feedback that was synchronized with key-presses but matched the pitch of other sequence events. Feedback direction was manipulated by presenting pitches that matched events intended for the past (delays; Experiments 1 and 3) or the future (prelays; Experiments 2 and 3). Feedback distance was manipulated by varying the absolute separation between the current event and the location of the feedback pitch. All alterations disrupted the accuracy of performance (pitch errors) more so than timing. Serial-ordering errors indicated confusions among proximal and metrically similar events, consistent with the predictions of an incremental planning model (Palmer & Pfordresher, 2003). Patterns of serial-ordering errors suggested that performers compensate for the disruptive effects of altered feedback by changing event activations during planning.     

Auditory feedback in music performance: Evidence for a dissociation of sequencing and timing

Four experiments examined temporal relationships between actions and auditory feedback in music performance. Experiment 1 incorporated phase shifts of feedback, which disrupted produced timing but not overall accuracy. Experiment 2 incorporated period shifts of pitch contents for synchronized feedback that primarily disrupted accuracy more than timing. Experiment 3 incorporated combined phase and period shifts, which caused moderate disruption to timing and accuracy and revealed interactive effects of period and phase shifts on production. A 4th experiment included all feedback conditions in the same session to confirm differences across Experiments 1-3. These results are consistent with the view that actions and their perceptual consequences are coordinated in a way that distinguishes timing (phase shifts) from sequencing (period shifts). ((c) 2003 APA, all rights reserved)     

Absolute pitch in infant auditory learning: evidence for developmental reorganization

To what extent do infants represent the absolute pitches of complex auditory stimuli? Two experiments with 8-month-old infants examined the use of absolute and relative pitch cues in a tone-sequence statistical learning task. The results suggest that, given unsegmented stimuli that do not conform to the rules of musical composition, infants are more likely to track patterns of absolute pitches than of relative pitches. A 3rd experiment tested adults with or without musical training on the same statistical learning tasks used in the infant experiments. Unlike the infants, adult listeners relied primarily on relative pitch cues. These results suggest a shift from an initial focus on absolute pitch to the eventual dominance of relative pitch, which, it is argued, is more useful for both music and speech processing.     

Infants’ perception of musical patterns

This paper examines infants’ ability to perceive various aspects of musical material that are significant in music in general and in Western European music in particular: contour, intervals, exact pitches, diatonic structure, and rhythm. For the most part, infants focus on relational aspects of melodies, synthesizing global representations from local details. They encode the contour of a melody across variations in exact pitches and intervals. They extract information about pitch direction from the smallest musically relevant pitch change in Western music, the semitone. Under certain conditions, infants detect interval changes in the context of transposed sequences, their performance showing enhancement for sequences that conform to Western musical structure. Infants have difficulty retaining exact pitches except for sets of pitches that embody important musical relations. In the temporal domain, they group the elements of auditory sequences on the basis of similarity and they extract the temporal structure of a melody across variations in tempo.     

A grammar of action generates predictions in skilled musicians

The present study investigates shared representations of syntactic knowledge in music and action. We examined whether expectancy violations in musical harmonic sequences are also perceived as violations of the movement sequences necessary to produce them. Pianists imitated silent videos showing one hand playing chord sequences on a muted keyboard. Results indicate that, despite the absence of auditory feedback, imitation of a chord is fastest when it is congruent with the preceding harmonic context. This suggests that the harmonic rules implied by observed actions induce expectations that influence action execution. As evidence that these predictions are derived at a high representational level, imitation was more accurate for harmonically incongruent chords than for congruent chords executed with unconventional fingering. The magnitude of the effects of context and goal prioritization increased with musical training. Thus, musical training may lead to a domain-general representation of musical grammar, i.e., to a grammar of action.     

Perception of timing is more context sensitive than sensorimotor synchronization

In three experiments, the effects of contextual temporal variation on the perception of timing and on sensorimotor synchronization were examined. Experiment 1 showed that exposure to a variably timed auditory precursor sequence reduces the detectability of deviations from isochrony in a musical test sequence. By contrast, in Experiment 2 there was only a small and transient effect of identical precursor sequences on the variability of finger taps that were synchronized with a similar test sequence. Moreover, the precursor did not impede phase error correction following deviations from isochrony in the test sequence. Experiment 3 employed a within-subjects design that required simultaneous detection of irregularities in and synchronization with nonmusical auditory sequences. Precursor variability impaired only detection, not synchronization performance. These results suggest that perception of deviations from regularity engages context-sensitive tining processes, probably related to conscious awareness, that are not involved in sensorimotor synchronization.     

Sequence Memory in Music Performance

Abstract— How do people remember and produce complex sequences like music or speech? Music provides an example of excellent sequence memory under fast performance conditions; novices as well as skilled musicians can perform memorized music rapidly, without making mistakes. In addition, musical pitches repeat often within a melodic sequence in different orders, yet people do not confuse the sequential ordering; temporal properties of musical pitches aid sequence memory. I describe a contextual model of sequence memory that is sensitive to the rate at which musical sequences are produced and to individual differences among performers. Age and musical experience differentiate adults' and children's memory for musical sequences during performance. Performers' memory for the sequential structure of one melody transfers or generalizes to other melodies in terms of the sequence of pitch events, their temporal properties, and their movements. Motion-analysis techniques provide further views of the time course of the cognitive processes that make sequence memory for music so accurate.     

Melody and pitch processing in five musical savants with congenital blindness

We examined absolute-pitch (AP) and short-term musical memory abilities of five musical savants with congenital blindness, seven musicians, and seven non-musicians with good vision and normal intelligence in two experiments. In the first, short-term memory for musical phrases was tested and the savants and musicians performed statistically indistinguishably, both significantly outperforming the non-musicians and remembering more material from the C major scale sequences than random trials. In the second experiment, participants learnt associations between four pitches and four objects using a non-verbal paradigm. This experiment approximates to testing AP ability. Low statistical power meant the savants were not statistically better than the musicians, although only the savants scored statistically higher than the non-musicians. The results are evidence for a musical module, separate from general intelligence; they also support the anecdotal reporting of AP in musical savants, which is thought to be necessary for the development of musical-savant skill.     

The dynamics of disruption from altered auditory feedback: Further evidence for a dissociation of sequencing and timing

Three experiments were designed to test whether perception and action are coordinated in a way that distinguishes sequencing from timing (Pfordresher, 2003). Each experiment incorporated a trial design in which altered auditory feedback (AAF) was presented for varying lengths of time and then withdrawn. Experiments 1 and 2 included AAF that resulted in action-effect asynchronies (delayed auditory feedback) during simple tapping (Experiment 1) and melody production (Experiment 2). Asynchronous AAF immediately slowed production; this effect then diminished rapidly after removal of AAF. By contrast, sequential alterations of feedback pitch during melody production (Experiment 3) had an effect that varied over successive presentations of AAF (by increasing error rates) that lasted after its withdrawal. The presence of auditory feedback after withdrawal of asynchronous AAF (Experiments 1 and 2) led to overcompensation of timing, whereas the presence of auditory feedback did not influence performance after withdrawal of AAF in Experiment 3. Based on these results, we suggest that asynchronous AAF perturbs the phase of an internal timekeeper, whereas alterations to feedback pitch over time degrade the internal representation of sequence structure.     

The experience of agency in sequence production with altered auditory feedback

When speaking or producing music, people rely in part on auditory feedback - the sounds associated with the performed action. Three experiments investigated the degree to which alterations of auditory feedback (AAF) during music performances influence the experience of agency (i.e., the sense that your actions led to auditory events) and the possible link between agency and the disruptive effect of AAF on production. Participants performed short novel melodies from memory on a keyboard. Auditory feedback during performances was manipulated with respect to its pitch contents and/or its synchrony with actions. Participants rated their experience of agency after each trial. In all experiments, AAF reduced judgments of agency across conditions. Performance was most disrupted (measured by error rates and slowing) when AAF led to an ambiguous experience of agency, suggesting that there may be some causal relationship between agency and disruption. However, analyses revealed that these two effects were probably independent. A control experiment verified that performers can make veridical judgments of agency.     

Multi-Encoding for Pitch Information of Tone Sequences

This study was designed to investigate the possibilities that subjects would visualize an auditory contour as a visual contour (visual imagery) to encode pitch information of tone sequences (Experiment 1), and that subjects would be motivated to attempt to engage in covert rehearsal with multi-code (Experiment 2). The findings from these experiments suggest that: (a) Whereas the highly musically trained subjects were able to encode pitches as accurate notes on a staff, the less well musically trained subjects encoded the pitch sequence as a contour. It is quite evident that there is an intermodal analogy between the perception of pitch relationships and that of relationships in visual space. (b) Pitch rehearsal of auditory information along with note names (dual-code) and staff notation accompanied by pitch rehearsal with note names (triple-code) were the most effective strategies for highly trained subjects with pitches of tonal sequences; melodic contour accompanied by pitch rehearsal (dual-code) was used by highly trained subjects with atonal sequences and by less well trained subjects with both types of sequences.     

Contingent auditory feedback of arm movement facilitates reaching behavior in infancy

The purpose of this study was to investigate the influence of contingent auditory feedback on the development of infant reaching. Eleven full-term infants were observed biweekly from the age of 10 weeks to 16 weeks, and their arm kinematics were recorded. Auditory feedback that was contingent on arm kinematics was provided in the form of: (a) the mother's voice; and (b) musical tones. Results showed that providing auditory feedback (mother's voice or musical tones): (i) increased the amplitude of exploratory arm movements before the onset of reaching; and (ii) increased the number of reaches at the onset of reaching. These results show that infants are able to use contingent auditory feedback to explore the relevant possibilities for action that are subsequently shaped into goal-directed movements.     

Perceptual interactions between musical pitch and timbre.

These experiments examined perceptual interactions between musical pitch and timbre. Experiment 1, through the use of the Garner classification tasks, found that pitch and timbre of isolated tones interact. Classification times showed interference from uncorrelated variation in the irrelevant attribute and facilitation from correlated variation; the effects were symmetrical. Experiments 2 and 3 examined how musical pitch and timbre function in longer sequences. In recognition memory tasks, a target tone always appeared in a fixed position in the sequences, and listeners were instructed to attend to either its pitch or its timbre. For successive tones, no interactions between timbre and pitch were found. That is, changing the pitches of context tones did not affect timbre recognition, and vice versa. The tendency to perceive pitch in relation to other context pitches was strong and unaffected by whether timbre was constant or varying. In contrast, the relative perception of timbre was weak and was found only when pitch was constant. These results suggest that timbre is perceived more in absolute than in relative terms. Perceptual implications for creating patterns in music with timbre variations are discussed.     

Staying Together: A Bidirectional Delay–Coupled Approach to Joint Action

To understand how individuals adapt to and anticipate each other in joint tasks, we employ a bidirectional delay–coupled dynamical system that allows for mutual adaptation and anticipation. In delay–coupled systems, anticipation is achieved when one system compares its own time‐delayed behavior, which implicitly includes past information about the other system’s behavior, with the other system’s instantaneous behavior. Applied to joint music performance, the model allows each system to adapt its behavior to the dynamics of the other. Model predictions of asynchrony between two simultaneously produced musical voices were compared with duet pianists’ behavior; each partner performed one voice while auditory feedback perturbations occurred at unpredictable times during live performance. As the model predicted, when auditory feedback from one musical voice was removed, the asynchrony changed: The pianist’s voice that was removed anticipated (preceded) the actions of their partner. When the auditory feedback returned and both musicians could hear each other, they rapidly returned to baseline levels of asynchrony. To understand how the pianists anticipated each other, their performances were fitted by the model to examine change in model parameters (coupling strength, time‐delay). When auditory feedback for one or both voices was removed, the fits showed the expected decrease in coupling strength and time‐delay between the systems. When feedback about the voice(s) returned, the coupling strength and time‐delay returned to baseline. These findings support the idea that when people perform actions together, they do so as a coupled bidirectional anticipatory system.     

Temporal coordination between actions and sound during sequence production

Delayed auditory feedback (DAF) causes asynchronies between perception and action that disrupt sequence production. Different delay lengths cause differing amounts of disruption that may reflect the phase location of feedback onsets relative to produced inter-response intervals, or the absolute temporal separation between actions and sounds. Two experiments addressed this issue by comparing the effects of traditional DAF, which uses a constant temporal separation, with delays that adjust temporal separation to maintain the phase location of feedback onsets within inter-response intervals. Participants played simple isochronous melodies on a keyboard, or tapped an isochronous beat, at three production rates. Disruption was best predicted by the phase location of feedback onsets, and diminished when feedback onsets formed harmonic phase ratios (phase synchrony). Both delay types led to similar effects. Different movement tasks (melody production versus tapping) led to slightly different patterns of disruption across phase that may relate to differing task demands. In general, these results support the view that perception and action are coordinated in relative rather than absolute time.     

Improving left spatial neglect through music scale playing

The study assessed whether the auditory reference provided by a music scale could improve spatial exploration of a standard musical instrument keyboard in right‐brain‐damaged patients with left spatial neglect. As performing music scales involves the production of predictable successive pitches, the expectation of the subsequent note may facilitate patients to explore a larger extension of space in the left affected side, during the production of music scales from right to left. Eleven right‐brain‐damaged stroke patients with left spatial neglect, 12 patients without neglect, and 12 age‐matched healthy participants played descending scales on a music keyboard. In a counterbalanced design, the participants' exploratory performance was assessed while producing scales in three feedback conditions: With congruent sound, no‐sound, or random sound feedback provided by the keyboard. The number of keys played and the timing of key press were recorded. Spatial exploration by patients with left neglect was superior with congruent sound feedback, compared to both Silence and Random sound conditions. Both the congruent and incongruent sound conditions were associated with a greater deceleration in all groups. The frame provided by the music scale improves exploration of the left side of space, contralateral to the right hemisphere, damaged in patients with left neglect. Performing a scale with congruent sounds may trigger at some extent preserved auditory and spatial multisensory representations of successive sounds, thus influencing the time course of space scanning, and ultimately resulting in a more extensive spatial exploration. These findings offer new perspectives also for the rehabilitation of the disorder.     

The effect of gender on the N1-P2 auditory complex while listening and speaking with altered auditory feedback

The effect of gender on the N1-P2 auditory complex was examined while listening and speaking with altered auditory feedback. Fifteen normal hearing adult males and 15 females participated. N1-P2 components were evoked while listening to self-produced nonaltered and frequency shifted /a/ tokens and during production of /a/ tokens during nonaltered auditory feedback (NAF), frequency altered feedback (FAF), and delayed auditory feedback (DAF; 50 and 200 ms). During speech production, females exhibited earlier N1 latencies during 50 ms DAF and earlier P2 latencies during 50 ms DAF and FAF. There were no significant differences in N1-P2 amplitudes across all conditions. Comparing listening to active speaking, N1 and P2 latencies were earlier among females, with speaking, and under NAF. N1-P2 amplitudes were significantly reduced during speech production. These findings are consistent with the notions that speech production suppresses auditory cortex responsiveness and males and females process altered auditory feedback differently while speaking.