Auditory feedback in music performance: the role of melodic structure and musical skill

Five experiments explored whether fluency in musical sequence production relies on matches between the contents of auditory feedback and the planned outcomes of actions. Participants performed short melodies from memory on a keyboard while musical pitches that sounded in synchrony with each keypress (feedback contents) were altered. Results indicated that altering pitch contents can disrupt production, but only when altered pitches form a sequence that is structurally similar to the planned sequence. These experiments also addressed the role of musical skill: Experiments 1 and 3 included trained pianists; other experiments included participants with little or no musical training. Results were similar across both groups with respect to the disruptive effects of auditory feedback manipulations. These results support the idea that a common hierarchical representation guides sequences of actions and the perception of event sequences and that this coordination is not acquired from learned associations formed by musical skill acquisition.     

Activation of learned action sequences by auditory feedback

The present research addressed whether auditory feedback associated with a learned action sequence can activate the action representation of that sequence. Nonpianist participants learned to perform two melodies at a piano keyboard repeatedly during a trial. The participants heard feedback either from the melody they were performing (normal feedback) or from the other learned melody (termed alternate feedback). An additional tone functioned as an instruction cue to either switch melodies or continue the current melody. Following the instruction cues, participants typically paused just before switching, and paused similarly during trials with a continue cue. Participants paused longer after a continue cue when they experienced alternate rather than normal feedback. This effect was specific to instruction cues positioned at weak metrical accents—positions at which participants were less likely to switch overall. Feedback did not influence timing on switch trials. These findings indicate that influences of auditory feedback can activate learned action sequences, leading to longer latencies associated with cue evaluation.     

Mapping musical thought to musical performance

Expressive timing methods are described that map pianists' musical thoughts to sounded performance. In Experiment 1, 6 pianists performed the same musical excerpt on a computer-monitored keyboard. Each performance contained 3 expressive timing patterns: chord asynchronies, rubato patterns, and overlaps (staccato and legato). Each pattern was strongest in experienced pianists' performances and decreased when pianists attempted to play unmusically. In Experiment 2 pianists performed another musical excerpt and notated their musical intentions on an unedited score. The notated interpretations correlated with the presence of the 3 methods: The notated melody preceded other events in chords (chord asynchrony); events notated as phase boundaries showed greatest tempo changes (rubato); and the notated melody showed most consistent amount of overlap between adjacent events (staccato and legato). These results suggest that the mapping of musical thought to musical action is rule-governed, and the same rules produce different interpretations.     

Reduced Memory Representations for Music

We address the problem of musical variation (identification of different musical sequences as variations) and its implications for mental representations of music. According to reductionist theories, listeners judge the structural importance of musical events while forming mental representations. These judgments may result from the production of reduced memory representations that retain only the musical gist. In a study of improvised music performance, pianists produced variations on melodies. Analyses of the musical events retained across variations provided support for the reductionist account of structural importance. A neural network trained to produce reduced memory representations for the same melodies represented structurally important events more efficiently than others. Agreement among the musicians' improvisations, the network model, and music-theoretic predictions suggest that perceived constancy across musical variation is a natural result of a reductionist mechanism for producing memory representations.     

Maximum key-profile correlation (MKC) as a measure of tonal structure in music

Tonal structure is musical organization on the basis of pitch, in which pitches vary in importance and rate of occurrence according to their relationship to a tonal center. Experiment 1 evaluated the maximum key-profile correlation (MKC), a product of Krumhansl and Schmuckler’s key-finding algorithm (Krumhansl, 1990), as a measure of tonal structure. The MKC is the maximum correlation coefficient between the pitch class distribution in a musical sample and key profiles,which indicate the stability of pitches with respect to particular tonal centers. The MKC values of melodies correlated strongly with listeners’ ratings of tonal structure. To measure the influence of the temporal order of pitches on perceived tonal structure, three measures (fifth span, semitone span, and pitch contour) taken from previous studies of melody perception were also correlated with tonal structure ratings. None of the temporal measures correlated as strongly or as consistently with tonal structure ratings as did the MKC, and nor did combining them with the MKC improve prediction of tonal structure ratings. In Experiment 2, the MKC did not correlate with recognition memory of melodies. However, melodies with very low MKC values were recognized less accurately than melodies with very high MKC values. Although it does not incorporate temporal, rhythmic, or harmonic factors that may influence perceived tonal structure, the MKC can be interpreted as a measure of tonal structure, at least for brief melodies.     

Temporal coordination between actions and sound during sequence production

Delayed auditory feedback (DAF) causes asynchronies between perception and action that disrupt sequence production. Different delay lengths cause differing amounts of disruption that may reflect the phase location of feedback onsets relative to produced inter-response intervals, or the absolute temporal separation between actions and sounds. Two experiments addressed this issue by comparing the effects of traditional DAF, which uses a constant temporal separation, with delays that adjust temporal separation to maintain the phase location of feedback onsets within inter-response intervals. Participants played simple isochronous melodies on a keyboard, or tapped an isochronous beat, at three production rates. Disruption was best predicted by the phase location of feedback onsets, and diminished when feedback onsets formed harmonic phase ratios (phase synchrony). Both delay types led to similar effects. Different movement tasks (melody production versus tapping) led to slightly different patterns of disruption across phase that may relate to differing task demands. In general, these results support the view that perception and action are coordinated in relative rather than absolute time.     

Tonal strength and melody recognition after long and short delays

In a continuous-running-memory task, subjects heard novel seven-note melodies that were tested after delays of 11 sec (empty) or 39 sec (filled). Test items were transposed to new pitch levels (to moderately distant keys in the musical sense)and included exact transpositions (targets), same-contour lures with altered pitch intervals, and new-contour lures. Melodies differed in tonal strength (degree of conformity to a musical key) and were tonally strong, tonally weak, or atonal. False alarms to same-contour lures decreased over the longer delay period, but only for tonal stimuli. In agreement with previous studies, discrimination of detailed changes in pitch intervals improved with increased delay, whereas discrimination of more global contour information declined, again only for tonal stimuli. These results suggest that poor short-delay performance in rejecting same-contour lures arises from confusion that is based on the similarity of tonality between standard stimuli and lures. If a test item has the same contour and a similar tonality to a just-presented item, subjects tend to accept it. After a delay filled with melodies in other tonalities, the salience of key information recedes, and subjects base their judgments on more detailed pattern information (namely, exact pitch intervals). The fact that tonality affects judgments of melodic contour indicates that contour is not an entirely separable feature of melodies but rather that a melody with its contour constitutes an integrated perceptual whole.     

Tonal strength and melody recognition after long and short delays.

In a continuous-running-memory task, subjects heard novel seven-note melodies that were tested after delays of 11 sec (empty) or 39 sec (filled). Test items were transposed to new pitch levels (to moderately distant keys in the musical sense) and included exact transpositions (targets), same-contour lures with altered pitch intervals, and new-contour lures. Melodies differed in tonal strength (degree of conformity to a musical key) and were tonally strong, tonally weak, or atonal. False alarms to same-contour lures decreased over the longer delay period, but only for tonal stimuli. In agreement with previous studies, discrimination of detailed changes in pitch intervals improved with increased delay, whereas discrimination of more global contour information declined, again only for tonal stimuli. These results suggest that poor short-delay performance in rejecting same-contour lures arises from confusion that is based on the similarity of tonality between standard stimuli and lures. If a test item has the same contour and a similar tonality to a just-presented item, subjects tend to accept it. After a delay filled with melodies in other tonalities, the salience of key information recedes, and subjects base their judgments on more detailed pattern information (namely, exact pitch intervals). The fact that tonality affects judgments of melodic contour indicates that contour is not an entirely separable feature of melodies but rather that a melody with its contour constitutes an integrated perceptual whole.     

Infants’ perception of musical patterns

This paper examines infants’ ability to perceive various aspects of musical material that are significant in music in general and in Western European music in particular: contour, intervals, exact pitches, diatonic structure, and rhythm. For the most part, infants focus on relational aspects of melodies, synthesizing global representations from local details. They encode the contour of a melody across variations in exact pitches and intervals. They extract information about pitch direction from the smallest musically relevant pitch change in Western music, the semitone. Under certain conditions, infants detect interval changes in the context of transposed sequences, their performance showing enhancement for sequences that conform to Western musical structure. Infants have difficulty retaining exact pitches except for sets of pitches that embody important musical relations. In the temporal domain, they group the elements of auditory sequences on the basis of similarity and they extract the temporal structure of a melody across variations in tempo.     

Effects of delayed auditory feedback on timing of music performance

Three experiments examined effects of delayed auditory feedback (DAF) on music performance as a function of the temporal location of feedback onsets within produced inter-onset intervals (IOIs). In Experiment 1, pianists performed isochronous melodies at two production rates with different amounts of DAF. Timing variability decreased for DAF amounts that caused feedback onsets to occur halfway through IOIs (binary subdivisions) in a 500-ms, but not 600-ms, IOI rate condition. In Experiment 2, pianists performed melodies with DAF delays and chose a preferred rate. Performers chose slower rates for larger delays; preferred rates approximated twice the amount of DAF. Experiment 3 tested the possibility that subjects deliberately conceptualized subdivisions in Experiments 1 and 2. Performers were given (1) no instructions, (2) instructions to mentally subdivide produced events in two, or (3) instructions to mentally subdivide produced events in three, in different blocks. Instructions to subdivide reduced timing variability for larger feedback delays. These experiments indicate that DAF disruption is reduced by subdividing instructions and sometimes by coincidences of feedback onsets with subdivisions of produced intervals. Such facilitation may reflect the use of hierarchical cognitive plans in production.     

Control of Expressive and Metronomic Timing in Pianists

In 12 tasks, each including 10 repetitions, 6 skilled pianists performed or responded to a musical excerpt. In the first 6 tasks, expressive timing was required; in the last 6 tasks, metronomic timing. The pianists first played the music on a digital piano (Tasks 1 and 7), then played it without auditory feedback (Tasks 2 and 8), then tapped on a response key in synchrony with one of their own performances (Tasks 3 and 9), with an imagined performance (Tasks 4 and 10), with a computer-generated performance (Tasks 5 and 11), and with a computer-generated sequence of clicks (Tasks 6 and 12). The results demonstrated that pianists are capable of generating the expressive timing pattern of their performance in the absence of auditory and kinaesthetic (piano keyboard) feedback. They can also synchronize their finger taps quite well with expressively timed music or clicks (while imagining the music), although they tend to underestimate long interonset intervals and to compensate on the following tap. Expressive timing is thus shown to be generated from an internal representation of the music. In metronomic performance, residual expressive timing effects were evident. Those did not depend on auditory feedback, but they were much reduced or absent when kinaesthetic feedback from the piano keyboard was eliminated. Thus, they seemed to arise from the pianist's physical interaction with the instrument. Systematic timing patterns related to expressive timing were also observed in synchronization with a metronomic computer performance and even in synchronization with metronomic clicks. These results shed light on intentional and unintentional, structurally governed processes of timing control in music performance.     

Phrasing influences the recognition of melodies

One critical step in the processing of complex auditory information (i.e., language and music) involves organizing such information into hierarchical units, such as phrases. In this study, musically trained and untrained listeners' recognition memory for short, naturalistic melodies varying in their phrase structure was tested. For musically trained subjects, memory for information preceding a phrase boundary was disrupted and memory for information subsequent to a phrase boundary was enhanced relative to memory in similar temporal locations for excerpts not containing a phrase boundary. Musically untrained listeners, in contrast, showed no such differences as a function of the phrasing of the melody. These findings conform with previous results in both psycholinguistics and musical cognition and suggest that the phrase serves as a functional unit in musical processing, guiding the parsing of musical sequences during perception, along with the structuring of memory for musical passages.     

The experience of agency in sequence production with altered auditory feedback

When speaking or producing music, people rely in part on auditory feedback - the sounds associated with the performed action. Three experiments investigated the degree to which alterations of auditory feedback (AAF) during music performances influence the experience of agency (i.e., the sense that your actions led to auditory events) and the possible link between agency and the disruptive effect of AAF on production. Participants performed short novel melodies from memory on a keyboard. Auditory feedback during performances was manipulated with respect to its pitch contents and/or its synchrony with actions. Participants rated their experience of agency after each trial. In all experiments, AAF reduced judgments of agency across conditions. Performance was most disrupted (measured by error rates and slowing) when AAF led to an ambiguous experience of agency, suggesting that there may be some causal relationship between agency and disruption. However, analyses revealed that these two effects were probably independent. A control experiment verified that performers can make veridical judgments of agency.     

Effects of hearing the past, present, or future during music performance

Three experiments were performed to explore the effects of mismatches between actions (key-presses) and the contents of auditory feedback (pitch events) during music performance. Pianists performed melodies from memory during altered auditory feedback that was synchronized with key-presses but matched the pitch of other sequence events. Feedback direction was manipulated by presenting pitches that matched events intended for the past (delays; Experiments 1 and 3) or the future (prelays; Experiments 2 and 3). Feedback distance was manipulated by varying the absolute separation between the current event and the location of the feedback pitch. All alterations disrupted the accuracy of performance (pitch errors) more so than timing. Serial-ordering errors indicated confusions among proximal and metrically similar events, consistent with the predictions of an incremental planning model (Palmer & Pfordresher, 2003). Patterns of serial-ordering errors suggested that performers compensate for the disruptive effects of altered feedback by changing event activations during planning.     

“Deafness” effects in detecting alterations to auditory feedback during sequence production

Past research has shown that when discrete responses are associated with a perceptual goal, performers may have difficulty detecting stimuli that are commensurate with that goal. Three experiments are reported here that test whether such effects extend to sequence production. In Experiment 1, participants performed 8-note melodies repeatedly, and on each trial a single tone could be altered with respect to its pitch and/or synchrony with actions. Results suggested a selective deficit of detection when feedback pitch was unchanged and the event was slightly delayed. Experiment 2 showed that this “deafness” to feedback is limited to rhythmic motor tasks that require sequencing, in that similar effects did not emerge when participants produced pitch sequences by tapping a single key repeatedly. A third experiment demonstrated similar results to Experiment 1 when the mapping of keys to pitches on the keyboard was reversed. Taken together, results suggest a selective deafness to response-congruent delayed feedback, consistent with the idea that performers suppress previously planned events during production.     

The influence of height and key on the perceptual similarity of transposed melodies

In two experiments, the perceptual similarity between a strong tonal melody and various transpositions was investigated using a paradigm in which listeners compared the perceptual similarity of a melody and its transposition with that of the same melody and another transposition. The paradigm has the advantage that it provides a direct judgment regarding the similarity of transposed melodies. The experimental results indicate that the perceptual similarity of a strong tonal melody and its transposition is mainly determined by two factors: (1) the distance on the height dimension between the original melody and its transposition (pitch distance), and (2) the distance between keys as inferred from the circle of fifths (key distance). The major part of the variance is explained by the factor pitch distance, whereas key distance explains only a small part.     

Representation and execution of vocal motor programs for expert singing of tonal melodies

Three experiments were conducted to study motor programs used by expert singers to produce short tonal melodies. Each experiment involved a response-priming procedure in which singers prepared to sing a primary melody but on 50% of trials had to switch and sing a different (secondary) melody instead. In Experiment 1, secondary melodies in the same key as the primary melody were easier to produce than secondary melodies in a different key. Experiment 2 showed that it was the initial note rather than key per se that affected production of secondary melodies. In Experiment 3, secondary melodies involving exact transpositions were easier to sing than secondary melodies with a different contour than the primary melody. Also, switches between the keys of C and G were easier than those between C and E. Taken together, these results suggest that the initial note of a melody may be the most important element in the motor program, that key is represented in a hierarchical form, and that melodic contour is represented as a series of exact semitone offsets.     

Music perception and octave generalization in rhesus monkeys

Two rhesus monkeys were tested for octave generalization in 8 experiments by transposing 6- and 7-note musical passages by an octave and requiring same or different judgments. The monkeys showed no octave generalization to random-synthetic melodies, atonal melodies, or individual notes. They did show complete octave generalization to childhood songs (e.g., "Happy Birthday") and tonal melodies (from a tonality algorithm). Octave generalization was equally strong for 2-octave transpositions but not for 0.5- or 1.5-octave transpositions of childhood songs. These results combine to show that tonal melodies form musical gestalts for monkeys, as they do for humans, and retain their identity when transposed with whole octaves so that chroma (key) is preserved. This conclusion implicates similar transduction, storage, processing, and relational memory of musical passages in monkeys and humans and has implications for nature-nurture origins of music perception.     

Laterality effects in processing tonal and atonal melodies with affective and nonaffective task instructions

Right-handed university subjects were presented with monaural melodies that either conformed to the rules of the Western tonal system (tonal melodies) or that systematically deviated from it (atonal melodies) while containing similar contours and pitch skips. Subjects were tested under two different task instructions. One group was requested to judge whether each melody sounded correct or not (the nonaffective task); the other group had to judge whether each melody sounded pleasant or not (the affective task). The nonaffective task was found to elicit essentially no ear difference. In contrast, the affective instruction induced opposite and reliable laterality effects, depending on the valence of the response. The pleasant responses were indicative of a left hemisphere predominance and the unpleasant responses of a right hemisphere predominance. The results are consistent with the claim that the left hemisphere is biased toward positive emotions and the right to negative emotions. Moreover, the results suggest that affective appreciation of melodies is dissociable from their nonaffective judgment.     

Repetition priming in music

The authors explore priming effects of pitch repetition in music in 3 experiments. Musically untrained participants heard a short melody and sang the last pitch of the melody as quickly as possible. Each experiment manipulated (a) whether or not the tone to be sung (target) was heard earlier in the melody (primed) and (b) the prime-target distance (measured in events). Experiment 1 used variable-length melodies, whereas Experiments 2 and 3 used fixed-length melodies. Experiment 3 changed the timbre of the target tone. In all experiments, fast-responding participants produced repeated tones faster than nonrepeated tones, and this repetition benefit decreased as prime-target distances increased. All participants produced expected tonic endings faster than less expected nontonic endings. Repetition and tonal priming effects are compared with harmonic priming effects in music and with repetition priming effects in language.