Temporal coordination between actions and sound during sequence production

Delayed auditory feedback (DAF) causes asynchronies between perception and action that disrupt sequence production. Different delay lengths cause differing amounts of disruption that may reflect the phase location of feedback onsets relative to produced inter-response intervals, or the absolute temporal separation between actions and sounds. Two experiments addressed this issue by comparing the effects of traditional DAF, which uses a constant temporal separation, with delays that adjust temporal separation to maintain the phase location of feedback onsets within inter-response intervals. Participants played simple isochronous melodies on a keyboard, or tapped an isochronous beat, at three production rates. Disruption was best predicted by the phase location of feedback onsets, and diminished when feedback onsets formed harmonic phase ratios (phase synchrony). Both delay types led to similar effects. Different movement tasks (melody production versus tapping) led to slightly different patterns of disruption across phase that may relate to differing task demands. In general, these results support the view that perception and action are coordinated in relative rather than absolute time.     

Variability of timing in expressive piano performance increases with interval duration

In simple motor tasks such as finger tapping at different constant rates, within-trial variability of response interonset intervals (IOIs) increases with IOI duration (which varies between trials). In expressive piano performance, the rate of key depressions is not constant, in part due to compositional structure and in part due to expressive timing, so that IOIs of many different durations occur within a single “trial.” Nevertheless, across repeated performances of the same music (Schumann’s “Träumerei” and Debussy’s “La fille aux cheveux de lin”) at the same intended tempo, the standard deviations of individual IOIs tend to increase linearly with their average duration. This is also true when the variation is due to expressive timing alone and when unintended differences in basic tempo between performances are taken into account. In the music studied here, at least, there was no evidence of compensatory timing. The results suggest that the pianists employed a continuously variable tempo governed by a flexible internal timekeeper whose variability follows a generalized Weber’s law (for IOIs longer than about 300 msec).     

Rate limits in sensorimotor synchronization with auditory and visual sequences: the synchronization threshold and the benefits and costs of interval subdivision

Synchronization of finger taps with an isochronous event sequence becomes difficult when the event rate exceeds a certain limit. In Experiment 1, the synchronization threshold was reached at interonset intervals (IOIs) above 100 ms with auditory tone sequences (in a 1:4 tapping task) but at IOIs above 400 ms with visual flash sequences (1:1 tapping). Using IOIs above those limits, the author investigated in Experiment 2 the reduction in the variability of asynchronies that tends to occur when the intervals between target events are subdivided by additional identical events (1:1 vs. 1:n tapping). The subdivision benefit was found to decrease with IOI duration and to turn into a cost at IOIs of 200-250 ms in auditory sequences and at IOIs of 450-500 ms in visual sequences. The auditory results are relevant to the limits of metrical subdivision and beat rate in music. The visual results demonstrate the remarkably weak rhythmicity of (nonmoving) visual stimuli.     

Rate Limits in Sensorimotor Synchronization With Auditory and Visual Sequences: The Synchronization Threshold and the Benefits and Costs of Interval Subdivision

Synchronization of finger taps with an isochronous event sequence becomes difficult when the event rate exceeds a certain limit. In Experiment 1, the synchronization threshold was reached at interonset intervals (IOIs) above 100 ms with auditory tone sequences (in a 1:4 tapping task) but at IOIs above 400 ms with visual flash sequences (1:1 tapping). Using IOIs above those limits, the author investigated in Experiment 2 the reduction in the variability of asynchronies that tends to occur when the intervals between target events are subdivided by additional identical events (1:1 vs 1:n tapping). The subdivision benefit was found to decrease with IOI duration and to turn into a cost at IOIs of 200–250 ms in auditory sequences and at IOIs of 450–500 ms in visual sequences. The auditory results are relevant to the limits of metrical subdivision and beat rate in music. The visual results demonstrate the remarkably weak rhythmicity of (nonmoving) visual stimuli.     

Auditory feedback in music performance: the role of transition-based similarity

Past research has suggested that the disruptive effect of altered auditory feedback depends on how structurally similar the sequence of feedback events is to the planned sequence of actions. Three experiments pursued one basis for similarity in musical keyboard performance: matches between sequential transitions in spatial targets for movements and the melodic contour of auditory feedback. Trained pianists and musically untrained persons produced simple tonal melodies on a keyboard while hearing feedback sequences that either matched the planned melody or were contour-preserving variations of that melody. Sequence production was disrupted among pianists when feedback events were serially shifted by one event, similarly for shifts of planned melodies and tonal variations but less so for shifts of atonal variations. Nonpianists were less likely to be disrupted by serial shifts of variations but showed similar disruption to pianists for shifts of the planned melody. Thus, transitional properties and tonal schemata may jointly determine perception-action similarity during musical sequence production, and the tendency to generalize from a planned sequence to variations of it may develop with the acquisition of skill.     

Delayed auditory feedback and movement

It is well known that timing of rhythm production is disrupted by delayed auditory feedback (DAF), and that disruption varies with delay length. We tested the hypothesis that disruption depends on the state of the movement trajectory at the onset of DAF. Participants tapped isochronous rhythms at a rate specified by a metronome while hearing DAF (for piano tones) of differing lengths. Motion capture was used to analyze movement trajectories. Mean Inter-Response Intervals (IRIs) varied as an approximately sinusoidal function of feedback condition, with DAF causing slowed production for shorter delays and speeded production for faster delays. Motion capture analyses revealed that finger velocity at the time of DAF predicted the effect of DAF on mean IRI whereas finger position predicted the variability of IRIs. A second experiment in which participants were instructed to vary the timing of peak finger height confirmed that the effect of DAF on timing variability is directly influenced by the finger trajectory.     

Self-control and choice in humans: Effects of video game playing as a positive reinforcer

Participants chose between reinforcement schedules differing in delay of reinforcement (interval between a choice response and onset of a video game) and/or amount of reinforcement (duration of access to the game). Experiment 1 showed that immediate reinforcement was preferred to delayed reinforcement with amount of reinforcement held constant, and a large reinforcer was preferred to a small reinforcer when both were obtainable immediately. Imposing a delay before the large reinforcer produced a preference for the immediate, small reinforcer in 40% of participants. This suggested a limited degree of “impulsivity.” In Experiment 2, unequal delays were extended by equal intervals, the amounts being kept equal. Preference for the shorter delay decreased, an effect that presumably makes possible the “preference reversal” phenomenon in studies of self-control. Overall, the results demonstrate that video game playing can produce useful, systematic data when used as a positive reinforcer for choice behavior in humans.     

Auditory–motor learning influences auditory memory for music

In two experiments, we investigated how auditory–motor learning influences performers’ memory for music. Skilled pianists learned novel melodies in four conditions: auditory only (listening), motor only (performing without sound), strongly coupled auditory–motor (normal performance), and weakly coupled auditory–motor (performing along with auditory recordings). Pianists’ recognition of the learned melodies was better following auditory-only or auditory–motor (weakly coupled and strongly coupled) learning than following motor-only learning, and better following strongly coupled auditory–motor learning than following auditory-only learning. Auditory and motor imagery abilities modulated the learning effects: Pianists with high auditory imagery scores had better recognition following motor-only learning, suggesting that auditory imagery compensated for missing auditory feedback at the learning stage. Experiment 2 replicated the findings of Experiment 1 with melodies that contained greater variation in acoustic features. Melodies that were slower and less variable in tempo and intensity were remembered better following weakly coupled auditory–motor learning. These findings suggest that motor learning can aid performers’ auditory recognition of music beyond auditory learning alone, and that motor learning is influenced by individual abilities in mental imagery and by variation in acoustic features.     

The time course of recognition of novel melodies

Seven experiments explored the time course of recognition of brief novel melodies. In a continuous-running-memory task, subjects recognized melodic transpositions following delays up to 2.0 min. The delays were either empty or filled with other melodies. Test items included exact transpositions (T), same-contour lures (SC) with altered pitch intervals, and different-contour lures (DC). DCs differed from Ts in the pattern of ups and downs of pitch. With this design, we assessed subjects’ discrimination of detailed changes in pitch intervals (T/SC discrimination) as well as their discrimination of contour changes (T/DC). We used both artificial and “real” melodies. Artificial melodies differed in conformity to a musical key, being tonal or atonal. After empty delays, T/DC discrimination was superior to T/SC discrimination. Surprisingly, after filled delays, T/SC discrimination was superior to T/DC. When only filled delays were tested, T/SC discrimination did not decline over the longest delays. T/DC performance declined more than did T/SC performance across both empty and filled delays. Tonality was an important factor only for T/SC discrimination after filled delays. T/DC performance was better with rhythmically intact folk melodies than with artificial isochronous melodies. Although T/SC performance improved over filled delays, it did not overtake T/DC performance. These results suggest that (1) contour and pitch-interval information make different contributions to recognition, with contour dominating performance after brief empty delays and pitch intervals dominating after longer filled delays; (2) a coherent tonality facilitates the encoding of pitch-interval patterns of melodies; and (3) the rich melodic—rhythmic contours of real melodies facilitate T/DC discrimination. These results are discussed in terms of automatic and controlled processing of melodic information.     

When hearing turns into playing: Movement induction by auditory stimuli in pianists

In this study, pianists were tested for learned associations between actions (movements on the piano) and their perceivable sensory effects (piano tones). Actions were examined that required the playing of two-tone sequences (intervals) in a four-choice paradigm. In Experiment 1, the intervals to be played were denoted by visual note stimuli. Concurrently with these imperative stimuli, task-irrelevant auditory distractor intervals were presented (“potential” action effects, congruent or incongruent). In Experiment 2, imperative stimuli were coloured squares, in order to exclude possible influences of spatial relationships of notes, responses, and auditory stimuli. In both experiments responses in the incongruent conditions were slower than those in the congruent conditions. Also, heard intervals actually “induced” false responses. The reaction time effects were more pronounced in Experiment 2. In nonmusicians (Experiment 3), no evidence for interference could be observed. Thus, our results show that in expert pianists potential action effects are able to induce corresponding actions, which demonstrates the existence of acquired action-effect associations in pianists.     

Pianists exhibit enhanced memory for vocal melodies but not piano melodies

Nonmusicians remember vocal melodies (i.e., sung to la la) better than instrumental melodies. If greater exposure to the voice contributes to those effects, then long-term experience with instrumental timbres should elicit instrument-specific advantages. Here we evaluate this hypothesis by comparing pianists with other musicians and nonmusicians. We also evaluate the possibility that absolute pitch (AP), which involves exceptional memory for isolated pitches, influences melodic memory. Participants heard 24 melodies played in four timbres (voice, piano, banjo, marimba) and were subsequently required to distinguish the melodies heard previously from 24 novel melodies presented in the same timbres. Musicians performed better than nonmusicians, but both groups showed a comparable memory advantage for vocal melodies. Moreover, pianists performed no better on melodies played on piano than on other instruments, and AP musicians performed no differently than non-AP musicians. The findings confirm the robust nature of the voice advantage and rule out explanations based on familiarity, practice, and motor representations.     

Token reinforcement, choice, and self-control in pigeons.

Pigeons were exposed to self-control procedures that involved illumination of light-emitting diodes (LEDs) as a form of token reinforcement. In a discrete-trials arrangement, subjects chose between one and three LEDs; each LED was exchangeable for 2-s access to food during distinct posttrial exchange periods. In Experiment 1, subjects generally preferred the immediate presentation of a single LED over the delayed presentation of three LEDs, but differences in the delay to the exchange period between the two options prevented a clear assessment of the relative influence of LED delay and exchange-period delay as determinants of choice. In Experiment 2, in which delays to the exchange period from either alternative were equal in most conditions, all subjects preferred the delayed three LEDs more often than in Experiment-1. In Experiment 3, subjects preferred the option that resulted in a greater amount of food more often if the choices also produced LEDs than if they did not. In Experiment 4, preference for the delayed three LEDs was obtained when delays to the exchange period were equal, but reversed in favor of an immediate single LED when the latter choice also resulted in quicker access to exchange periods. The overall pattern of results suggests that (a) delay to the exchange period is a more critical determinant of choice than is delay to token presentation; (b) tokens may function as conditioned reinforcers, although their discriminative properties may be responsible for the self-control that occurs under token reinforcer arrangements; and (c) previously reported differences in the self-control choices of humans and pigeons may have resulted at least in part from the procedural conventions of using token reinforcers with human subjects and food reinforcers with pigeon subjects.     

Pigeons may not remember the stimuli that reinforced their recent behavior

In two experiments the conditioned reinforcing and delayed discriminative stimulus functions of stimuli that signal delays to reinforcement were studied. Pigeons' pecks to a center key produced delayed-matching-to-sample trials according to a variable-interval 60-s (or 30-s in 1 pigeon) schedule (Experiment 1) or a multiple variable-interval 20-s variable-interval 120-s schedule (Experiment 2). The trials consisted of a 2-s illumination of one of two sample key colors followed by delays ranging across phases from 0.1 to 27.0 s followed in turn by the presentation of matching and nonmatching comparison stimuli on the side keys. Pecks to the key color that matched the sample were reinforced with 4-s access to grain. Under some conditions of Experiment 1, pecks to nonmatching comparison stimuli produced a 4-s blackout and the start of the next interval. Under other conditions of Experiment 1 and each condition of Experiment 2, pecks to nonmatching stimuli had no effect and trials ended only when pigeons pecked the other, matching stimulus and received food. The functions relating pretrial response rates to delays differed markedly from those relating matching-to-sample accuracy to delays. Specifically, response rates remained relatively high until the longest delays (15.0 to 27.0 s) were arranged, at which point they fell to low levels. Matching accuracy was high at short delays, but fell to chance at delays between 3.0 and 9.0 s. In Experiment 2, both matching accuracy and response rates remained high over a wider range of delays in the variable-interval 120-s component relative to the variable-interval 20-s component. The difference in matching accuracy between the components was not due to an increased tendency in the variable-interval 20-s component toward proactive interference following short intervals. Thus, under these experimental conditions the conditioned reinforcing and the delayed discriminative functions of the sample stimulus depended on the same variables (delay and variable-interval value), but were nevertheless dissociated.     

The time course of attentional capture under dual-task conditions

Several recent studies have shown that attentional capture is not an automatic process. For example, abrupt peripheral onsets do not affect the processing of targets presented subsequently at that location when participants have to concurrently perform a perceptually demanding task elsewhere. This result leaves open the question of whether peripheral onsets lose their effectiveness in capturing attention or whether, instead, the performance of a perceptually demanding task entails a faster disengagement of attention from the cued location. Here, we measured exogenous spatial attentional-orienting effects either while participants performed a concurrent perceptually demanding central-monitoring task (a rapid serial visual presentation of letters for a to-be-detected digit target; Experiments 1 and 2) or in isolation (the baseline condition in Experiment 2). The results showed that peripheral onsets captured participants' attention at both the 80- and 190-ms stimulus onset asynchronies (SOAs) in the baseline condition. Crucially, however, during concurrent central monitoring, peripheral onsets were effective in capturing attention only at an 80-ms SOA, while the orienting effect disappeared as soon as a changing letter drew participants' attention back to the central stream (at an SOA of 190 ms). These findings demonstrate that task-irrelevant abrupt onsets cannot be entirely overridden by top-down attentional control, although attentional capture effects are dramatically reduced by an ongoing perceptually demanding task.     

The role of symmetry in the good continuation ratings of two-part tonal melodies

Subjects heard a series of two-part melodies, in which each part was a random sequence of eight pitches (Experiment 1) or five pitches(Experiment 2) from the diatonic scale. The task was to rate each melody on how well the second part followed the first. It was predicted that the presence of symmetry between the two parts would increase the perception of good continuation. In Experiment 1, two symmetrical relat/lons—(itinversion) and (itretrograde)—yielded melodies that were more highly rated than the control melodies, which consisted of nominally (itdifferent) parts. A third symmetry, the (itretrograde inversion), did not enhance good continuation ratings. In Experiment 2, inversions and retrograde inversions were compared with control melodies, using shorter sequences and pitches with equal durations. Again, inversions, but not retrograde inversions, were significantly preferred. The results suggest that the aesthetic judgment of good continuation depends at least partially on a cognitive analysis of the relation between the melody parts. The positive symmetry effects are further discussed in relation to other studies of symmetry transformations in the contexts of both musical sequences and visual arrays.     

The role of learning in remembered duration

In two experiments, the effects of learning on both the accuracy and bias of duration judgments were examined. In Experiment 1, subjects learned one of two tasks (i.e., using a computer software package, building a model car), containing a varying number of action steps, over a one-, three-, or five-trial period. Retrospective judgments of a task’s total duration revealed that accuracy was high at intermediate stages of learning but was low at early stages due to an overestimation bias and low at later stages due to an underestimation bias. The number of action steps within a task influenced behavior only at early learning stages where more action steps led to significantly longer duration estimates. Experiment 2 acted as a converging operation in which novice and experienced pianists were asked to estimate, in advance, how long they thought it would take them to play melodies that varied in their degree of familiarity (i.e. recently learned, well learned, extremely well learned). When these estimates were compared with the melodies’ actual playing times, results revealed a similar pattern of accuracy and bias as found in Experiment 1. These findings are discussed in terms of a “structural remembering model” that emphasizes the role of event predictability in time estimation behavior.     

Auditory feedback in music performance: the role of melodic structure and musical skill

Five experiments explored whether fluency in musical sequence production relies on matches between the contents of auditory feedback and the planned outcomes of actions. Participants performed short melodies from memory on a keyboard while musical pitches that sounded in synchrony with each keypress (feedback contents) were altered. Results indicated that altering pitch contents can disrupt production, but only when altered pitches form a sequence that is structurally similar to the planned sequence. These experiments also addressed the role of musical skill: Experiments 1 and 3 included trained pianists; other experiments included participants with little or no musical training. Results were similar across both groups with respect to the disruptive effects of auditory feedback manipulations. These results support the idea that a common hierarchical representation guides sequences of actions and the perception of event sequences and that this coordination is not acquired from learned associations formed by musical skill acquisition.     

Disruption of kinematic coordination in throwing under stress

The present study tested the conscious-control theory of the relationship between stress and performance. Performers under stress conditions consciously attempted to control their movements, disrupting the automaticity of control. Twenty-two male subjects (11 in Experiment 1 and 11 in Experiment 2) performed an underhand ball-throwing task using the non-dominant hand. The inter-trial variability of two kinematic measures was analyzed, namely arm-joint coordination during the throw and hand position at release (release point). Experiment 1 confirmed the validity of regarding these variability measures as indices of automaticity, as they did not vary in spite of resource shortage induced by a dual-task paradigm. In Experiment 2, in which stress led to a detriment in performance, the variability of joint coordination increased, whereas the release point became more fixed. These findings imply that throwing performance is impaired when the coordination is disrupted as a result of inflexible movement executed by conscious control.     

Memory for the absolute pitch of familiar songs

Four experiments were conducted to examine the ability of people without "perfect pitch" to retain the absolute pitch of familiar tunes. In Experiment 1, participants imagined given tunes, and then hummed their first notes four times either between or within sessions. The variability of these productions was very low. Experiment 2 used a recognition paradigm, with results similar to those in Experiment 1 for musicians, but with some additional variability shown for unselected subjects. In Experiment 3, subjects rated the suitability of various pitches to start familiar tunes. Previously given preferred notes were rated high, as were notes three or four semitones distant from the preferred notes, but not notes one or two semitones distant. In Experiment 4, subjects mentally transformed the pitches of familiar tunes to the highest and lowest levels possible. These experiments suggest some retention of the absolute pitch of tunes despite a paucity of verbal or visual cues for the pitch.     

Reduced Memory Representations for Music

We address the problem of musical variation (identification of different musical sequences as variations) and its implications for mental representations of music. According to reductionist theories, listeners judge the structural importance of musical events while forming mental representations. These judgments may result from the production of reduced memory representations that retain only the musical gist. In a study of improvised music performance, pianists produced variations on melodies. Analyses of the musical events retained across variations provided support for the reductionist account of structural importance. A neural network trained to produce reduced memory representations for the same melodies represented structurally important events more efficiently than others. Agreement among the musicians' improvisations, the network model, and music-theoretic predictions suggest that perceived constancy across musical variation is a natural result of a reductionist mechanism for producing memory representations.