The sequential view: From rapidly fading stimulus traces to the organization of memory and the abstract concept of number

The development of the sequential approach to instrumental learning from about 1958 to the present is described. The sequential model began as an attempt to explain a particular class of neglected partial reward phenomena, those in which performance in acquisition and extinction is influenced by the particular sequence in which rewarded and nonrewarded trials occur in acquisition, and it was subsequently applied to a variety of other phenomena. Over time, the sequential model grew, sometimes through the replacement of older assumptions by novel ones, as when retrieved memories replaced stimulus traces, and sometimes simply through the addition of novel assumptions, such as that animals are capable of remembering retrospectively one, two, three or more prior nonrewarded outcomes—the N-length assumption. The most recent assumption added to the sequential model is that on a given trial the animal may utilize its memory of prior reward outcomes to anticipate both the current reward outcome and one or more subsequent reward outcomes. One way to view the sequential model is to say that it is a specific theory in various degrees of competition with other specific theories. Several examples of this are provided. Another way to view the sequential model, a more important way in my opinion, is to see it as a representative of a general theoretical approach, intertrial theory, which differs in fundamental respects from another much more generally utilized theoretical approach, intra-trial theory. I suggest that there is a substantial body of data that can be explained by inter-trial mechanisms but not by intratrial mechanisms. The future may well reveal that the inter-trial mechanisms have greater explanatory potential than the currently more popular intratrial mechanisms.     

Memories and anticipations control responding by rats (<Emphasis Type="Italic">Rattus norvegicus</Emphasis>) in a Pavlovian procedure

In Experiment 1 each rat received two different fixed series of three trials each. The unconditioned stimulus occurred on Trial 1 of one series and on Trial 3 of the other series, all other trials being nonreinforced. Previous Pavlovian investigations have shown that rats can remember the immediately prior reward outcome and anticipate the immediately subsequent reward outcome. Experiment 1 demonstrated that rats could remember and anticipate even more remote reward outcomes. In Experiment 2 two groups received a series of two nonrewarded trials followed by a rewarded trial. It was demonstrated that a change in the conditioned stimulus (CS) from Trial 2 to Trial 3, which occurred in one group, produced weaker responding than in the other group that did not experience such CS change. On the basis of these findings it was suggested that the rats organized the trials of a series into a unit or chunk. This was concluded for two reasons. First, remembering and anticipating remote reward outcomes strongly suggests that responding is being controlled by events extending beyond the current trial. Secondly, the experimental manipulations employed in the Pavlovian situation here are similar to those used in prior human learning and animal instrumental learning investigations concerned with chunking. Thus, it would appear that chunking is a ubiquitous phenomenon appearing in human serial learning (e.g., Bower and Winzenz 1969; Crowder 1976), in animal instrumental learning (e.g., Capaldi 1992; Hulse and Dorsky 1977; Terrace 1987), and now in Pavlovian learning.     

How transitions from nonrewarded to rewarded trials regulate responding in Pavlovian and instrumental learning following extensive acquisition training

In both discrimination learning and partial reinforcement, transitions may occur from nonrewarded to rewarded trials (NR transition). In discrimination learning, NR transitions may occur in two different stimulus alternatives (NR different transitions). In partial reward, NR transitions may occur in a single stimulus alternative (NR same transitions). Available instrumental learning data indicate that resistance to extinction is increased by both types of NR transitions following limited acquisition training. Following more extensive acquisition training, resistance to extinction appears to be increased by NR same transitions but not by NR different transitions. In Experiment 1, it was shown for the first time that following extensive acquisition training the effects of the two types of transitions are the same in the Pavlovian situation as in the instrumental situation. This finding indicates that on the current trial the rat remembered the conditioned stimulus (CS) and the unconditioned stimulus (US) from the prior trial and those memories along with the CS on the current trial become the signal for the current US. Experiments 2 and 3, which tested hypotheses about instrumental learning, identified why NR different transitions lose their capacity to promote vigorous responding following extensive acquisition training. This is because cues occurring on the rewarded trials of NR different transitions more validly signal reward than other situational cues and thus overshadow them. Finally, some implications of the present findings for understanding the role of NR different transitions in discrimination learning situations were discussed.     

Grouping, chunking, memory, and learning

An example of grouping is dividing a long series of digits into two or more smaller units by, for example, pausing for a short time between items within groups but for a longer time between groups. Grouping, virtually neglected in animal learning, was examined in each of five rat investigations reported here in which a series of two or more runway trials was either grouped together with or apart from a terminal large reward trial. It was found that running speed on early small reward trials in the series was greater when prior trials were grouped together with rather than apart from the terminal large reward trial and this when the intertrial interval was short, 20 sec, or long, 20 to 40 min. Three possible explanations of the present findings were examined: a reward schedule view, a rule-learning view, and an anticipation view based on memory. The most feasible of these explanations, it was suggested, was the anticipation view. According to this view, when prior trials are grouped together with a terminal large reward, there is a tendency for the rat to anticipate the terminal large reward well before its scheduled occurrence, elevating running speed on earlier small reward trials. Such anticipation occurs, it was suggested, because when trials are grouped together the memory of each reward event in the series is retrieved on each subsequent trial, including the remote terminal large reward trial, where it becomes a signal for large reward. Thus the present results implicate not merely adjacent associations—associations between adjacent events—but also the highly controversial remote associations—associations between two events separated not only in time but by one or more intervening events as well.     

Reward schedule effects in extinction: Intertrial interval, memory, and memory retrieval

Rats were trained in a runway such that partial reward occurred on Trial 1 of the day and consistent reward on subsequent massed trials (Group PRT1), or consistent reward occurred on Trial 1 of the day and partial reward on subsequent massed trials (Group PRTM). Under spaced (24-hr) extinction, Group PRT1 was more resistant to extinction than Group PRTM and under massed (1-min) extinction, Group PRTM was more resistant to extinction than Group PRT1. These findings suggest that (a) distinctive stimuli are associated with Trial 1 of the day and with subsequent massed trials, (b) these distinctive stimuli function as retrieval cues for memories, memory retrieval being independent of intertrial interval, and (c) behavior in extinction is controlled by a stimulus compound consisting of the memory of nonreward plus stimuli which accompany the memory of nonreward on rewarded acquisition trials.     

Duration of extinction trials as a determinant of instrumental extinction in terrestrial toads (<Emphasis Type="Italic">Rhinella arenarum</Emphasis>)

Instrumental learning guides behavior toward resources. When such resources are no longer available, approach to previously reinforced locations is reduced, a process called extinction. The present experiments are concerned with factors affecting the extinction of acquired behaviors in toads. In previous experiments, total reward magnitude in acquisition and duration of extinction trials were confounded. The present experiments were designed to test the effects of these factors in factorial designs. Experiment 1 varied reward magnitude (900, 300, or 100 s of water access per trial) and amount of acquisition training (5 or 15 daily trials). With total amount of water access equated in acquisition, extinction with large rewards was faster (longer latencies in 900/5 than 300/15), but with total amount of training equated, extinction with small rewards was faster (longer latencies in 100/15 than 300/15). Experiment 2 varied reward magnitude (1200 or 120 s of water access per trial) while holding constant the number of acquisition trials (5 daily trials) and the duration of extinction trials (300 s). Extinction performance was lower with small, rather than large reward magnitude (longer latencies in 120/300 than in 1200/300). Thus, instrumental extinction depends upon the amount of time toads are exposed to the empty goal compartment during extinction trials.     

Response persistence in the rat following intermittent punishment and partial reward: Effects of trial sequence

Thirty-six rats were given 16 days of partial reward training in a runway. During the final 12 days each of the animals received one foot-shock experience each day. One group received shock on an N trial preceding an R trial (P-R), a second group was shocked on N trials not followed by an R trial (R-P), and the third group received shock after completing all daily trials (Control). Following acquisition the rats were split within each group (one half received 24 trials of unpunished extinction and one half continued to receive partial reward but were punished on every trial). During consistent punishment the P-R animals were more persistent than the R-P or Control rats and during unpunished extinction the P-R and Control animals were equal in persistence but both were superior to the R-P animals. The results were discussed in terms of Capaldi's sequential trial theory.     

An analysis of sequential variables in Pavlovian conditioning employing extended and limited acquisition training

Sequential theory’s memory model of learning has been successfully applied in response contingent instrumental conditioning experiments (Capaldi, 1966, Capaldi, 1967, Capaldi, 1994 and Capaldi and Miller, 2003). However, it has not been systematically tested in nonresponse contingent Pavlovian conditioning experiments. The present experiments attempted to determine if several sequential variables affect responding in Pavlovian situations as they do in instrumental ones. Of primary concern here were the effects on extinction of number of NR transitions (the number of times a nonreinforced trial is followed by a reinforced trial), N-length (the number of successive nonreinforced trials that precede a reinforced trial), and percentage of reinforcement (50 versus 100%) following either extended acquisition training (Experiment 1, 720 trials) or limited acquisition training (Experiment 3, 24 trials). In agreement with a sequential analysis, N-length increased resistance to extinction more than number of NR transitions following extensive training with the opposite occurring following limited training. In Experiment 1, greater resistance to extinction was associated with 50% than with 100% reinforcement, a partial reinforcement extinction effect (PREE). Experiment 2 examined an anomalous finding obtained in Experiment 1. A major theoretical difference between instrumental and Pavlovian conditioning has been held to be the greater ease of producing a PREE in instrumental than in Pavlovian conditioning (Kimble, 1961 and Mackintosh, 1974). However, the findings obtained here suggest that the probability of obtaining a PREE and other Pavlovian extinction effects, as in instrumental conditioning, increases along with the effectiveness of the sequential variables employed.     

Autoshaping: further study of "negative automaintenance"

The key pecking of pigeons was autoshaped to three key colors paired with food in discrete trials. Then, the effects of three different color-correlated contingencies were compared: reward (presentation of food contingent on pecking), omission (presentation of food prevented by pecking), and extinction (no food). Two measures of performance were used: initial response (the number of trials with each color on which at least one peck was made) and multiple response (the total number of pecks per trial). In general, the reward color produced more pecking than the omission color, the omission color more than the extinction color, and the extinction color more than on blank trials with an unlighted key, although (relative to reward) omission produced a higher level of initial than of multiple responding. These results point clearly to the importance of stimulus-reinforcer continguity in the control of pecking.     

Acquisition and transfer effects of cued and noncued double alternation schedules with rats

In an attempt to establish patterning, rats were administered eight daily runway trials in a double alternation schedule with distinctive goal events in either a consistent or a varied sequence. Different magnitudes of reward differentiated the reinforced trials, whereas different lengths of confinement in the empty goal box distinguished the nonreinforced trials. During acquisition, rats given the consistent cue sequence “patterned”, whereas the group presented varied cues did not. Subsequently, the Consistent and Varied Cue Groups were subdivided and were administered for 2 days only one of the four acquisition training goal events. During the final 5 days of transfer testing, all subgroups were switched to receiving four daily trials with a second and then a third goal event. In agreement with expectations derived from stimulus aftereffects theory, performance was immediately in accord with the reinforcement contingencies previously established during acquisition training and not the goal event actually presented during testing.     

Sodium amylobarbitone and responses to nonreward

Three experiments are reported testing two alternative hypotheses concerning the behavioural effects of sodium amylobarbitone (SA): (1) that it blocks the after-effect of nonreward; (2) that it blocks conditioned frustration, elicited by stimuli associated with nonreward. In support of (2) Experiment I showed that SA given in acquisition abolished the partial reinforcement extinction effect (PREE) when rats were run at one trial a day in an alley for food reward on a continuous (CRF) or partial (PRF) reinforcement schedule. Experiment II showed that, in the goal section, the effect of the drug on the PREE was due to its presence during acquisition and was not due to state dependency; but the effect of the drug in the start section was consistent with state dependency of the PREE. In Experiment III, in opposition to (1) and again in support of (2), SA given to rats trained to show patterned running for water reward on a single alternation schedule blocked patterning by increasing running speeds on nonreward trials, not by decreasing running speeds on rewarded trials.     

Higher order sequential effects in psychophysical judgments

We found that the depth of sequential effects depends on the judgment task. An experiment with squares indicated that stimulus-response pairs up to two trials back were included in the judgment process when subjects were required to make category judgments of size, whereas only the immediately preceding event was incorporated when subjects were making magnitude estimations. In the case of category judgment, interactions between the current stimulus and prior stimuli as well as configural effects indicated that events one and two trials back meet an equivalent function in the judgment process and that these events may jointly operate in one trial. These findings can be explained by a class of models that assume that the position of preceding stimuli relative to the current stimulus is decisive in the judgment process. The multiple-standards model is a representative of this class according to which there are two types of standards: (1) the endpoints of the range as long-term standards and (2) traces of preceding stimuli as short-term standards.     

Higher order sequential effects in psychophysical judgments

We found that the depth of sequential effects depends on the judgment task. An experiment with squares indicated that stimulus-response pairs up to two trials back were included in the judgment process when subjects were required to make category judgments of size, whereas only the immediately preceding event was incorporated when subjects were making magnitude estimations. In the case of category judgment, interactions between the current stimulus and prior stimuli as well as configural effects indicated that events one and two trials back meet an equivalent function in the judgment process and that these events may jointly operate in one trial. These findings can be explained by a class of models that assume that the position of preceding stimuli relative to the current stimulus is decisive in the judgment process. The multiple-standards model is a representative of this class according to which there are two types of standards: (1) the endpoints of the range as long-term standards and (2) traces of preceding stimuli as short-term standards.     

Memories of reward and nonreward regulate the extinction and reacquisition of both excitatory and inhibitory associations

The memories of the unconditioned stimulus (US) and its absence (No US), symbolized as S^R and S^N, respectively, may be retrieved on US or No US trials giving rise to four types of associations, S^R → US, S^R → No US, S^N → US, and S^N → No US. Here, following acquisition under partial reward (PRF), rats were shifted either to different schedules of PRF (Experiment 1) or extinction (Experiments 1 and 2). Inhibitory S^R → No US associations formed in acquisition survived extinction and shifts to one, but not another type of PRF schedule (Experiments 1 and 2). Excitatory S^R → US associations also survived extinction (Experiment 2). These findings, as well as the acquisition findings of Experiment 2, are consistent with the sequential model but not with the only other two theories said to be able to explain PRF findings, the frustration hypothesis of Amsel and the attention hypothesis of Mackintosh [see Haselgrove, M., Aydin, A., & Pearce, J. M. (2004). A partial reinforcement extinction effect despite equal rates of reinforcement during Pavlovian conditioning. Journal of Experimental Psychology: Animal Behavior Processes, 30, 240–250]. Also, the reacquisition findings obtained here are inconsistent with two views applied to several learning phenomena, the rule-learning view and the position-item view.     

Fear conditioning and extinction in anxiety- and depression-prone persons

Anxiety and depression frequently co-occur and may share similar deficits in the processing of emotional stimuli. High anxiety is associated with a failure in the acquisition and extinction of fear conditioning. Despite the supposed common deficits, no research has been conducted on fear acquisition and extinction in depression. The main aim of the present study was to investigate and compare fear acquisition and extinction in anxiety- and depression-prone participants. Non-clinical anxious, depressive, anxious-depressive and control participants performed a fear discrimination task. During acquisition, the CS+ predicted an aversive event (unconditioned stimulus, US) and the CS? safety (no US). During extinction, the CS+ was no longer followed by the US, rendering it (temporarily) into a safety signal. On each CS participants rated their US expectancy; skin conductance responses (SCRs) were measured throughout. The expectancy scores indicated that high anxiety resulted in less safety learning during acquisition and extinction; no effect of depression was observed. SCRs showed that high-anxiety persons displayed less discrimination learning (CS+ minus CS?) during acquisition than low-anxiety persons. During extinction, high-depression persons demonstrated more discriminative SCR than low-depression persons. The observed discrepancies in response patterns of high-anxiety and -depression persons seem to indicate distinctive information processing of emotional stimuli.     

A COMPARATIVE STUDY OF INSTRUMENTAL LEARNING BY PRESCHOOL CHILDREN IN PAPUA NEW GUINEA AND AUSTRALIA1

Instrumental learning of preschool children in Papua New Guinea (PNG) and Australia (AUST) was compared using two tasks (imitative and nonimitative) and two rewards (social and nonsocial). There were no differences between the two groups in the rate of acquisition measure of trials to criterion. PNG children made more late responses during acquisition and, for nil responses, there were group x task and group x reward x task effects. In the extinction phase, there were two main effects for trials to criterion: PNG children were more resistant to extinction than AUST children, and there was greater persistence in responding after social reward regardless of nationality. Reward x group, reward x task, and reward x group x task interactions also were observed in the extinction trials to criterion. In addition, there were three main task effects during extinction for other responses: on the imitative task, more wrong responses were made, and on the nonimitative task, more extra responses and more paired responses were made. A subsidiary analysis compared the two culturally different but educationally similar groups comprising the PNG sample: no major differences were isolated in acquisition or extinction.     

Verbal instruction abolishes fear conditioned to racial out-group faces

Previous research has shown resistance to extinction of fear conditioned to racial out-group faces, suggesting that these stimuli may be subject to prepared fear learning. The current study replicated and extended previous research by using a different racial out-group, and testing the prediction that prepared fear learning is unaffected by verbal instructions. Four groups of Caucasian participants were trained with male in-group (Caucasian) or out-group (Chinese) faces as conditional stimuli; one paired with an electro-tactile shock (CS+) and one presented alone (CS−). Before extinction, half the participants were instructed that no more shocks would be presented. Fear conditioning, indexed by larger electrodermal responses to, and blink startle modulation during the CS+, occurred during acquisition in all groups. Resistance to extinction of fear learning was found only in the racial out-group, no instruction condition. Fear conditioned to a racial out-group face was reduced following verbal instructions, contrary to predictions for the nature of prepared fear learning.     

Effects of fornix transection upon associative memory in monkeys: Role of the hippocampus in learned action

Thirty-three monkeys took part in seven experiments designed to elucidate further the effect of fornix transection on learning and memory. In the first experiment the monkeys had to remember whether stimulus objects had previously been paired with reward or no reward, and they had to use this memory to guide choice between stimulus objects at retention tests according to an arbitrary rule which they had learned: to choose objects previously paired with no reward in preference to objects previously paired with reward. Fornix transection produced a severe and permanent impairment in this task. In the second experiment the monkeys also had to remember object-reward associations but the performance rule was more natural: to choose objects previously paired with reward. Here fornix transection had no effect. The third experiment required the monkeys to remember, given a stimulus object, which of two events of equal valence had previously been the outcome of displacing that object. The two events were either a peanut and a sultana or a black penny and a white penny of equal secondary reinforcing value. Performance was unimpaired by fornix transection. The fourth experiment also demonstrated, in a different paradigm, unimpaired recall of sensory events. The fifth experiment demonstrated an impairment following fornix transection in acquisition of simultaneous spatial-visual conditional discriminations; the sixth demonstrated normal learning by fornix-transected monkeys of a successive spatial-visual conditional discrimination and the seventh demonstrated unimpaired acquisition of a simultaneous auditory-visual conditional discrimination. These results, when considered in detail and together, are incompatible with existing hypotheses of hippocampal function. A new hypothesis is discussed.