Memories and anticipations control responding by rats (<Emphasis Type="Italic">Rattus norvegicus</Emphasis>) in a Pavlovian procedure

In Experiment 1 each rat received two different fixed series of three trials each. The unconditioned stimulus occurred on Trial 1 of one series and on Trial 3 of the other series, all other trials being nonreinforced. Previous Pavlovian investigations have shown that rats can remember the immediately prior reward outcome and anticipate the immediately subsequent reward outcome. Experiment 1 demonstrated that rats could remember and anticipate even more remote reward outcomes. In Experiment 2 two groups received a series of two nonrewarded trials followed by a rewarded trial. It was demonstrated that a change in the conditioned stimulus (CS) from Trial 2 to Trial 3, which occurred in one group, produced weaker responding than in the other group that did not experience such CS change. On the basis of these findings it was suggested that the rats organized the trials of a series into a unit or chunk. This was concluded for two reasons. First, remembering and anticipating remote reward outcomes strongly suggests that responding is being controlled by events extending beyond the current trial. Secondly, the experimental manipulations employed in the Pavlovian situation here are similar to those used in prior human learning and animal instrumental learning investigations concerned with chunking. Thus, it would appear that chunking is a ubiquitous phenomenon appearing in human serial learning (e.g., Bower and Winzenz 1969; Crowder 1976), in animal instrumental learning (e.g., Capaldi 1992; Hulse and Dorsky 1977; Terrace 1987), and now in Pavlovian learning.     

Remembrance of places lasts: Proactive inhibition and patterns of choice in rat spatial memory

A series of experiments was carried out to evaluate the notion that rats given a sequence of massed daily trials on the radial maze reset working memory at the end of each trial by deleting its contents. Although curves presented by D. S. Olton [Scientific American, 1977, 236, 82–98; In S. H. Hulse, H. Fowler, & W. K. Honig (Eds.), Cognitive processes in animal behavior, Hillsdale, N.J.: Erlbaum, 1978] show that rats return to errorless performance at the beginning of each trial after the first, the fact that accuracy falls less rapidly over choices on Trial 1 than on subsequent trials suggests a proactive inhibition (PI) effect. In Experiment 1, Olton's findings were replicated, and a PI effect was observed on Days 1–2 of testing. On Days 3–5, overall accuracy improved significantly and was associated with the development of a strong tendency for rats to enter adjacent alleys, which became particularly marked on the final trials of a day's testing. In order to prevent rats from achieving accurate performance by using an adjacent alleys pattern, a procedure was used in Experiment 2 which involved initial forced random choices followed by a retention test consisting of free choices. Repeated daily trials with this procedure yielded a significant PI effect, which was more marked at a 60-sec delay than at a 0-sec delay. Experiments 3 and 4 showed this PI effect to be robust and resistant to manipulations designed to produce release from PI. Both the PI effect and a strong tendency found in Experiment 1 for animals to avoid on the initial choices of Trial n those alleys most recently entered on Trial n?1 argue that rats do not reset working memory between trials.     

Transfer between downshift in reward magnitude and continuous delay of reward

Two separate experiments were conducted to investigate transfer of persistence between delay and a downshift in reward magnitude. In the first experiment, experimental rats were initially downshifted in reward magnitude and later tested for persistence to continuous delay of large reward. It was found that these rats were more persistent to the effects of delay than control rats which did not receive prior experience with a downshift in reward magnitude. In the second experiment, experimental rats were first trained to receive large reward under delayed conditions and then tested for persistence to a downshift in reward magnitude. Compared to control rats which received no prior experience with delay, the experimental rats showed a significantly smaller negative contrast effect. The results were interpreted as supporting Amsel's theory of persistence as well as Capaldi's recent interpretation of contrast effects.     

Reward schedule effects in extinction: Intertrial interval, memory, and memory retrieval

Rats were trained in a runway such that partial reward occurred on Trial 1 of the day and consistent reward on subsequent massed trials (Group PRT1), or consistent reward occurred on Trial 1 of the day and partial reward on subsequent massed trials (Group PRTM). Under spaced (24-hr) extinction, Group PRT1 was more resistant to extinction than Group PRTM and under massed (1-min) extinction, Group PRTM was more resistant to extinction than Group PRT1. These findings suggest that (a) distinctive stimuli are associated with Trial 1 of the day and with subsequent massed trials, (b) these distinctive stimuli function as retrieval cues for memories, memory retrieval being independent of intertrial interval, and (c) behavior in extinction is controlled by a stimulus compound consisting of the memory of nonreward plus stimuli which accompany the memory of nonreward on rewarded acquisition trials.     

Positive contrast following gradual and abrupt increases in reward magnitude using delay of reinforcement

In Experiment 1, rats were given a 1-pellet reward for 48 preshift trials. During a subsequent 20-trial postshift phase, one group was shifted to a 12-pellet reward on Trial 1, a second was shifted on Trial 11, and a third was given 1 more pellet each trial and then 12 pellets for the last 10 trials. The speeds of all three groups increased to a level above that of a control group given a 12-pellet food reward throughout training (positive contrast). In experiment 2, rats were shifted from 1 to 12 pellets either gradually or abruptly following either abbreviated training (9 trials) or extended training (20 trials). One group of control subjects received 12 pellets throughout training. The results revealed a positive contrast effect for gradually shifted subjects following extended training but not following abbreviated training. The abrupt shift procedure produced positive contrast following abbreviated training but only a marginal effect following extended training. These results indicate that, contingent upon the amount of preshift training, either gradual or abrupt reward increases may produce positive contrast.     

The effects of reward on tutor's behaviors in a cross-age tutoring context

Utilizing a cross-age tutoring context, this study examined the effects of reward on the teaching behaviors of the tutor, the tutor's subsequent motivation to continue to teach during a free choice period, and the social interaction between a tutor and a tutee. Third-grade boys and girls (n = 96) who exhibited a positive reinforcement style were asked to teach six addition problems to a first-grade boy or girl (n = 96). The children were randomly assigned to pairs and to one of the three reward conditions. In the performance-contingent reward condition, the tutors were promised a toy if the first-grader learned all of the arithmetic problems. In the noncontingent reward condition, the tutors were promised a toy for teaching the first-grader. In the no reward condition, the tutors taught the first-grader without promise of a toy. The results indicated that the social interaction was rated lower for the children in the performance-contingent group and that the tutors in this group spent less time teaching during the free choice period. However, neither the tutor's teaching style nor the tutee's post-test performance was adversely affected by the reward.     

Familial sinistrality, sex, and laterality differences in naming and Lexical Decision Latencies of right-handers

Sixty-four subjects, divided into four groups of 16 according to sex and familial sinistrality (FS), were given a lateralized Object-Naming Latency Task (ONLT) and a Lexical Decision Latency Task (LDLT). Both tasks showed RVF superiorities. On the ONLT a sex × FS × visual field interaction was obtained, with FS− females and FS+ males showing comparably smaller RVF superiorities than FS+ females and FS− males. On the LDLT an FS by Stimulus Type (word-nonword) interaction was found, FS− subjects being more RVF superior for nonword trials and FS+ subjects more RVF superior for word trials. If one compared only FS− males and females, as some studies have done, greater RVF superiorities obtained for males than females on both the ONLT and word trials of the LDLT. However, FS+ subjects showed the opposite pattern. The data support neither the simple model of FS nor sex influence on language laterality. Data aslso suggest that a history of maternal versus paternal left-handedness may affect lateralization differently in the sexes.     

Patterning,partial reinforcement,and N-length effects at spaced trials as a function of reinstatement of retrieval cues

Four experiments are reported which demonstrate the importance of the reinstatement of retrieval cues in partial-reinforcement experiments using spaced trials. Reinstatement occurs when the goalbox and startbox are of the same brightness (gray). Nonreinstatement occurs when the goalbox and startbox are of different brightnesses (black-and-white striped vs gray). Under reinstatement conditions, both a partial-reinforcement effect (PRE) and N-length effects were observed whether small reward or large reward was used. Under nonreinstatement conditions, a PRE was observed when large reward was used but not when small reward was used; N-length effects were not observed either with large or small reward. Finally, using a 24-hr intertrial interval, single alternation patterning was found only with a group receiving large reward, a long nonreward confinement duration, and reinstatement. These results are not consistent with the notion that massed and spaced trials are governed by separate mechanisms, and support an explanation of both massed and spaced trials based on E. J. Capaldi's sequential theory.     

Maintenance of Self-Imposed Delay of Gratification by Four Chimpanzees (Pan troglodytes) and an Orangutan (Pongo pygmaeus)

Delay maintenance, which is the continuance over time of the choice to forgo an immediate, less preferred reward for a future, more preferred reward, was examined in 4 chimpanzees (Pan troglodytes) and 1 orangutan (Pongo pygmaeus). In the 1st experiment, the apes were presented with 20 chocolate pieces that were placed, one at a time, into a bowl that was within their reach. The apes could consume the available chocolate pieces at any time during a trial, but no additional pieces would be given. The total length of time taken to place the 20 items into the bowl ranged from 60 s to 180 s. All 5 apes delayed gratification on a majority of trials until all 20 chocolate pieces were presented. Unlike in most experiments with human children using this test situation, attention by the apes to the reward was not detrimental to delay maintenance. In a 2nd experiment with the chimpanzees only, 4 foods of differing incentive value were presented in different trials in the same manner as in Experiment 1. The chimpanzees were highly successful in obtaining all food pieces, and there was no difference in performance as a function of food type.     

The effect of frustrative nonreward on the attractiveness of the omitted reward

Two experiments were performed to test the hypothesis that omission of a previously received reward is followed by greater motivation to obtain that specific reward, as compared with the case where the reward had been present. In Experiment 1, experimental rats regularly received food rewards (R_a) at one goal, followed by a choice between food and milk (R_b). Control animals received the same choice without food preceding it. Reward magnitudes were manipulated so as to produce a preference for R_b, and additional forced trials were given each day to equalise experience of the two rewards. In the test phase both groups received trials with R_a present (R) or absent (N) at the first goal. The experimental Ss shifted their choice towards R_a on N trials; the controls did not behave differently after R and N. Experiment 2 was similar, except that R_a was milk and R_b food. The same effect of N on experimental animals was obtained. It is argued that the results are relevant to the double-runway ‘frustration effect’, and cannot be explained by existing theories of frustration.     

Control of observing by antecedent stimuli

Five pigeons were run on a one-key discrete trials observing procedure. Trial onset was signaled by a white or yellow key light. Pecks in white or yellow intermittently produced S⁺ and S⁻, green and red key lights that signaled whether the trial would end with response-independent grain reinforcement or nonreinforcement. In the Redundant conditions, white and yellow were correlated with trial outcome, making S⁺ and S⁻ redundant. In the Informative condition, white and yellow were uncorrelated with trial outcome, so that S⁺ and S⁻ provided new information. During the Informative condition, all birds responded in the formerly positive, now uncorrelated color at higher rates than they did during the preceding or succeeding Redundant conditions, in which that same color was positively correlated with primary reinforcement. This result confirmed the prediction that an animal will observe at higher rates in the absence of reinforcement-correlated cues than in their presence.     

The differential reinforcement of reward-produced and response-produced stimuli

In two experiments, groups received successive large-reward trials on odd-numbered days and successive small-reward trials on even-numbered days in the same gray alleyway. This produced a discrimination problem in which the memory of large reward (S^L) was reliably discriminative of large reward and the memory of small reward (S^S) was reliably discriminative of small reward. Intertrial interval (ITI) was varied both between and within groups. In the within-groups manipulation, ITI separating S+ trials differed from that separating S- trials. Experimental groups learned the discrimination, running slower to S^S (S- cue) than to S^L (S+ cue), and showed a negative contrast effect, running slower to S- than a small-reward control group. Discrimination was somewhat faster at massed than at spaced trials. The within-groups manipulation of ITI suggested that the effects of ITI were mediated by time-dependent changes in internal cues produced by reward events and by the instrumental response. The control exercised by internal cues was shown to be associative rather than nonassociative, e.g., motivational. Similarities with, and implications for, conventional brightness differential conditioning were discussed.     

Response persistence in the rat following intermittent punishment and partial reward: Effects of trial sequence

Thirty-six rats were given 16 days of partial reward training in a runway. During the final 12 days each of the animals received one foot-shock experience each day. One group received shock on an N trial preceding an R trial (P-R), a second group was shocked on N trials not followed by an R trial (R-P), and the third group received shock after completing all daily trials (Control). Following acquisition the rats were split within each group (one half received 24 trials of unpunished extinction and one half continued to receive partial reward but were punished on every trial). During consistent punishment the P-R animals were more persistent than the R-P or Control rats and during unpunished extinction the P-R and Control animals were equal in persistence but both were superior to the R-P animals. The results were discussed in terms of Capaldi's sequential trial theory.     

Early stimulation and patterned responding in rats

This experiment was designed to determine the extent to which performance differences on learning tasks arising from infantile handling reflect learning or emotionality differences. Handled and nonhandled rats were trained and tested in a controlled operant situation where a response on every other trial yielded reinforcement. A key light was turned on for 10 sec during each trial and was off during the 10-sec intertrial interval. The reinforced (S+) trials alternated in a regular sequence with nonreinforced (S–) trials. The pattern of S+ and S– trials recycled continuously throughout the session of 160 trials. The handled rats emitted more responses during both S+ and S– trials than did the nonhandled rats. However, when they were compared on the percentage of emitted responses that were reinforced, there was no significant differences between the two groups. These results were interpreted to suggest that infantile handling does not directly facilitate associative learning in rats.This research was aided by a grant from the National Research Council of Canada, 67-0247. I thank Linda Easton and Beverly Clark for their help in collecting and analyzing the data.     

Preference for unpredictable food rewards occurs with high proportion of reinforced trials or alcohol if rewards are not delayed

Organisms typically prefer situations where reward and nonreward are predictable rather than unpredictable. Although many theories can account for this result (e.g., information theory and delay-reduction theory), a recently developed mathematical model (DMOD) also predicts that subjects prefer the unpredictable reward situation under conditions that substantially decrease aversiveness of unpredictable nonreward (Daly & Daly, 1982). Because a high proportion of reinforced trials (lenient schedule) and alcohol injections decrease aversive conditioning, these variables were tested with rats in five E-maze experiments. A choice to one side of the maze resulted in a stimulus uncorrelated with reward outcome (unpredictable situation). A choice to the other side resulted in stimuli correlated with reward and nonreward (predictable situation). The stimuli were not visible until after the choice was made. A lenient reinforcement schedule resulted in preference for the unpredictable reward situation if rewards were not delayed. Alcohol resulted in preference for the unpredictable reward situation if a medium five-pellet reward was given. A lenient reinforcement schedule combined with an alcohol injection resulted in faster acquisition of the preference for the unpredictable reward situation than did a lenient schedule combined with a saline control injection. These results pose a major challenge to most theories, yet were predicted by DMOD.     

Unidirectional Mazes for use with Animals

One hundred twenty goldfish were differentially trained with large and small reward in a shuttle swimway. The end compartments and the swimming directions for a given S were consistently identifiable over all trials with either a large or small magnitude of reward. Negative contrast effects were obtained in the start measure for Ss with short intervals between trials. The results suggest similar mechanisms for rat and goldfish performance in differential conditioning studies.     

The temporal course of the frustration effect

General activity subsequent to reward (R) and nonreward (N) was monitored at 5-sec intervals with a stabilimeter in the runway goal box. Activity of never-rewarded control Ss was also measured. In Expt 1 it was found that the frustration effect (difference between N- and R-trial activities) disappeared after about 40 sec of goal box confinement. This disappearance of the frustration effect was due to activity increase on R trials rather than activity decrease on N trials as a function of time. Comparison of N-trial activity with control group activity indicated that frustration does not dissipate within 60 sec. Expt 2 investigated activity following reward and nonreward as a function of reward magnitude. Evidence from these experiments suggests that the late R-trial activity increase results from frustration, possibly conditioned to apparatus cues on N trials.     

Children on the autism spectrum update their behaviour in response to a volatile environment

Typical adults can track reward probabilities across trials to estimate the volatility of the environment and use this information to modify their learning rate (Behrens et al., 2007). In a stable environment, it is advantageous to take account of outcomes over many trials, whereas in a volatile environment, recent experience should be more strongly weighted than distant experience. Recent predictive coding accounts of autism propose that autistic individuals will demonstrate atypical updating of their behaviour in response to the statistics of the reward environment. To rigorously test this hypothesis, we administered a developmentally appropriate version of Behrens et al.'s (2007) task to 34 cognitively able children on the autism spectrum aged between 6 and 14 years, 32 age‐ and ability‐matched typically developing children and 19 typical adults. Participants were required to choose between a green and a blue pirate chest, each associated with a randomly determined reward value between 0 and 100 points, with a combined total of 100 points. On each trial, the reward was given for one stimulus only. In the stable condition, the ratio of the blue or green response being rewarded was fixed at 75:25. In the volatile condition, the ratio alternated between 80:20 and 20:80 every 20 trials. We estimated the learning rate for each participant by fitting a delta rule model and compared this rate across conditions and groups. All groups increased their learning rate in the volatile condition compared to the stable condition. Unexpectedly, there was no effect of group and no interaction between group and condition. Thus, autistic children used information about the statistics of the reward environment to guide their decisions to a similar extent as typically developing children and adults. These results help constrain predictive coding accounts of autism by demonstrating that autism is not characterized by uniform differences in the weighting of prediction error.     

Long delay learning in the T-maze

Rats were trained to go to one side of a T-maze with delays of reward lasting 1, 20, or 60 min in Expt 1 and 1 or 60 min in Expt 2. Mediation by secondary reward was prevented by administering the same delay treatment regardless of whether the response was correct or incorrect: after a response, the rat was removed from the choice alley and placed in its home cage to spend the delay. Feedback for the response was given in the startbox after the delay interval ended. The rats learned and there were no significant differences in performance among groups trained with different delays. These results had been expected on the basis of Revusky's (1971) hypothesis that removal of the rat from the learning situation to spend the delay elsewhere facilitates long delay learning by reducing associative interference. In Expt 3, this notion was tested explicitly by varying the amount of a 2-min delay to be spent in the experimental situation. Different groups of rats were left in the choice alley after the response for 0, 15, or 60 sec; then the rats were removed to spend the remainder of the 2-min delay in the home cage As predicted, the level of performance decreased as the length of time in the choice alley was increased.     

Motivational bases in mixed-motive interactions: The effects of comparison levels

The relationship between reward level and the basic motives which underlie strategic choices (competitive and noncompetitive) in a mixed-motive game was examined from two theoretical perspectives. The “regret” interpretation proposed by McClintock and McNeel was compared to an approach based upon Thibout and Kelley's concept of comparison level (CL). Two independent variables, CL and Reward, were manipulated by varying the payoff rates in two playing series of the MDG. Initially, 50 iterations were played for high (4¢ per point) or low ($\text{1}\text{5¢}$ per point) incentive to provide subjects with outcomes upon which to base their CL's. The Reward manipulation was introduced in 150 subsequent trials of the game with subjects playing for 8¢ or 1¢ per point. The results show that the absolute magnitude of reward did not influence the extent of cooperative or competitive behavior. On the other hand, the magnitude of reward relative to CL was significantly associated with the degree of cooperative behavior—supra-CL outcomes yielding a higher frequency of cooperative behavior than infra-CL outcomes.