Immediate reinforcement in delayed reward learning in pigeons

Pigeons were trained on simultaneous red-green discrimination procedures with delayed reward and sequences of stimuli during the delay. In Experiment 1, three stimuli appeared during the 60-second intervals between the correct responses and reward, and the incorrect responses and nonreward. The stimulus that immediately followed a correct response also preceded nonreward, and the stimulus that followed an incorrect response preceded reward. These stimuli were 10 or .33 second in duration for different groups. Stimuli during the remainder of the delay interval differed following correct and incorrect responses. Group 10 initially persisted in the nonrewarded choice, but shifted to a preponderance of rewarded responses after further training. Group .33 rapidly acquired the correct response. Similar results were obtained in Experiment 2 where delay intervals consisted of opposite sequences of two stimuli of equal duration and total delays were 6, 20, or 60 seconds. Early in training, generalization of differential conditioned-reinforcing properties from the conditions preceding reward and nonreward to postchoice conditions had a greater effect relative to backchaining than it did later. It was concluded that delayed-reward learning is best analyzed in terms of the conditioned-reinforcing value of the patterns of cues that follow immediately after rewarded and nonrewarded responses.     

How transitions from nonrewarded to rewarded trials regulate responding in Pavlovian and instrumental learning following extensive acquisition training

In both discrimination learning and partial reinforcement, transitions may occur from nonrewarded to rewarded trials (NR transition). In discrimination learning, NR transitions may occur in two different stimulus alternatives (NR different transitions). In partial reward, NR transitions may occur in a single stimulus alternative (NR same transitions). Available instrumental learning data indicate that resistance to extinction is increased by both types of NR transitions following limited acquisition training. Following more extensive acquisition training, resistance to extinction appears to be increased by NR same transitions but not by NR different transitions. In Experiment 1, it was shown for the first time that following extensive acquisition training the effects of the two types of transitions are the same in the Pavlovian situation as in the instrumental situation. This finding indicates that on the current trial the rat remembered the conditioned stimulus (CS) and the unconditioned stimulus (US) from the prior trial and those memories along with the CS on the current trial become the signal for the current US. Experiments 2 and 3, which tested hypotheses about instrumental learning, identified why NR different transitions lose their capacity to promote vigorous responding following extensive acquisition training. This is because cues occurring on the rewarded trials of NR different transitions more validly signal reward than other situational cues and thus overshadow them. Finally, some implications of the present findings for understanding the role of NR different transitions in discrimination learning situations were discussed.     

Stimulus control of responding in the early trials of differential conditioning

In Experiment 1 two groups of rats were given 12 differential conditioning trials, seven to the rewarded alley (S+) and five to the nonrewarded alley (S?), prior to being extinguished in both alleys. Group S?S+ received S+ trials, following S? trials in acquisition, while Group S+S? did not receive S+ trials following S? trials in acquisition. In extinction S+ and S? trials were presented according to a quasi-random sequence for both groups. Running on the last 3 trials of acquisition was found to be faster following S+ than following S? trials. Group S?S+ showed greater resistance to extinction and less discriminative responding in extinction than Group S+S?. These results suggest that responding in differential conditioning is controlled not merely by S+ and S? but by the memories of reward (S^R) and of nonreward (S^N) as well. When the joint effects of both classes of cues were considered, e.g., S^R+S+, responding in the early trials of differential conditioning was shown to be highly orderly. Experiment 2 was highly similar to Experiment 1 except that Groups S?S+ and S+S? were equated along dimensions not equated in Experiment 1. The results obtained in Experiment 2 were highly similar to those obtained in Experiment 1.     

Memories and anticipations control responding by rats (<Emphasis Type="Italic">Rattus norvegicus</Emphasis>) in a Pavlovian procedure

In Experiment 1 each rat received two different fixed series of three trials each. The unconditioned stimulus occurred on Trial 1 of one series and on Trial 3 of the other series, all other trials being nonreinforced. Previous Pavlovian investigations have shown that rats can remember the immediately prior reward outcome and anticipate the immediately subsequent reward outcome. Experiment 1 demonstrated that rats could remember and anticipate even more remote reward outcomes. In Experiment 2 two groups received a series of two nonrewarded trials followed by a rewarded trial. It was demonstrated that a change in the conditioned stimulus (CS) from Trial 2 to Trial 3, which occurred in one group, produced weaker responding than in the other group that did not experience such CS change. On the basis of these findings it was suggested that the rats organized the trials of a series into a unit or chunk. This was concluded for two reasons. First, remembering and anticipating remote reward outcomes strongly suggests that responding is being controlled by events extending beyond the current trial. Secondly, the experimental manipulations employed in the Pavlovian situation here are similar to those used in prior human learning and animal instrumental learning investigations concerned with chunking. Thus, it would appear that chunking is a ubiquitous phenomenon appearing in human serial learning (e.g., Bower and Winzenz 1969; Crowder 1976), in animal instrumental learning (e.g., Capaldi 1992; Hulse and Dorsky 1977; Terrace 1987), and now in Pavlovian learning.     

Memory retrieval and discrimination learning

A better reinforcement schedule may be associated with one exteroceptive stimulus (S+) than with another exteroceptive stimulus (S−). While all theories agree that performance in such discrimination tasks is affected by the reinforcement schedules associated with the exteroceptive S+ and S− cues, some theories suggest that performance is also affected by the reinforcement schedule associated with interoceptive reward produced cues. The two experiments reported here were concerned with identifying the conditions under which the interoceptive cues produced by nonreward come to affect performance. According to one hypothesis, such cues will acquire control over approach responding if animals make an approach response in their presence, but not otherwise. According to the memory hypothesis, the memory of nonreinforcement will become a signal for reinforcement, thus invigorating performance, if it is retrieved on a rewarded trial. In two experiments, two groups made strong approach responses to the nonreinforced cues on S− trials, but the memory of nonreward was better retrieved on a subsequent rewarded trial in one group than in the other. Subsequently, both groups were extinguished. The extinction findings support the memory view.     

Disruption in Pavlovian-Instrumental Transfer as a Function of Depression and Anxiety

The Pavlovian-Instrumental Transfer (PIT) paradigm examines probabilistic and reinforcement learning. Disruptions in mechanisms that mediate PIT (i.e., cues not triggering adaptive behaviors) are thought to be contributors to psychopathology, making the study of probabilistic and reinforcement learning clinically relevant. The current study evaluated an appetitive PIT effect and its relationship with symptom dimensions spanning depression and anxiety, with a particular focus on anhedonia. Forty young adults ranging in scores across dimensions of depression and anxiety symptoms completed the PIT paradigm and self-report symptom measures. The PIT paradigm consisted of three phases. The instrumental phase consisted of a contingent association in which participants squeezed a handgrip for monetary reward. The Pavlovian phase established a purely predictive association between three visual stimuli (CS?+?, CS-, baseline) and presence or absence of monetary reward. In the transfer phase, participants’ responses allowed for examination of whether motivational characteristics of Pavlovian predictors influenced the vigor of their handgrip squeezes (instrumental action), which were formerly independent of Pavlovian associations. Analyses revealed a baseline-reward PIT effect, whereby a reward-associated Pavlovian cue enhanced instrumental responding in the transfer phase. However, there were no significant differences between CS?+?and CS- or CS- and baseline cues, suggesting a disrupted interaction of Pavlovian and instrumental learning. Further, the appetitive PIT effect captured in this paradigm was not associated with anhedonia, fears, or general distress. Future work should investigate the influence of mood states using more specific appetitive PIT paradigms to further understanding of the implications of disrupted reflexive and instrumental responding.

     

Reward schedule effects in extinction: Intertrial interval, memory, and memory retrieval

Rats were trained in a runway such that partial reward occurred on Trial 1 of the day and consistent reward on subsequent massed trials (Group PRT1), or consistent reward occurred on Trial 1 of the day and partial reward on subsequent massed trials (Group PRTM). Under spaced (24-hr) extinction, Group PRT1 was more resistant to extinction than Group PRTM and under massed (1-min) extinction, Group PRTM was more resistant to extinction than Group PRT1. These findings suggest that (a) distinctive stimuli are associated with Trial 1 of the day and with subsequent massed trials, (b) these distinctive stimuli function as retrieval cues for memories, memory retrieval being independent of intertrial interval, and (c) behavior in extinction is controlled by a stimulus compound consisting of the memory of nonreward plus stimuli which accompany the memory of nonreward on rewarded acquisition trials.     

Effect of signaling intertrial unconditioned stimuli in autoshaping

Context-unconditioned-stimulus (US) associations have been suggested as the mediator of the response decrement that occurs when extra USs are added to the intertrial intervals (ITIs) of an otherwise standard Pavlovian conditioning situation. The present autoshaping experiments were concerned with the effect of signaling those extra USs, since such signaling might be expected to lessen their ability to condition the context. Experiments 1 and 2 showed that signaling the ITI USs did reduce their detrimental effects on responding to the conditioned stimulus (CS). To determine whether that reduction was due to an impact of signaling on the target-CS-US association or on performance to the target-CS, Experiment 3 examined responding to differentially trained CSs in a common context, as well as responding to identically trained CSs in differentially trained contexts. Whether the CS was tested in a context of relatively high or low associative strength, more responding occurred to the CS trained with signaled, as compared with unsignaled, ITI USs; further, there was more responding to that CS in the more highly valued context. The pattern of results suggests that contextual value does interact with CS-US learning and may also affect performance to the CS.     

Acquisition and forgetting in monkeys' memory of informational object-reward associations

The first experiment showed the monkeys could recall whether an object had been rewarded with peanuts or with sultanas, two equally preferred foods. The second investigated the effect of rewarded trials with an object on monkeys' ability to recall a nonrewarded trial with the same object. The third demonstrated that monkeys could use the memory of reward to predict nonreward and the memory of nonreward to predict reward, in a Win-Shift Lose-Stay paradigm. The fourth found differences between Win-Shift Lose-Stay and Win-Stay Lose-Shift in the rate at which associations between objects and reward events were forgotten. These results are discussed in relation to D. L. Medin's (In A. M. Schrier, Ed., Behavioral primatology, Hillsdale, N.J.: Erlbaum, 1977, Vol. I, pp. 33–69) distinction between informational and hedonic effects of reward in monkeys. It is argued that the association between an object and a reward event is represented in memory by many independent traces, different traces recording the object's association with different attributes of the reward event.     

Patterning,partial reinforcement,and N-length effects at spaced trials as a function of reinstatement of retrieval cues

Four experiments are reported which demonstrate the importance of the reinstatement of retrieval cues in partial-reinforcement experiments using spaced trials. Reinstatement occurs when the goalbox and startbox are of the same brightness (gray). Nonreinstatement occurs when the goalbox and startbox are of different brightnesses (black-and-white striped vs gray). Under reinstatement conditions, both a partial-reinforcement effect (PRE) and N-length effects were observed whether small reward or large reward was used. Under nonreinstatement conditions, a PRE was observed when large reward was used but not when small reward was used; N-length effects were not observed either with large or small reward. Finally, using a 24-hr intertrial interval, single alternation patterning was found only with a group receiving large reward, a long nonreward confinement duration, and reinstatement. These results are not consistent with the notion that massed and spaced trials are governed by separate mechanisms, and support an explanation of both massed and spaced trials based on E. J. Capaldi's sequential theory.     

Pavlovian influences on goal-directed behavior in mice: the role of cue-reinforcer relations

Two experiments refined procedures to study Pavlovian influences on goal-directed behavior in mice and studied the effects of CS-US relations in Pavlovian-instrumental interactions. Independent groups of mice underwent Pavlovian training to associate either a 10-sec or 2-min auditory stimulus (CS) with reward. We next assessed the ability of the response-contingent CS presentations to reinforce novel instrumental responding (conditioned reinforcement; CRf) or the ability of noncontingent CS presentations to increase ongoing instrumental responding (Pavlovian-instrumental transfer; PIT). Whereas 10-sec training conditions produced strong CRf (and no PIT), 2-min training conditions produced robust PIT (but no CRf).     

Memories of reward and nonreward regulate the extinction and reacquisition of both excitatory and inhibitory associations

The memories of the unconditioned stimulus (US) and its absence (No US), symbolized as S^R and S^N, respectively, may be retrieved on US or No US trials giving rise to four types of associations, S^R → US, S^R → No US, S^N → US, and S^N → No US. Here, following acquisition under partial reward (PRF), rats were shifted either to different schedules of PRF (Experiment 1) or extinction (Experiments 1 and 2). Inhibitory S^R → No US associations formed in acquisition survived extinction and shifts to one, but not another type of PRF schedule (Experiments 1 and 2). Excitatory S^R → US associations also survived extinction (Experiment 2). These findings, as well as the acquisition findings of Experiment 2, are consistent with the sequential model but not with the only other two theories said to be able to explain PRF findings, the frustration hypothesis of Amsel and the attention hypothesis of Mackintosh [see Haselgrove, M., Aydin, A., & Pearce, J. M. (2004). A partial reinforcement extinction effect despite equal rates of reinforcement during Pavlovian conditioning. Journal of Experimental Psychology: Animal Behavior Processes, 30, 240–250]. Also, the reacquisition findings obtained here are inconsistent with two views applied to several learning phenomena, the rule-learning view and the position-item view.     

The reaction to frustration of middle- and lower-class children

Preschool age, white children from lower and middle socioeconomic backgrounds were exposed separately to the same operant analog of the Amsel “double alley” procedure. Two measures of responding, latency of press and pressing rate, were taken following frustration and reward of prior responding. The middle-class children responded slower, in both measures, on nonrewarded compared to rewarded trials, a reversal of the more frequently observed frustration effect (FE). The lower-class children, by contrast, showed the usual FE, shorter response latencies following frustrative nonreward. The FE was not observed with the rate measure. The obtained socioeconomic class difference in reaction to frustration was discussed in terms of the possible relationship to class differences in perceived locus of control of reinforcement. An alternative explanation relating the results to the maximizing vs response patterning behavior in three-choice probability learning tasks was also discussed.     

Contingency awareness,partial reinforcement effects,and curiosity motivation in children

Third- and fourth-grade children were given a two-choice discrimination learning task, designed to associate each of three syllables with a different reward schedule: 100% reward, 50% reward, and 0% reward. Subsequent to the conditioning phase, measures were made of the subject's awareness of the reward contingencies associated with the syllables, as well as the pleasantness and curiosity which these syllables had acquired during the conditioning. The results showed that a partially rewarded cue was evaluated as somewhat less “pleasant” than a continuously rewarded cue, but evoked more “curiosity” than either the continuous or nonrewarded cues. These effects emerged only for the contingency aware groups.     

The temporal course of the frustration effect

General activity subsequent to reward (R) and nonreward (N) was monitored at 5-sec intervals with a stabilimeter in the runway goal box. Activity of never-rewarded control Ss was also measured. In Expt 1 it was found that the frustration effect (difference between N- and R-trial activities) disappeared after about 40 sec of goal box confinement. This disappearance of the frustration effect was due to activity increase on R trials rather than activity decrease on N trials as a function of time. Comparison of N-trial activity with control group activity indicated that frustration does not dissipate within 60 sec. Expt 2 investigated activity following reward and nonreward as a function of reward magnitude. Evidence from these experiments suggests that the late R-trial activity increase results from frustration, possibly conditioned to apparatus cues on N trials.     

Short-term memory for temporal intervals: Contrasting explanations of the choose-short effect in pigeons

To better understand short-term memory for temporal intervals, we re-examined the choose-short effect. In Experiment 1, to contrast the predictions of two models of this effect, the subjective shortening and the coding models, pigeons were exposed to a delayed matching-to-sample task with three sample durations (2, 6 and 18 s) and retention intervals ranging from 0 to 20 s. Consistent with the coding model, the results suggested a sudden forgetting of memories for duration. In Experiment 2, to test the confusion hypothesis, the characteristics of the ITI and the retention interval differed. Contrary to the confusion hypothesis, a choose-short effect was obtained. In both experiments, a test with only two of the three comparison keys was performed. The results suggest three effects that may be controlling the birds’ responses: stimulus generalization when no retention interval is present; an increase in random responding at longer retention intervals; and, similarly, an increase in preference for the “short-sample” key at longer retention intervals.     

On the distinction between open and closed economies.

Open and closed economies have been assumed to produce opposite relations between responding and the programmed density of reward (the amount of reward divided by its cost). Experimental procedures that are treated as open economies typically dissociate responding and total reward by providing supplemental income outside the experimental session; procedures construed as closed economies do not. In an open economy responding is assumed to be directly related to reward density, whereas in a closed economy responding is assumed to be inversely related to reward density. In contrast to this predicted correlation between response-reward relations and type of economy, behavior regulation theory predicts both direct and inverse relations in both open and closed economies. Specifically, responding should be a bitonic function of reward density regardless of the type of economy and is dependent only on the ratio of the schedule terms rather than on their absolute size. These predictions were tested by four experiments in which pigeons' key pecking produced food on fixed-ratio and variable-interval schedules over a range of reward magnitudes and under several open- and closed-economy procedures. The results better supported the behavior regulation view by showing a general bitonic function between key pecking and food density in all conditions. In most cases, the absolute size of the schedule requirement and the magnitude of reward had no effect; equal ratios of these terms produced approximately equal responding.     

Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus

Three experiments examined appetitive Pavlovian-instrumental interactions by presenting separately trained conditioned stimuli (CSs) during reinforced instrumental responding in rabbits. Intra-oral reinforcement was used to minimize interference from peripheral responses such as magazine approach. In experiment 1, the rabbits were first trained to perform an instrumental head-raising response for sucrose reward. A conditioned jaw movement response was then established to a 2-sec CS by pairing it with sucrose; a control stimulus was unpaired with sucrose. Instrumental responding maintained by a variable-interval 40-sec schedule was enhanced during 10-sec presentations of the paired, but not the unpaired, CS. Responding on a variable-ratio 15 schedule was unaffected except on trials on which the pre-CS baseline response rate was low; in such cases the paired CS caused a long-lasting acceleration of responding. Noncontingent presentation of the sucrose reinforcer itself briefly suppressed responding but had no long-term effect. In Experiment 2, a CS that had been conditioned at a 10-sec duration produced the same pattern of effects as in the first study, indicating that facilitation resulted from CS presentation rather than from the frustrative effects of non-reinforcement of the CS. In Experiment 3 an inhibitory CS blocked facilitation by the excitatory CS but did not itself affect instrumental responding. These results support the view that Pavlovian processes play a positive role in instrumental performance and suggest that previous findings of suppression by a short-duration CS reflect peripheral interference. The dependence of facilitation on the baseline level of responding is discussed in terms of associative and motivational theories of Pavlovian mediation.     

Associations Among Smoking,Anhedonia, and Reward Learning in Depression

Depression and cigarette smoking co-occur at high rates. However, the etiological mechanisms that contribute to this relationship remain unclear. Anhedonia and associated impairments in reward learning are key features of depression, which also have been linked to the onset and maintenance of cigarette smoking. However, few studies have investigated differences in anhedonia and reward learning among depressed smokers and depressed nonsmokers. The goal of this study was to examine putative differences in anhedonia and reward learning in depressed smokers (n = 36) and depressed nonsmokers (n = 44). To this end, participants completed self-report measures of anhedonia and behavioral activation (BAS reward responsiveness scores) and as well as a probabilistic reward task rooted in signal detection theory, which measures reward learning (Pizzagalli, Jahn, & O'Shea, 2005). When considering self-report measures, depressed smokers reported higher trait anhedonia and reduced BAS reward responsiveness scores compared to depressed nonsmokers. In contrast to self-report measures, nicotine-satiated depressed smokers demonstrated greater acquisition of reward-based learning compared to depressed nonsmokers as indexed by the probabilistic reward task. Findings may point to a potential mechanism underlying the frequent co-occurrence of smoking and depression. These results highlight the importance of continued investigation of the role of anhedonia and reward system functioning in the co-occurrence of depression and nicotine abuse. Results also may support the use of treatments targeting reward learning (e.g., behavioral activation) to enhance smoking cessation among individuals with depression.     

To think or not to think,that is the question: Individual differences in suppression and rebound effects in autobiographical memory

Two studies explored the effects of forget instructions on autobiographical memory at immediate test and following delays of either 12–13 months, or 3–4 months. Using the Autobiographical Think/No-Think procedure (cf., Noreen & MacLeod, 2013), 24 never-depressed participants (Study 1) first generated 12 positive and 12 negative autobiographical memories and associated cues. Participants were then asked to recall the memory associated with some of the cues (i.e., ‘think’ condition), or to avoid saying or thinking about the memory associated with other cues (i.e., ‘no-think’ condition). Participants were then asked to recall the memories associated with all the cues at immediate test and following a delay of 12–13 months. Participants were found to be successful at forgetting both positive and negative autobiographical memories following ‘no-think’ instructions at immediate test but this forgetting effect did not persist following a 12–13 month delay. This pattern of remembering and forgetting was replicated in a second study (using 27 never-depressed participants) following a 3–4 month delay. Participants who had been less successful at forgetting ‘no-think’ memories at immediate test, were more likely to show rebound effects for those memories following a delay compared to memories which received neither ‘think’ nor ‘no-think’ instructions. Individual differences in inhibitory control and the efficacy of potential therapeutic interventions of ‘no-think’ instructions are considered.