Associative learning versus fear habituation as predictors of long-term extinction retention

Violation of unconditioned stimulus (US) expectancy during extinction training may enhance associative learning and result in improved long-term extinction retention compared to within-session habituation. This experiment examines variation in US expectancy (i.e., expectancy violation) as a predictor of long-term extinction retention. It also examines within-session habituation of fear-potentiated startle (electromyography, EMG) and fear of conditioned stimuli (CS) throughout extinction training as predictors of extinction retention. Participants (n?=?63) underwent fear conditioning, extinction and retention and provided continuous ratings of US expectancy and EMG, as well as CS fear ratings before and after each phase. Variation in US expectancy throughout extinction and habituation of EMG and fear was entered into a regression as predictors of retention and reinstatement of levels of expectancy and fear. Greater variation in US expectancy throughout extinction training was significantly predictive of enhanced extinction performance measured at retention test, although not after reinstatement test. Slope of EMG and CS fear during extinction did not predict retention of extinction. Within-session habituation of EMG and self-reported fear is not sufficient for long-term retention of extinction learning, and models emphasizing expectation violation may result in enhanced outcomes.     

Greater reinforcement rate during training increases spontaneous recovery

Spontaneous recovery occurs when a previously reinforced and recently extinguished response reemerges over the course of time, often at the beginning of a new session of extinction. Spontaneous recovery could underlie instances of treatment relapse that threaten otherwise effective behavioral interventions for problem behavior. In two experiments, we arranged multiple schedules with pigeons and a human child to assess the effects of different training reinforcer rates on spontaneous recovery. In both experiments, responding was both more resistant to extinction and more likely to relapse following training with greater reinforcement rates upon returning to extinction after time off from extinction testing. A quantitative model based on behavioral momentum theory accounted well for the data, which suggests reexposure to the extinction context following time off during extinction resulted in (1) the failure of extinction learning to generalize, and (2) greater generalization of original learning during training. The present model attempts to quantify theories attributing spontaneous recovery to changes in temporal context.     

Behavioral momentum and resistance to extinction across repeated extinction tests

The present experiments assessed whether resistance to extinction of pigeons' key pecking decreased across repeated extinction tests. An additional impetus for this research was to determine how the quantitative framework provided by behavioral momentum theory might be used to describe any such changes across tests. Pigeons pecked keys in two‐component multiple schedules (one component associated with a higher reinforcer rate and the other with a lower rate) in which baseline and extinction conditions alternated. In Experiment 1, baseline and extinction conditions alternated every session, and, in Experiment 2, these conditions lasted for 10 and 7 sessions, respectively. Resistance to extinction decreased across successive extinction conditions in both experiments. Fits of the behavioral‐momentum based model of extinction to the data returned uncertain results in Experiment 1 but implicated both generalization decrement and response–reinforcer contingency termination as the possible mechanisms responsible for behavior change in Experiment 2. Thus, these data suggest that experimental manipulations that affect discrimination of changes in reinforcement contingencies may influence resistance to extinction by modulating the disruptive impacts of removing reinforcers from the experimental context and of suspending response–reinforcer contingencies.     

Extinction revisited: similarities between extinction and reductions in US intensity in classical conditioning of the rabbit's nictitating membrane response

The mechanisms of extinction were examined by reducing the intensity of the unconditioned stimulus (US) after acquisition training to determine whether such reductions lie on a continuum with CS-alone extinction. The experiments revealed that reductions in US intensity yielded extinction-like effects. Specifically, there were proportional reductions in the daily mean level of responding across sessions. There were also persistent within-session declines and between-session increases of responding analogous to spontaneous recovery. Surprisingly, even when US intensity was held constant, within-session declines and between-session increases were apparent. The results are discussed with respect to possible contributions from unlearning, new learning, generalization decrement, and nonassociative loss, especially CS-specific attentional changes and CR-specific reactive inhibition.     

Stimulus control and delayed reinforcement

Pigeons were tested for generalization along the line-orientation dimension, after being trained on various two-component multiple schedules. The first component contained either a variable-interval 1-min schedule of immediate reinforcement or an extinction schedule and was associated with a plain white key (S1). The second component contained a variable-interval 1-min schedule of delayed reinforcement and was associated with a black line on a white background (S2). The major results showed that (a) decremental gradients were obtained around the stimulus associated with the delayed reinforcement component when S1 was associated with extinction, but that incremental gradients were obtained when S1 was associated with immediate reinforcement, (b) the subjects' pretraining did not affect the generalization gradients if sufficient training on the terminal multiple schedule was provided, and (c) changing the S1 schedule from immediate reinforcement to extinction produced behavioral contrast if reinforcement was delayed for 10 sec during S2, but not if it was delayed for 20 sec.     

Backward inhibitory learning in honeybees: a behavioral analysis of reinforcement processing

One class of theoretical accounts of associative learning suggests that reinforcers are processed according to learning rules that minimize the predictive error between the expected strength of future reinforcement and its actual strength. The omission of reinforcement in a situation where it is expected leads to inhibitory learning of stimuli indicative for such a violation of the prediction. There are, however, results indicating that inhibitory learning can also be induced by other mechanisms. Here, we present data from olfactory reward conditioning in honeybees that show that (1) one- and multiple-trial backward conditioning results in conditioned inhibition (CI); (2) the inhibition is maximal for a 15-sec interval between US and CS; (3) there is a nonmonotonic dependency on the degree of CI from the US-CS interval during backward pairing; and (4) the prior association of context stimuli with reinforcement is not necessary for the development of CI. These results cannot be explained by models that only minimize a prediction error. Rather, they are consistent with models of associative learning that, in addition, assume that learning depends on the temporal overlap of a CS with two processes, a fast excitatory and a slow inhibitory one, both evoked by a reinforcer. The fmdings from this behavioral analysis of reinforcement processing are compared with the known properties of an individual, identified neuron involved in reinforcement processing in the bee brain, to further understand the mechanisms underlying predictive reward learning.     

The L-type calcium channel blocker nifedipine impairs extinction, but not reduced contingency effects, in mice

We recently reported that fear extinction, a form of inhibitory learning, is selectively blocked by systemic administration of L-type voltage-gated calcium channel (LVGCC) antagonists, including nifedipine, in mice. We here replicate this finding and examine three reduced contingency effects after vehicle or nifedipine (40 mg/kg) administration. In the first experiment, contingency reduction was achieved by adding USs to the training protocol (degraded contingency), a phenomenon thought to be independent of behavioral inhibition. In the second experiment, contingency reduction was achieved by varying the percentage of CS-US pairing, a phenomenon thought to be weakly dependent on behavioral inhibition. In the third and fourth experiments, contingency reduction was achieved by adding CSs to the training protocol (partial reinforcement), a phenomenon thought to be completely dependent on behavioral inhibition. We found that none of these reduced contingency effects was impaired by nifedipine. In a final experiment, we found that extinction conducted 1 or 3 h post-acquisition, but not immediately, was LVGCC-dependent. Taken together, the results suggest that reduced contingency effects and extinction depend on different molecular mechanisms and that LVGCC dependence of behavioral inhibition develops with time after associative CS-US learning.     

Generalization decrement in human contingency learning

An association between a cue and an outcome will generalize to a similar novel cue to some extent, but not completely. Learning theorists refer to the discrepancy between responding elicited by the original cue and the novel cue as a generalization decrement. Two experiments used a contingency learning task with human participants to compare the size of a generalization decrement between configurations of cues that were altered by adding or subtracting compositional elements. The results suggest that adding elements to a configuration can produce a generalization decrement, but removing elements produces a more robust generalization decrement. Furthermore, the generalization decrement caused by adding elements was not likely to be caused by competing orienting responses. The results are used to contrast Pearce's (1987, 1994) and Wagner's (2003) models of stimulus generalization.     

Generalization decrement in human contingency learning

An association between a cue and an outcome will generalize to a similar novel cue to some extent, but not completely. Learning theorists refer to the discrepancy between responding elicited by the original cue and the novel cue as a generalization decrement. Two experiments used a contingency learning task with human participants to compare the size of a generalization decrement between configurations of cues that were altered by adding or subtracting compositional elements. The results suggest that adding elements to a configuration can produce a generalization decrement, but removing elements produces a more robust generalization decrement. Furthermore, the generalization decrement caused by adding elements was not likely to be caused by competing orienting responses. The results are used to contrast Pearce's (1987, 1994) and Wagner's (2003) models of stimulus generalization.     

An investigation of peak shift and behavioral contrast for autoshaped and operant behavior.

Instrumental treadle press and nonreinforced key peck responses were monitored during discrimination training and generalization testing in pigeons on positive and negative reinforcement schedules. In Experiment 1, six pigeons pressed a treadle for food on a multiple variable-interval extinction schedule. In Experiment 2, three pigeons pressed a treadle to avoid shock on a multiple free-operant avoidance extinction schedule. Different color keylights signaled S+ and S- components. Some positive behavioral contrast occurred during discrimination training, but the effect was small. Pecking occurred to the S+ keylight in Experiment 1 but not in Experiment 2. On stimulus generalization tests, all subjects displayed a positive peak shift when pressing the treadle for food or to avoid shock. However, peak shift was not found for nonreinforced "autopecks" on the stimulus key, although an area shift was observed in Experiment 1. This is the first demonstration of peak shift for pigeons pressing treadles and the only reliable demonstration of peak shift when negative reinforcement maintained responding. These results, in combination with previous demonstrations of peak shift for rats pressing levers and pigeons pecking keys, indicate that peak shift is a general by-product of operant discrimination learning, since it occurs across a variety of the organisms, responses, and reinforcers.     

Some determinants of inhibitory stimulus control

Interspersed reinforcement and extinction during discrimination learning generate a U-shaped gradient of inhibition about the stimulus correlated with extinction. The present work showed that extinction is not a necessary determinant of inhibitory stimulus control. In Exp. I, a reduction in the rate of reinforcement, through a shift from a multiple variable-interval 1-min variable-interval 1-min schedule to a multiple variable-interval 1-min variable-interval 5-min schedule, resulted in a post-discrimination line orientation gradient of inhibition about the stimulus correlated with the variable-interval 5-min schedule. In Exp. II, the rates of reinforcement, correlated with a pair of stimuli, were held constant during a shift from a multiple variable-interval 1-min variable-interval 1-min schedule to a multiple variable-interval 1-min differential-reinforcement-of-low-rate schedule. Inhibitory stimulus control about the stimulus correlated with the differential reinforcement of low rate was obtained. In both experiments, a reduction in the rate of responding during one stimulus and behavioral contrast during the other stimulus preceded the observation of inhibitory stimulus control.     

Learning and memory in conditioned fear extinction: effects of D-cycloserine

This review addresses the effects of the cognitive enhancer D-cycloserine (DCS) on the memory processes that occur in conditioned fear extinction, which is the experimental model for exposure techniques to reduce clinical anxiety. All reported rat studies show an enhanced fear extinction effect when DCS is administered acutely before or shortly after extinction training. DCS also promotes the generalization of this fear extinction effect. In addition, DCS reduces some forms of relapse (reduced reinstatement, reduced spontaneous recovery), but not others (contextual renewal, rapid reacquisition). It is argued that this pattern of results is best explained by assuming that DCS promotes extinction learning to the background context, resulting in enhanced contextual inhibition. Four human studies have produced mixed results, but some methodological issues complicate the reported failures. It is concluded that DCS is a promising tool as an adjunct to extinction techniques in exposure treatment, but that more pre-clinical and clinical research is needed to fully characterize its behavioral consequences.     

On Stimulus Generalization and Stimulus Classes

Stimulus generalization has been defined as the spread of effect of reinforcement for responses emitted in the presence of one stimulus to different stimuli presented under extinction conditions. As a result of stimulus generalization, novel stimuli come to exert stimulus control over members of the response class. Studies in the applied behavior analysis literature, however, have reported experimental preparations that included prompting and reinforcement procedures during what were claimed to be stimulus generalization conditions. These studies violated the procedural requirement that stimulus generalization be tested under extinction conditions. Responses that come under the control of a class of stimuli may do so by direct training or by stimulus generalization. It is desirable for organisms to respond in the presence of members of an appropriately constructed stimulus class, but we should understand the mechanism of entry into the class by its members. If inaccurate claims of stimulus generalization are made when training procedures are used in the ostensible generalization conditions, the robustness of the original training procedures will be over estimated. By adhering to the operational requirements of behavioral definitions, we could better understand the power and limits of our educational and training procedures.     

Generalization gradients in human predictive learning: Effects of discrimination training and within-subjects testing

Generalization gradients have been investigated widely in animal conditioning experiments, but much less so in human predictive learning tasks. Here, we apply the experimental design of a recent study on conditioned fear generalization in humans (Lissek et al., 2008) to a predictive learning task, and examine the effects of a number of relevant procedural parameters drawn from the generalization literature in animal conditioning. Experiment 1 shows that prior discrimination learning and steady-state testing procedures sharpen the gradient; Experiment 2 shows that within-subjects testing of the range of generalization stimuli also sharpens the gradient. In addition, Experiment 2 shows that, in case of very flat initial generalization, an orderly gradient can reveal itself through differential rates of extinction learning. Finally, Experiment 2 also evidenced an orderly gradient of generalization-of-extinction. These results suggest that discrimination processes have an important effect on the generalization of predictive learning in humans, and highlight behavioral analogies between animal conditioning and human predictive learning.     

Does the context of reinforcement affect resistance to change?

Eight pigeons were trained on multiple schedules of reinforcement where pairs of components alternated in blocks on different keys to define 2 local contexts. On 1 key, components arranged 160 and 40 reinforcers/hr; on the other, components arranged 40 and 10 reinforcers/hr. Response rates in the 40/hr component were higher in the latter pair. Within pairs, resistance to prefeeding and resistance to extinction were generally greater in the richer component. The two 40/hr components did not differ in resistance to prefeeding, but the 40/hr component that alternated with 10/hr was more resistant to extinction. This discrepancy was interpreted by an algebraic model relating response strength to component reinforcer rate, including generalization decrement. According to this model, strength is independent of context, consistent with research on schedule preference.     

Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling

Because learned associations are quickly renewed following extinction, the extinction process must include processes other than unlearning. However, reinforcement learning models, such as the temporal difference reinforcement learning (TDRL) model, treat extinction as an unlearning of associated value and are thus unable to capture renewal. TDRL models are based on the hypothesis that dopamine carries a reward prediction error signal; these models predict reward by driving that reward error to zero. The authors construct a TDRL model that can accommodate extinction and renewal through two simple processes: (a) a TDRL process that learns the value of situation-action pairs and (b) a situation recognition process that categorizes the observed cues into situations. This model has implications for dysfunctional states, including relapse after addiction and problem gambling.     

Rate of response as a visual social stimulus

In Exp. 1, a high rate of responding (chain pulling) of a stimulus monkey was established as a visual positive discriminative stimulus for the operant behavior (bar pressing) of an observer monkey. The terminal performance of the observer under conditions in which a high rate of response of the stimulus monkey alternated in a variable temporal arrangement with a zero rate of response of the stimulus monkey (negative discriminative stimulus) was essentially the same as when nonbehavioral stimuli are correlated with the availability of reinforcement. By manipulating the schedule of reinforcement to change the rate of responding of the stimulus subject without changing its rate of reinforcement, Exp. 2 showed that the effective behavioral stimulus for the observer was the rate of chain pulling by the stimulus subject. A novel intermediate rate of responding by the stimulus monkey resulted in an intermediate rate (generalization) on the part of the observer during an extinction test. These experiments demonstrated that the rate of responding of one organism can function as a discriminative stimulus to control the rate of responding of another organism; and that the rate of responding is similar to other physical stimuli in terms of discrimination and generalization.     

Contrast and stimulus generalization following prolonged discrimination training

Different groups of pigeons received discrimination training in which the reinforcement-associated and extinction-associated stimuli were respectively either (a) a line tilt vs a blank key, (b) a blank key vs a line tilt, or (c) two different line tilts. The high response rates that developed to the positive stimulus in all groups during discrimination learning were maintained over 64 sessions of training. After these sessions, all subjects were tested for stimulus generalization along the line-tilt dimension. Gradients of relative (per cent) generalization around the stimulus associated with reinforcement (so-called excitatory gradients) and around the stimulus associated with extinction (so-called inhibitory gradients) were as steep as they typically are after much briefer training periods. These results do not support several of Terrace's predictions on the basis of the hypothesis that emotional responses develop to the stimulus associated with extinction during discrimination learning with errors, but eventually dissipate after extended training.     

Extinction after partial reinforcement: predicted vs. judged persistence

The partial reinforcement extinction effect (PREE) (i.e., increased persistence following partial reward) is one of the most important generalizations from experimental studies of learning. Many theories of PREE assume that it involves cognitive and emotional mechanisms, but investigations of PREE have focused almost exclusively on behavioral measures. Four experiments with human adults investigated whether PREE is also reflected in cognitive measures. Independent groups of subjects learned an instrumental response under CRF vs. PRF contingencies, and then predicted (Experiments 1, 2, and 3) and/or judged (Experiments 3 and 4) their own persistence under extinction conditions. Predictions of persistcnce were unrelated to prior continuous or partial reinforcement contingencies (Experiments 1, 2, and 3), but subsequent judgments of persistence behavior were accurate (Experiments 3 and 4). These results indicate that increased persistence due to occasional reward is not well represented cognitively prior to its behavioral manifestation, but it is well represented after that manifestation. Possible explanations and implications of this apparent behavior cognition dissociation are discussed.