Rethinking reinforcement: allocation, induction, and contingency

The concept of reinforcement is at least incomplete and almost certainly incorrect. An alternative way of organizing our understanding of behavior may be built around three concepts: allocation, induction, and correlation. Allocation is the measure of behavior and captures the centrality of choice: All behavior entails choice and consists of choice. Allocation changes as a result of induction and correlation. The term induction covers phenomena such as adjunctive, interim, and terminal behavior-behavior induced in a situation by occurrence of food or another Phylogenetically Important Event (PIE) in that situation. Induction resembles stimulus control in that no one-to-one relation exists between induced behavior and the inducing event. If one allowed that some stimulus control were the result of phylogeny, then induction and stimulus control would be identical, and a PIE would resemble a discriminative stimulus. Much evidence supports the idea that a PIE induces all PIE-related activities. Research also supports the idea that stimuli correlated with PIEs become PIE-related conditional inducers. Contingencies create correlations between "operant" activity (e.g., lever pressing) and PIEs (e.g., food). Once an activity has become PIE-related, the PIE induces it along with other PIE-related activities. Contingencies also constrain possible performances. These constraints specify feedback functions, which explain phenomena such as the higher response rates on ratio schedules in comparison with interval schedules. Allocations that include a lot of operant activity are "selected" only in the sense that they generate more frequent occurrence of the PIE within the constraints of the situation; contingency and induction do the "selecting."     

Summation of food and shock based responding

In a multiphase experiment, dogs first received discriminative, discretetrial, barrier-jumping training with two tones (S^D, S^Δ) in a shuttle box reinforced by either shock avoidance (Group I) or by food (Group II). Then the dogs were trained on free-operant barrier-jumping reinforced by the qualitatively opposite reinforcer—food in Group I and shock avoidance in Group II. Finally, test presentations of the tone stimuli were superimposed on the free-operant behavior. The tone S^Ds markedly facilitated responding in all animals. This experiment demonstrates a summation of responding maintained by shock avoidance and food reinforcement and casts doubt on explanations of conditioned suppression outcomes that appeal solely to incompatible motivational states within the organism.     

Density and delay of punishment of free-operant avoidance

In two experiments, the free-operant shock-avoidance behavior of rats was punished by electric shock. Two aspects of the schedule of response-produced shock were varied: the frequency of punishment over time (punishment density) and the temporal interval between the punished response and the punishment (punishment delay). The general finding was that response-produced shock suppressed avoidance responding under most of the density-delay combinations studied, and suppression increased as a function of increases in density and decreases in delay. Rate increases of small magnitude also were observed, usually as an initial reaction to the lesser densities and longer delays. Response suppression, while decreasing the number of punishment shocks received, also increased the number of avoidance shocks, so that the total number of shocks received usually was greater than the minimal number possible. The results were discussed from the standpoint of similarities between the effects of punishing positively and negatively reinforced behavior. The finding that subjects did not minimize the total number of shocks suggested that when avoidance behavior is punished, responding is controlled more by the local consequences of responding than by overall shock frequencies during the course of the session.     

Effects of cycle length on performance on a temporally defined avoidance schedule

Three rats were trained on a temporally defined avoidance schedule logically similar to a fixed-interval, limited-hold positive reinforcement schedule. This avoidance schedule was composed of time periods during which responses had no scheduled consequences alternating with time periods during which a response precluded shock. As with fixed-interval length and response rate on positive reinforcement schedules, an inverse relationship was obtained between the length of the no-consequence interval and response rate during the no-consequence interval. An inverse relationship was also obtained between the length of the no-consequence interval and the per cent of shocks avoided. A rate increase within the no-consequence interval, similar to that typically produced by fixed-interval positive reinforcement procedures, was displayed by one of the rats where the no-consequence interval was at intermediate values and frequency of shock was relatively high. The introduction of a discriminative stimulus correlated with the avoidance interval produced typical discriminated avoidance behavior as well as alterations in temporal patterning of responses during the no-consequence interval in the two rats exposed to this procedure. These alterations in temporal patterning disappeared when the discriminative stimulus was removed. The results were consonant with those reported in the literature involving food reinforcement and fixed-interval, limited-hold schedules.     

Acquisition and extinction of signaled avoidance as a function of intermittent reinforcement

Signaled, shuttle-box avoidance responding in female rats of the Fischer₃₄₄ strain was examined as a function of four separate contingencies of intermittent reinforcement. In Experiment 1, when avoidance responses during acquisition were reinforced 25% of the time with prompt CS termination, animals responded equally often during acquisition and significantly more often during extinction than animals who received such reinforcement on a 100% schedule. Similar results were found under a trace procedure in Experiment 2 when avoidance responses were reinforced 25% of the time with informational feedback stimuli. In contrast, during Experiment 3, when animals were shocked on only 25% of the trials on which they failed to respond, the level of avoidance responding during both acquisition and extinction was significantly less than it was when animals were shocked on a 100% schedule. Comparable results were found in Experiment 4 when avoidance responses during acquisition averted shock on only 25% of the trials. Thus, intermittent reinforcement contingencies involving response-contingent feedback stimuli and shock have differential effects on avoidance responding during both acquisition and extinction trials under the signaled avoidance procedure.     

Parameters affecting the acquisition of Sidman avoidance

Four studies were conducted to investigate the source of reinforcement of Sidman avoidance. First it was found that the acquisition of avoidance was seriously impaired if there was no immediate stimulus consequence of responding, i.e., proprioceptive and auditory feedback. The second study showed that if the shock-shock interval was split so that only responses occurring in one half of the interval were effective in postponing shock, then learning was impaired. It made little difference which half of the interval was used. The third study demonstrated that Ss learn Sidman avoidance more quickly with a variable shock-shock interval than with the usual fixed-interval procedure. The results of the second and third experiments argue against the view that avoidance is reinforced by the decrease in conditioned aversiveness that occurs at long post-shock times. The final experiment indicated that Ss do not learn Sidman avoidance if the response-produced delay of shock is preceded by a shock, hence it seems unlikely that the crucial source of reinforcement is merely an overall reduction in shock density. All of the findings are consistent with the hypothesis that the avoidance response is reinforced by the decrease in conditioned aversiveness of stimuli at short post-response times. This seems to be the case even at the beginning of acquisition.     

Human, free-operant avoidance of “time out” from monetary reinforcement

To assess the aversive effects of withdrawing monetary reinforcement, human subjects were exposed to a free-operant avoidance procedure in which periods of no reinforcement occurred if the subject failed to respond, and each response postponed withdrawal of reinforcement. Avoidance behavior was developed either through specific instructions about the consequence of responding or through preliminary escape-avoidance training. In all cases, rates of response were found to be a positively accelerated function of decreases in the duration by which responding postponed reinforcement withdrawal. The findings with respect to the function relating avoidance behavior to the interval of postponement were viewed as similar to those obtained when shock is used as the aversive event in free-operant avoidance conditioning.     

Brain, emotion and decision making: the paradigmatic example of regret

Human decisions cannot be explained solely by rational imperatives but are strongly influenced by emotion. Theoretical and behavioral studies provide a sound empirical basis to the impact of the emotion of regret in guiding choice behavior. Recent neuropsychological and neuroimaging data have stressed the fundamental role of the orbitofrontal cortex in mediating the experience of regret. Functional magnetic resonance imaging data indicate that reactivation of activity within the orbitofrontal cortex and amygdala occurring during the phase of choice, when the brain is anticipating possible future consequences of decisions, characterizes the anticipation of regret. In turn, these patterns reflect learning based on cumulative emotional experience. Moreover, affective consequences can induce specific mechanisms of cognitive control of the choice processes, involving reinforcement or avoidance of the experienced behavior.     

Still No Evidence For Temporally Extended Shock-frequency Reduction As A Reinforcer.

There is no consensus and very little overlap in the criticisms of my target article. Because the primary consequences of avoidance behavior are by definition alterations in the distribution of shocks in time, any theory about the reinforcement of such behavior necessarily must begin with that dimension. However, the safety‐signal version of two‐process theory calls on positively and negatively correlated stimuli, including the responses themselves serving as stimuli, to transmit the effects of those alterations to the relevant behavior. Meanwhile, the Herrnstein—Hineline single‐process theory hypothesizes an additional source of reinforcement: a direct effect of reduction in the density of shock over some extended period of time. I can find no data that selectively support that hypothesis.     

Persistence during and resurgence following noncontingent reinforcement implemented with and without extinction

Noncontingent reinforcement (NCR) is typically implemented with extinction (EXT) for destructive behavior reinforced by social consequences and without EXT for destructive behavior reinforced by sensory consequences. Behavioral momentum theory (BMT) predicts that responding will be more persistent, and treatment relapse in the form of response resurgence more likely, when NCR is implemented without EXT due to the greater overall rate of reinforcement associated with this intervention. We used an analogue arrangement to test these predictions of BMT by comparing NCR implemented with and without EXT. For two of three participants, we observed more immediate reductions in responding during NCR without EXT. However, for all participants, NCR without EXT produced greater resurgence than NCR with EXT when we discontinued all reinforcers during an EXT Only phase, although there was variability in response patterns across and within participants. Implications for treatment of destructive behavior using NCR are discussed.     

Actual versus potential shock in making shock situations function as negative reinforcers

The relative importance of potential and actual shocks in making shock situations function as negative reinforcers was studied. Shocks were scheduled to occur at the same rate during two stimuli. During one, squirrel monkeys could avoid the shocks; during the other, they were unavoidable. For the two stimuli the potential rate of shocks was the same, but the actual rate was lower during avoidance because of avoidance responding. Fixed-ratio responding was maintained by the change from unavoidable shock to avoidance, indicating that the change was reinforcing when it resulted in a reduction in actual shock rate with no reduction in potential shock rate. Further increases in the rate of potential shock during avoidance had little effect upon the fixed-ratio responding until the rate was increased to the point that the actual shock rate during avoidance was comparable with that during unavoidable shock. At that point, the fixed-ratio response rate decreased nearly to zero. These findings show that actual shocks are more important than potential shocks in determining whether or not a shock situation will function as a negative reinforcer; this explains why the change from unavoidable shock to avoidable shock is reinforcing.     

Punishment of an extinguishing shock-avoidance response by time-out from positive reinforcement

Two experiments were conducted to determine the effects of punishment by time-out from positive reinforcement on the extinction of discriminated shock-avoidance responding. Subjects were trained initially to bar press for food on an intermittent schedule of reinforcement and, concurrently, to avoid shock at the onset of a warning signal. Experiment I compared avoidance extinction performance under no punishment and when avoidance responding resulted in a 30-sec TO from reinforced appetitive responding. In Exp II, the contingent use of TO punishment was compared with its random, or noncontingent use. The results of both experiments showed that in the absence of punishment, avoidance extinction was characterized by short latencies and nearly 100% avoidance responding. Avoidance responding in extinction was little affected by noncontingent TO punishment. When TO was made contingent upon avoidance responding, however, avoidance latencies immediately increased and the frequency of avoidance responses subsequently decreased to zero.     

The effect of incongruent instructions and consequences on imitation in retarded children

Three retarded boys served as subjects in a 13-phase experiment. In eight of these phases, the instructions administered by the experimenter before demonstrating a behavior and the consequences for imitative behavior were incongruent (the consequences were not those indicated by the instructions). Consequences rather than instructions controlled imitative behavior when (a) subjects were instructed not to imitate but received reinforcers if they imitated; (b) subjects were instructed to imitate but were differentially reinforced for other behavior; (c) subjects were instructed to imitate but were verbally reprimanded for imitation. Although subjects were highly imitative at the beginning of the study, when there was no reinforcement for imitation subjects gradually stopped imitating when instructed not to imitate. Instructions seemed to control imitative behavior when there was no reinforcement for imitation and subjects were instructed to imitate. These results indicated a need for further investigation of antecedent and consequent variables in imitation experiments and pointed out that certain techniques may be more efficient than others in eliminating well-established responses.     

The effects of aversive conditioned stimuli on the timing of unsignalled avoidance responding in rats

Rats received unsignalled avoidance training in a shuttlebox. After acquisition, classical conditioning sessions were administered. The Excitatory Group received signals paired with shock, the Inhibitory Group received signals explicitly unpaired with shock, while the Random Control Group received signals and shocks randomly in time. When subsequently superimposed on the avoidance baseline, the excitatory signal increased response rate, the inhibitory signal decreased rate and the random signal had no effect compared to signal-only control data. Unsignalled avoidance generates temporal gradients of responding; the major purpose of this study was to determine how these rate changes were reflected in the baseline temporal behavior. Very clear and simple summation rules emerged: In the presence of the excitatory signal, rats acted as if they were ahead of their actual location in the response-shock interval regardless of where the signal occurred; the inhibitory signal had the opposite effect while the random signal did not change the temporal behavior relative to signal-only control data. These results were interpreted within a motivational framework.     

The influence of positive and negative reinforcement on selective attention in the rat

In Experiments 1 and 2 rats were trained under two multiple schedules of reinforcement. In one, bar pressing during a tone-light compound stimulus was reinforced under a variable-interval food reinforcement schedule. In the other multiple schedule, bar pressing avoided grid shock on a free-operant schedule. In both multiple schedules, a discrimination was maintained by an extinction schedule that was operative during the absence of the tone-light compound. In Experiments 1 and 2 the intensity of the tone-light compound was manipulated over three levels. Subsequent extinction tests revealed that light was attended to, almost exclusively of the tone, when food reinforcement had maintained bar pressing. On the other hand, the tone gained considerable attentional control under the shock avoidance schedule. This stimulus-reinforcer interaction was maintained for all three levels of the compound intensity. In Experiment 3 it was investigated whether this interaction was associative by presenting shock during the absence of the tone-light compound when food reinforcement maintained responding, and food during the absence of the compound when shock avoidance maintained responding. Since both food and shock were presented during a single session for both schedules, nonassociative effects of the reinforcing stimuli were equivalent across the schedules. Nevertheless, the stimulus-reinforcer interaction was maintained, indicating that the interaction was an associative effect.     

Punishment of an alternative behavior generates resurgence of a previously extinguished target behavior

Resurgence is often defined as the recurrence of an extinguished behavior when a more recently reinforced alternative behavior is also extinguished. Resurgence has also been observed when the alternative behavior is devalued by other means (e.g., reinforcement rate or magnitude reductions). The present study investigated whether punishment of an alternative behavior would generate resurgence. A target response was reinforced during Phase 1 and then extinguished in Phase 2 while an alternative response was reinforced. During Phase 3, response‐dependent foot shocks were superimposed on the schedule of reinforcement for the alternative response and shock intensity was escalated gradually across sessions. Resurgence of the target response was reliably observed, mostly at higher intensities. The effect was replicated in two subsequent exposures to the sequence of conditions, with resurgence tending to occur at the lowest foot shock intensity. These results suggest that devaluation of an alternative behavior via punishment can generate resurgence. Although it is difficult to reconcile the overall pattern of results with Bouton's context account, these findings are consistent with the suggestion that resurgence results from a “worsening of conditions” for the alternative behavior and with the formalization of that suggestion in terms of a choice‐based matching‐law account (i.e., Resurgence as Choice).     

Remote effects of aversive contingencies: Disruption of appetitive behavior by adjacent avoidance sessions

Disruption of ongoing appetitive behavior before and after daily avoidance sessions was examined. After baselines of appetitive responding were established under a fixed-interval 180-s schedule of food presentation, 4 rats were exposed to 40-min sessions of the appetitive schedule just prior to 100-min sessions of electric shock postponement, while another 4 rats received the 40-min appetitive sessions just following daily sessions of shock postponement. In all 8 subjects, fixed-interval response rates decreased relative to baseline levels, the effect being somewhat more pronounced when the avoidance sessions immediately followed. The disruption of fixed-interval responding was only partially reversed when avoidance sessions were discontinued. During the initial exposure to the avoidance sessions, patterns of responding under the fixed-interval schedule were differentially sensitive to disruption, with high baseline response rates generally more disturbed than low rates. These disruptions were not systematically related to changes in reinforcement frequency, which remained fairly high and invariant across all conditions of the experiment; they were also not systematically related to the response rates or to the shock rates of the adjacent avoidance sessions. The results, while qualitatively resembling patterns of conditioned suppression as typically studied, occurred on a greatly expanded time scale. As disruption of behavior extending over time, the present data suggest that some forms of conditioned suppression are perhaps best viewed within a larger temporal context.     

Persistent behavior maintained by unavoidable shocks

Squirrel monkeys were trained on a multiple schedule in which 10-min periods on a continuous shock avoidance schedule, indicated by a yellow light, alternated with 10-min periods on a 1.5-min variable interval schedule of food reinforcement (VI 1.5). A white light indicated that VI 1.5 was in effect, except for the middle 2 min of the period on VI 1.5, in which a blue light appeared and terminated with the delivery of a 0.5-sec unavoidable shock. Stable response rates developed in the avoidance and VI 1.5 components. However, the highest response rates occurred in the blue, preshock stimulus. A series of experiments showed that responding in the blue stimulus persisted even when responding had been extinguished on both the VI schedule of food reinforcement and the shock avoidance schedule. Responding in the blue stimulus ceased when the blue stimulus terminated without shock or when it terminated with a response-contingent shock. Each time responding ceased, it was restored by terminating the blue stimulus with an unavoidable shock. When the blue stimulus was on throughout each session and unavoidable shocks were delivered at regular 10-min intervals, responding was well maintained. These results show that in monkeys that have been trained on a continuous avoidance schedule, unavoidable shocks can maintain responding even under conditions where responses have no programmed consequences.     

Variable-ratio schedules of timeout from avoidance: effects of d-amphetamine and morphine.

Rats were trained on concurrent schedules in which pressing one lever postponed shock and pressing the other lever produced periods of signaled timeout from avoidance on variable-ratio schedules. These procedures generated high rates of timeout-reinforced responding and provided a baseline for studying the effects of drugs on behavior maintained by different types of negative reinforcement (shock postponement vs. timeout). Morphine (2.5 to 10.0 mg/kg) reduced behavior maintained by timeout at doses that increased or had no effect on avoidance responding. In contrast, d-amphetamine (0.125 to 2.0 mg/kg) produced large increases in timeout responding at doses that had minimal effect on avoidance in rats trained on variable-interval and variable-ratio schedules. Thus, the event-dependent effects of morphine, observed in previous studies in which timeout responding was maintained at low rates by interval schedules, were replicated with high timeout rates maintained by variable-ratio schedules. The effects of d-amphetamine could also be described as "event dependent" because timeout responding was stimulated more than avoidance regardless of the maintenance schedule or baseline rate.     

Immobility as an avoidance response, and its disruption by drugs

Much of the available literature on avoidance behavior is based on responses which require the animal to run, lever-press, or to make some active response to avoid noxious stimulation. The purpose of Experiment I reported in this paper was to determine whether animals can learn to sit or stand motionless in order to escape or avoid electric shock. Five experimental rats were given escape-avoidance training, while five yoked control animals received electric shocks without any response-related contingency. It was shown that an immobility avoidance response, as distinct from the unconditioned “freezing” response to shock, can be trained. The results of Experiment II (30 rats) revealed that this response is more readily acquired at higher shock intensities than at lower ones, provided escape by jumping is prevented at the high shock intensities. The effects of six doses of each of three drugs on the immobility avoidance response were studied in Experiment III (13 rats). Methylphenidate, chlorpromazine, and imipramine all produced a decrement in the immobility response, but the pattern and amount of the effects of the three drugs were quite different, one from the other. The implications of these findings for a general theory of avoidance behavior and for drug screening are discussed.