The law of effect and avoidance: a quantitative relationship between response rate and shock-frequency reduction

Two experiments were conducted to investigate the quantitative relationship between response rate and reinforcement frequency in single and multiple variable-interval avoidance schedules. Responses cancelled delivery of shocks that were scheduled by variable-interval schedules. When shock-frequency reduction was taken as the measure of reinforcement, the relationship between response rate and reinforcement frequency on single variable-interval avoidance schedules was accurately described by Herrnstein's (1970) equation for responding on single variable-interval schedules of positive reinforcement. On multiple variable-interval avoidance schedules with brief components, asymptotic relative response rate matched relative shock-frequency reduction. The results suggest that many interactions between response rates and shock-frequency reduction in avoidance can be understood within the framework of the generalized matching relation, as applied by Herrnstein (1970) to positive reinforcement.     

Reductions in shock frequency and response effort as factors in reinforcement by timeout from avoidance

Rats' presses on one lever canceled shocks programmed after variable cycles, while presses on a second lever occasionally produced a 2-min timout during which the shock-delection schedule was suspended and its correlated stimuli removed. These concurrent schedules of avoidance and timeout were embedded in a multiple schedule whose components differed, within and across conditions, in terms of the programmed shock rate associated with the shock-deletion schedule. Analyses based on the generalized matching law suggest that the reduction in the response requirement correlated with termination of the avoidance schedule was a more important factor in the reinforcing effectiveness of timeout than was shock-frequency reduction, at least in 2 of 3 rats. After training in each condition, responding on the timeout lever was extinguished by withholding timeouts in both components over seven sessions. Resistance to extinction varied directly with the rates of both shock-frequency reduction and avoidance-response reduction experienced during training. Although reduction in response effort appeared to dominate shock-frequency reduction in the maintenance of responding, neither factor had a clear advantage in predicting the course of extinction.     

The discriminative control of free-operant avoidance despite exposure to shock during the stimulus correlated with nonreinforcement

Four rats were trained in darkness on a free-operant avoidance procedure in which shocks occurred randomly, but lever presses could reduce their frequency. Discrimination training followed, during which responses in light continued to reduce shock frequency, but responses in darkness had no effect. During each cycle, the light period was 4 min, while darkness lasted only until a 20-sec interval had elapsed without a response. This no-response requirement was increased to 40 sec for three animals and eventually to 60 sec for two of them. Discriminative control developed, despite a greater shock density in the dark, with response rate and number of responses per shock maintained or increasing during light and decreasing to very low values in darkness. Two animals were later exposed to a procedure in which shock density was unaffected by responding either in light or darkness. A 60-sec no-response requirement was continued in the dark. Discriminative control persisted through 42 sessions for one animal and required 45 sessions to approach extinction for the other animal. The role of the light as a potential conditioned reinforcer of other behavior in the dark was implicated in the development and persistence of discriminative control. These data support shock-frequency reduction as reinforcement for avoidance behavior.     

Timing in free-operant and discrete-trial avoidance

A procedure (“discrete-trial” avoidance) was devised to differentiate between the two main theories of responding in Sidman's “free-operant” avoidance procedure. One theory, a version of two-factor theory, holds that responding is reinforced by the removal of a conditioned aversive stimulus. The conditioned aversive stimulus is held to be temporal, which accounts for the spaced responding, or timing, that Sidman's procedure produces. The other theory holds that the reinforcement for both responding and timing is shock-frequency reduction. The new procedure eliminated this reinforcement for timing, but retained the conditions for the formation of conditioned aversive temporal stimuli. According to one theory, the new procedure should have sustained timing as well as Sidman's, while according to the other, it should have sustained no timing. The results confirmed neither theory. Timing was found with both procedures, but unequally in degree and kind. Large variations in the precision of timing did not appear to be correlated with successful avoidance for either procedure.     

Effects of partial avoidance and punishment contingencies on shuttlebox-avoidance learning in Fischer 344 rats

An unsignaled, escapable shock was presented contingent on an avoidance response. Fischer 344 rats responded less to the warning signal in proportion to its temporal distance from the avoidance response. Partial contingency effects were further obtained by variation in the instrumental conditioning space for an aversive stimulus. However, the arbitrary omission of an imminent shock on half the trial in which the rats failed to avoid a shock, led to little avoidance acquisition, and shock-frequency reduction was thus not sufficient to produce the acquisition of the avoidance response. Because early avoidance responses were initiated by escape from shock, a stimulus contingency may be essential for response initiation, and an explicit response contingency is important in maintaining successful avoidance responses.     

Avoidance conditioning with shock contingent upon the avoidance response

Rats learned either a lever-press response, a shuttle response or a one-way crossing response, which produced one immediate shock but was instrumental in avoiding five identical shocks scheduled to occur later. These responses were acquired both with and without support of an escape contingency. These results support shock-frequency reduction as a sufficient condition for the acquisition and maintenance of avoidance.     

Reinforcement and response rate interaction in multiple random-interval avoidance schedules

Two experiments were conducted to investigate the interaction between response rate and reinforcement frequency in multiple random-interval avoidance schedules. Responses cancelled delivery of shocks that could be scheduled at different random intervals in each component. When shock-frequency reduction was taken as the measure of reinforcement, the relationship between response rate and frequency of reinforcement was described by the same equations used by Herrnstein (1970) to describe responding with positive reinforcement.     

Stimuli inevitably generated by behavior that avoids electric shock are inherently reinforcing.

A molecular analysis based on the termination of stimuli that are positively correlated with shock and the production of stimuli that are negatively correlated with shock provides a parsimonious count for both traditional discrete-trial avoidance behavior and the data derived from more recent free-operant procedures. The necessary stimuli are provided by the intrinsic feedback generated by the subject's behavior, in addition to those presented by the experimenter. Moreover, all data compatible with the molar principle of shock-frequency reduction as reinforcement are also compatible with a delay-of-shock gradient, but some data compatible with the delay gradient are not compatible with frequency reduction. The delay gradient corresponds to functions relating magnitude of behavioral effect to the time between conditional and unconditional stimuli, the time between conditioned and primary reinforcers, and the time between responses and positive reinforcers.     

Behavioral variability in an evolutionary theory of behavior dynamics

McDowell’s evolutionary theory of behavior dynamics (McDowell, 2004) instantiates populations of behaviors (abstractly represented by integers) that evolve under the selection pressure of the environment in the form of positive reinforcement. Each generation gives rise to the next via low‐level Darwinian processes of selection, recombination, and mutation. The emergent patterns can be analyzed and compared to those produced by biological organisms. The purpose of this project was to explore the effects of high mutation rates on behavioral variability in environments that arranged different reinforcer rates and magnitudes. Behavioral variability increased with the rate of mutation. High reinforcer rates and magnitudes reduced these effects; low reinforcer rates and magnitudes augmented them. These results are in agreement with live‐organism research on behavioral variability. Various combinations of mutation rates, reinforcer rates, and reinforcer magnitudes produced similar high‐level outcomes (equifinality). These findings suggest that the independent variables that describe an experimental condition interact; that is, they do not influence behavior independently. These conclusions have implications for the interpretation of high levels of variability, mathematical undermatching, and the matching theory. The last part of the discussion centers on a potential biological counterpart for the rate of mutation, namely spontaneous fluctuations in the brain's default mode network.     

Interresponse-time punishment: a basis for shock-maintained behavior.

Lever pressing of squirrel monkeys postponed brief electric shock according to a free-operant shock-postponement procedure. Pressing also produced shock with a probability proportional to the duration of the current interresponse time in some conditions, or to the fifth ordinally-preceding interresponse time in others. These conditions provided equal frequencies and temporal distributions of response-produced shocks either contingent on or independent of the current interresponse-time duration, respectively. Shock delivered contingent on the current interresponse-time duration resulted in shorter mean interresponse times and higher overall response rates that shock delivered independent of the current interresponse time. In subsequent conditions, response-produced shocks were sufficient to maintain responding following suspension of the postponement procedure only when those shocks were contingent on the current interresponse time. Presenting shock independent of the current interresponse time, conversely, suppressed response rate and ultimately led to cessation of responding in the absence of a conjoint shock-postponement procedure. These results demonstrate interresponse-time punishment in the absence of any indirect avoidance contingencies based on overall shock-frequency reduction, and strongly support similar interpretation at the more local level of shock-frequency reduction correlated with particular interresponse times. Differential punishment of long interresponse times also provides both an a priori basis for predicting whether a schedule of shock presentation will maintain or suppress responding and a framework for interpreting many of the functional relations between overall response rate and parameters of consequent shock presentation. Finally, these results and others indicate the importance of response-consequence contiguity above and beyong any notion of noncontiguous contingency in the control of behavior.     

Feedback in discriminative avoidance learning: Mechanistic versus cognitive views

Presentation of a feedback stimulus, a non-aversive event never paired with shock, following an avoidance response has been found to be an effective reinforcer in avoidance learning. Alternative formulations of the feedback effect, one a version of the traditional S-R mechanistic, two-factor theory and the other a cognitive (informational) theory, were evaluated with a view toward determining which type of formulation provides a more viable explanation of avoidance learning. The conclusion reached, however, was that both informational and fear-reduction positions operationalize their critical concepts in the same manner, and to date there are no empirical grounds for distinguishing these different views.     

A theory of attending, remembering, and reinforcement in delayed matching to sample

A theory of attending and reinforcement in conditional discriminations is extended to working memory in delayed matching to sample by adding terms for disruption of attending during the retention interval. Like its predecessor, the theory assumes that reinforcers and disruptors affect the independent probabilities of attending to sample and comparison stimuli in the same way as the rate of overt free-operant responding as suggested by Nevin and Grace, and that attending is translated into discriminative performance by the model of Davison and Nevin. The theory accounts for the effects of sample-stimulus discriminability and retention-interval disruption on the levels and slopes of forgetting functions, and for the diverse relations between accuracy and sensitivity to reinforcement reported in the literature. It also accounts for the effects of reinforcer probability in multiple schedules on the levels and resistance to change of forgetting functions; for the effects of reinforcer probabilities signaled within delayed-matching trials; and for the effects of reinforcer delay, sample duration, and intertrial-interval duration. The model accounts for some data that have been problematic for previous theories, and makes testably different predictions of the effects of reinforcer probabilities and disruptors on forgetting functions in multiple schedules and signaled trials.     

Biasing the pacemaker in the behavioral theory of timing

In the behavioral theory of timing, pacemaker rate is determined by overall rate of reinforcement. A two-alternative free-operant psychophysical procedure was employed to investigate whether pacemaker period was also sensitive to the differential rate of reinforcement. Responding on a left key during the first 25 s and on a right key during the second 25 s of a 50-s trial was reinforced at variable intervals, and variable-interval schedule values during the two halves of the trials were varied systematically. Responding on the right key during the first 25 s and on the left key during the second 25 s was not reinforced. Estimates of pacemaker period were derived from fits of a function predicted by the behavioral theory of timing to right-key response proportions in consecutive 5-s bins of the 50-s trial. Estimates of pacemaker period were shortest when the differential reinforcer rate most strongly favored right-key responses, and were longest when the differential reinforcer rate most strongly favored left-key responses. The results were consistent with the conclusion that pacemaker rate is influenced by relative reinforcer rate.     

Stimulus equivalence: testing Sidman's (2000) theory

Sidman's (2000) theory regarding the origin of equivalence relations predicts that a reinforcing stimulus common to distinct equivalence classes must drop out of the equivalence relations. This prediction was tested in the present study by arranging class-specific reinforcers, R1 and R2, following correct responding on the prerequisite conditional discriminations (Ax-Bx, Cx-Bx) for two stimulus classes, A1B1C1 and A2B2C2. A class-common reinforcer, R3, was presented following correct responding on the prerequisite conditional discriminations for a further two stimulus classes, A3B3C3 and A4B4C4. Sidman's theory predicts reinforcer inclusion within Classes 1 and 2 only, given this training arrangement. Experiment 1 tested for the emergence of four equivalence classes and of stimulus-reinforcer and reinforcer-stimulus relations in each class. Four of the 6 subjects demonstrated the reinforcer-based relations in all four equivalence classes, rather than in only those classes with a class-specific reinforcer, as Sidman's theory predicts. One of the remaining 2 subjects showed the reinforcer-based relations in three of the four classes. Experiment 2 extended these findings to document the emergence of interclass matching relations based on the common reinforcer R3, in 5 of 6 subjects, such that a Class 3 sample occasioned the selection of a Class 4 sample when the Class 3 comparison was absent, and similarly, a Class 4 sample occasioned the selection of a Class 3 comparison when the Class 4 comparison was absent. These interclass relations emerged despite the simultaneous maintenance of Class 3 and 4 baseline conditional discriminations, so that the Class 3 and 4 stimuli and reinforcer simultaneously were, and were not, part of a single larger equivalence class. These data are irreconcilable with Sidman's theory, and question the utility of the application of the equivalence relation in describing derived stimulus relations.     

Integrated delays to shock as negative reinforcement

Rats were shocked at the rate of two per minute until they pressed a lever. In Experiment I, shocks were delivered at variable-time intervals averaging 30 sec; in Experiment II, shocks were delivered at fixed-time intervals of 30 sec. A response produced an alternate condition for a fixed-time period. The shock frequency following a response, calculated over the whole alternate condition, was two per minute. The pattern of shocks in the alternate condition was controlled so that the first shock occurred at the same time as it would have occurred had the response not been emitted; the remaining shocks were delayed until near the end of the alternate condition. Bar pressing was acquired in both experiments. This finding is not explained by two-factor theories of avoidance and is inconsistent with the notion that overall shock-frequency reduction is necessary for negative reinforcement. The data imply that responding is determined by the integrated delays to each shock following a response versus the integrated delays to shock in the absence of a response.     

Fixed-interval limited-hold avoidance with and without signalled reinforcement

Rats trained to lever press on a fixed-interval limited-hold avoidance schedule maintained a pattern of responding similar to that maintained by fixed-interval limited-hold schedules of positive reinforcement. But this positively accelerated pattern of behavior was maintained only when the occurrence of reinforcement was signalled by the presentation of a brief flash of light. This result suggests that the discriminative function of the reinforcer in avoidance is less pronounced than the discriminative function of the reinforcer in escape or positive reinforcement. It also suggests that the distinction between positive reinforcement and avoidance is not superfluous. Although the schedule of reinforcement is an important variable in determining the pattern of behavior, other variables, such as the nature (i.e., stimulus presentation, termination, or omission) of the reinforcer, are also potent determinants of behavior.     

Reinforcement Probability and Delay as Determinants of Human Impulsiveness

In behavior theory, “impulsiveness” refers to the choice of an immediate, small reinforcer over a delayed, large reinforcer. Such behavior generally is attributed to a reduction in the value of the large reinforcer as a function of the duration of delay. In contrast, social learning theorists have suggested that human impulsiveness can result from a lowered “expectancy” (subjective probability) of reinforcement. Effects of probability and delay were assessed by asking adults to make repeated choices between reinforcement schedules in which the reinforcers were slides of entertainment figures. An immediate, 5-s reinforcer was consistently chosen over an immediate, 40-s reinforcer if the probability of receiving the large reinforcer had previously been low (.20), implying that impulsiveness can occur without time-based discounting. However, reinforcement delay was also influential: Choice between a certain, small reinforcer and an uncertain, large reinforcer varied according to which reinforcer was immediate and which delayed.

     

Renewal of fear and avoidance in humans to escalating threat: Implications for translational research on anxiety disorders

Exposure-based treatment for threat avoidance in anxiety disorders often results in fear renewal. However, little is known about renewal of avoidance. This multimodal laboratory-based treatment study used an ABA renewal design and an approach–avoidance (AP–AV) task to examine renewal of fear/threat and avoidance in twenty adults. In Context A, 9 visual cues paired with increases in probabilistic money loss (escalating threats) produced increases in ratings of feeling threatened and loss expectancies and skin-conductance responses (SCR). During the AP–AV task, a monetary reinforcer was available concurrently with threats. Approach produced the reinforcer or probabilistic loss, while avoidance prevented loss and forfeited reinforcement. Escalating threat produced increasing avoidance and ratings. In Context B with Pavlovian extinction, threats signaled no money loss and SCR declined. During the AP–AV task, avoidance and ratings also declined. In a return to Context A with Pavlovian threat extinction in effect during the AP–AV task, renewal was observed. Escalating threat was associated with increasing ratings and avoidance in most participants. SCR did not show renewal. These are the first translational findings to highlight renewal of avoidance in humans. Further research should identify individual difference variables and altered neural mechanisms that may confer increased risk of avoidance renewal.     

Actual versus potential shock in making shock situations function as negative reinforcers

The relative importance of potential and actual shocks in making shock situations function as negative reinforcers was studied. Shocks were scheduled to occur at the same rate during two stimuli. During one, squirrel monkeys could avoid the shocks; during the other, they were unavoidable. For the two stimuli the potential rate of shocks was the same, but the actual rate was lower during avoidance because of avoidance responding. Fixed-ratio responding was maintained by the change from unavoidable shock to avoidance, indicating that the change was reinforcing when it resulted in a reduction in actual shock rate with no reduction in potential shock rate. Further increases in the rate of potential shock during avoidance had little effect upon the fixed-ratio responding until the rate was increased to the point that the actual shock rate during avoidance was comparable with that during unavoidable shock. At that point, the fixed-ratio response rate decreased nearly to zero. These findings show that actual shocks are more important than potential shocks in determining whether or not a shock situation will function as a negative reinforcer; this explains why the change from unavoidable shock to avoidable shock is reinforcing.