DELAYED REINFORCEMENT OF OPERANT BEHAVIOR

The experimental analysis of delay of reinforcement is considered from the perspective of three questions that seem basic not only to understanding delay of reinforcement, but, also, by implication, the contributions of temporal relations between events to operant behavior. The first question is whether effects of the temporal relation between responses and reinforcers can be isolated from other features of the environment that often accompany delays, such as stimuli or changes in the temporal distribution or rate of reinforcement. The second question is that of the effects of delays on operant behavior. Beyond the common denominator of a temporal separation between reinforcers and the responses that produce them, delay of reinforcement procedures differ from one another along several dimensions, making delay effects circumstance dependent. The final question is one of interpreting delay of reinforcement effects. It centers on the role of the response—reinforcer temporal relation in the context of other, concurrently operating behavioral processes.     

Short-term memory in the pigeon: the previously reinforced response

Eighteen pigeons served in a discrete-trials short-term memory experiment in which the reinforcement probability for a peck on one of two keys depended on the response reinforced on the previous trial: either the probability of reinforcement on a trial was 0.8 for the same response reinforced on the previous trial and was 0.2 for the other response (Group A), or, it was 0 or 0.2 for the same response and 1.0 or 0.8 for the other response (Group B). A correction procedure ensured that over all trials reinforcement was distributed equally across the left and right keys. The optimal strategy was either a winstay, lose-shift strategy (Group A) or a win-shift, lose-stay strategy (Group B). The retention interval, that is the intertrial interval, was varied. The average probability of choosing the optimal alternative reinforced 80% of the time was 0.96, 0.84, and 0.74 after delays of 2.5, 4.0, and 6.0 sec, respectively for Group A, and was 0.87, 0.81, and 0.55 after delays of 2.5, 4.0, and 6.0 sec, respectively, for Group B. This outcome is consistent with the view that behavior approximated the optimal response strategy but only to an extent permitted by a subject's short-term memory for the cue correlated with reinforcement, that is, its own most-recently reinforced response. More generally, this result is consistent with “molecular” analyses of operant behavior, but is inconsistent with traditional “molar” analyses holding that fundamental controlling relations may be discovered by routinely averaging over different local reinforcement contingencies. In the present experiment, the molar results were byproducts of local reinforcement contingencies involving an organism's own recent behavior.     

The dynamics of operant conditioning

Existing models of operant learning are relatively insensitive to historical properties of behavior and applicable to only limited data sets. This article proposes a minimal set of principles based on short-term and long-term memory mechanisms that can explain the major static and dynamic properties of operant behavior in both single-choice and multiresponse situations. The critical features of the theory are as follows: (a) The key property of conditioning is assessment of the degree of association between responses and reinforcement and between stimuli and reinforcement; (b) the contingent reinforcement is represented by learning expectancy, which is the combined prediction of response-reinforcement and stimulus-reinforcement associations; (c) the operant response is controlled by the interplay between facilitatory and suppressive variables that integrate differences between expected (long-term) and experienced (short-term) events; and (d) very-long-term effects are encoded by a consolidated memory that is sensitive to the entire reinforcement history. The model predicts the major qualitative features of operant phenomena and then suggests an experimental test of theoretical predictions about the joint effects of reinforcement probability and amount of training on operant choice. We hypothesize that the set of elementary principles that we propose may help resolve the long-standing debate about the fundamental variables controlling operant conditioning.     

Sub-Optimal Choice by Pigeons: Failure to Support The Allais Paradox

Pigeons show a preference for an alternative that provides them with discriminative stimuli (sometimes a stimulus that predicts reinforcement and at other times a stimulus that predicts the absence of reinforcement) over an alternative that provides them with nondiscriminative stimuli, even if the nondiscriminative stimulus alternative is associated with 2.5 times as much reinforcement (Stagner & Zentall, 2010). In Experiment 1 we found that the delay to reinforcement associated with the nondiscriminative stimuli could be reduced by almost one half before the pigeons were indifferent between the two alternatives. In Experiment 2 we tested the hypothesis that the preference for the discriminative stimulus alternative resulted from the fact that, like humans, the pigeons were attracted by the stimulus that consistently predicted reinforcement (the Allais paradox). When the probability of reinforcement associated with the discriminative stimulus that predicted reinforcement was reduced from 100% to 80% the pigeons still showed a strong preference for the discriminative stimulus alternative. Thus, under these conditions, the Allais paradox cannot account for the sub-optimal choice behavior shown by pigeons. Instead we propose that sub-optimal choice results from positive contrast between the low expectation of reinforcement associated with the discriminative stimulus alternative and the much higher obtained reinforcement when the stimulus associated with reinforcement appears. We propose that similar processes can account for sub-optimal gambling behavior by humans.     

Stimulus–reinforcer relations established during training determine resistance to extinction and relapse via reinstatement

The baseline rate of a reinforced target response decreases with the availability of response‐independent sources of alternative reinforcement; however, resistance to disruption and relapse increases. Because many behavioral treatments for problem behavior include response‐dependent reinforcement of alternative behavior, the present study assessed whether response‐dependent alternative reinforcement also decreases baseline response rates but increases resistance to extinction and relapse. We reinforced target responding at equal rates across two components of a multiple schedule with pigeons. We compared resistance to extinction and relapse via reinstatement of (1) a target response trained concurrently with a reinforced alternative response in one component with (2) a target response trained either concurrently or in separate components from the alternative response across conditions. Target response rates trained alone in baseline were higher but resistance to extinction and relapse via reinstatement tests were greater after training concurrently with the alternative response. In another assessment, training target and alternative responding together, but separating them during extinction and reinstatement tests, produced equal resistance to extinction and relapse. Together, these findings are consistent with behavioral momentum theory—operant response–reinforcer relations determined baseline response rates but Pavlovian stimulus–reinforcer relations established during training determined resistance to extinction and relapse. These findings imply that reinforcing alternative behavior to treat problem behavior could initially reduce rates but increase persistence.     

The development of functional response units: the role of demarcating stimuli.

An experiment with rats examined the roles of demarcating stimuli and differential reinforcement probability on the development of functional response units. It examined the development of units in a probabilistic, free-operant situation in which the presence of demarcating stimuli was manipulated. In all conditions, behavior became organized into two-response sequences framed by changes in local reinforcement probability. A tone demarcating the beginning and end of contingent response sequences facilitated the development of functional response units, as in chunking, but the same units developed slowly in the absence of the tone. Complex functional response units developed even though reinforcement contigencies remained constant. These findings demonstrate that models of operant learning must include a mechanism for changing the response unit as a function of reinforcement history. Markov models may seem to be a natural technique for modeling response sequences because of their ability to predict individual responses as a function of reinforcement history; however, no class of Markov chain can incorporate changing response units in their predictions.     

Undermatching and overmatching as deviations from the matching law

A model of performance under concurrent variable-interval reinforcement schedules that takes as its starting point the hypothetical “burst” structure of operant responding is presented. Undermatching and overmatching are derived from two separate, and opposing, tendencies. The first is a tendency to allocate a certain proportion of response bursts randomly to a response alternative without regard for the rate of reinforcement it provides, others being allocated according to the simple matching law. This produces undermatching. The second is a tendency to prolong response bursts that have a high probability of initiation relative to those for which initiation probability is lower. This process produces overmatching. A model embodying both tendencies predicts (1) that undermatching will be more common than overmatching, (2) that overmatching, when it occurs, will tend to be of limited extent. Both predictions are consistent with available data. The model thus accounts for undermatching and overmatching deviations from the matching law in terms of additional processes added on to behavior allocation obeying the simple matching relation. Such a model thus enables processes that have been hypothesized to underlie matching, such as some type of reinforcement rate or probability optimization, to remain as explanatory mechanisms even though the simple matching law may not generally be obeyed.     

Predator videos and electric shock function as punishers for zebrafish (Danio rerio)

Zebrafish (Danio rerio) are a promising animal model for studying the effects of gene–environment interactions on behavior. Two experiments were conducted to assess punishment effects of presenting predator videos (Indian leaf fish; Nandus nandus) and electric shock on operant approach responses in zebrafish. In Experiment 1, the predator video and shock stimuli were presented upon a response maintained by a single variable‐interval schedule of food reinforcement in different groups of fish. In Experiment 2, the predator video and shock stimuli were presented upon one of two response alternatives maintain by concurrently available variable‐interval schedules of food reinforcement in different groups of fish. Responding decreased when the predator video and shock stimuli were presented relative to their absence in both experiments. Moreover, responding on an unpunished alternative did not reliably decrease in Experiment 2. These results indicate that the decrease in responding resulted from the punishment contingency rather than from elicited species‐specific defense responses or conditioned avoidance. Thus, the predator video and electric shock functioned as punishers of operant behavior for zebrafish. Identifying punishers for this species could lead to research on how gene–environment interactions influence individual differences in sensitivity to punishment.     

Variation, repetition, and choice

Experiment 1 investigated the controlling properties of variability contingencies on choice between repeated and variable responding. Pigeons were exposed to concurrent-chains schedules with two alternatives. In the REPEAT alternative, reinforcers in the terminal link depended on a single sequence of four responses. In the VARY alternative, a response sequence in the terminal link was reinforced only if it differed from the n previous sequences (lag criterion). The REPEAT contingency generated low, constant levels of sequence variation whereas the VARY contingency produced levels of sequence variation that increased with the lag criterion. Preference for the REPEAT alternative tended to increase directly with the degree of variation required for reinforcement. Experiment 2 examined the potential confounding effects in Experiment 1 of immediacy of reinforcement by yoking the interreinforcer intervals in the REPEAT alternative to those in the VARY alternative. Again, preference for REPEAT was a function of the lag criterion. Choice between varying and repeating behavior is discussed with respect to obtained behavioral variability, probability of reinforcement, delay of reinforcement, and switching within a sequence.     

Differences in the effect of Pavlovian contingencies upon behavioral momentum using auditory versus visual stimuli.

We examined the role of Pavlovian and operant relations in behavioral momentum by arranging response-contingent alternative reinforcement in one component of a three-component multiple concurrent schedule with rats. This permitted the simultaneous arranging of different response-reinforcer (operant) and stimulus-reinforcer (Pavlovian) contingencies during three baseline conditions. Auditory or visual stimuli were used as discriminative stimuli within the multiple concurrent schedules. Resistance to change of a target response was assessed during a single session of extinction following each baseline condition. The rate of the target response during baseline varied inversely with the rate of response-contingent reinforcement derived from a concurrent source, regardless of whether the discriminative stimuli were auditory or visual. Resistance to change of the target response, however, did depend on the discriminative-stimulus modality. Resistance to change in the presence of visual stimuli was a positive function of the Pavlovian contingencies, whereas resistance to change was unrelated to either the operant or Pavlovian contingencies when the discriminative stimuli were auditory. Stimulus salience may be a factor in determining the differences in resistance to change across sensory modalities.     

Human behavioral momentum in a sample of older adults.

Behavioral momentum, the persistence of behavior under altered environmental contingencies, is derived from Newtonian physics and operant psychology. It has relevance to behavior analysis in terms of shaping strong behaviors and ensuring effective relapse prevention strategies in behavior modification and therapy. The authors investigated whether changing the operant schedule contingencies affects the responses of older humans to different stimuli when reinforcement density is systematically manipulated. Fifteen older adults participated in a computer study in which each of 2 keys in a baseline condition was associated with the same schedule of reinforcement and multiple variable intervals; the only difference was that 1 reinforcer was 10 times larger than the other. After 6 sessions, the authors changed the contingency schedule to either an extinction condition, a variable-time schedule, or a different variable-interval schedule, to assess how participants' responses persisted when reinforcement contingencies were systematically changed. The results were consistent with the predictions of behavioral momentum. The participants not only biased their responses in favor of the more densely reinforcing key, but when contingencies changed, they showed significantly biased responses. Results supported the conclusion that healthy older adults allocate their behaviors in a manner very sensitive to training stimuli conditions; consistent with the basic principles of behavioral momentum, they show a degree of resistance to change in their behaviors when the behavioral contingencies are altered.     

Rethinking reinforcement: allocation, induction, and contingency

The concept of reinforcement is at least incomplete and almost certainly incorrect. An alternative way of organizing our understanding of behavior may be built around three concepts: allocation, induction, and correlation. Allocation is the measure of behavior and captures the centrality of choice: All behavior entails choice and consists of choice. Allocation changes as a result of induction and correlation. The term induction covers phenomena such as adjunctive, interim, and terminal behavior-behavior induced in a situation by occurrence of food or another Phylogenetically Important Event (PIE) in that situation. Induction resembles stimulus control in that no one-to-one relation exists between induced behavior and the inducing event. If one allowed that some stimulus control were the result of phylogeny, then induction and stimulus control would be identical, and a PIE would resemble a discriminative stimulus. Much evidence supports the idea that a PIE induces all PIE-related activities. Research also supports the idea that stimuli correlated with PIEs become PIE-related conditional inducers. Contingencies create correlations between "operant" activity (e.g., lever pressing) and PIEs (e.g., food). Once an activity has become PIE-related, the PIE induces it along with other PIE-related activities. Contingencies also constrain possible performances. These constraints specify feedback functions, which explain phenomena such as the higher response rates on ratio schedules in comparison with interval schedules. Allocations that include a lot of operant activity are "selected" only in the sense that they generate more frequent occurrence of the PIE within the constraints of the situation; contingency and induction do the "selecting."     

Male versus female Siamese fighting fish as reinforcing stimuli for conspecific males in single- and two-choice operant situations

Two experiments attempted to determine the reinforcing effectiveness of visual exposure to a female versus a male stimulus in male Siamese fighting fish (Betta splendens). This was accomplished by making male or female presentation contingent upon an operant ring swimming response in single- (Experiment 1) and two-choice (Experiment 2) continuous reinforcement operant situations for 1 hr per day. It was found that visual exposure to a female stimulus was behaviorally reinforcing in a way comparable to that typically found in intermale studies. If given a choice, male test subjects were observed to demonstrate a consistent preference for the female as opposed to the male stimulus. The results are discussed in terms of stimulus discrimination and approach-avoidance conflict as differentially elicited by male and female target stimuli.     

Further experiments on probability-matching in the pigeon

In a discrete-trials, two-key choice situation, probability-learning by pigeons was studied under a variety of training conditions. Matching was found in simultaneous and in successive problems, but a spatial problem produced only maximizing. In the simultaneous problem, noncorrection produced maximizing, while correction produced matching. Guidance produced maximizing when the animals were required to earn each opportunity for choice by pecking a center key on FR-5, but matching when the center key was not used. In a discrete-trials one-key situation, with latency as the measure, frequency and probability of reinforcement were varied independently. Differences in probability produced differences in latency of response, but differences in frequency did not.     

Timing in choice experiments

In Experiment 1, pigeons chose between variable- and fixed-interval schedules. The timer for 1 schedule was reset by a reinforcement on that schedule or on either schedule. In both cases, the pigeons timed reinforcement on each schedule from trial onset. The data further suggest that their behavior reflects 2 independent processes: 1 deciding when a response should be emitted and responsible for the timing of the overall activity, and the other determining what this response should be and responsible for the allocation of behavior between the 2 response keys. Results from Experiment 2, which studied choice between 2 fixed-interval schedules, support those 2 conclusions. These results have implications for the study of operant choice in general.     

Treatment of resistance to change in children with autism

“Resistance to change” represents a core symptom of autism that we conceptualized and assessed as resulting in part due to factors known to govern free‐operant choice. During a free‐choice baseline, participants chose between problematic, resistive responses and an appropriate alternative response. During the asymmetrical‐choice condition, we delivered their most highly preferred item if the participant chose the alternative response (i.e., differential reinforcement of alternative behavior [DRA]). During the guided‐ (Experiment 1) and singular‐ (Experiment 2) choice conditions, we prompted participants to choose the alternative response and then delivered their most highly preferred item (i.e., DRA with escape extinction). All participants learned to tolerate (Experiment 1) or choose (Experiment 2) the alternative response when we combined DRA with escape extinction. After exposure to escape extinction, two participants showed strong maintenance effects with DRA alone. We discuss these finding relative to the effects of DRA and escape extinction on resistance to change.     

Response elimination: A comparison of four procedures

Four procedures for eliminating an operant response were compared within a multiple schedule. Response reduction was most rapid when reinforcement was provided for a specific alternative response. The decline in responding produced by extinction and differential reinforcement of other behavior was similar. Responding initially increased and then gradually decreased when reinforcers were delivered independently of responding at fixed times. Removal of reinforcement from the alternative-response and DRO procedures did not result in recovery of the target response. However, the shift from response-independent dilivery of reinforcers to extinction caused an initial increase in responding.     

Operant conditioning of behavioral variability using a percentile reinforcement schedule.

The present investigation developed and tested a new percentile reinforcement schedule suited to study pattern variability, whose main feature was the relative dissociation it provided between the variability requirement defining criterional responses and overall probability of reinforcement. In a discrete-trials procedure, pigeons produced patterns of four pecks on two response keys. If the pattern emitted on the current trial differed from the N preceding patterns, reinforcement was delivered with probability mu. The schedule continuously adjusted the criterion N such that the probability of a criterional response, estimated from the subject's recent behavior, was always constant. In these circumstances, the criterion corresponded to an invariant percentile in the distribution of recent responses. Using a between-subjects design, Experiment 1 manipulated the variability requirement--the percentile--while keeping overall reinforcement probability constant. The degree of variability varied directly with the requirement. In addition, an inverse relationship existed between the requirement and within-group variance. Experiment 2 manipulated probability of reinforcement while maintaining the variability requirement constant. No consistent relationship was found between variability and reinforcement probability. A tentative hypothesis was advanced ascribing the operant conditioning of behavioral variability to a process of probability-dependent selection.     

Probability learning and Piagetian probability conceptions in children 5 to 12 years old

This study focused on the relations between performance on a three-choice probability-learning task and conceptions of probability as outlined by Piaget concerning mixture, normal distribution, random selection, odds estimation, and permutations. The probability-learning task and four Piagetian tasks were administered randomly to 100 male and 100 female, middle SES, average IQ children in three age groups (5 to 6, 8 to 9, and 11 to 12 years old) from different schools. Half the children were from Middle Eastern backgrounds, and half were from European or American backgrounds. As predicted, developmental level of probability thinking was related to performance on the probability-learning task. The more advanced the child's probability thinking, the higher his or her level of maximization and hypothesis formulation and testing and the lower his or her level of systematically patterned responses. The results suggest that the probability-learning and Piagetian tasks assess similar cognitive skills and that performance on the probability-learning task reflects a variety of probability concepts.     

Rate of conditioned reinforcement affects observing rate but not resistance to change

The effects of rate of conditioned reinforcement on the resistance to change of operant behavior have not been examined. In addition, the effects of rate of conditioned reinforcement on the rate of observing have not been adequately examined. In two experiments, a multiple schedule of observing-response procedures was used to examine the effects of rate of conditioned reinforcement on observing rates and resistance to change. In a rich component, observing responses produced a higher frequency of stimuli correlated with alternating periods of random-interval schedule primary reinforcement or extinction. In a lean component, observing responses produced similar schedule-correlated stimuli but at a lower frequency. The rate of primary reinforcement in both components was the same. In Experiment 1, a 4:1 ratio of stimulus production was arranged by the rich and lean components. In Experiment 2, the ratio of stimulus production rates was increased to 6:1. In both experiments, observing rates were higher in the rich component than in the lean component. Disruptions in observing produced by presession feeding, extinction of observing responses, and response-independent food deliveries during intercomponent intervals usually were similar in the rich and lean components. When differences in resistance to change did occur, observing tended to be more resistant to change in the lean component. If resistance to change is accepted as a more appropriate measure of response strength than absolute response rates, then the present results provide no evidence that higher rates of stimuli generally considered to function as conditioned reinforcers engender greater response strength.