Performances on ratio and interval schedules of reinforcement: Data and theory

Two differences between ratio and interval performance are well known: (a) Higher rates occur on ratio schedules, and (b) ratio schedules are unable to maintain responding at low rates of reinforcement (ratio “strain”). A third phenomenon, a downturn in response rate at the highest rates of reinforcement, is well documented for ratio schedules and is predicted for interval schedules. Pigeons were exposed to multiple variable-ratio variable-interval schedules in which the intervals generated in the variable-ratio component were programmed in the variable-interval component, thereby “yoking” or approximately matching reinforcement in the two components. The full range of ratio performances was studied, from strained to continuous reinforcement. In addition to the expected phenomena, a new phenomenon was observed: an upturn in variable-interval response rate in the midrange of rates of reinforcement that brought response rates on the two schedules to equality before the downturn at the highest rates of reinforcement. When the average response rate was corrected by eliminating pausing after reinforcement, the downturn in response rate vanished, leaving a strictly monotonic performance curve. This apparent functional independence of the postreinforcement pause and the qualitative shift in response implied by the upturn in variable-interval response rate suggest that theoretical accounts will require thinking of behavior as partitioned among at least three categories, and probably four: postreinforcement activity, other unprogrammed activity, ratio-typical operant behavior, and interval-typical operant behavior.     

Choice for aperiodic versus periodic ratio schedules: A comparison of concurrent and concurrent-chain procedures

Choice between mixed-ratio schedules, consisting of equiprobable ratios of 1 and 99 responses per reinforcement, and fixed-ratio schedules of food reinforcement was assessed by two commonly used procedures: concurrent schedules and concurrent-chains schedules. Rats were trained under concurrent fixed-ratio mixed-ratio schedules, in which both ratio schedules were simultaneously available, and under a concurrent-chains schedule, in which access to one of the mutually exclusive ratio schedules comprising the terminal links was contingent on a single “choice” response. The distribution of responses between the two ratio schedules was taken as the choice proportion under the concurrent procedure, and the distribution of “choice” responses was taken as the choice proportion under the concurrent-chains procedure. Seven of eight rats displayed systematic choice; of those, each displayed nearly exclusive choice for fixed-ratio 35 to the mixed-ratio schedule under the concurrent procedure, but each displayed nearly exclusive choice for the mixed-ratio schedule to fixed-ratio 35 under the concurrent-chains procedure. Thus, preference for a fixed or a mixed schedule of reinforcement depended on the procedure used to assess preference.     

Schedule-induced mirror responding in the pigeon

Two pigeons that were previously exposed to a multiple schedule of reinforcement in the presence of a stuffed and a live pigeon, and two of three naive pigeons, responded on a mirror during exposure to multiple fixed-ratio, fixed-ratio schedules of reinforcement for key pecking. Both the topography and temporal pattern of mirror responding were comparable to schedule-induced “attack” on live and stuffed targets. Rate of target responding was reduced when either the mirror was covered with paper or when the multiple schedule was removed. A reversal in the relationship between reinforcement schedules and discriminative stimuli demonstrated that mirror responding was controlled by the stimulus correlated with the higher fixed-ratio schedule. With one component of the multiple schedule held constant at fixed ratio 25 and the ratio requirement of the other component varying from 25 to 150, there was an inverted U-shaped relationship between rate of mirror responding and fixed-ratio schedule in the varied component. As in Flory's study (1969b) there was an inverted U-shaped relationship between target responding and inter-food intervals. The combined results of these studies suggest that the relationship between rate of target responding and reinforcement schedules is controlled primarily by the inter-food intervals resulting from the schedules.     

Conditioning of within-trial patterns of key pecking in pigeons

The possibility of conditioning systematic patterns of responding during brief discrete trials was studied by requiring hungry pigeons to key peck and then pause or to pause and then key peck in order to gain access to food. These schedules were highly effective in promoting decelerated and accelerated rates of responding, respectively, within individual trials; indeed, performance was quite similar to that observed when explicit external stimuli were correlated with “peck” and “pause” portions of the daily trials. Finally, schedules of reinforcement that did not selectively reinforce peck-pause or pause-peck patterns neither generated these patterns nor maintained them at the previous high levels. The results, therefore, confirm Shimp's (1976) proposal that organized groupings of discrete responses may function as operants—even in the absence of strict response-reinforcer contiguity.     

Reinforcement by an imprinting stimulus versus water on simple schedules in ducklings

Ducklings (5 to 28 days old) were trained to peck a pole on fixed-ratio, fixed-interval, and multiple schedules using brief presentation of an imprinting stimulus as the response-contingent event. Other ducklings of the same age were trained similarly except that reinforcement consisted of access to water. With water reinforcement the typical fixed-ratio (“break-run”), fixed-interval (“scallop”), and multiple schedule response patterns were readily established and consistently maintained. With the imprinting stimulus these schedule effects were inconsistent in some subjects and virtually nonexistent in others, despite extended training. Schedule control with the imprinting stimulus was not improved by the use of a reinforcement signaling procedure which enhances responding reinforced by electrical brain stimulation on intermittent schedules. However, the overall rates of responding and the extinction functions generated after reinforcement with water versus the imprinting stimulus were comparable. These findings imply that control by temporal and discriminative stimuli may be relatively weak when a young organism's behavior is reinforced by presentation of an imprinting stimulus.     

Choice: Effects of changeover schedules on concurrent performance

The components of concurrent schedules were separated temporally by placing interval schedules on the changeover key. The rates of responding on both the main and changeover keys were examined as a function of the reinforcement rates. In the first experiment, the sensitivity of main-key performance to changing reinforcement rates was inversely related to the temporal separation of components, and changeover performance was monotonically related to the ratio of the reinforcement rates. In the second experiment, when the ratio of the reinforcement rates was scheduled to remain constant while the frequency of reinforcement was varied, changeover performance did not remain constant. A “sampling” interpretation of changeover responding was proposed and subsequently tested in a third experiment where extinction was always scheduled in one component and the frequency of reinforcement was varied in the second component. It was concluded that changeover performance can be interpreted using molar measures of reinforcement and that animals sample activities available to them at rates which are controlled by relative reinforcement rates.     

Development of a complex multiple schedule in the chimpanzee

The development of chimpanzee behavior on a four-component, three-lever multiple schedule is described. Component schedules included the Sidman avoidance procedure with a concurrent discriminated avoidance schedule on a second lever, fixed ratio performance for food, differential reinforcement of low rate for water requiring a dual response chain, and a symbol discrimination task for continuous food reinforcement using three levers. The avoidance component of this schedule was employed during the January 31, 1961 suborbital space flight of the chimpanzee “Ham.” On November 29, 1961, the chimpanzee “Enos” performed on the multiple schedule during three orbits around the earth in a Mercury capsule.     

Competitive fixed-interval performance in humans

Two persons responded in the same session in separate cubicles, but under a single schedule of reinforcement. Each time reinforcement was programmed, only the first response to occur, that is, the response of only one of the subjects, was reinforced. “Competitive” behavior that developed under these conditions was examined in three experiments. In Experiment 1 subjects responded under fixed-interval (FI) 30-s, 60-s, and 90-s schedules of reinforcement. Under the competition condition, relative to baseline conditions, the response rates were higher and the pattern was “break-and-run.” In Experiment 2, subjects were exposed first to a conventional FI schedule and then to an FI competition schedule. Next, they were trained to respond under either a differential-reinforcement-of-low-rate (DRL) or fixed-ratio (FR) schedule, and finally, the initial FI competition condition was reinstated. In this second exposure to the FI competition procedure, DRL subjects responded at lower rates than were emitted during the initial exposure to that condition and FR subjects responded at higher rates. For all subjects, however, responding gradually returned to the break-and-run pattern that had occurred during the first FI competition condition. Experiment 3 assessed potential variables contributing to the effects of the competitive FI contingencies during Experiments 1 and 2. Subjects were exposed to FI schedules where (a) probability of reinforcement at completion of each fixed interval was varied, or (b) a limited hold was in effect for reinforcement. Only under the limited hold was responding similar to that observed in previous experiments.     

Responding under chained and tandem fixed-ratio schedules

The role of stimuli in chained fixed-ratio schedules of reinforcement was examined. At various ratio values, responding on schedules consisting of three or five equal components, with a different colored light in each component (“block counter”) was compared with responding on tandem or simple fixed-ratio schedules having the same color present throughout the entire ratio. At all ratio values except the smallest, the chain stimuli resulted in longer pauses after reinforcement. The magnitude of this effect became greater as the size of the ratio was increased. Post-reinforcement pause durations were longer under five-component schedules than under three-component schedules. Running rates in the first component were lower on the chained schedules than on the tandem schedules; on both kinds of schedule, rates were lower in the first component than in the rest of the ratio. When the sequence of stimuli was reversed, the duration of the post-reinforcement pause dropped markedly and the running rate in the initial component increased, but these effects gradually disappeared after the first reversal session. When the final chain stimulus was substituted for the first component stimulus but continued to appear in the final chain component as well, the pause duration dropped and remained at this lower level during subsequent sessions.     

The matching law in and within groups of rats

In each of the two experiments, a group of five rats lived in a complex maze containing four small single-lever operant chambers. In two of these chambers, food was available on variable-interval schedules of reinforcement. In Experiment I, nine combinations of variable intervals were used, and the aggregate lever-pressing rates (by the five rats together) were studied. The log ratio of the rates in the two chambers was linearly related to the log ratio of the reinforcement rates in them; this is an instance of Herrnstein's matching law, as generalized by Baum. Summing over the two food chambers, food consumption decreased, and response output increased, as the time required to earn each pellet increased. In Experiment II, the behavior of individual rats was observed by time-sampling on selected days, while different variable-interval schedules were arranged in the two chambers where food was available. Individual lever-pressing rates for the rats were obtained, and their median bore the same “matching” relationship to the reinforcement rates as the group aggregate in Experiment I. There were differences between the rats in their distribution of time and responses between the two food chambers; these differences were correlated with differences in the proportions of reinforcements the rats obtained from each chamber.     

Contrast and reallocation of extraneous reinforcers between multiple-schedule components

Four pigeons responded in components of multiple schedules in which two responses were available and reinforced with food. Pecks on the left key (“main” key) were reinforced at a constant rate in one component and at a rate that varied over conditions in the other component. When reinforcer rate was varied, behavioral contrast occurred in the constant component. On the right key (“extra” key), five variable-interval schedules and one variable-ratio schedule, presented conjointly, arranged reinforcers for responses in all conditions. These conjoint schedules were common to both multiple-schedule components—rather than unique to particular components—and reinforcers from these schedules could therefore be arranged in one component and obtained during the other component. In this way, the additional reinforcers were analogous to the “extraneous” reinforcers thought to maintain behavior other than pecking in conventional multiple schedules. Response rate on the extra key did not change systematically over conditions in the constant component, and in the varied component extra responding was inversely related to main-key reinforcement. All subjects obtained more extra-key reinforcers in whichever component arranged fewer main-key reinforcers. Consistent with the theory that reallocation of extraneous reinforcers may cause behavioral contrast, absolute reinforcer rate for the extra key in the constant component was low in conditions that produced positive contrast on the main key and high in those that produced negative contrast. Also consistent with this theory, behavioral contrast was reduced in two conditions that canceled extra-key reinforcers that had been arranged but not obtained at the end of components. Thus, a constraint on reallocation markedly reduced the extent of contrast.     

What 50 years of research tell us about pausing under ratio schedules of reinforcement

Textbooks in learning and behavior commonly describe performance on fixed-ratio schedules as “break and run,” indicating that after reinforcement subjects typically pause and then respond quickly to the next reinforcement. Performance on variable-ratio schedules, on the other hand, is described as steady and fast, with few long pauses. Beginning with Ferster and Skinner''s magnum opus, Schedules of Reinforcement (1957), the literature on pausing under ratio schedules has identified the influences on pausing of numerous important variables, in particular ratio size and reinforcement magnitude. As a result, some previously held assumptions have been called into question. For example, research has shown that the length of the pause is controlled not only by the preceding ratio, as Ferster and Skinner and others had assumed (and as implied by the phrase postreinforcement pause), but by the upcoming ratio as well. Similarly, despite the commonly held belief that ratio pausing is unique to the fixed-ratio schedule, there is evidence that pausing also occurs under variable-ratio schedules. If such widely held beliefs are incorrect, then what about other assumptions? This article selectively examines the literature on pausing under ratio schedules over the past 50 years and concludes that although there may indeed be some common patterns, there are also inconsistencies that await future resolution. Several accounts of pausing under ratio schedules are discussed along with the implications of the literature for human performances, most notably the behaviors termed procrastination.     

Pictorial target control of schedule-induced attack in White Carneaux pigeons

Three pigeons with a history of attacking a mirror target, and two of six pigeons with no prior exposure to targets, attacked a colored photograph of a conspecific during exposure to intermittent schedules of reinforcement for key pecking. Rate of attack on the photograph decreased when the reinforcement schedule was removed. The topography, temporal pattern, and locus of attack on the picture were comparable to schedule-induced attack on live, stuffed, and mirror targets. When silhouette, outline, and plain paper targets were used, schedule-induced attack was more sensitive to a change in target characteristics with a concurrent target-preference procedure than with an analogous successive-testing procedure. The combined results of the two testing procedures indicated that an “upright” white-on-black silhouette of a pigeon with or without an eye was more effective in controlling attack than was a comparable “inverted” silhouette, an outline of a pigeon, or a piece of colored paper.     

Conditioned reinforcement and discrimination in second-order schedules

Pigeons were exposed to multiple second-order schedules in which responding on the “main key” was reinforced according to either a variable-interval or fixed-interval schedule by production of a brief stimulus on the “brief-stimulus key”. A response was required to the brief stimulus during its fourth (final) presentation to produce food; responses to the earlier brief stimuli indicated the extent to which the final brief stimulus was discriminated from preceding ones. Main-key response rates were higher in early components of paired brief-stimulus schedules, in which each brief stimulus was the same as that paired with reinforcement, than in comparable unpaired brief-stimulus or tandem schedules. Poor discrimination occurred between paired brief stimuli (Experiment I). When chain stimuli on the main key induced a discrimination between the first two and second two brief stimuli, the response-rate enhancement in the paired brief-stimulus schedule persisted (Experiment II). Rate enhancement diminished when the initial link of the chain included the first three components (Experiment IV). Eliminating the contingency between responding and brief-stimulus production also diminished rate enhancement (Experiment III). The results show that the discriminative and conditioned reinforcing effects of food-paired brief stimuli may be selectively manipulated and suggest that the reinforcing effects are modulated by other reinforcers in the situation.     

On Herrnstein's equation and related forms

In 1970, Herrnstein proposed a simple equation to describe the relation between response and reinforcement rates on interval schedules. Its empirical basis is firm, but its theoretical foundation is still uncertain. Two approaches to the derivation of Herrnstein's equation are discussed. It can be derived as the equilibrium solution to a process model equivalent to familiar linear-operator learning models. Modifications of this approach yield competing power-function formulations. The equation can also be derived from the assumption that response strength is proportional to reinforcement rate, given that there is a ceiling on response rate. The proportional relation can, in turn, be derived from a threshold assumption equivalent to Shimp's “momentary maximizing”. This derivation implies that the two parameters of Herrnstein's equation should be correlated, and may explain its special utility in application to internal schedules.     

Operant and nonoperant vocal responding in the mynah: Complex schedule control and deprivation-induced responding

Several recent studies have been concerned with operant responses that are also affected by nonoperant factors, (e.g., biological constraints, innate behavior patterns, respondent processes). The major reason for studying mynah vocal responding concerned the special relation of avian vocalizations to nonoperant emotional and reflexive systems. The research strategy was to evaluate operant and nonoperant control by comparing the schedule control obtained with the vocal response to that characteristic of the motor responses of other animals. We selected single, multiple, and chain schedules that ordinarily produce disparate response rates at predictable times. In multiple schedules with one component where vocal responding (“Awk”) was reinforced with food (fixed-ratio or fixed-interval schedule) and one where the absence of vocal responding was reinforced (differential reinforcement of other behavior), response rates never exceeded 15 responses per minute, but clear schedule differences developed in response rate and pause time. Nonoperant vocal responding was evident when responding endured across 50 extinction sessions at 25% to 40% of the rate during reinforcement. The “enduring extinction responding” was largely deprivation induced, because the operant-level of naive mynahs under food deprivation was comparable in magnitude, but without deprivation the operant level was much lower. Food deprivation can induce vocal responding, but the relatively precise schedule control indicated that operant contingencies predominate when they are introduced.     

Stimulus-reinforcer contingencies and local behavioral contrast

Four pigeons were exposed to a series of multiple schedules of variable-interval reinforcement in which pecks were required on one key (operant key) and components were signalled on a second key (signal key). Four additional pigeons experienced identical conditions, except that a yoking procedure delivered food on variable-time schedules, with no key pecks required. One of the components of the multiple schedule was constant throughout the experiment as a variable-interval (or variable-time) 30-second schedule. Operant-key responding during the constant component was uniform throughout the component, uninfluenced by changes in the duration of the variable component, and only slightly influenced by changes in reinforcement frequency correlated with the variable component. By comparison, signal-key response rate during the constant component was highest at the onset of the component, was higher when the variable component was 60-sec long than when it was 1-sec long, and was higher when no reinforcement occurred in the variable component than when reinforcement was scheduled in the variable component. These characteristics of signal-key pecking matched characteristics of local positive behavioral contrast. These data are taken to support the “additivity theory” of behavioral contrast and to suggest that Pavlovian stimulus-reinforcer relations contribute primarily to the phenomenon of local positive contrast.     

College students' responding to and rating of contingency relations: The role of temporal contiguity

Two experiments investigated the role of temporal contiguity in college students' responding to and rating of contingency relations during operant conditioning. Schedules were devised that determined when but not whether appetitive or aversive events would occur. Subjects' reports concerning the schedules were obtained by means of a 200-point rating scale, anchored by the phrases “prevents the light from occurring” (−100) and “causes the light to occur” (+100). When tapping a telegraph key advanced the time of point gain, responding was maintained or increased and subjects gave positive ratings. When tapping a telegraph key advanced the time of point loss, subjects also gave positive ratings, but responding now decreased. When key tapping delayed the time of point gain, responding decreased and subjects gave negative ratings. When key tapping delayed the time of point loss, subjects also gave negative ratings, but responding now increased. These findings implicate response-outcome contiguity as an important contributor to causal perception and to reinforcement and punishment effects. Other accounts—such as those stressing the local probabilistic relation between response and outcome or the molar correlation between response rate and outcome rate—were seen to be less preferred interpretations of these and other results.     

Response strength in multiple periodic and aperiodic schedules

Responding in multiple periodic and aperiodic schedules of equal mean reinforcement rate was examined during extinction, satiation, and in the presence of various free-food schedules. In Experiments I and II, pigeons were trained on multiple variable-interval–fixed-interval schedules. Decreases in the rate of responding due to extinction, satiation, or food schedules were approximately equal regardless of the temporal pattern of reinforcer presentation. In Experiment III, pigeons responded on a two-component multiple schedule in which each component was a two-member homogeneous response chain terminating in a fixed-interval schedule during one component and in a variable-interval schedule during the other. The length of both terminal links was varied over a series of conditions. Initial-link responding in the fixed-interval component was reduced more by increasing terminal-link length than was initial-link responding in the variable-interval component. However, no differences in resistance to satiation and extinction were obtained across the fixed and variable components. If the relative decrease in responding produced by satiation and extinction is used as an index of the “value” of the conditions maintaining responding, then these data suggest that fixed and variable schedules of equal mean length are equally valued. This conclusion, however, is not consistent with findings of preference for variable over fixed schedules obtained in studies using concurrent-chain procedures.     

Towards an empirical calculus of reinforcement value

Only one of two keys reinforces the subject with food. This key can assume one of two colors, each associated with a different fixed-ratio schedule for obtaining reinforcement. The function of the second key is to permit the animal to switch from the long schedule to the short schedule. If the difference between the ratio schedules is large enough, a preference for the shorter schedule is demonstrable. A quantitative index of preference is obtained as follows: each time the animal switches to the shorter schedule, the number of pecks required to produce the next switch is increased. As the “ante” on the switching key increases, the effective difference between the two ratio schedules decreases. After each food reinforcement, when the bird is exposed to the choice-situation, it takes longer before the bird switches again. This is used to “titrate” the bird's preference. If it does not switch within x sec, the progressively increasing ratio schedule of the switching key is decreased. A specific value, in terms of a rather specific number of responses the bird settles at on the choice key, is obtained. This equilibrium is employed as a dependent variable. Several variables of which it is a function are explored.