Second-order schedules of token reinforcement: comparisons of performance under fixed-ratio and variable-ratio exchange schedules

Rats' lever pressing produced tokens according to a 20-response fixed-ratio schedule. Sequences of token schedules were reinforced under a second-order schedule by presentation of periods when tokens could be exchanged for food pellets. When the exchange period schedule was a six-response fixed ratio, patterns of completing the component token schedules were bivalued, with relatively long and frequent pauses marking the initiation of each new sequence. Altering the exchange period schedule to a six-response variable ratio resulted in sharp reductions in the frequency and duration of these initial pauses, and increases in overall rates of lever pressing. These results are comparable to those ordinarily obtained under simple fixed-ratio and variable-ratio schedules.     

Schedule-induced defecation by rats during ratio and interval schedules of food reinforcement.

Lever pressing in rats was maintained by continuous and intermittent schedules of food while defecation was monitored. In Experiment 1, reinforcement densities were matched across variable-ratio and variable-interval schedules for three pairs of rats. Defecation occurred in all 3 rats on the variable-ratio schedule and in all 3 rats on the yoked variable-interval schedule. In Experiment 2, fixed-ratio and fixed-interval schedules with similar reinforcement densities maintained lever pressing. Defecation occurred in 3 of 4 rats on the fixed-ratio schedule and in 4 of 4 rats on the fixed-interval schedule. Almost no defecation occurred during continuous reinforcement in either experiment. These results demonstrate that defecation may occur during both ratio and interval schedules and that the inter-reinforcement interval is more important than the behavioral requirements of the schedule in generating schedule-induced defecation.     

Concurrent ratio schedules: Fixed vs. variable response requirements

Rats were trained on concurrent fixed-ratio variable-ratio or concurrent fixed-ratio mixed-ratio schedules of food reinforcement. The variable-ratio schedule was composed of an arithmetic sequence of 11 ratios that averaged 50; the mixed-ratio schedule consisted of equiprobable ratios of 1 and 99. Fixed-ratio values, varied over experimental conditions, included 25, 35, 50, 60, and 99. The proportion of responses and time allocated to the variable- or mixed-ratio schedule increased as the size of the fixed ratio increased. For most subjects, higher proportions of responses and time were maintained on the fixed-ratio schedule at fixed-ratio values of 25 and 35; higher proportions of responses and time were maintained on the variable- or mixed-ratio schedule at fixed-ratio values of 50 or higher. On concurrent variable-ratio fixed-ratio schedules, the tendency for responding to be maintained exclusively by one schedule was related to the difference in local reinforcement rates obtained from those schedules. Exclusive responding was approximated when the difference in local reinforcement rates obtained from those schedules was large; responding was more evenly distributed between the schedules as the difference in the rates at which reinforcement was obtained from each decreased.     

On the primacy of molecular processes in determining response rates under variable-ratio and variable-interval schedules

This study focused on variables that may account for response-rate differences under variable-ratio (VR) and variable-interval (VI) schedules of reinforcement. Four rats were exposed to VR, VI, tandem VI differential-reinforcement-of-high-rate, regulated-probability-interval, and negative-feedback schedules of reinforcement that provided the same rate of reinforcement. Response rates were higher under the VR schedule than the VI schedule, and the rates on all other schedules approximated those under the VR schedule. The median reinforced interresponse time (IRT) under the VI schedule was longer than for the other schedules. Thus, differences in reinforced IRTs correlated with differences in response rate, an outcome suggestive of the molecular control of response rate. This conclusion was complemented by the additional finding that the differences in molar reinforcement-feedback functions had little discernible impact on responding.     

Effects of variations in local reinforcement rate on local response rate in variable interval schedules

Rats trained to lever press for sucrose were exposed to variable-interval schedules in which (i) the probability of reinforcement in each unit of time was a constant, (ii) the probability was high in the first ten seconds after reinforcement and low thereafter, (iii) the probability was low for ten seconds and high thereafter, (iv) the probability increased with time since reinforcement, or (v) the probability was initially zero and then increased with time since reinforcement. All schedules generated similar overall reinforcement rates. A peak in local response rate occurred several seconds after reinforcement under those schedules where reinforcement rate at this time was moderate or high ([i], [ii], and [iv]). Later in the inter-reinforcement interval, local response rate was roughly constant under those schedules with a constant local reinforcement rate ([i], [ii], and [iii]), but increased steadily when local reinforcement rate increased with time since reinforcement ([iv] and [v]). Postreinforcement pauses occurred on all schedules, but were much longer when local reinforcement rate was very low in the ten seconds after reinforcement ([iii]). The interresponse time distribution was highly correlated with the distribution of reinforced interresponse times, and the distribution of postreinforcement pauses was highly correlated with the distribution of reinforced postreinforcement pauses on some schedules. However, there was no direct evidence that these correlations resulted from selective reinforcement of classes of interresponse times and pauses.     

Conditioned reinforcement dynamics in three-link chained schedules.

In two experiments rats were trained on three-link concurrent-chains schedules of reinforcement. In Experiment 1, additional entries to one terminal link were added during one of the middle links to a baseline schedule that was otherwise equal for the two chains, and, depending on the condition, these additional terminal-link presentations ended either in food or in no food. When food occurred, preference was always in favor of the chain with the additional terminal-link presentations (which also entailed a higher rate of reinforcement). When no food occurred at the end of the additional terminal links, the outcome depended on the nature of the stimuli associated with these additional terminal links. When stimuli different from the reinforced baseline terminal links were used for the no-food terminal links, preference was against the choice alternative that led to the extra periods of extinction. When the same stimulus was used for the two kinds of terminal links, preference was near indifference, that is, significantly greater than when different stimuli were used. In Experiment 2, rats learned repeated reversals of a simultaneous discrimination under a three-link concurrent-chains schedule, in which the food or no-food choice outcomes were delayed until the end of the chain. Different conditions were defined by the point in the chain at which differential stimuli occurred. When the middle and terminal links provided no differential stimuli, discrimination was acquired more slowly than when differential stimuli occurred in both links. When differential stimuli occurred in the middle but not the terminal links, acquisition rates were intermediate. Both experiments together show that the effects of stimuli in a chain schedule are due partly to the time to food correlated with the stimuli and partly to the time to the next conditioned reinforcer in the sequence.     

The effects of reinforcement frequency and response requirements on the maintenance of behavior.

Six rats responded under fixed-interval and tandem fixed-interval fixed-ratio schedules of food reinforcement. Basic fixed-interval schedules alternated over experimental conditions with tandem fixed-interval fixed-ratio schedules with the same fixed-interval value. Fixed-interval length was varied within subjects over pairs of experimental conditions; the ratio requirement of the tandem schedules was varied across subjects. For both subjects with a ratio requirement of 10, overall response rates and running response rates typically were higher under the tandem schedules than under the corresponding basic fixed-interval schedules. For all subjects with ratio requirements of 30 or 60, overall response rates and running response rates were higher under the tandem schedules than under the corresponding basic fixed-interval schedules only with relatively short fixed intervals. At longer fixed intervals, higher overall response rates and running rates were maintained by the basic fixed-interval schedules than by the tandem schedules. These findings support Zeiler and Buchman's (1979) reinforcement-theory account of response strength as an increasing monotonic function of both the response requirement and reinforcement frequency. Small response requirements added in tandem to fixed-interval schedules have little effect on reinforcement frequency and so their net effect is to enhance responding. Larger response requirements reduce reinforcement frequency more substantially; therefore their net effect depends on the length of the fixed interval, which limits overall reinforcement frequency. At the longest fixed intervals studied in the present experiment, reinforcement frequency under the tandem schedules was sufficiently low that responding weakened or ceased altogether.     

Concurrent fixed-interval variable-ratio schedules and the matching relation

Five rats responded under concurrent fixed-interval variable-ratio schedules of food reinforcement. Fixed-interval values ranged from 50-seconds to 300-seconds and variable-ratio values ranged from 30 to 360; a five-second changeover delay was in effect throughout the experiment. The relations between reinforcement ratios obtained from the two schedules and the ratios of responses and time spent on the schedules were described by Baum's (1974) generalized matching equation. All subjects undermatched both response and time ratios to reinforcement ratios, and all subjects displayed systematic bias in favor of the variable-ratio schedules. Response ratios undermatched reinforcement ratios less than did time ratios, but response ratios produced greater bias than did time ratios for every subject and for the group as a whole. Local rates of responding were generally higher on the variable-ratio than on the fixed-interval schedules. When responding was maintained by both schedules, a period of no responding on either schedule immediately after fixed-interval reinforcement typically was followed by high-rate responding on the variable-ratio schedule. At short fixed-interval values, when a changeover to the fixed-interval schedule was made, responding usually continued until fixed-interval reinforcement was obtained; at longer values, a changeover back to the variable-ratio schedule usually occurred when fixed-interval reinforcement was not forthcoming within a few seconds, and responding then alternated between the two schedules every few seconds until fixed-interval reinforcement finally was obtained.     

Multiple determinants of the effects of reinforcement magnitude on free-operant response rates

Four experiments examined the effects of increasing the number of food pellets given to hungry rats for a lever-press response. On a simple variable-interval 60-s schedule, increased number of pellets depressed response rates (Experiment 1). In Experiment 2, the decrease in response rate as a function of increased reinforcement magnitude was demonstrated on a variable-interval 30-s schedule, but enhanced rates of response were obtained with the same increase in reinforcement magnitude on a variable-ratio 30 schedule. In Experiment 3, higher rates of responding were maintained by the component of a concurrent variable-interval 60-s variable-interval 60-s schedule associated with a higher reinforcement magnitude. In Experiment 4, higher rates of response were produced in the component of a multiple variable-interval 60-s variable-interval 60-s schedule associated with the higher reinforcement magnitude. It is suggested that on simple schedules greater reinforcer magnitudes shape the reinforced pattern of responding more effectively than do smaller reinforcement magnitudes. This effect is, however, overridden by another process, such a contrast, when two magnitudes are presented within a single session on two-component schedules.     

Variable-interval schedules of timeout from avoidance

Rats were trained on concurrent schedules in which pressing one lever postponed shock and pressing the other occasionally produced a 2-min timeout during which the shock-postponement schedule was suspended and its correlated stimuli were removed. Throughout, the shock-postponement schedule maintained proficient levels of avoidance. Nevertheless, in Experiment 1 responding on the timeout lever was established rapidly, was maintained at stable levels on variable-interval schedules, was extinguished by withholding timeout, was reestablished when timeout was reintroduced, and was brought under discriminative control with a multiple variable-interval extinction schedule of timeout. These results are in contrast with Verhave's (1962) conclusion that timeout is an ineffective reinforcer when presented to rats on intermittent schedules. In Experiment 2 the consequence of responding on the timeout lever was altered so that the shock-postponement schedule remained in effect even though the stimulus conditions associated with timeout were produced for 2 min. Responding extinguished, indicating that suspension of the shock-postponement schedule, not stimulus change, was the source of reinforcement. By establishing the reinforcing efficacy of timeout with standard variable-interval schedules, these experiments illustrate a procedure for studying negative reinforcement in the same way as positive reinforcement.     

Response-rate differences in variable-interval and variable-ratio schedules: An old problem revisited

In Experiment 1, a variable-ratio 10 schedule became, successively, a variable-interval schedule with only the minimum interreinforcement intervals yoked to the variable ratio, or a variable-interval schedule with both interreinforcement intervals and reinforced interresponse times yoked to the variable ratio. Response rates in the variable-interval schedule with both interreinforcement interval and reinforced interresponse time yoking fell between the higher rates maintained by the variable-ratio schedule and the lower rates maintained by the variable-interval schedule with only interreinforcement interval yoking. In Experiment 2, a tandem variable-interval 15-s variable-ratio 5 schedule became a yoked tandem variable-ratio 5 variable-interval x-s schedule, and a tandem variable-interval 30-s variable-ratio 10 schedule became a yoked tandem variable-ratio 10 variable-interval x-s schedule. In the yoked tandem schedules, the minimum interreinforcement intervals in the variable-interval components were those that equated overall interreinforcement times in the two phases. Response rates did not decline in the yoked schedules even when the reinforced interresponse times became longer. Experiment 1 suggests that both reinforced interresponse times and response rate–reinforcement rate correlations determine response-rate differences in variable-ratio 10 and yoked variable-interval schedules in rats. Experiment 2 suggests a minimal role for the reinforced interresponse time in determining response rates on tandem variable-interval 30-s variable-ratio 10 and yoked tandem variable-ratio 10 variable-interval x-s schedules in rats.     

Yoked variable-ratio and variable-interval responding in pigeons

Pigeons' key pecks were maintained by variable-ratio or variable-interval schedules of food reinforcement. For pairs of pigeons in one group, variable-ratio reinforcement was arranged for one pigeon's pecks; for the second pigeon, reinforcement was arranged according to a variable-interval schedule yoked to the interreinforcement times produced by the first pigeon. For pairs of pigeons in another group, variable-interval reinforcement was arranged for one pigeon's pecks; for the second pigeon, reinforcement was arranged according to a variable-ratio schedule yoked to the interreinforcement responses produced by the first pigeon. For each pair, the yoking procedure was maintained for four or five consecutive sessions of 50 reinforcements each. In more than three-quarters of the pairs, variable-ratio response rates were higher than variable-interval rates within two sessions; in all cases, the rate difference developed within four sessions.     

Second-order schedules of token reinforcement: effects of varying the schedule of food presentation

In the initial link of a complex schedule, one discriminative stimulus was presented and lever pressing produced tokens on fixed-ratio schedules. In the terminal link, signalled by a second discriminative stimulus, deposits of the tokens produced food. With two rats, the terminal link was presented after each sixth component schedule of token reinforcement was completed. With the other two rats, the terminal link was presented following the first component schedule completed after a fixed interval. During the terminal link, each token deposit initially produced food. The schedule of food presentation was subsequently increased such that an increasing number of token deposits in the terminal link was required for each food presentation. Rates of lever pressing in the initial link were inversely related to the schedule of food presentation in the terminal link. These results are similar to those of experiments that have varied schedules of food presentation in chained schedules. Rates and patterns of responding controlled throughout the initial link were more similar to those ordinarily controlled by second-order brief-stimulus schedules than to those controlled by comparable extended chained schedules.     

Temporal control by signals of interval duration within variable-interval schedules

Rats' lever pressing produced sucrose reinforcers on a variable-interval schedule where, in different conditions, the duration of a stimulus presented immediately after reinforcement was either correlated or uncorrelated with the duration of the current interreinforcement interval. Under the baseline schedule, in which no stimulus was presented, the minimum interreinforcement interval was 8 s and the mean postreinforcement pause of each subject approximated this value. Response rates increased slowly over the first 10 to 15 s and then remained roughly constant throughout the remainder of the interval. In both the correlated and uncorrelated conditions, the added stimulus resulted in the postreinforcement pauses lengthening to values in excess of the duration of the preceding stimulus. This resulted in a poststimulus pause which was, in most cases, roughly constant irrespective of the duration of the preceding stimulus, or of the reinforcement contingencies prevailing immediately after stimulus offset. Local response-rate patterns in the uncorrelated conditions were similar to those obtained under the baseline schedule in which no stimulus was presented. However, in the correlated condition local response rates increased across the remainder of the interreinforcer interval. Further, the rate of acceleration was inversely related to the duration of the preceding stimulus. These results show that a correlation between stimulus duration and the ensuing time to reinforcement can control behavior—a type of temporal control not previously reported.     

Molar And Molecular Control In Variable-interval And Variable-ratio Schedules

Response rates are typically higher under variable-ratio than under variable-interval schedules of reinforcement, perhaps because of differences in the dependence of reinforcement rate on response rate or because of differences in the reinforcement of long interresponse times. A variable-interval-with-added-linear-feedback schedule is a variable-interval schedule that provides a response rate/reinforcement rate correlation by permitting the minimum interfood interval to decrease with rapid responding. Four rats were exposed to variable-ratio 15, 30, and 60 food reinforcement schedules, variable-interval 15-, 30-, and 60-s food reinforcement schedules, and two versions of variable-interval-with-added-linear-feedback 15-, 30-, and 60-s food reinforcement schedules. Response rates on the variable-interval-with-added-linear-feedback schedule were similar to those on the variable-interval schedule; all three schedules led to lower response rates than those on the variable-ratio schedules, especially when the schedule values were 30. Also, reinforced interresponse times on the variable-interval-with-added-linear-feedback schedule were similar to those on variable interval and much longer than those produced by variable ratio. The results were interpreted as supporting the hypothesis that response rates on variable-interval schedules in rats are lower than those on comparable variable-ratio schedules, primarily because the former schedules reinforce long interresponse times.     

Absence of anticipatory contrast in rats trained on multiple schedules.

Rats were trained on three- and four-component multiple schedules in which two of the components were correlated with identical reinforcement schedules that remained unchanged throughout training. These target components differed in terms of whether their respective following schedules were either higher or lower in value. Unlike corresponding experiments previously reported with pigeons, higher response rates occurred in the target component followed by a higher valued schedule than in the target component followed by the lower valued schedule. Overall contrast effects occurred independently of these sequential effects, but were inconsistent across subjects. The results suggest that the effects of a following schedule of reinforcement are opposite for pigeons and rats, and that one reason previous studies have often failed to show contrast effects with rats is that the effects of the following schedule in rats are in competition with contrast dynamics.     

Techniques for establishing schedules with wheel running as reinforcement in rats.

In three experiments, access to wheel running was contingent on lever pressing. In each experiment, the duration of access to running was reduced gradually to 4, 5, or 6 s, and the schedule parameters were expanded gradually. The sessions lasted 2 hr. In Experiment 1, a fixed-ratio 20 schedule controlled a typical break-and-run pattern of lever pressing that was maintained throughout the session for 3 rats. In Experiment 2, a fixed-interval schedule of 6 min maintained lever pressing throughout the session for 3 rats, and for 1 rat, the rate of lever pressing was positively accelerated between reinforcements. In Experiment 3, a variable-ratio schedule of 20 or 35 was in effect and maintained lever pressing at a very stable pace throughout the session for 2 of 3 rats; for 1 rat, lever pressing was maintained at an irregular rate. When the session duration was extended to successive 24-hr periods, with food and water accessible in Experiment 3, lever pressing settled into a periodic pattern occurring at a high rate at approximately the same time each day. In each experiment, the rats that developed the highest local rates of running during wheel access also maintained the most stable and highest rates of lever pressing.     

Matching and maximizing with variable-time schedules.

Pigeons were offered choices between a variable-time schedule that arranged reinforcers throughout the session and a variable-time schedule that arranged reinforcers only when the pigeon was spending time on it. The subjects could maximize the overall rate of reinforcement in this situation by biasing their time allocation towards the latter schedule. This arrangement provides an alternative to concurrent variable-interval variable-ratio schedules for testing whether animals maximize overall rates or match relative rates, and has the advantage of being free of the asymmetrical response requirements present with those schedules. The results were contrary to those predicted by maximizing: The bias it predicts did not appear.     

Matching and maximizing with concurrent ratio-interval schedules.

Animals exposed to standard concurrent variable-ratio variable-interval schedules could maximize overall reinforcement rate if, in responding, they showed a strong response bias toward the variable-ratio schedule. Tests with the standard schedules have failed to find such a bias and have been widely cited as evidence against maximization as an explanation of animal choice behavior. However, those experiments were confounded in that the value of leisure (behavior other than the instrumental response) partially offsets the value of reinforcement. The present experiment provides another such test using a concurrent procedure in which the confounding effects of leisure were mostly eliminated while the critical aspects of the concurrent variable-ratio variable-interval contingency were maintained: Responding in one component advanced only its ratio schedule while responding in the other component advanced both ratio schedules. The bias toward the latter component predicted by maximization theory was found.     

Conditioned reinforcement versus time to reinforcement in chain schedules.

Pigeons were trained on three-component chain schedules in which the initial component was either a fixed-interval or variable-interval schedule. The middle and terminal components were varied among fixed-interval fixed-interval, variable-interval variable-interval, and an interdependent variable-interval variable-interval schedule in which the sum of the durations of the two variable-interval components was always equal to the sum of the fixed-interval fixed-interval components. At issue was whether the response rate in the initial component was controlled by its time to primary reinforcement or by the temporal parameters of the stimulus correlated with the middle terminal link. The fixed-interval initial-link schedule maintained much lower response rates than the variable-interval initial-link schedule regardless of the schedules in the middle and terminal links. Nevertheless, the intervening schedules played some role: With fixed-interval schedules in the initial links, response rates were consistently highest with independent variable-interval schedules in the middle and terminal links and intermediate with the interdependent variable-interval schedules; these initial-link differences were predicted by the response rates in the middle link of the chain. With variable-interval schedules in the initial links, response rates were lowest with the fixed-interval fixed-interval schedules following the initial link and were not systematically different for the two types of variable-interval variable-interval schedules. The results suggest that time to reinforcement itself accounts for little if any variance in initial-link responding.