Concurrent schedules of interresponse time reinforcement: probability of reinforcement and the lower bounds of the reinforced interresponse time intervals

Data were obtained with rats on the effects of interresponse time contingent reinforcement of the lever press response using schedules in which interresponse times falling within either of two temporal intervals could be reinforced. Some of the findings were (a) the mode of the interresponse time distribution generally occurred near the first lower bound when the maximum reinforcement rate for the two lower bounds was equal; this also frequently occurred even when the reinforcement rate was less for the first lower bound; (b) as is the case with schedules using a single interval of reinforced interresponse times the values of the lower bounds partially determined the location and spread of the distributions; but the particular pair of values used did not seem to influence the effects of the probabilities of reinforcement; (c) although the modal interresponse time was usually at the lower bound of one of the two intervals of reinforced interresponse times, no simple relation existed between either the probability or rate of reinforcement of interresponse times in these two intervals and the location of this mode.     

The role of reinforcement in controlling sequential IRT dependencies

Sequential dependencies were investigated with two rats in a mixed and in a tandem differential-reinforcement-of-low-rate-responding schedule. In each schedule, 5-sec and 15-sec components were presented in fixed alternation. In the mixed schedule, a 5-sec interresponse time followed a 15-sec interresponse time and a 15-sec interresponse time followed a 5-sec interresponse time in predictable sequence. The correlation between prior and subsequent interresponse times, however, existed only when the prior interresponse time resulted in reinforcement. In the tandem schedule, an interresponse time greater than 5 sec in the differential-reinforcement-of-low-rate 5-sec component was not associated directly with reinforcement. One subject demonstrated sequential response patterns similar to those noted in the mixed schedule, even though the prior 5-sec interresponse time was not reinforced in the tandem schedule. The results indicate that the prior interresponse time length alone is not sufficient to influence the subsequent interresponse time length. Implications are, however, that a temporal response pattern arises when an interresponse interacts with schedule contingencies to control the interreinforcement interval.     

Local patterns of responding maintained by concurrent and multiple schedules

Local patterns of responding were studied when pigeons pecked for food in concurrent variable-interval schedules (Experiment I) and in multiple variable-interval schedules (Experiment II). In Experiment I, similarities in the distribution of interresponse times on the two keys provided further evidence that responding on concurrent schedules is determined more by allocation of time than by changes in local pattern of responding. Relative responding in local intervals since a preceding reinforcement showed consistent deviations from matching between relative responding and relative reinforcement in various postreinforcement intervals. Response rates in local intervals since a preceding changeover showed that rate of responding is not the same on both keys in all postchangeover intervals. The relative amount of time consumed by interchangeover times of a given duration approximately matched relative frequency of reinforced interchangeover times of that duration. However, computer simulation showed that this matching was probably a necessary artifact of concurrent schedules. In Experiment II, when component durations were 180 sec, the relationship between distribution of interresponse times and rate of reinforcement in the component showed that responding was determined by local pattern of responding in the components. Since responding on concurrent schedules appears to be determined by time allocation, this result would establish a behavioral difference between multiple and concurrent schedules. However, when component durations were 5 sec, local pattern of responding in a component (defined by interresponse times) was less important in determining responding than was amount of time spent responding in a component (defined by latencies). In fact, with 5-sec component durations, the relative amount of time spent responding in a component approximately matched relative frequency of reinforcement in the component. Thus, as component durations in multiple schedules decrease, multiple schedules become more like concurrent schedules, in the sense that responding is affected by allocation of time rather than by local pattern of responding.     

The copyist model of response emission

The variety of different performances maintained by schedules of reinforcement complicates comprehensive model creation. The present account assumes the simpler goal of modeling the performances of only variable reinforcement schedules because they tend to maintain steady response rates over time. The model presented assumes that rate is determined by the mean of interresponse times (time between two responses) between successive reinforcers, averaged so that their contribution to that mean diminishes exponentially with distance from reinforcement. To respond, the model randomly selects an interresponse time from the last 300 of these mean interresponse times, the selection likelihood arranged so that the proportion of session time spent emitting each of these 300 interresponse times is the same. This interresponse time defines the mean of an exponential distribution from which one is randomly chosen for emission. The response rates obtained approximated those found on several variable schedules. Furthermore, the model reproduced three effects: (1) the variable ratio maintaining higher response rates than does the variable interval; (2) the finding for variable schedules that when the reinforcement rate varies from low to high, the response rate function has an ascending and then descending limb; and (3) matching on concurrent schedules. Because these results are due to an algorithm that reproduces reinforced interresponse times, responding to single and concurrent schedules is viewed as merely copying what was reinforced before.     

Synthetic variable-interval schedules of reinforcement

Three pigeons pecked for food on a synthetic variable-interval schedule of reinforcement that had two independent parts: a variable-interval schedule that arranged a distribution of interreinforcement intervals, and a device that randomly assigned each reinforcement to one of 10 classes of interresponse times. The frequencies of reinforcement for the 10 classes of interresponse times were systematically varied, while the overall frequency of reinforcement was held within a comparatively narrow range. The 10 classes extended either from 0.1 to 0.6 sec in 0.05-sec intervals, or from 1.0 to 6.0 sec in 0.5-sec intervals. In the former case, some control by reinforcement was obtained, but it was weak and no simple relationships were discernible. In the latter case, the relative frequency of an interresponse time was a generally increasing function of its relative frequency of reinforcement, and two simple controlling relationships were found. First, the function relating interresponse times per opportunity to reinforcements per opportunity was, over a restricted range, approximately linear with a slope of unity. Second, when all 10 classes of interresponse times were reinforced equally often, the relative frequency of an interresponse time approximately equalled the relative reciprocal of its length.     

Effects of variations in local reinforcement rate on local response rate in variable interval schedules

Rats trained to lever press for sucrose were exposed to variable-interval schedules in which (i) the probability of reinforcement in each unit of time was a constant, (ii) the probability was high in the first ten seconds after reinforcement and low thereafter, (iii) the probability was low for ten seconds and high thereafter, (iv) the probability increased with time since reinforcement, or (v) the probability was initially zero and then increased with time since reinforcement. All schedules generated similar overall reinforcement rates. A peak in local response rate occurred several seconds after reinforcement under those schedules where reinforcement rate at this time was moderate or high ([i], [ii], and [iv]). Later in the inter-reinforcement interval, local response rate was roughly constant under those schedules with a constant local reinforcement rate ([i], [ii], and [iii]), but increased steadily when local reinforcement rate increased with time since reinforcement ([iv] and [v]). Postreinforcement pauses occurred on all schedules, but were much longer when local reinforcement rate was very low in the ten seconds after reinforcement ([iii]). The interresponse time distribution was highly correlated with the distribution of reinforced interresponse times, and the distribution of postreinforcement pauses was highly correlated with the distribution of reinforced postreinforcement pauses on some schedules. However, there was no direct evidence that these correlations resulted from selective reinforcement of classes of interresponse times and pauses.     

The effect upon simple animal behavior of different frequencies of reinforcement, Part II: separate control of the reinforcement of different IRTs

Rats' responding was stabilized for over 35 days on 4-min variable-interval reinforcement. Reinforcements per hour for 4-sec wide classes of interresponse times were then separately controlled by adjusting those for each class to the variable-interval values that had just prevailed. This produced little or no change in interresponse times, indicating that the new procedure was substantially equivalent to a variable-interval schedule. The variable-interval schedule produced a high and stable conditional probability of interresponse times in the 0- to 4-sec class, associated with a peak in reinforcements per hour for this class. Reducing the reinforcements per hour for this class while raising that for another class (by 3.3 reinforcements per hour) significantly reduced the conditional probability of 0- to 4-sec interresponse times. Restoring the 3.3 reinforcements per hour to the 0- to 4-sec class significantly elevated the conditional probability of interresponse times in this class. Hence, it is concluded that the distribution of interresponse times produced by a subject during some variable-interval schedules is determined partly by the relative reinforcement of different interresponse times that the variable-interval schedule provided.Reprinted from Part II of the Final Report of Research under Contract DA-49-007-MD-408 with the Medical Research and Development Board, Office of the Surgeon General, Department of the Army, 31 December 1954. Edwin B. Newman, Responsible Investigator; Douglas Anger, Research Assistant and author of report. Experimental work done in the Psychological Laboratories of Harvard University.     

Studies on responding under fixed-interval schedules of reinforcement: the effects on the pattern of responding of changes in requirements at reinforcement

In pigeons responding under a 180-sec fixed-interval schedule of reinforcement, the frequency distribution of the duration of the final interresponse time before the reinforcer was compared with the distribution of the preceding two interresponse times. The results confirmed qualitatively and quantitatively the expected preferential reinforcement of longer interreinforcement times under fixed-interval reinforcement. Requirements at reinforcement were then changed to eliminate the preferential reinforcement of longer interresponse times. Local patterns and mean rate of responding could change, without the characteristic fixed-interval pattern of increasing responding through the interval (scalloping) being much affected. It is concluded that this characteristic pattern of fixed-interval responding does not depend crucially on effects of the reinforcer at the moment of reinforcement, but rather to effects extending over much longer periods of time than just the last interresponse time.     

Molar And Molecular Control In Variable-interval And Variable-ratio Schedules

Response rates are typically higher under variable-ratio than under variable-interval schedules of reinforcement, perhaps because of differences in the dependence of reinforcement rate on response rate or because of differences in the reinforcement of long interresponse times. A variable-interval-with-added-linear-feedback schedule is a variable-interval schedule that provides a response rate/reinforcement rate correlation by permitting the minimum interfood interval to decrease with rapid responding. Four rats were exposed to variable-ratio 15, 30, and 60 food reinforcement schedules, variable-interval 15-, 30-, and 60-s food reinforcement schedules, and two versions of variable-interval-with-added-linear-feedback 15-, 30-, and 60-s food reinforcement schedules. Response rates on the variable-interval-with-added-linear-feedback schedule were similar to those on the variable-interval schedule; all three schedules led to lower response rates than those on the variable-ratio schedules, especially when the schedule values were 30. Also, reinforced interresponse times on the variable-interval-with-added-linear-feedback schedule were similar to those on variable interval and much longer than those produced by variable ratio. The results were interpreted as supporting the hypothesis that response rates on variable-interval schedules in rats are lower than those on comparable variable-ratio schedules, primarily because the former schedules reinforce long interresponse times.     

DISCRIMINATION OF VARIABLE SCHEDULES IS CONTROLLED BY INTERRESPONSE TIMES PROXIMAL TO REINFORCEMENT

In Experiment 1, food‐deprived rats responded to one of two schedules that were, with equal probability, associated with a sample lever. One schedule was always variable ratio, while the other schedule, depending on the trial within a session, was: (a) a variable‐interval schedule; (b) a tandem variable‐interval, differential‐reinforcement‐of‐low‐rate schedule; or (c) a tandem variable‐interval, differential‐reinforcement‐of‐high‐rate schedule. Completion of a sample‐lever schedule, which took approximately the same time regardless of schedule, presented two comparison levers, one associated with each sample‐lever schedule. Pressing the comparison lever associated with the schedule just presented produced food, while pressing the other produced a blackout. Conditional‐discrimination accuracy was related to the size of the difference in reinforced interresponse times and those that preceded it (predecessor interresponse times) between the variable‐ratio and other comparison schedules. In Experiment 2, control by predecessor interresponse times was accentuated by requiring rats to discriminate between a variable‐ratio schedule and a tandem schedule that required emission of a sequence of a long, then a short interresponse time in the tandem's terminal schedule. These discrimination data are compatible with the copyist model from Tanno and Silberberg (2012) in which response rates are determined by the succession of interresponse times between reinforcers weighted so that each interresponse time's role in rate determination diminishes exponentially as a function of its distance from reinforcement.     

Operant acceleration during a pre-reward stimulus

Stimuli of 20, 40, and 80 sec duration terminated with five non-response-contingent food pellets were superimposed upon lever pressing reinforced with single pellets on a DRL 30-sec schedule. Two rhesus monkeys served as subjects. No change in response frequency was observed during the 20- and 40-sec stimuli. During the 80-sec pre-food stimulus, overall response frequency increased to approximately 150% and 220% of pre-stimulus levels, and the temporal distributions of interresponse times shifted toward the shorter intervals. When the 80-sec stimulus was no longer terminated with food, the response frequency decreased and the temporal distributions of interresponse times gradually approached pre-stimulus levels. An increased frequency of short interresponse times and an increase in response rate was again observed when the pellet termination procedure was reinstituted with the 80-sec stimulus. No change in response frequency or interresponse times was observed in the absence of the conditioning stimulus, and performance efficiency, as reflected in the ratio of responses to reinforcements during non-stimulus periods, remained stable throughout the experiment.     

Interresponse time duration in fixed-interval schedules of reinforcement: control by ordinal position and time since reinforcement

The times between each of the first thirteen responses after reinforcement (the first twelve interresponse times) were determined for two pigeons whose pecking was reinforced on fixed-interval schedules of food reinforcement ranging from 0.5 min to 5 min. These interresponse times were classified with respect to their ordinal position in the sequence of responses and with respect to the time since the preceding reinforcement at which the initiating response occurred. The median interresponse time durations were essentially constant after the sixth response after reinforcement regardless of the time at which the interresponse time was initiated. The durations of the first few interresponse times after reinforcement decreased as the number of preceding responses increased and as the time since the preceding reinforcement increased.     

Temporal patterns of responding in small fixed-ratio schedules

Pigeons were exposed to an ascending series of small fixed-ratio schedules from fixed-ratio 1 to 7. Two of those pigeons were later placed on a fixed-ratio 30 schedule. The two primary dependent variables were the postreinforcement pause and the interresponse time. Changes in these variables under small fixed ratios were sometimes opposite to changes reported with large fixed ratios. For example, postreinforcement pauses decreased in length as the fixed-ratio requirement increased from fixed-ratio 1 to fixed-ratio 3. Also, the interresponse times early in the small fixed-ratio schedule were shorter than those immediately preceding reinforcement. These findings question the role of interresponse-time reinforcement in determining temporal patterns of responding under small fixed-ratio schedules. They also suggest that there may be a limited region in which the independent variable, fixed-ratio size, does not operate as previously described.     

Two-key concurrent paced variable-interval paced variable-interval schedules of reinforcement

Nine pigeons were used in two experiments in which a response was reinforced if a variable-interval schedule had assigned a reinforcement and if the response terminated an interresponse time within a certain interval, or class, of interresponse times. One such class was scheduled on one key, and a second class was scheduled on a second key. The procedure was, therefore, a two-key concurrent paced variable-interval paced variable-interval schedule. In Exp. I, the lengths of the two reinforced interresponse times were varied. The relative frequency of responding on a key approximately equalled the relative reciprocal of the length of the interresponse time reinforced on that key. In Exp. II, the relative frequency and relative magnitude of reinforcement were varied. The relative frequency of responding on the key for which the shorter interresponse time was reinforced was a monotonically increasing, negatively accelerated function of the relative frequency of reinforcement on that key. The relative frequency of responding depended on the relative magnitude of reinforcement in approximately the same way as it depended on the relative frequency of reinforcement. The relative frequency of responding on the key for which the shorter interresponse time was reinforced depended on the lengths of the two reinforced interresponse times and on the relative frequency and relative magnitude of reinforcement in the same way as the relative frequency of the shorter interresponse time depended on these variables in previous one-key concurrent schedules of reinforcement for two interresponse times.     

Effects of a DRL contingency added to a fixed-interval reinforcement schedule

Following 30 days of reinforcement for the bar press response of two white rats on 30-sec fixed-interval (FI), a DRL component was added so that a minimal interresponse time (IRT) for the reinforced response, in addition to the FI variable, was necessary for reinforcement. Marked control over response rate by the superimposed DRL requirement was demonstrated by an inverse hyperbolic function as the DRL component was increased from 1 to 24 sec within the constant 30-sec FI interval. Interresponse time and post-reinforcement (post-S^R) “break” distributions taken at one experimental point (DRL = 24 sec) suggested that a more precise temporal discrimination was initiated by an S^R than by a response, since the relative frequency of a sequence of two reinforced responses appeared greater than that of a sequence of a non-reinforced response followed by a reinforced one. This latter finding was confirmed with new animals in a follow-up experiment employing a conventional 24-sec DRL schedule.     

The concurrent reinforcement of two interresponse times: the relative frequency of an interresponse time equals its relative harmonic length

The relative lengths of two concurrently reinforced interresponse times were varied in an experiment in which three pigeons obtained food by pecking on a single key. Visual discriminative stimuli accompanied the two time intervals in which reinforcements were scheduled according to a one-minute variable-interval. The steady-state relative frequency of an interresponse time approximately equalled the complement of its relative length, that is, its relative harmonic length. Thus, lengths of interresponse times and delays of reinforcement have the same effect on the relative frequencies of interresponse times and choices in one-key and two-key concurrent variable-interval schedules, respectively. A second experiment generalized further the functional equivalence between the effects of these one-key and two-key concurrent schedules by revealing that the usual matching-to-relative-immediacy in two-key concurrent schedules is undisturbed if reinforcement depends upon the occurrence of a response at the end of the delay interval, as it does in the one-key schedules. The results of both experiments are consistent with a quantitative theory of concurrent operant behavior.     

Parametric manipulation of interresponse-time contingency independent of reinforcement rate

Pecking of pigeons was reinforced under a modified interval-percentile procedure that allowed independent manipulation of overall reinforcement rate and the degree to which reinforcement depended on interresponse-time duration. Increasing the contingency, as measured by the phi coefficient, between reinforcement and long interresponse times while controlling the overall rate of reinforcement systematically increased the frequency of those interresponse times and decreased response rate under both of the reinforcement rates studied. Increasing reinforcement rate also generally increased response rate, particularly under weaker interresponse-time contingencies. Random-interval schedules with comparable reinforcement rates generated response rates and interresponse-time distributions similar to those obtained with moderate-to-high interresponse-time reinforcement contingencies. These results suggest that interresponse-time reinforcement contingencies inherent in random-interval and constant-probability variable-interval schedules exercise substantial control over responding independent of overall reinforcement rate effects. The interresponse-time reinforcement contingencies inherent in these schedules may actually mask the effects of overall reinforcement rate; thus differences in response rate as a function of reinforcement rate when interresponse-time reinforcement is eliminated may be underestimated.     

Children's reactions to failure as a function of interresponse interval

Several studies have reported that nonreward facilitates subsequent performance; other studies have found that failure is followed by a performance decrement. Experiments that have shown facilitating effects of nonreward have used interresponse intervals of at least 5 sec, whereas those that have shown decremental effects of failure have used 0-sec interresponse intervals. The present study examined the effects of .5-, 1.0- and 5.0-sec interresponse intervals on children's lever-pulling responses following success and failure on a ball tower task. Second- and third-grade children responded slower following failure relative to success with .5- and 1.0-sec interresponse intervals. Speeds following success and failure did not differ when the interresponse interval was 5.0 sec. Increases in interresponse interval were associated with increases in speeds following failure, but speeds following success were not related to interresponse interval. These results were discussed in terms of frustration-produced competing responses and Elliott (1970) analysis of the influence of preparatory intervals on children's reaction-time performance.     

An interresponse time analysis of variable-ratio punishment

An interresponse time analysis was used to study the effects of variable-ratio punishment schedules on the temporal pattern of reinforced responding. Twelve pigeons responded on a baseline variable-interval schedule of food reinforcement. A variable-ratio ten schedule of electric shock punishment was then introduced. The shock intensity was systematically increased to the highest intensity at which responding could be maintained. At this intensity, the mean variable-ratio value was increased and then decreased. Variable-ratio punishment resulted in an increased relative frequency of very short unreinforced interresponse times (response bursting). Increased response bursting accounted for instances of response rate facilitation. In addition, shock was followed by interresponse times of decreasing mean length over the first several responses after shock.     

Time allocation and response rate

Food reinforcement for key pecking by three pigeons was arranged by a variable-interval schedule and a device that assigned each reinforcement to one of 10 component response rates corresponding to 10 classes of equally reinforced interresponse times ranging from 1.0 to 6.0 sec in 0.5-sec classes. The overall number of reinforcements per hour was varied from one to more than 60. Overall response rate was a monotonically increasing, negatively accelerated function of the overall number of reinforcements per hour. This function was decomposed into two time-allocation functions: (1) the time allocated to all of the reinforced component response rates as a function of the total reinforcement rate, and, (2) the time allocated to a particular reinforced component response rate as a function of the reinforcement rate for that component. Asymptotic response rate was predicted by combining the asymptotes of the two separate time-allocation functions: virtually all of the time was spent responding, and the percentage of the time spent responding that was allocated to a particular reinforced component response rate roughly equalled the relative reinforcements per hour for that component.