The copyist model of response emission

The variety of different performances maintained by schedules of reinforcement complicates comprehensive model creation. The present account assumes the simpler goal of modeling the performances of only variable reinforcement schedules because they tend to maintain steady response rates over time. The model presented assumes that rate is determined by the mean of interresponse times (time between two responses) between successive reinforcers, averaged so that their contribution to that mean diminishes exponentially with distance from reinforcement. To respond, the model randomly selects an interresponse time from the last 300 of these mean interresponse times, the selection likelihood arranged so that the proportion of session time spent emitting each of these 300 interresponse times is the same. This interresponse time defines the mean of an exponential distribution from which one is randomly chosen for emission. The response rates obtained approximated those found on several variable schedules. Furthermore, the model reproduced three effects: (1) the variable ratio maintaining higher response rates than does the variable interval; (2) the finding for variable schedules that when the reinforcement rate varies from low to high, the response rate function has an ascending and then descending limb; and (3) matching on concurrent schedules. Because these results are due to an algorithm that reproduces reinforced interresponse times, responding to single and concurrent schedules is viewed as merely copying what was reinforced before.     

Concurrent schedules of interresponse time reinforcement: probability of reinforcement and the lower bounds of the reinforced interresponse time intervals

Data were obtained with rats on the effects of interresponse time contingent reinforcement of the lever press response using schedules in which interresponse times falling within either of two temporal intervals could be reinforced. Some of the findings were (a) the mode of the interresponse time distribution generally occurred near the first lower bound when the maximum reinforcement rate for the two lower bounds was equal; this also frequently occurred even when the reinforcement rate was less for the first lower bound; (b) as is the case with schedules using a single interval of reinforced interresponse times the values of the lower bounds partially determined the location and spread of the distributions; but the particular pair of values used did not seem to influence the effects of the probabilities of reinforcement; (c) although the modal interresponse time was usually at the lower bound of one of the two intervals of reinforced interresponse times, no simple relation existed between either the probability or rate of reinforcement of interresponse times in these two intervals and the location of this mode.     

Interresponse time duration in fixed-interval schedules of reinforcement: control by ordinal position and time since reinforcement

The times between each of the first thirteen responses after reinforcement (the first twelve interresponse times) were determined for two pigeons whose pecking was reinforced on fixed-interval schedules of food reinforcement ranging from 0.5 min to 5 min. These interresponse times were classified with respect to their ordinal position in the sequence of responses and with respect to the time since the preceding reinforcement at which the initiating response occurred. The median interresponse time durations were essentially constant after the sixth response after reinforcement regardless of the time at which the interresponse time was initiated. The durations of the first few interresponse times after reinforcement decreased as the number of preceding responses increased and as the time since the preceding reinforcement increased.     

Effects of variations in local reinforcement rate on local response rate in variable interval schedules

Rats trained to lever press for sucrose were exposed to variable-interval schedules in which (i) the probability of reinforcement in each unit of time was a constant, (ii) the probability was high in the first ten seconds after reinforcement and low thereafter, (iii) the probability was low for ten seconds and high thereafter, (iv) the probability increased with time since reinforcement, or (v) the probability was initially zero and then increased with time since reinforcement. All schedules generated similar overall reinforcement rates. A peak in local response rate occurred several seconds after reinforcement under those schedules where reinforcement rate at this time was moderate or high ([i], [ii], and [iv]). Later in the inter-reinforcement interval, local response rate was roughly constant under those schedules with a constant local reinforcement rate ([i], [ii], and [iii]), but increased steadily when local reinforcement rate increased with time since reinforcement ([iv] and [v]). Postreinforcement pauses occurred on all schedules, but were much longer when local reinforcement rate was very low in the ten seconds after reinforcement ([iii]). The interresponse time distribution was highly correlated with the distribution of reinforced interresponse times, and the distribution of postreinforcement pauses was highly correlated with the distribution of reinforced postreinforcement pauses on some schedules. However, there was no direct evidence that these correlations resulted from selective reinforcement of classes of interresponse times and pauses.     

Local patterns of responding maintained by concurrent and multiple schedules

Local patterns of responding were studied when pigeons pecked for food in concurrent variable-interval schedules (Experiment I) and in multiple variable-interval schedules (Experiment II). In Experiment I, similarities in the distribution of interresponse times on the two keys provided further evidence that responding on concurrent schedules is determined more by allocation of time than by changes in local pattern of responding. Relative responding in local intervals since a preceding reinforcement showed consistent deviations from matching between relative responding and relative reinforcement in various postreinforcement intervals. Response rates in local intervals since a preceding changeover showed that rate of responding is not the same on both keys in all postchangeover intervals. The relative amount of time consumed by interchangeover times of a given duration approximately matched relative frequency of reinforced interchangeover times of that duration. However, computer simulation showed that this matching was probably a necessary artifact of concurrent schedules. In Experiment II, when component durations were 180 sec, the relationship between distribution of interresponse times and rate of reinforcement in the component showed that responding was determined by local pattern of responding in the components. Since responding on concurrent schedules appears to be determined by time allocation, this result would establish a behavioral difference between multiple and concurrent schedules. However, when component durations were 5 sec, local pattern of responding in a component (defined by interresponse times) was less important in determining responding than was amount of time spent responding in a component (defined by latencies). In fact, with 5-sec component durations, the relative amount of time spent responding in a component approximately matched relative frequency of reinforcement in the component. Thus, as component durations in multiple schedules decrease, multiple schedules become more like concurrent schedules, in the sense that responding is affected by allocation of time rather than by local pattern of responding.     

Intermittent reinforcement of an interresponse time

Rats were exposed to schedules in which reinforcement was contingent upon the emission of a 1.0- to 2.0-sec interresponse time. The rate of emission and the temporal distribution of this interresponse time was recorded. Several different contingencies between the emission of the interresponse time and reinforcement were examined. Both the rate of emission and the temporal distribution of the 1.0- to 2.0-sec interresponse time varied as a function of the schedule on which it was reinforced. This finding, which suggests that an interresponse time behaves as other operants, has implications for the analysis of conventional reinforcement schedules in terms of the differential reinforcement of interresponse times.     

The concurrent reinforcement of two interresponse times: the relative frequency of an interresponse time equals its relative harmonic length

The relative lengths of two concurrently reinforced interresponse times were varied in an experiment in which three pigeons obtained food by pecking on a single key. Visual discriminative stimuli accompanied the two time intervals in which reinforcements were scheduled according to a one-minute variable-interval. The steady-state relative frequency of an interresponse time approximately equalled the complement of its relative length, that is, its relative harmonic length. Thus, lengths of interresponse times and delays of reinforcement have the same effect on the relative frequencies of interresponse times and choices in one-key and two-key concurrent variable-interval schedules, respectively. A second experiment generalized further the functional equivalence between the effects of these one-key and two-key concurrent schedules by revealing that the usual matching-to-relative-immediacy in two-key concurrent schedules is undisturbed if reinforcement depends upon the occurrence of a response at the end of the delay interval, as it does in the one-key schedules. The results of both experiments are consistent with a quantitative theory of concurrent operant behavior.     

Two-key concurrent paced variable-interval paced variable-interval schedules of reinforcement

Nine pigeons were used in two experiments in which a response was reinforced if a variable-interval schedule had assigned a reinforcement and if the response terminated an interresponse time within a certain interval, or class, of interresponse times. One such class was scheduled on one key, and a second class was scheduled on a second key. The procedure was, therefore, a two-key concurrent paced variable-interval paced variable-interval schedule. In Exp. I, the lengths of the two reinforced interresponse times were varied. The relative frequency of responding on a key approximately equalled the relative reciprocal of the length of the interresponse time reinforced on that key. In Exp. II, the relative frequency and relative magnitude of reinforcement were varied. The relative frequency of responding on the key for which the shorter interresponse time was reinforced was a monotonically increasing, negatively accelerated function of the relative frequency of reinforcement on that key. The relative frequency of responding depended on the relative magnitude of reinforcement in approximately the same way as it depended on the relative frequency of reinforcement. The relative frequency of responding on the key for which the shorter interresponse time was reinforced depended on the lengths of the two reinforced interresponse times and on the relative frequency and relative magnitude of reinforcement in the same way as the relative frequency of the shorter interresponse time depended on these variables in previous one-key concurrent schedules of reinforcement for two interresponse times.     

Molar And Molecular Control In Variable-interval And Variable-ratio Schedules

Response rates are typically higher under variable-ratio than under variable-interval schedules of reinforcement, perhaps because of differences in the dependence of reinforcement rate on response rate or because of differences in the reinforcement of long interresponse times. A variable-interval-with-added-linear-feedback schedule is a variable-interval schedule that provides a response rate/reinforcement rate correlation by permitting the minimum interfood interval to decrease with rapid responding. Four rats were exposed to variable-ratio 15, 30, and 60 food reinforcement schedules, variable-interval 15-, 30-, and 60-s food reinforcement schedules, and two versions of variable-interval-with-added-linear-feedback 15-, 30-, and 60-s food reinforcement schedules. Response rates on the variable-interval-with-added-linear-feedback schedule were similar to those on the variable-interval schedule; all three schedules led to lower response rates than those on the variable-ratio schedules, especially when the schedule values were 30. Also, reinforced interresponse times on the variable-interval-with-added-linear-feedback schedule were similar to those on variable interval and much longer than those produced by variable ratio. The results were interpreted as supporting the hypothesis that response rates on variable-interval schedules in rats are lower than those on comparable variable-ratio schedules, primarily because the former schedules reinforce long interresponse times.     

An interresponse time analysis of variable-ratio punishment

An interresponse time analysis was used to study the effects of variable-ratio punishment schedules on the temporal pattern of reinforced responding. Twelve pigeons responded on a baseline variable-interval schedule of food reinforcement. A variable-ratio ten schedule of electric shock punishment was then introduced. The shock intensity was systematically increased to the highest intensity at which responding could be maintained. At this intensity, the mean variable-ratio value was increased and then decreased. Variable-ratio punishment resulted in an increased relative frequency of very short unreinforced interresponse times (response bursting). Increased response bursting accounted for instances of response rate facilitation. In addition, shock was followed by interresponse times of decreasing mean length over the first several responses after shock.     

Parametric manipulation of interresponse-time contingency independent of reinforcement rate

Pecking of pigeons was reinforced under a modified interval-percentile procedure that allowed independent manipulation of overall reinforcement rate and the degree to which reinforcement depended on interresponse-time duration. Increasing the contingency, as measured by the phi coefficient, between reinforcement and long interresponse times while controlling the overall rate of reinforcement systematically increased the frequency of those interresponse times and decreased response rate under both of the reinforcement rates studied. Increasing reinforcement rate also generally increased response rate, particularly under weaker interresponse-time contingencies. Random-interval schedules with comparable reinforcement rates generated response rates and interresponse-time distributions similar to those obtained with moderate-to-high interresponse-time reinforcement contingencies. These results suggest that interresponse-time reinforcement contingencies inherent in random-interval and constant-probability variable-interval schedules exercise substantial control over responding independent of overall reinforcement rate effects. The interresponse-time reinforcement contingencies inherent in these schedules may actually mask the effects of overall reinforcement rate; thus differences in response rate as a function of reinforcement rate when interresponse-time reinforcement is eliminated may be underestimated.     

Choice between response units: The rate constancy model

In a conjoint schedule, reinforcement is available simultaneously on two or more schedules for the same response. The present experiments provided food for key pecking on both a random-interval and a differential-reinforcement-of-low-rate (DRL) schedule. Experiment 1 involved ordinary DRL schedules; Experiment 2 added an external stimulus to indicate when the required interresponse time had elapsed. In both experiments, the potential reinforcer frequency from each component was varied by means of a second-order fixed-ratio schedule, and the DRL time parameter was changed as well. Response rates were described by a model stating that time allocation to each component matches the relative frequency of reinforcement for that component. When spending time in a given component, the subject is assumed to respond at the rate characteristic of baseline performance. This model appeared preferable to the absolute-rate version of the matching law. The model was shown to be applicable to multiple-response concurrent schedules as well as to conjoint schedules, and it described some of the necessary conditions for response matching, undermatching, and bias. In addition, the pigeons did not optimize reinforcer frequency.     

Interresponse-time shaping by variable-interval-like interresponse-time reinforcement contingencies

The interresponse-time reinforcement contingencies and distributions of interreinforcement intervals characteristic of certain variable-interval schedules were mimicked by reinforcing each key peck with a probability equal to the duration of the interresponse time it terminated, divided by the scheduled mean interreinforcement interval. The interresponse-time reinforcement contingency was then eliminated by basing the probability of reinforcement on the fifth interresponse time preceding the key peck. Even though distributions of interreinforcement intervals were unaffected by this manipulation, response rates consistently increased. A second experiment replicated this effect and showed it to combine additively with that of mean reinforcement rate. These results provide strong support for the contention that current analyses of variable-interval response rates that ignore the inherent interresponse-time reinforcement contingency may be seriously in error.     

Signalled reinforcement in differential-reinforcement-of-low rate schedules

At several fixed and variable minimum reinforced interresponse times, a stimulus was added to differential-reinforcement-of-low-rate schedules to signal the availability or nonavailability of reinforcement. As the minimum reinforced interresponse time increased, the rate of unreinforced responding decreased. Changing from fixed to variable minimum interresponse time in the basic differential-reinforcement-of-low-rate schedule further decreased the rate of unreinforced responding. Both effects were to some degree reversible. For fixed minimum reinforced interresponse times of 30 sec or shorter, most unreinforced responses terminated interresponse times just short of that required for reinforcement. The minimum reinforced interresponse time and the number of short response latencies (≤0.5 sec) to the onset of the signal were negatively correlated. Both of these analyses suggested that at values of 30 sec or shorter, the subjects discriminated the availability of the reinforcer more on the basis of time than on the basis of presence or absence of the signal.     

The role of reinforcement in controlling sequential IRT dependencies

Sequential dependencies were investigated with two rats in a mixed and in a tandem differential-reinforcement-of-low-rate-responding schedule. In each schedule, 5-sec and 15-sec components were presented in fixed alternation. In the mixed schedule, a 5-sec interresponse time followed a 15-sec interresponse time and a 15-sec interresponse time followed a 5-sec interresponse time in predictable sequence. The correlation between prior and subsequent interresponse times, however, existed only when the prior interresponse time resulted in reinforcement. In the tandem schedule, an interresponse time greater than 5 sec in the differential-reinforcement-of-low-rate 5-sec component was not associated directly with reinforcement. One subject demonstrated sequential response patterns similar to those noted in the mixed schedule, even though the prior 5-sec interresponse time was not reinforced in the tandem schedule. The results indicate that the prior interresponse time length alone is not sufficient to influence the subsequent interresponse time length. Implications are, however, that a temporal response pattern arises when an interresponse interacts with schedule contingencies to control the interreinforcement interval.     

Changes in functional response units with briefly delayed reinforcement

In two experiments, key-peck responding of pigeons was compared under variable-interval schedules that arranged immediate reinforcement and ones that arranged unsignaled delays of reinforcement. Responses during the nominal unsignaled delay periods had no effect on the reinforcer presentations. In Experiment 1, the unsignaled delays were studied using variable-interval schedules as baselines. Relative to the immediate reinforcement condition, 0.5-s unsignaled delays decreased the duration of the reinforced interresponse times and increased the overall frequency of short (<0.5-s) interresponse times. Longer, 5.0-s unsignaled delays increased the duration of the reinforced interresponse times and decreased the overall frequency of the short interresponse times. In Experiment 2, similar effects to those of Experiment 1 were obtained when the 0.5-s unsignaled delays were imposed upon a baseline schedule that explicitly arranged reinforcement of short interresponse times and therefore already generated a large number of short interresponse times. The results support earlier suggestions that the unsignaled 0.5-s delays change the functional response unit from a single key peck to a multiple key-peck unit. These findings are discussed in terms of the mechanisms by which contingencies control response structure in the absence of specific structural requirements.     

DISCRIMINATION OF VARIABLE SCHEDULES IS CONTROLLED BY INTERRESPONSE TIMES PROXIMAL TO REINFORCEMENT

In Experiment 1, food‐deprived rats responded to one of two schedules that were, with equal probability, associated with a sample lever. One schedule was always variable ratio, while the other schedule, depending on the trial within a session, was: (a) a variable‐interval schedule; (b) a tandem variable‐interval, differential‐reinforcement‐of‐low‐rate schedule; or (c) a tandem variable‐interval, differential‐reinforcement‐of‐high‐rate schedule. Completion of a sample‐lever schedule, which took approximately the same time regardless of schedule, presented two comparison levers, one associated with each sample‐lever schedule. Pressing the comparison lever associated with the schedule just presented produced food, while pressing the other produced a blackout. Conditional‐discrimination accuracy was related to the size of the difference in reinforced interresponse times and those that preceded it (predecessor interresponse times) between the variable‐ratio and other comparison schedules. In Experiment 2, control by predecessor interresponse times was accentuated by requiring rats to discriminate between a variable‐ratio schedule and a tandem schedule that required emission of a sequence of a long, then a short interresponse time in the tandem's terminal schedule. These discrimination data are compatible with the copyist model from Tanno and Silberberg (2012) in which response rates are determined by the succession of interresponse times between reinforcers weighted so that each interresponse time's role in rate determination diminishes exponentially as a function of its distance from reinforcement.     

Reinforcement frequency and contingency as factors in fixed-ratio behavior

Two variables often confounded in fixed-ratio schedules are reinforcement frequency and response requirement. These variables were isolated by a technique that yoked the distributions of reinforcements in time for one group of pigeons to those of pigeons responding on various fixed-ratio schedules. The contingencies for the yoked birds were then manipulated by adding various tandem fixed-ratio requirements to their schedules. Post-reinforcement pause was approximately equal for the yoked and ratio pigeons, and was relatively insensitive to changes in the tandem requirement. Terminal response rate increased with increases in the tandem requirement, even though reinforcement rate was invariant. This increase was attributed to the progressive interference of the tandem requirement with the differential reinforcement of long interresponse times.     

Steady-state performance on fixed-, mixed-, and random-ratio schedules

Three groups of rats pressed a lever for milk reinforcers on various simple reinforcement schedules (one schedule per condition). In Group M, each pair of conditions included a mixed-ratio schedule and a fixed-ratio schedule with equal average response:reinforcer ratios. On mixed-ratio schedules, reinforcement occurred with equal probability after a small or a large response requirement was met. In Group R, fixed-ratio and random-ratio schedules were compared in each pair of conditions. For all subjects in these two groups, the frequency distributions of interresponse times of less than one second were very similar on all ratio schedules, exhibiting a peak at about .2 seconds. For comparison, subjects in Group V responded on variable-interval schedules, and few interresponse times as short as .2 seconds were recorded. The results suggest that the rate of continuous responding is the same on all ratio schedules, and what varies among ratio schedules is the frequency, location, and duration of pauses. Preratio pauses were longer on fixed-ratio schedules than on mixed-ratio or random-ratio schedules, but there was more within-ratio pausing on mixed-ratio and random-ratio schedules. Across a single trial, the probability of an interruption in responding decreased on fixed-ratio schedules, was roughly constant on random-ratio schedules, and often increased and then decreased on mixed-ratio schedules. These response patterns provided partial support for Mazur's (1982) theory that the probability of instrumental responding is directly related to the probability of reinforcement and the proximity of reinforcement.     

The concurrent reinforcement of two interresponse times: absolute rate of reinforcement

Three pigeons obtained food on a one-key schedule of reinforcement for two concurrent, discriminated interresponse times. The overall rate of reinforcement was determined by a family of variable-interval schedules and by a continuous reinforcement schedule. The average frequency of reinforcement varied from 1.1 to 300 reinforcements per hour; the relative frequency of reinforcement for each of the two interresponse times was 0.5 throughout the experiment. The number of responses per minute increased sharply as the number of reinforcements per hour increased from 1 to 20. Beyond 30 reinforcements per hour, the curve was approximately flat, although it sometimes decreased slightly at the highest reinforcement rates. The relative frequency of the shorter interresponse time also increased sharply as the number of reinforcements per hour increased from 1 to 20. The asymptote of the relative frequency function approximately equalled the relative reciprocal of the length of the shorter interresponse time for reinforcement rates greater than 30 or 40 reinforcements per hour. This approximation was obscured by the response-rate function.