The schedule dependency of the signaled reinforcement effect

In Experiment 1, rats leverpressed for food reinforcement on either a variable ratio (VR) 30 schedule or a variable interval (VI) 15-s schedule. One group in each condition received a signal filling a 500-ms delay of reinforcement. This treatment enhanced rates on the VR schedule, and attenuated rates on the VI schedule, relative to the rate seen in an unsignaled control condition. In Experiment 2 there was no delay of reinforcement and the signal and food were presented simultaneously. Attenuated rates of responding were observed on VI schedules with a range of mean interval values (15 to 300 s). Experiment 3 used a range of VR schedules (10 to 150) with simultaneous presentations of signal and food. A signal-induced enhancement of response rate was found at all VR values. In Experiment 4, a signal elevated response rates on a tandem VI VR schedule, but depressed rates on a tandem VR VI schedule, compared to control conditions receiving unsignaled delayed reinforcement. These results are taken to show that the effect of a signal accompanying reinforcement depends upon the nature of the behavior that is reinforced during exposure to a given schedule.     

Hill-climbing by pigeons

Pigeons were exposed to two types of concurrent operant-reinforcement schedules in order to determine what choice rules determine behavior on these schedules. In the first set of experiments, concurrent variable-interval, variable-interval schedules, key-peck responses to either of two alternative schedules produced food reinforcement after a random time interval. The frequency of food-reinforcement availability for the two schedules was varied over different ranges for different birds. In the second series of experiments, concurrent variable-ratio, variable-interval schedules, key-peck responses to one schedule produced food reinforcement after a random time interval, whereas food reinforcement occurred for an alternative schedule only after a random number of responses. Results from both experiments showed that pigeons consistently follow a behavioral strategy in which the alternative schedule chosen at any time is the one which offers the highest momentary reinforcement probability (momentary maximizing). The quality of momentary maximizing was somewhat higher and more consistent when both alternative reinforcement schedules were time-based than when one schedule was time-based and the alternative response-count based. Previous attempts to provide evidence for the existence of momentary maximizing were shown to be based upon faulty assumptions about the behavior implied by momentary maximizing and resultant inappropriate measures of behavior.     

Rats' performance on variable-interval schedules with a linear feedback loop between response rate and reinforcement rate

Three experiments investigated whether rats are sensitive to the molar properties of a variable-interval (VI) schedule with a positive relation between response rate and reinforcement rate (i.e., a VI+ schedule). In Experiment 1, rats responded faster on a variable ratio (VR) schedule than on a VI+ schedule with an equivalent feedback function. Reinforced interresponse times (IRTs) were shorter on the VR as compared to the VI+ schedule. In Experiments 2 and 3, there was no systematic difference in response rates maintained by a VI+ schedule and a VI schedule yoked in terms of reinforcement rate. This was found both when the yoking procedure was between-subject (Experiment 2) and within-subject (Experiment 3). Mean reinforced IRTs were similar on both the VI+ and yoked VI schedules, but these values were more variable on the VI+ schedule. These results provided no evidence that rats are sensitive to the feedback function relating response rate to reinforcement rate on a VI+ schedule.     

Sensitivity to relative reinforcer rate in concurrent schedules: independence from relative and absolute reinforcer duration

Twelve pigeons responded on two keys under concurrent variable-interval (VI) schedules. Over several series of conditions, relative and absolute magnitudes of reinforcement were varied. Within each series, relative rate of reinforcement was varied and sensitivity of behavior ratios to reinforcer-rate ratios was assessed. When responding at both alternatives was maintained by equal-sized small reinforcers, sensitivity to variation in reinforcer-rate ratios was the same as when large reinforcers were used. This result was observed when the overall rate of reinforcement was constant over conditions, and also in another series of concurrent schedules in which one schedule was kept constant at VI ached 120 s. Similarly, reinforcer magnitude did not affect the rate at which response allocation approached asymptote within a condition. When reinforcer magnitudes differred between the two responses and reinforcer-rate ratios were varied, sensitivity of behavior allocation was unaffected although response bias favored the schedule that arranged the larger reinforcers. Analysis of absolute response rates ratio sensitivity to reinforcement occurrred on the two keys showed that this invariance of response despite changes in reinforcement interaction that were observed in absolute response rates on the constant VI 120-s schedule. Response rate on the constant VI 120-s schedule was inversely related to reinforcer rate on the varied key and the strength of this relation depended on the relative magnitude of reinforcers arranged on varied key. Independence of sensitivity to reinforcer-rate ratios from relative and absolute reinforcer magnitude is consistent with the relativity and independence assumtions of the matching law.     

More on concurrent interval-ratio schedules: a replication and review.

It has been suggested that the failure to maximize reinforcement on concurrent variable-interval, variable-ratio schedules may be misleading. Inasmuch as response costs are not directly measured, it is possible that subjects are optimally balancing the benefits of reinforcement against the costs of responding. To evaluate this hypothesis, pigeons were tested in a procedure in which interval and ratio schedules had equal response costs. On a concurrent variable time (VT), variable ratio-time (VRT) schedule, the VT schedule runs throughout the session and the VRT schedule is controlled by responses to a changeover key that switches from one schedule to the other. Reinforcement is presented independent of response. This schedule retains the essential features of concurrent VI VR, but eliminates differential response costs for the two alternatives. It therefore also eliminates at least one significant ambiguity about the reinforcement maximizing performance. Pigeons did not maximize rate of reinforcement on this procedure. Instead, their times spent on the alternative schedules matched the relative rates of reinforcement, even when schedule parameters were such that matching earned the lowest possible overall rate of reinforcement. It was further shown that the observed matching was not a procedural artifact arising from the constraints built into the schedule.     

On the primacy of molecular processes in determining response rates under variable-ratio and variable-interval schedules

This study focused on variables that may account for response-rate differences under variable-ratio (VR) and variable-interval (VI) schedules of reinforcement. Four rats were exposed to VR, VI, tandem VI differential-reinforcement-of-high-rate, regulated-probability-interval, and negative-feedback schedules of reinforcement that provided the same rate of reinforcement. Response rates were higher under the VR schedule than the VI schedule, and the rates on all other schedules approximated those under the VR schedule. The median reinforced interresponse time (IRT) under the VI schedule was longer than for the other schedules. Thus, differences in reinforced IRTs correlated with differences in response rate, an outcome suggestive of the molecular control of response rate. This conclusion was complemented by the additional finding that the differences in molar reinforcement-feedback functions had little discernible impact on responding.     

Human sensitivity to reinforcement feedback functions

Two experiments investigated human sensitivity to the temporally extended aspects of reinforcement schedules. Experiment 1 investigated human sensitivity to the extended and local aspects of three reinforcement schedules: variable ratio (VR), variable interval (VI), and variable-interval-plus-linear-feedback (VI+) schedules. Experiment 2 investigated this sensitivity on two reinforcement schedules: VI and VI+ schedules. In both experiments, there was evidence of sensitivity to the temporally extended aspects of the schedule: There were differences between the response rate on the VI+ and yoked-VI schedules, but no statistical difference in rates of response between the VR and VI+ schedules. The VI+ versus VI difference was much more pronounced when a lower response force needed to depress a lever was used. These results suggest that human subjects do show some sensitivity to temporally extended aspects of schedules of reinforcement.     

Response rate and sensitivity to the molar feedback function relating response and reinforcement rate on VI+ schedules of reinforcement

In 5 experiments, the author examined rats' sensitivity to the molar feedback function relating response rate to reinforcement rate on schedules of reinforcement. These studies demonstrated that, at lower rates of responding, rats' performance on variable ratio (VR), variable interval (VI), and variable interval with linear feedback loop (VI+) schedules was determined largely by reinforcement of interresponse times; response rates were faster on VR than on both VI and VI+ schedules. In contrast, when procedures were adopted to maintain high rates of response, rats showed sensitivity to the molar characteristics of the schedules; they responded as fast on a VI+ schedule as on a VR schedule and faster on both of these schedules than on a yoked VI schedule. When the variance of response rate was manipulated, this factor was noted as an important element in determining sensitivity to the molar characteristics of the schedule.     

Human choice in concurrent ratio-interval schedules of reinforcement.

Seven undergraduates participated in a concurrent-choice experiment with monetary reinforcers. Response-independent analogues of variable-interval and variable-ratio schedules were used to assess whether subjects would maximize reinforcement rate. The optimal pattern of behavior, in terms of maximizing reinforcement rate, involved a large bias toward the ratio alternative, with only occasional sampling of the interval schedule. Most experiments with pigeons, however, demonstrate matching of response rates to reinforcement rates, with only slight biases for the ratio schedule. Although subjects in the present experiment allocated more time to the ratio alternative than required by matching, the magnitude of the bias did not approximate that predicted by a maximizing account. After exposure to clock stimuli correlated with the operation of each schedule, 1 subject's behavior did show a substantial level of bias, increasing the total number of reinforcers obtained, and lay at a point between the predictions of matching and maximizing. The other subjects, however, continued to respond less optimally. The present results can be accounted for by a view of matching that incorporates the effects of delayed reinforcement.     

Dissociation of theories of choice by temporal spacing of choice opportunities.

Rats were trained on a discrete-trial version of a concurrent variable-interval (VI) variable-ratio (VR) schedule in which the relative reinforcement rates were varied across conditions. Subjects with the shorter intertrial interval (ITI) had a significant bias toward the VR alternative, as predicted by optimality theory, and were also more likely to choose the VI alternative with longer times since responses to the VI alternative, as predicted by momentary-maximizing theory. Subjects with the longer ITI failed to show either of these effects. Approximation to the matching law was greater with the longer ITI. Thus, matching is not derivative of the processes postulated by optimality or momentary-maximizing theory but instead is in competition with those processes.     

Schedules of reinforcement as determinants of human causality judgments and response rates

Experiments examined the effect of relationships between a response and an outcome on human judgments of causal effectiveness. In Experiment 1, the time between outcomes obtained on a variable ratio (VR) schedule became the intervals for a yoked variable interval (VI) schedule. Response rates were higher on the VR than on the VI schedule. In Experiment 2, the number of responses required per outcome on a VR schedule were matched to that on a master VI 20-s schedule. Both ratings of causal effectiveness and response rates were higher in the VR schedule. In Experiment 3, tandem VI fixed-ratio (FR) schedules produced higher rates and judgments than equivalent conjunctive VI FR schedule. In Experiment 4, a VI schedule with a reinforcement requirement for a short interresponse time (IRT) produced higher rates and judgments than a simple VI schedule. These results corroborate the view that schedules are a determinant of both response rates and causal judgments. Few current theories of causal judgment predict this pattern of results.     

Free-operant performance on variable interval schedules with a linear feedback loop: no evidence for molar sensitivities in rats

Four experiments examined rats' sensitivity to molar and molecular factors on instrumental schedules of reinforcement. Rats were exposed to a variable interval schedule with a positive feedback loop (VI+), such that faster responding led to a shorter interreinforcement interval. In Experiments 1 and 2, rats responded faster on a variable response (VR) schedule than on either a VI schedule matched for reinforcement rate or a VI+ schedule matched for the feedback function. In Experiment 3, rats responded no differently on a VI schedule than they did on a VI+ schedule with equated rates of reinforcement. In Experiment 4, rats responded faster on a VI+ schedule with an interresponse time requirement yoked to that experienced on a VR schedule, than on a VI+ schedule with the same feedback function as the VR schedule. Taken together these results suggest that rats are more sensitive to the molecular than the molar properties of the schedules.     

A local model of concurrent performance

Concurrent procedures may be conceptualized as consisting of two pairs of schedules with only one pair operating at a time. One schedule of each pair arranges reinforcers for staying in the current alternative, and the other schedule arranges reinforcers for switching to the other alternative. These pairs alternate operation as the animal switches between choices. This analysis of the contingencies suggests that variables operating within an alternative produce behavior that conforms to the generalized matching law. Rats were exposed to one pair of stay and switch schedules in each condition, and the probabilities of reinforcement varied across conditions. Both run length and visit duration were power functions of the ratio of the probabilities of reinforcement for staying and switching. The local model, a model of performance on concurrent procedures, was derived from this power function. Performance on concurrent schedules was synthesized from the performances on the separate pairs. Both the generalized matching law and the local model fitted the synthesized concurrent performances. These results are consistent with the view that the contingencies in the alternative, the probability of stay and switch reinforcement, are responsible for performance consistent with the generalized matching law. These results are compatible with momentary maximizing and molar maximizing accounts of concurrent performance. Models of concurrent performance that posit comparisons among the alternatives are not easily applied to these results.     

Cooperative responding in rats maintained by fixed‐ and variable‐ratio schedules

The present study investigated the effects of fixed‐ratio (FR) and variable‐ratio (VR) reinforcement schedules on patterns of cooperative responding in pairs of rats. Experiment 1 arranged FR 1, FR 10, and VR 10 schedules to establish cooperative responding (water delivery depended on the joint responding of two rats). Cooperative response rates and proportions were higher under intermittent schedules than under continuous reinforcement. The FR 10 schedule generated a break‐and‐run pattern, whereas the VR 10 schedule generated a relatively high and constant rate pattern. Experiment 2 evaluated the effects of parametric manipulations of FR and VR schedules on cooperative responding. Rates and proportions of cooperative responding generally increased between ratio sizes of 1 and 5 but showed no consistent trend as the ratio increased from 5 to 10. Experiment 3 contrasted cooperative responding between an FR6 schedule and a yoked control schedule. Coordinated behavior occurred at a higher rate under the former schedule. The present study showed that external consequences and the schedules under which the delivery of these consequences are based, select patterns of coordinated behavior.     

The role of the response-reinforcer relation in delay-of-reinforcement effects

The role of the response-reinforcer relation in maintaining operant behavior under conditions of delayed reinforcement was investigated by using a two-operandum (i.e., two-key) procedure with pigeons. Responding on one key was reinforced under a tandem variable-interval differential-reinforcement-of-other-behavior (tandem VI DRO) schedule. The schedule defined a resetting unsignaled delay-of-reinforcement procedure in that a response was required when the interfood interval of the VI schedule lapsed, but further responding during the DRO component on either key reset the time interval. This ensured a fixed delay duration between any response and reinforcement. Responding on another key, physically identical to the first one except for spatial location, otherwise was without consequence. The location of the key correlated with the delay-of-reinforcement procedure varied between sessions according to a semirandom sequence. Differences in response rates between the two keys were greater, with proportionally higher rates on the key correlated with the delay-of-reinforcement procedure, the longer the delay-of-reinforcement procedure remained correlated with the same key. Differences in responding on the two keys also increased within individual sessions. These results suggest that the response-reinforcer relation is the primary determinant of responding when responding is acquired and maintained with delayed reinforcement.     

A direct comparison of four methods for eliminating a response

Thirty-two rats pressed one lever (lever A) on a VI 30-sec schedule of food reinforcement and were then shifted to one of four procedures for eliminating the lever A response: extinction, differential reinforcement of other behavior, reinforcement of a different response (pole pushing), and reinforcement of a similar response (pressing lever B). Effectiveness of a response-elimination procedure was measured by (1) how quickly lever A response rate fell to a low level when the procedure was in effect, (2) how much lever A responding recovered when the procedure was discontinued, and (3) how resistant lever A responding was to reinstatement when the VI reinforcement schedule was reimposed. No one method was superior by all three measures. Extinction produced the most variable behavior, while differential reinforcement of other behavior produced the least. Reinforcing alternative behavior produced the greatest recovery in the original lever A response when the response-elimination procedure was discontinued.     

Concurrent schedules: Maximization versus reinforcement of changeover behaviour

Pigeons responded on concurrent variable-interval 180-sec variable-interval 36-sec schedules during Conditions 1 and 3 of Experiment 1. Condition 2 arranged variable-interval 60-sec schedules for both response alternatives. The schedule assigned to the alternative that was associated with the variable-interval 36-sec schedule in Conditions 1 and 3 operated only when the subject responded on that alternative. The proportion of time spent responding on the alternative with the conventional variable-interval 60-sec schedule increased during Condition 2, but exclusive choice of that alternative did not develop. This result is inconsistent with maximization of the overall reinforcement rate and with maximization of the momentary probability of reinforcement (momentary maximizing). Increasing time proportions were also found in Experiment 2, which arranged similar conditions, except that reinforcement was provided on a variable-time basis. The time proportions were close to the momentary maximizing prediction in Experiment 2. The results of both experiments can be explained if it is assumed that time allocation is controlled by delayed reinforcement of changeovers between alternatives.     

Variable-ratio schedules as variable-interval schedules with linear feedback loops.

The mathematical theory of linear systems has been used successfully to describe responding on variable-interval (VI) schedules. In the simplest extension of the theory to the variable-ratio (VR) case, VR schedules are treated as if they were VI schedules with linear feedback loops. The assumption entailed by this approach, namely, that VR and VI-plus-linear-feedback schedules are equivalent, was tested by comparing responding on the two types of schedule. Four human subjects' lever pressing produced monetary reinforcers on five VR schedules, and on five VI schedules with linear feedback loops that reproduced the feedback properties of the VR schedules. Pressing was initiated by instructions in 2 subjects, and was shaped by successive approximation in the other 2. The different methods of response initiation did not have differential effects on behavior. For each of the 4 subjects, the VR and the comparable VI-plus-linear-feedback schedules generated similar average response rates and similar response patterns. The subjects' behavior on both types of schedule was similar to that of avian and rodent species on VR schedules. These results indicate that the assumption entailed by the VI-plus-linear-feedback approach to the VR case is valid and, consequently, that the approach is worth pursuing. The results also confute interresponse-time theories of schedule performance, which require interval and ratio contingencies to produce different response rates.     

Influence of the cost of responding

In three experiments, the effect of costs associated with responding on judgments of the causal effectiveness of the response was examined. In Experiment 1, the temporal interval between outcomes was matched on a variable interval (VI) and a variable ratio (VR) schedule. When each response was made at some “cost,” and outcomes represented some “gain” for the subject, the rating of causal effectiveness for responses was higher on the VR than on the VI schedule. This relationship was absent when the outcome was a triangle Hash. In Experiment 2, the number of responses required per outcome on a VI and a VR schedule were matched, and responses on the VR schedule were rated as more causally effective. In Experiment 3, a VI-to-VR yoking procedure was used. With minimal response costs, judgments were similar on the VI and VR schedules, but with greater response costs, responses performed on the VR schedule were rated as more causally effective than those emitted on the VI schedule.     

A simple BASIC program to generate values for variable-interval schedules of reinforcement

A BASIC program to generate values for variable-interval (VI) schedules of reinforcement is presented. A VI schedule should provide access to reinforcement with a constant probability over a time horizon. If the values in a VI schedule are calculated from an arithmetic progression, the probability of reinforcement is positively correlated with the time since the last reinforcer was delivered. Fleshler and Hoffman (1962) developed an iterative equation to calculate VI schedule values so that the probability of reinforcement remains constant. This easy-to-use program generates VI schedule values according to the Fleshler and Hoffman equation, randomizes the values, and saves the values in ASCII to a disk file.