How to teach a pigeon to maximize overall reinforcement rate

In two experiments deviations from matching earned higher overall reinforcement rates than did matching. In Experiment 1 response proportions were calculated over a 360-response moving average, updated with each response. Response proportions that differed from the nominal reinforcement proportions, by a criterion that was gradually increased, were eligible for reinforcement. Response proportions that did not differ from matching were not eligible for reinforcement. When the deviation requirement was relatively small, the contingency proved to be effective. However, there was a limit as to how far response proportions could be pushed from matching. Consequently, when the deviation requirement was large, overall reinforcement rate decreased and pecking was eventually extinguished. In Experiment 2 a discriminative stimulus was added to the procedure. The houselight was correlated with the relationship between response proportions and the nominal (programmed) reinforcement proportions. When the difference between response and reinforcement proportions met the deviation requirement, the light was white and responses were eligible for reinforcement. When the difference between response and reinforcement proportions failed to exceed the deviation requirement, the light was blue and responses were not eligible for reinforcement. With the addition of the light, it proved to be possible to shape deviations from matching without any apparent limit. Thus, in Experiment 2 overall reinforcement rate predicted choice proportions and relative reinforcement rate did not. In contrast, in previous experiments on the relationship between matching and overall reinforcement maximization, relative reinforcement rate was usually the better predictor of responding. The results show that whether overall or relative reinforcement rate better predicts choice proportions may in part be determined by stimulus conditions.     

Matching under concurrent fixed-ratio variable-interval schedules of food presentation

Four pigeons were exposed to concurrent fixed-ratio, variable-interval schedules of food presentation. The fixed-ratio requirement was either 25, 50, 75, or 100 responses, with the variable-interval schedule parameter held constant at 4 minutes. A delay time was imposed between a changeover from one schedule to the other and subsequent food availability. The delay time was varied at each ratio requirement over four values; no delay, 0-second delay, 1.5-second delay, and 5.0-second delay. As the fixed-ratio requirement or the delay time increased, a greater proportion of the total responses and time spent responding occurred under the variable-interval schedule relative to the proportion of food deliveries under that schedule. Neither relative overall response rate nor relative time spent responding equalled the relative frequency of food presentation, as would be predicted by a linear “matching” model. Rather, these data were described by power functions with slopes of approximately 1.0 and intercepts greater than 1.0. In the terms of Baum's (1974) analysis, these deviations from linear matching represent bias in favor of responding under the interval schedule. Bias, as reflected in the intercept of the power function, was greater for the ratio of time than the ratio of responses.     

Human choice on concurrent variable-interval variable-ratio schedules.

Each of 5 adult male humans sat in a 4 degrees C room where they could warm themselves by illuminating six heat lamps for 10-second periods according to a concurrent variable-interval variable-ratio schedule. Left-button presses on a response panel switched between the schedules and started a 2-second changeover delay. Right-button presses illuminated the heat lamps if assigned by the associated schedule and if the changeover delay had timed out. Panel lights identified the schedule in effect and each effective right-button press. A discrimination procedure--either a multiple variable-interval variable-ratio schedule or the presentation of each schedule individually on alternate days--preceded exposure to the choice procedure for some subjects. For subjects not exposed to a discrimination procedure prior to exposure to choice, or if such exposure failed to result in higher rates to the ratio than to the interval schedule, relative response rates matched relative reinforcement rates. However, if subjects responded at higher rates to the ratio schedule than to the interval schedule during a prior discrimination procedure, relative rates on a subsequent choice procedure deviated from matching in the direction of reinforcement-rate maximizing. In eight of 11 conditions, choice appeared to be governed by maximizing processes. In all cases, human concurrent ratio-interval performances differed from those of nonhumans in that matching was never obtained with local ratio-interval rate differences.     

Response rate and changeover performance on concurrent variable-interval schedules

Six pigeons were exposed to variable-interval schedules arranged on one, two, three, and four response keys. The reinforcement rate was also varied across conditions. Numbers of responses, the time spent responding, the number of reinforcements, and the number of changeovers between keys were recorded. Response rates on each key were an increasing function of reinforcement rate on that key and a decreasing function of the reinforcement rate on other keys. Response and time-allocation ratios under-matched ratios of obtained reinforcements. Three sets of equations were developed to express changeover rate as a function of response rate, time allocation, and reinforcement rate respectively. These functions were then applied to a broad range of experiments in the literature in order to test their generality. Further expressions were developed to account for changeover rates reported in experiments where changeover delays were varied.     

Undermatching: a reappraisal of performance on concurrent variable-interval schedules of reinforcement

The extant data for pigeons' performance on concurrent variable-interval schedules were examined in detail. Least-squares lines relating relative pecks and time to the corresponding relative reinforcements were obtained for four studies. The between-study group slopes for time and pecks and five of seven within-study group slopes from individual studies were less than 1.00. This suggested the generality that pigeons respond less to the richer reinforcement schedule than predicted by matching. For pecks, a nonparametric test for distribution of points also supported this concept of undermatching (to the richer reinforcement schedule). In addition, using mean squared error as the criterion, a cubic curve fit the peck proportion data better than any line or other polynomial. This indicates that the relation between peck and reinforcement proportions may be nonlinear.     

Performance of rats under concurrent variable-interval schedules of negative reinforcement

The behavior of rats under concurrent variable-interval schedules of negative reinforcement was examined. A single one-minute variable-interval programmer determined the availability of 30-second timeouts from electric shock. These were assigned to one or the other of the two component schedules with a probability of 0 to 1.0. The response requirement for the component schedules was standing to the right or left of the center of the experimental chamber. With a six-second changeover delay, the relative time spent under one component schedule varied directly and linearly with the relative number of timeouts earned under that component schedule. The absolute number of changeovers was highest when a similar number of timeouts was earned under each component schedule, and lowest when all or nearly all timeouts were earned under one component schedule. In general, these relations are similar to those reported with concurrent variable-interval schedules of positive reinforcement.     

Maximizing versus matching on concurrent variable-interval schedules

Maximization and matching predictions were examined for a time-based analogue of the concurrent variable-interval variable-ratio schedule. One alternative was a variable interval whose time base operated relatively independent of the schedule chosen, and the other was a discontinuous variable interval for which timing progressed only when selected. Pigeons switched between schedules by pecking a changeover key. The maximization hypothesis predicts that subjects will show a bias toward the discontinuous variable interval and undermatching; however the obtained results conformed closely to the predictions of the matching law. Finally, a quantitative comparison was made of the bias and sensitivity estimates obtained in published concurrent variable-interval variable-ratio analogue studies. Results indicated that only the ratio-based analogue of the concurrent variable interval variable ratio studied by Green, Rachlin, and Hanson (1983) produced significant bias toward the variable-ratio alternative and undermatching, as predicted by reinforcement maximization.     

Effects of variable-interval value and amount of training on stimulus generalization.

In Experiment 1 pigeons pecked a key that was illuminated with a 501-nm light and obtained food by doing so according to a variable-interval (VI) schedule of reinforcement, the mean value of which differed across groups: either 30 s, 120 s, or 240 s. The pigeons in all three groups were trained for 10 50-min sessions. Generalization testing was conducted in extinction with different wavelengths of light. Absolute and relative generalization gradients were similar in shape for the three groups. Experiment 2 was a systematic replication of Experiment 1 using line orientation as the stimulus dimension and a mean VI value of either 30 s or 240 s. Again, gradients of generalization were similar for the two groups. In Experiment 3 pigeons pecked a key that was illuminated with a 501-nm light and obtained food reinforcers according to either a VI 30-s or a 240-s schedule. Training continued until response rates stabilized (> 30 sessions). For subjects trained with the 30-s schedule, generalization gradients were virtually identical regardless of whether training was for 10 sessions (Experiment 1) or until response rates stabilized. For subjects trained with the VI 240-s schedule, absolute generalization gradients for subjects trained to stability were displaced upward relative to gradients for subjects trained for only 10 sessions (Experiment 1), and relative generalization gradients were slightly flatter. These results indicate that the shape of a generalization gradient does not necessarily depend on the rate of reinforcement during 10-session single-stimulus training but that the effects of prolonged training on stimulus generalization may be schedule dependent.     

Response-rate differences in variable-interval and variable-ratio schedules: An old problem revisited

In Experiment 1, a variable-ratio 10 schedule became, successively, a variable-interval schedule with only the minimum interreinforcement intervals yoked to the variable ratio, or a variable-interval schedule with both interreinforcement intervals and reinforced interresponse times yoked to the variable ratio. Response rates in the variable-interval schedule with both interreinforcement interval and reinforced interresponse time yoking fell between the higher rates maintained by the variable-ratio schedule and the lower rates maintained by the variable-interval schedule with only interreinforcement interval yoking. In Experiment 2, a tandem variable-interval 15-s variable-ratio 5 schedule became a yoked tandem variable-ratio 5 variable-interval x-s schedule, and a tandem variable-interval 30-s variable-ratio 10 schedule became a yoked tandem variable-ratio 10 variable-interval x-s schedule. In the yoked tandem schedules, the minimum interreinforcement intervals in the variable-interval components were those that equated overall interreinforcement times in the two phases. Response rates did not decline in the yoked schedules even when the reinforced interresponse times became longer. Experiment 1 suggests that both reinforced interresponse times and response rate–reinforcement rate correlations determine response-rate differences in variable-ratio 10 and yoked variable-interval schedules in rats. Experiment 2 suggests a minimal role for the reinforced interresponse time in determining response rates on tandem variable-interval 30-s variable-ratio 10 and yoked tandem variable-ratio 10 variable-interval x-s schedules in rats.     

Changeover ratio effects on concurrent variable-interval performance

Rats' bar-pressing was maintained by concurrent variable-interval schedules of reinforcement. A fixed-ratio of pulls on a chain (the changeover ratio) was required for switching between schedules. The first experiment employed equal variable-interval schedules and symmetrical changeover ratios. Increasing these ratios resulted in a decrease in the rate of switching between schedules and an increase in local response rate. In the second experiment, a range of asymmetrical changeover ratios was used with equal variable-interval schedules, and a preference was found for the schedule associated with the larger switching-into ratio. Both the distributions of responses and time between the two schedules deviated from those expected on the basis of obtained reinforcers. In the third experiment, the switching-out-of ratio was dependent on the amount of time spent in a variable-interval 2-minute schedule; a constant ratio permitted switching out of the alternative variable-interval 1-minute schedule. A strong preference was shown for the variable-interval 2-minute schedule. The fourth experiment used equal variable-interval schedules; one changeover ratio was varied while the second remained constant. The results failed to show systematic differences in local response rates immediately after a changeover.     

Control rate of response or reinforcement and amphetamine's effect on behavior.

The roles of control response rate and reinforcement frequency in producing amphetamine's effect on operant behavior were evaluated independently in rats. Two multiple schedules were arranged in which one variable, either response rate or reinforcement frequency, was held constant and the other variable manipulated. A multiple differential-reinforcement-of-low-rate seven-second yoked variable-interval schedule was used to equate reinforcement frequencies at different control response rates between multiple-schedule components. Amphetamine increased responding under the variable-interval component. In contrast, amphetamine decreased responding equivalently between components of a multiple random-ratio schedule that produced similar control response rates at different reinforcement frequencies. The results provide experimental support to the rate-dependency principle that control rate of responding is an important determinant of amphetamine's effect on operant behavior.     

Performance on variable-interval schedules arranged singly and concurrently

Extensive parametric data were obtained from pigeons responding on variable-interval schedules arranged on three, two, and one response keys. Number of responses on the keys, the time spent responding on the keys, and the number of reinforcements obtained on the keys were measured. Response rates on each key were an increasing function of the reinforcement rate on that key, and an inverse function of the reinforcement rate on the other keys. In terms of preference, both response and time-allocation ratios undermatched ratios of obtained reinforcements, and the degree of undermatching was consistent both within, and between, two- and three-schedule data. When absolute response-rate data were analyzed according to Herrnstein's (1970) quantitative account, obtained values of assumed constants were not consistent either within or between conditions. However, a power-function modification of Herrnstein's account fitted the data well and provided similar exponent values to those obtained for the undermatching of preference ratios.     

Stock optimizing: maximizing reinforcers per session on a variable-interval schedule.

In Experiment 1, 2 monkeys earned their daily food ration by pressing a key that delivered food according to a variable-interval 3-min schedule. In Phases 1 and 4, sessions ended after 3 hr. In Phases 2 and 3, sessions ended after a fixed number of responses that reduced food intake and body weights from levels during Phases 1 and 4. Monkeys responded at higher rates and emitted more responses per food delivery when the food earned in a session was reduced. In Experiment 2, monkeys earned their daily food ration by depositing tokens into the response panel. Deposits delivered food according to a variable-interval 3-min schedule. When the token supply was unlimited (Phases 1, 3, and 5), sessions ended after 3 hr. In Phases 2 and 4, sessions ended after 150 tokens were deposited, resulting in a decrease in food intake and body weight. Both monkeys responded at lower rates and emitted fewer responses per food delivery when the food earned in a session was reduced. Experiment 1's results are consistent with a strength account, according to which the phases that reduced body weights increased food's value and therefore increased subjects' response rates. The results of Experiment 2 are consistent with an optimizing strategy, because lowering response rates when food is restricted defends body weight on variable-interval schedules. These contrasting results may be attributed to the discriminability of the contingency between response number and the end of a session being greater in Experiment 2 than in Experiment 1. In consequence, subjects lowered their response rates in order to increase the number of reinforcers per session (stock optimizing).     

Effect of variable-interval punishment on the behavior of humans in variable-interval schedules of monetary reinforcement

One male and three female human subjects pressed a button for monetary reinforcement under a range of variable-interval schedules specifying different frequencies of reinforcement. On alternate days, responding was also punished (by subtraction of money) according to a variable-interval 170-second schedule. In the absence of punishment, the rate of responding was an increasing negatively accelerated function of reinforcement frequency, as predicted by Herrnstein's equation. The effect of the punishment schedule was to suppress responding under lower frequencies of reinforcement; responding under higher reinforcement frequencies was much less affected. This was reflected in an increase in the value of K_H (the constant expressing the reinforcement frequency corresponding to the half-maximal response rate), whereas there was no significant change in the value of R_max (the constant expressing the maximum response rate). Previous results had shown that variable-ratio punishment resulted in a change in the values of both constants (Bradshaw, Szabadi, and Bevan, 1977). The results of the present study were consistent with the concept that the suppressive effects of punishment on responding depend on the nature of the punishment schedule.     

Effect of punishment on human variable-interval performance

Three female human subjects pressed a button for monetary reinforcement in a range of variable-interval schedules specifying different frequencies of reinforcement. On alternate days, responding was also punished (by subtracting money) according to a variable-ratio 34 schedule. In the absence of punishment, rate of responding was an increasing negatively accelerated function of reinforcement frequency; the relationship between response rate and reinforcement frequency conformed to Herrnstein's equation. The effect of the punishment schedule was to suppress responding at all frequencies of reinforcement. This was reflected in a change in the values of both constants in Herrnstein's equation: the value of the theoretical maximum response-rate parameter was reduced, while the parameter describing the reinforcement frequency corresponding to the half-maximal response rate was increased.     

Rats' performance on variable-interval schedules with a linear feedback loop between response rate and reinforcement rate

Three experiments investigated whether rats are sensitive to the molar properties of a variable-interval (VI) schedule with a positive relation between response rate and reinforcement rate (i.e., a VI+ schedule). In Experiment 1, rats responded faster on a variable ratio (VR) schedule than on a VI+ schedule with an equivalent feedback function. Reinforced interresponse times (IRTs) were shorter on the VR as compared to the VI+ schedule. In Experiments 2 and 3, there was no systematic difference in response rates maintained by a VI+ schedule and a VI schedule yoked in terms of reinforcement rate. This was found both when the yoking procedure was between-subject (Experiment 2) and within-subject (Experiment 3). Mean reinforced IRTs were similar on both the VI+ and yoked VI schedules, but these values were more variable on the VI+ schedule. These results provided no evidence that rats are sensitive to the feedback function relating response rate to reinforcement rate on a VI+ schedule.     

Local response-rate constancy on concurrent variable-interval schedules of reinforcement

Concurrent variable-interval schedules were arranged with a main key that alternated in color and schedule assignment, along with a changeover key on which a small fixed ratio was required to changeover. Acceptable matching was observed with pigeons in two replications, but there was a tendency toward overmatching. Local response rates were found to differ for unequal schedules of a concurrent pair: local response rate was greater for the variable-interval schedule with the smaller average interreinforcement interval, but qualifications based on an interresponse-time analysis were discussed. In a second experiment, two 3-minute variable-interval schedules were arranged concurrently, and the experimental variable was the changeover procedure: either a changeover delay was incurred by each changeover or a small fixed ratio on a changeover key was required to complete a changeover. Changeover delays of 2 and 5 seconds were compared with a fixed-ratio changeover of five responses. The response output on the main key (associated with the variable-interval schedules) was greater when a changeover delay was arranged than when a fixed ratio was required to changeover. A detailed analysis of stripchart records showed that a 2-second delay generated an increased response rate for 3 seconds after a changeover, while the fixed-ratio requirement generated an increased rate during the first second only, followed by a depressed response rate for 2 seconds.