Contrast and reallocation of extraneous reinforcers between multiple-schedule components

Four pigeons responded in components of multiple schedules in which two responses were available and reinforced with food. Pecks on the left key (“main” key) were reinforced at a constant rate in one component and at a rate that varied over conditions in the other component. When reinforcer rate was varied, behavioral contrast occurred in the constant component. On the right key (“extra” key), five variable-interval schedules and one variable-ratio schedule, presented conjointly, arranged reinforcers for responses in all conditions. These conjoint schedules were common to both multiple-schedule components—rather than unique to particular components—and reinforcers from these schedules could therefore be arranged in one component and obtained during the other component. In this way, the additional reinforcers were analogous to the “extraneous” reinforcers thought to maintain behavior other than pecking in conventional multiple schedules. Response rate on the extra key did not change systematically over conditions in the constant component, and in the varied component extra responding was inversely related to main-key reinforcement. All subjects obtained more extra-key reinforcers in whichever component arranged fewer main-key reinforcers. Consistent with the theory that reallocation of extraneous reinforcers may cause behavioral contrast, absolute reinforcer rate for the extra key in the constant component was low in conditions that produced positive contrast on the main key and high in those that produced negative contrast. Also consistent with this theory, behavioral contrast was reduced in two conditions that canceled extra-key reinforcers that had been arranged but not obtained at the end of components. Thus, a constraint on reallocation markedly reduced the extent of contrast.     

Effects of the discriminability of alternatives in three-alternative concurrent-schedule performance

Six pigeons were trained on two- and three-alternative concurrent schedules in which the alternatives were signaled by different wavelengths of light on the main pecking key. The schedules were arranged according to a switching-key procedure in which pecks on a white side key produced a 3-s blackout and, intermittently, a change in the variable-interval schedule of food programmed on the main (center) key after the blackout. In Part 1, a two-alternative concurrent variable-interval schedule was arranged in which the alternatives were signaled by 560 nm and 630 nm. Parts 2 and 3 arranged three-alternative concurrent variable-interval schedules with the alternatives signaled by 560 nm, 600 nm, and 630 nm (Part 2) and 560 nm, 623 nm, and 630 nm (Part 3). Within each part, the relative rate of food reinforcers available on the alternatives was varied across a wide range. In all parts of the experiment, the ratios of responses emitted between pairs of alternatives were more extreme than the ratios of reinforcers obtained on the pairs of alternatives, a result termed overmatching. In Parts 2 and 3, generalized matching sensitivities between pairs of alternatives were found to be higher when the reinforcer rate on the third alternative was low than when it was high—an apparent failure of the constant-ratio rule. The data were well described by an extension of the Davison and Jenkins (1985) model, which assumes differing discriminabilities between concurrent-schedule alternatives in combination with a punishing effect of blackout following changeovers.     

Reporting contingencies of reinforcement in concurrent schedules

Five pigeons were trained on concurrent variable-interval schedules in which two intensities of yellow light served as discriminative stimuli in a switching-key procedure. A conditional discrimination involving a simultaneous choice between red and green keys followed every reinforcer obtained from both alternatives. A response to the red side key was occasionally reinforced if the prior reinforcer had been obtained from the bright alternative, and a response to the green side key was occasionally reinforced if the prior reinforcer had been obtained from the dim alternative. Measures of the discriminability between the concurrent-schedule alternatives were obtained by varying the reinforcer ratio for correct red and correct green responses across conditions in two parts. Part 1 arranged equal rates of reinforcement in the concurrent schedule, and Part 2 provided a 9:1 concurrent-schedule reinforcer ratio. Part 3 arranged a 1:9 reinforcer ratio in the conditional discrimination, and the concurrent-schedule reinforcer ratio was varied across conditions. Varying the conditional discrimination reinforcer ratio did not affect response allocation in the concurrent schedule, but varying the concurrent-schedule reinforcer ratio did affect conditional discrimination performance. These effects were incompatible with a contingency-discriminability model of concurrent-schedule performance (Davison & Jenkins, 1985), which implies a constant discriminability parameter that is independent of the obtained reinforcer ratio. However, a more detailed analysis of conditional discrimination performance showed that the discriminability between the concurrent-schedule alternatives decreased with time since changing over to an alternative. This effect, combined with aspects of the temporal distribution of reinforcers obtained in the concurrent schedules, qualitatively predicted the molar results and identified the conditions that operate whenever contingency discriminability remains constant.     

Leaving patches: An investigation of a laboratory analogue

Five pigeons were trained on a procedure that has been used as a laboratory analogue to natural patch residence. Trials commenced with two responses available. One of these might provide a reinforcer if the patch was a prey patch; the other ended the residence time in the patch and, after a fixed travel time in blackout, produced another patch that might or might not provide a reinforcer. Patch residence also ended, and was followed by the same travel time, after a reinforcer was obtained or after a fixed maximum time was spent in the patch. The dependent variable was patch residence time, from the commencement of the patch to the time at which the subject emitted a response to exit from the patch or until the maximum patch residence time had elapsed. In Parts 1 to 3, the duration of the imposed travel time was varied from 0.25 to 16 s at three different probabilities (.05, .1, and .2) of food per second (λ) in prey patches. As reported in previous research, both increasing travel time and decreasing probabilities of reinforcers per second increased patch residence time. In Parts 4 to 7, the probability of prey trials (ρ) was varied in an irregular order from .1, through .2, .5, and .7, to .9 for different combinations of λ and travel time. Respectively, these were in Part 4, .05 per second and 0.25 s; in Part 5, .05 per second and 16 s; in Part 6, .2 per second and 0.25 s; and in Part 7, .2 per second and 16 s. A previously offered model, based on optimization assumptions, substantially and consistently underpredicted patch residence time. However, a modification of that model, which assumes that the subjects could not accurately discriminate the residence time that provided the minimum interreinforcer interval, described the data well. The same model also described previously reported residence times in a different species with a uniform distribution of prey-arrival times.     

Choosing among multiple alternatives: Relative and overall reinforcer rates

Choice behavior among two alternatives has been widely researched, but fewer studies have examined the effect of multiple (more than two) alternatives on choice. Two experiments investigated whether changing the overall reinforcer rate affected preference among three and four concurrently scheduled alternatives. Experiment 1 trained six pigeons on concurrent schedules with three alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 9:3:1 with the configuration counterbalanced across pigeons. The overall rate of reinforcement was varied across conditions. Preference between the pair of keys arranging the 9:3 reinforcer ratio was less extreme than the pair arranging the 3:1 reinforcer ratio regardless of overall reinforcer rate. This difference was attributable to the richer alternative receiving fewer responses per reinforcer than the other alternatives. Experiment 2 trained pigeons on concurrent schedules with four alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 8:4:2:1, and the overall reinforcer rate was varied. Next, two of the alternatives were put into extinction and the random interval duration was changed from 60 s to 5 s. The ratio of absolute response rates was independent of interval length across all conditions. In both experiments, an analysis of sequences of visits following each reinforcer showed that the pigeons typically made their first response to the richer alternative irrespective of which alternative was just reinforced. Performance on these three‐ and four‐alternative concurrent schedules is not easily extrapolated from corresponding research using two‐alternative concurrent schedules.     

Choice in a variable environment: every reinforcer counts

Six pigeons were trained in sessions composed of seven components, each arranged with a different concurrent-schedule reinforcer ratio. These components occurred in an irregular order with equal frequency, separated by 10-s blackouts. No signals differentiated the different reinforcer ratios. Conditions lasted 50 sessions, and data were collected from the last 35 sessions. In Part 1, the arranged overall reinforcer rate was 2.22 reinforcers per minute. Over conditions, number of reinforcers per component was varied from 4 to 12. In Part 2, the overall reinforcer rate was six per minute, with both 4 and 12 reinforcers per component. Within components, log response-allocation ratios adjusted rapidly as more reinforcers were delivered in the component, and the slope of the choice relation (sensitivity) leveled off at moderately high levels after only about eight reinforcers. When the carryover from previous components was taken into account, the number of reinforcers in the components appeared to have no systematic effect on the speed at which behavior changed after a component started. Consequently, sensitivity values at each reinforcer delivery were superimposable. However, adjustment to changing reinforcer ratios was faster, and reached greater sensitivity values, when overall reinforcer rate was higher. Within a component, each successive reinforcer from the same alternative ("confirming") had a smaller effect than the one before, but single reinforcers from the other alternative ("disconfirming") always had a large effect. Choice in the prior component carried over into the next component, and its effects could be discerned even after five or six reinforcement and nonreinforcement is suggested.     

Examining the discriminative and strengthening effects of reinforcers in concurrent schedules

Reinforcers may increase operant responding via a response-strengthening mechanism whereby the probability of the preceding response increases, or via some discriminative process whereby the response more likely to provide subsequent reinforcement becomes, itself, more likely. We tested these two accounts. Six pigeons responded for food reinforcers in a two-alternative switching-key concurrent schedule. Within a session, equal numbers of reinforcers were arranged for responses to each alternative. Those reinforcers strictly alternated between the two alternatives in half the conditions, and were randomly allocated to the alternatives in half the conditions. We also varied, across conditions, the alternative that became available immediately after a reinforcer. Preference after a single reinforcer always favored the immediately available alternative, regardless of the local probability of a reinforcer on that alternative (0 or 1 in the strictly alternating conditions, .5 in the random conditions). Choice then reflected the local reinforcer probabilities, suggesting some discriminative properties of reinforcement. At a more extended level, successive same-alternative reinforcers from an alternative systematically shifted preference towards that alternative, regardless of which alternative was available immediately after a reinforcer. There was no similar shift when successive reinforcers came from alternating sources. These more temporally extended results may suggest a strengthening function of reinforcement, or an enhanced ability to respond appropriately to "win-stay" contingencies over "win-shift" contingencies.     

Contrast and reallocation of extraneous reinforcers as a function of component duration and baseline rate of reinforcement

Four pigeons responded on multiple schedules arranged on a “main” key in a two-key experimental chamber. A constant schedule component was alternated with another component that was varied over conditions. On an extra response key, conjoint schedules of reinforcement that operated in both components were arranged concurrently with the multiple schedule on the main key. On the main key, changes in reinforcement rate in the varied component were inversely related to changes in response rates in the constant component (behavioral contrast). On the extra key, some reinforcers were reallocated between components, depending on the schedules in effect on the main key in the varied component. In the varied component, the obtained rates of reinforcement on the extra key were inversely related to main-key reinforcement rate. In the constant component, extra-key reinforcer rates were positively related to main-key reinforcer rates obtained in the varied component, and were not a function of response rates on the extra key. In two comparisons, the rate at which components alternated and the value of the main-key schedule in the constant component were varied. Consistent with earlier work, long components reduced the extent of contrast. Reductions in contrast as a function of component duration were accompanied by similar reductions in the extent of reinforcer reallocation on the extra key. In the second comparison, lowering the rate of reinforcement in the constant component increased the rate at which extra-key reinforcers were obtained, reduced the extent of reinforcer reallocation, and reduced contrast. Overall, the results are consistent with the suggestion that some contrast effects are due to the changes in extraneous reinforcement during the constant component, and that manipulations of component duration, and manipulations of the rate of reinforcement in the constant component, affect contrast because they influence the extent of extraneous reinforcer real-location.     

Concurrent-schedule performance: Effects of relative and overall reinforcer rate

Six pigeons were trained to respond on two keys, each of which provided reinforcers on an arithmetic variable-interval schedule. These concurrent schedules ran nonindependently with a 2-s changeover delay. Six sets of conditions were conducted. Within each set of conditions the ratio of reinforcers available on the two alternatives was varied, but the arranged overall reinforcer rate remained constant. Each set of conditions used a different overall reinforcer rate, ranging from 0.22 reinforcers per minute to 10 reinforcers per minute. The generalized matching law fit the data from each set of conditions, but sensitivity to reinforcer frequency (a) decreased as the overall reinforcer rate decreased for both time allocation and response allocation based analyses of the data. Overall response rates did not vary with changes in relative reinforcer rate, but decreased with decreases in overall reinforcer rate. Changeover rates varied as a function of both relative and overall reinforcer rates. However, as explanations based on changeover rate seem unable to deal with the changes in generalized matching sensitivity, discrimination accounts of choice may offer a more promising interpretation.     

Concurrent schedules: short- and long-term effects of reinforcers

Five pigeons were trained on concurrent variable-interval schedules in a switching-key procedure. The overall rate of reinforcement was constant in all conditions, and the ratios of reinforcers obtainable on the two alternatives were varied over seven levels. Each condition remained in effect for 65 sessions, and the last 50 sessions of data from each condition were analyzed. The most recently obtained reinforcer had the largest effect on current preference, but each of the eight previously obtained reinforcers had a small measurable effect. These effects were larger when the reinforcer ratio was more extreme. A longer term effect of reinforcement was also evident, which changed as a function of the reinforcer ratio arranged. More local analyses showed regularities at a reinforcer-by-reinforcer level and large transient movements in preference toward the just-reinforced alternative immediately following reinforcers, followed by a return to stable levels that were related to the reinforcer ratio in effect. The present data suggest that the variables that control choice have both short- and long-term effects and that the short-term effects increased when the reinforcer ratios arranged were more extreme.     

The control of choice by its consequences

Five pigeons were trained on concurrent variable-interval schedules in which equal rates of reinforcement were always arranged for left- and right-key responses, but different overall rates were signaled by key colors. Sessions began with both keys lit yellow for the instrumental phase. If, after 20 s of this phase, the relative number of responses that had been made to the left key equaled or exceeded .75, both keys changed red for the contingent phase. The contingent phase arranged another concurrent variable-interval schedule for a further 20 s before the instrumental phase was reinstated. However, if preference in the instrumental phase did not exceed .75, the instrumental phase continued for a further 20 s before preference was again compared with the criterion. In Part 1, the reinforcer rate arranged in the instrumental phase was held constant at 4.8 reinforcers per minute, while the reinforcer rate arranged in the contingent phase was varied across conditions from 0 to 19.2 over five steps. In Part 2, reinforcer rates in the contingent phase were kept constant at 36 per minute, while reinforcer rates in the instrumental phase were varied from 0 to 36 over seven steps. Part 3 replicated Part 2 but used reinforcer rates in both phases that were one third of those arranged in Part 2. Measures of choice obtained by summing responses across presentations of the instrumental phase became more extreme toward the left key as the reinforcer rate obtained in the contingent phase was increased (Part 1) and as the reinforcer rate obtained in the instrumental phase was decreased (Parts 2 and 3). Changes in these measures of choice were accompanied by systematic changes in the relative frequency with which the criterion was exceeded. Changes in both these measures were correlated with changes in the relative frequency with which subjects responded exclusively to the left key. These results are discussed with respect to the two choices that were concurrently available in this procedure and the response alternatives that might constitute the concurrent operants in each choice.     

Concurrent schedules: reinforcer magnitude effects

Five pigeons were trained on pairs of concurrent variable-interval schedules in a switching-key procedure. The arranged overall rate of reinforcement was constant in all conditions, and the reinforcer-magnitude ratios obtained from the two alternatives were varied over five levels. Each condition remained in effect for 65 sessions and the last 50 sessions of data from each condition were analyzed. At a molar level of analysis, preference was described well by a version of the generalized matching law, consistent with previous reports. More local analyses showed that recently obtained reinforcers had small measurable effects on current preference, with the most recently obtained reinforcer having a substantially larger effect. Larger reinforcers resulted in larger and longer preference pulses, and a small preference was maintained for the larger-magnitude alternative even after long inter-reinforcer intervals. These results are consistent with the notion that the variables controlling choice have both short- and long-term effects. Moreover, they suggest that control by reinforcer magnitude is exerted in a manner similar to control by reinforcer frequency. Lower sensitivities when reinforcer magnitude is varied are likely to be due to equal frequencies of different sized preference pulses, whereas higher sensitivities when reinforcer rates are varied might result from changes in the frequencies of different sized preference pulses.     

Local preference in concurrent schedules: the effects of reinforcer sequences

We investigated the effects that sequences of reinforcers obtained from the same response key have on local preference in concurrent variable-interval schedules with pigeons as subjects. With an overall reinforcer rate of one every 27 s, on average, reinforcers were scheduled dependently, and the probability that a reinforcer would be arranged on the same alternative as the previous reinforcer was manipulated. Throughout the experiment, the overall reinforcer ratio was 1:1, but across conditions we varied the average lengths of same-key reinforcer sequences by varying this conditional probability from 0 to 1. Thus, in some conditions, reinforcer locations changed frequently, whereas in others there tended to be very long sequences of same-key reinforcers. Although there was a general tendency to stay at the just-reinforced alternative, this tendency was considerably decreased in conditions where same-key reinforcer sequences were short. Some effects of reinforcers are at least partly to be accounted for by their signaling subsequent reinforcer locations.     

Choice in a "self-control" paradigm: effects of a fading procedure

Pigeons chose between an immediate 2-second reinforcer (access to grain) and a 6-second reinforcer delayed 6 seconds. The four pigeons in the control group were exposed to this condition initially. The four experimental subjects first received a condition where both reinforcers were delayed 6 seconds. The small reinforcer delay was then gradually reduced to zero over more than 11,000 trials. Control subjects almost never chose the large delayed reinforcer. Experimental subjects chose the large delayed reinforcer significantly more often. Two experimental subjects showed preference for the large reinforcer even when the consequences for pecking the two keys were switched. The results indicate that fading procedures can lead to increased “self-control” in pigeons in a choice between a large delayed reinforcer and a small immediate reinforcer.     

Choice between single and multiple delayed reinforcers.

Pigeons chose between alternatives that differed in the number of reinforcers and in the delay to each reinforcer. A peck on a red key produced the same consequences on every trial within a condition, but between conditions the number of reinforcers varied from one to three and the reinforcer delays varied between 5 s and 30 s. A peck on a green key produced a delay of adjustable duration and then a single reinforcer. The green-key delay was increased or decreased many times per session, depending on a subject's previous choices, which permitted estimation of an indifference point, or a delay at which a subject chose each alternative about equally often. The indifference points decreased systematically with more red-key reinforcers and with shorter red-key delays. The results did not support the suggestion of Moore (1979) that multiple delayed reinforcers have no effect on preference unless they are closely grouped. The results were well described in quantitative detail by a simple model stating that each of a series of reinforcers increases preference, but that a reinforcer's effect is inversely related to its delay. The success of this model, which considers only delay of reinforcement, suggested that the overall rate of reinforcement for each alternative had no effect on choice between those alternatives.     

Olfactory learning in individually assayed Drosophila larvae

Insect and mammalian olfactory systems are strikingly similar. Therefore, Drosophila can be used as a simple model for olfaction and olfactory learning. The brain of adult Drosophila, however, is still complex. We therefore chose to work on the larva with its yet simpler but adult-like olfactory system and provide evidence for olfactory learning in individually assayed Drosophila larvae. We developed a differential conditioning paradigm in which odorants are paired with positive (“+” fructose) or negative (“-” quinine or sodium chloride) gustatory reinforcers. Test performance of individuals from two treatment conditions is compared—one received odorant A with the positive reinforcer and odorant B with a negative reinforcer (A+/B-); animals from the other treatment condition were trained reciprocally (A-/B+). During test, differences in choice between A and B of individuals having undergone either A+/B- or A-/B+ training therefore indicate associative learning. We provide such evidence for both combinations of reinforcers; this was replicable across repetitions, laboratories, and experimenters. We further show that breaks improve performance, in accord with basic principles of associative learning. The present individual assay will facilitate electrophysiological studies, which necessarily use individuals. As such approaches are established for the larval neuromuscular synapse, but not in adults, an individual larval learning paradigm will serve to link behavioral levels of analysis to synaptic physiology.     

DELAY‐AMOUNT TRADEOFFS IN CHOICES BY PIGEONS AND RATS: HYPERBOLIC VERSUS EXPONENTIAL DISCOUNTING

An adjusting‐delay procedure was used to study the choices of pigeons and rats when both delay and amount of reinforcement were varied. In different conditions, the choice alternatives included one versus two reinforcers, one versus three reinforcers, and three versus two reinforcers. The delay to one alternative (the standard alternative) was kept constant in a condition, and the delay to the other (the adjusting alternative) was increased or decreased many times a session so as to estimate an indifference point—a delay at which the two alternatives were chosen about equally often. Indifference functions were constructed by plotting the adjusting delay as a function of the standard delay for each pair of reinforcer amounts. The experiments were designed to test the prediction of a hyperbolic decay equation that the slopes of the indifference functions should increase as the ratio of the two reinforcer amounts increased. Consistent with the hyperbolic equation, the slopes of the indifference functions depended on the ratios of the two reinforcer amounts for both pigeons and rats. These results were not compatible with an exponential decay equation, which predicts slopes of 1 regardless of the reinforcer amounts. Combined with other data, these findings provide further evidence that delay discounting is well described by a hyperbolic equation for both species, but not by an exponential equation. Quantitative differences in the y‐intercepts of the indifference functions from the two species suggested that the rate at which reinforcer strength decreases with increasing delay may be four or five times slower for rats than for pigeons.     

Sensitivity to relative reinforcer rate in concurrent schedules: independence from relative and absolute reinforcer duration

Twelve pigeons responded on two keys under concurrent variable-interval (VI) schedules. Over several series of conditions, relative and absolute magnitudes of reinforcement were varied. Within each series, relative rate of reinforcement was varied and sensitivity of behavior ratios to reinforcer-rate ratios was assessed. When responding at both alternatives was maintained by equal-sized small reinforcers, sensitivity to variation in reinforcer-rate ratios was the same as when large reinforcers were used. This result was observed when the overall rate of reinforcement was constant over conditions, and also in another series of concurrent schedules in which one schedule was kept constant at VI ached 120 s. Similarly, reinforcer magnitude did not affect the rate at which response allocation approached asymptote within a condition. When reinforcer magnitudes differred between the two responses and reinforcer-rate ratios were varied, sensitivity of behavior allocation was unaffected although response bias favored the schedule that arranged the larger reinforcers. Analysis of absolute response rates ratio sensitivity to reinforcement occurrred on the two keys showed that this invariance of response despite changes in reinforcement interaction that were observed in absolute response rates on the constant VI 120-s schedule. Response rate on the constant VI 120-s schedule was inversely related to reinforcer rate on the varied key and the strength of this relation depended on the relative magnitude of reinforcers arranged on varied key. Independence of sensitivity to reinforcer-rate ratios from relative and absolute reinforcer magnitude is consistent with the relativity and independence assumtions of the matching law.     

Successive independence of multiple-schedule component performances

In three experiments, pigeons' responses were reinforced on two keys in each component of a series of multiple-schedule conditions. In each series, concurrent variable-interval schedules were constant in one component and were varied over conditions in the other component. In the first experiment both components arranged the same, constant total number of reinforcers, in the second the two components arranged constant but different totals, and in the third experiment the total was varied in one component and remained constant in the other. Relative reinforcer rate during the varied component was manipulated over conditions in all three experiments. In all these experiments, response and time allocation in the constant component were invariant when reinforcer ratios varied in the other component, demonstrating independence of behavior allocation in a multiple-schedule component from the relative reinforcer rate for the same alternatives in another component. In the two experiments which maintained constant reinforcer totals in components, sensitivity to reinforcement in the multiple schedules was the same as that in the concurrent schedules arranged during the varied component, with multiple-schedule bias in the experiment in which the totals were unequal.