Effects of delayed reinforcement in a concurrent situation

Pigeons were trained to peck either of two response keys for food reinforcement on equated aperiodic schedules. Delays of reinforcement for pecks at one key reduced the relative frequency of pecking exponentially as a function of the delay interval. Similar functions were obtained when other dependent variables were plotted against the delay interval.     

Concurrent performances: inhibition of one response by reinforcement of another

In an analysis of interactions between concurrent performances, variable-interval reinforcement was scheduled, in various sequences, for both keys, for only one key, or for neither key of a two-key pigeon chamber. With changeover delays of 0.5 or 1.0 sec, and with each key's reinforcements discriminated on the basis of key-correlated feeder stimuli, reinforcement of pecks on one key reduced the pecking maintained by reinforcement on the other key. The decrease in pecking early after reinforcement was discontinued on one key was not substantially affected by whether pecks on the other key were reinforced, but after reinforcement was discontinued on both keys, reinstatement of reinforcement for one key sometimes produced transient increases in pecking on the other key. Correlating the availability of right-key reinforcements with a stimulus, which maintained right-key reinforcement while reducing right-key pecking to negligible levels, demonstrated that these interactions depended on concurrent reinforcement, not concurrent responding. Thus, reinforcement of a response, but not necessarily the occurrence of the response, inhibits other reinforced responses. Compared with accounts in terms of excitatory effects of extinction, often invoked in treatments of behavioral contrast, this inhibitory account has the advantage of dealing only with observed dimensions of behavior.     

Delayed signal detection, differential reinforcement, and short-term memory in the pigeon.

In two discrete-trial delayed-detection experiments, six pigeons were trained on dependent concurrent variable-interval schedules. Pecking a red side key was reinforced when the brighter of two white lights (S1) had been presented on the center key, and pecking a green side key was reinforced when the duller of two white lights (S2) had been presented on the center key. Incorrect responses were red side-key pecks following S2 presentations and green side-key pecks following S1 presentations; these resulted in three-second blackouts. In Experiment 1, the time between presentation of S1 or S2 on the center key and the onset of the red and green side keys was varied nonsystematically from 0.06 seconds to 19.69 seconds across experimental conditions. Stimulus discriminability decreased as the stimulus-choice delay increased. A rectangular-hyperbolic function better described this decrease in discriminability over time than did a negative-exponential function. In Experiment 2, at each of three stimulus-choice delays (0.06, 3.85, and 10.36 seconds), relative reinforcer frequency for correct responses to the red and green side keys was varied by changing the values of the dependent concurrent variable-interval schedules. The sensitivity of choice to relative reinforcer frequency was independent of the decrease in stimulus discriminability with increasing stimulus-choice delay.     

Secondary reinforcement and number of primary reinforcements

Pigeons' pecks on either of two concurrently available response keys produced secondary reinforcers according to independent one-minute variable-interval schedules. Different secondary reinforcers, in the presence of which the rates of primary reinforcement were equal, were associated with each key. The rate of pecking maintained by each secondary reinforcer varied directly, but nonproportionally, with the number of primary reinforcements given in the presence of the secondary reinforcer.     

On the directionality of key pecking during signals for appetitive and aversive events

Pigeons were exposed to a signal paired with either blackout or blackout plus shock and to another signal paired with food superimposed on a baseline of concurrent variable-interval reinforcement of pecks on two keys. The signals were changes of color of one of the two keys. The rate of pecking both keys during the signal paired with blackout or blackout plus shock was lower than the baseline rate of pecking (a conditioned emotional response), but the decrease in pecking was greater on the signal key. When the intensity of shock was increased, the rate of pecking did not decrease further on the signal key but did decrease on the other key. Rate of pecking during the signal paired with food increased sharply on the signal key (an autoshaping effect) and decreased sharply on the other key. These results support a view that there are two effects of the interaction between classical and instrumental conditioning, a stimulus-directed effect and a generalemotional effect.     

Independence of stimulus discriminability from absolute rate of reinforcement in a signal-detection procedure.

Three experiments are reported in which two pigeons were trained to detect differences in stimulus duration under varying levels of absolute rate of reinforcement. Two red stimuli, differing in duration, were arranged probabilistically on the center key of a three-key chamber. On completion of the center-key duration, the center keylight was extinguished and the two side keys were illuminated white. Correct responses were left-key pecks following the shorter duration and right-key pecks following the longer duration. In Experiment 1, relative rate of reinforcement for correct responses was held constant and absolute rate of reinforcement was varied in seven conditions from continuous reinforcement to a variable-interval 90-second schedule. In Experiment 2, relative rate of reinforcement was manipulated across three different absolute rates of reinforcement (continuous reinforcement, variable-interval 15-second, and variable-interval 45-second). Stimulus discriminability was unaffected by changes in absolute or relative rates of reinforcement. Experiment 3 showed that discriminability was also unaffected by arranging the same consequences (three-second blackout) for unreinforced correct responses and errors.     

Two different kinds of key peck in the pigeon: some properties of responses maintained by negative and positive response-reinforcer contingencies

Pigeons emitted almost exclusively short-duration key pecks (shorter than 20 msec) when on negative automaintenance procedures, in which pecks prevented reinforcement. Peck durations under fixed-interval and fixed-ratio reinforcement schedules were generally two to five times longer than pecks under a negative automaintenance schedule. However, initial key pecks were of short duration, independent of procedure. The frequency of short-duration pecks was insensitive to differential reinforcement, while the frequency of long-duration pecks was sensitive to differential reinforcement. It is proposed that short-duration pecks arise from the pigeon's normal feeding pattern and are directly enhanced by food presentation, while long-duration pecks are controlled by the contingent effects of food presentation. The implications of the existence of two classes of pecks for the functional definition of operants and the separation of phylogenetic and ontogenetic sources of control of key pecking are discussed.     

Rates and patterns of responding with concurrent fixed-interval and variable-interval reinforcement

Pigeons were exposed to concurrent fixed-interval and variable-interval schedules of food reinforcement on two keys. The times between reinforcement were varied systematically on both keys. The overall relative frequency of responding on the fixed-interval key depended on the relative frequency of reinforcement, but did not match it. Instead, the ratio of responses on the fixed-interval key to responses on the variable-interval key was a power function of the ratio of reinforcements, with an exponent of 0.5. Patterns of responding between reinforcements on the fixed-interval key depended on both relative and absolute values of the reinforcement schedules. Similar overall relative responding was obtained at different absolute schedule values with equal relative reinforcement, despite some differences in patterns of responding.     

Conditioned reinforcement as a function of duration of stimulus

Pigeons were provided with three keys. Pecking the center key produced grain on a schedule that alternated at unpredictable times between a variable-interval component and extinction. On concurrent variable-interval schedules, pecking either side key produced a stimulus associated with the variable-interval component on the center key provided that said schedule was currently in effect. The independent variable was the length of time this stimulus remained on the keys. Pecking one side key produced the stimulus for 27 seconds, whereas the duration produced by pecking the other key varied for successive blocks of sessions. For the first four birds, the values tested were 3, 9, 27, and 81 seconds. For the second group, numbering three birds, the values tested were 1, 3, 9, and 27 seconds. The dependent variable was the proportion of total side key pecks that occurred on the variable key. For all birds, the function was positive in slope and negative in acceleration. This finding supports a formulation that ascribes the maintenance of observing responses in a normal setting to the fact that the subject exposes itself to the positive discriminative stimulus for a longer mean duration than it does to the negative stimulus.     

Reinforcement schedules: the role of responses preceding the one that produces the reinforcer

In a two-key pigeon chamber, variable-interval reinforcement was scheduled for a specified number of pecks, emitted either on a single key or in a particular sequence on the two keys. Although the distribution of pecks between the two keys was affected by whether pecks were required on one or on both keys, the total pecks emitted was not; the change from a one-key to a two-key requirement simply moved some pecks from one key to the other. Thus, each peck preceding the one that produced the reinforcer contributed independently to the subsequent rate of responding; the contribution of a particular peck in the sequence was determined by the time between its emission and the delivery of the reinforcer (delay of reinforcement), and was identified by the proportion of pecks moved from one key to the other when the response requirement at that point in the sequence was moved from one key to the other.     

Mechanisms underlying the effects of unsignaled delayed reinforcement on key pecking of pigeons under variable-interval schedules.

Three experiments were conducted to test an interpretation of the response-rate-reducing effects of unsignaled nonresetting delays to reinforcement in pigeons. According to this interpretation, rates of key pecking decrease under these conditions because key pecks alternate with hopper-observing behavior. In Experiment 1, 4 pigeons pecked a food key that raised the hopper provided that pecks on a different variable-interval-schedule key met the requirements of a variable-interval 60-s schedule. The stimuli associated with the availability of the hopper (i.e., houselight and keylight off, food key illuminated, feedback following food-key pecks) were gradually removed across phases while the dependent relation between hopper availability and variable-interval-schedule key pecks was maintained. Rates of pecking the variable-interval-schedule key decreased to low levels and rates of food-key pecks increased when variable-interval-schedule key pecks did not produce hopper-correlated stimuli. In Experiment 2, pigeons initially pecked a single key under a variable-interval 60-s schedule. Then the dependent relation between hopper presentation and key pecks was eliminated by arranging a variable-time 60-s schedule. When rates of pecking had decreased to low levels, conditions were changed so that pecks during the final 5 s of each interval changed the keylight color from green to amber. When pecking produced these hopper-correlated stimuli, pecking occurred at high rates, despite the absence of a peck-food dependency. When peck-produced changes in keylight color were uncorrelated with food, rates of pecking fell to low levels. In Experiment 3, details (obtained delays, interresponse-time distributions, eating times) of the transition from high to low response rates produced by the introduction of a 3-s unsignaled delay were tracked from session to session in 3 pigeons that had been initially trained to peck under a conventional variable-interval 60-s schedule. Decreases in response rates soon after the transition to delayed reinforcement were accompanied by decreases in eating times and alterations in interresponse-time distributions. As response rates decreased and became stable, eating times increased and their variability decreased. These findings support an interpretation of the effects of delayed reinforcement that emphasizes the importance of hopper-observing behavior.     

Concurrent performances: a baseline for the study of reinforcement magnitude

When a pigeon's pecking on a single key was reinforced by a variable-interval (VI) schedule of reinforcement, the rate of pecking was insensitive to changes in the duration of reinforcement from 3 to 6 sec. When, however, the pigeon's pecking on each of two keys was concurrently reinforced by two independent VI schedules, one for each key, the rate of pecking was directly proportional to the duration of reinforcement.     

Concurrent performances: stimulus-control gradients during schedules of signalled and unsignalled concurrent reinforcement

On one key, pigeons' pecks were reinforced according to a variable-interval schedule in the presence of vertical lines, and were not reinforced in the presence of oblique lines. On a second key, pecks were reinforced according to a variable-interval schedule in the presence of blue, according to a signalled variable-interval schedule in the presence of red, and were not reinforced in the presence of white. Subsequently, during extinction, stimulus-control gradients were obtained by presenting eight different line orientations on the first key concurrent with each of the three colors on the second key. On the first key, line-orientation gradients tended to be lower, narrower, and less shifted in peak or area when the second-key stimulus was blue or red, the stimuli respectively correlated with unsignalled and signalled reinforcement, than when it was white, the stimulus correlated with extinction. Thus, the effect on first-key line-orientation gradients depended on second-key stimuli correlated with concurrent reinforcement, whether or not these stimuli were also correlated with concurrent responding. As a function of first-key line orientation, an inverted gradient was obtained on the second key during blue; during both red and white, rates of pecking on the second key were near zero.     

Punishment of observing by the negative discriminative stimulus

To determine the effect of a negative discriminative stimulus on the response producing it, two pigeons were each studied in a three-key conditioning chamber. During alternating periods of unpredictable duration, pecking the center (food) key either was reinforced with grain on a variable-interval schedule or was never reinforced. On equal but independent variable-interval schedules, pecking either of the side (observing) keys changed the color of all keys for 30 sec from yellow to either green or red. When the schedule on the center key was variable-interval reinforcement, the color was green (positive discriminative stimulus); when no reinforcements were scheduled, the color was red (negative discriminative stimulus). Since pecking the side keys did not affect grain deliveries, changes in the rate of pecking could not be ascribed to changes in the frequency of primary reinforcement. In subsequent sessions, red was withheld as one of the possible consequences of pecking a given side key. When red was omitted, the rate on that key increased, and when red was restored, the rate decreased. It was concluded that red illumination of the keys, the negative discriminative stimulus, had a suppressive effect on the response that produced it.     

Concurrent performances: rate and accuracy of free-operant oddity responding

In pigeon's oddity performances, maintained by variable-interval reinforcement of pecks on the odd key of three keys in a triangular array, accuracy and response rate varied inversely with the rate of variable-interval reinforcement scheduled concurrently for pecks on a fourth, spatially isolated key. But when variable-interval and extinction components alternated in a multiple schedule for pecks on the spatially isolated key, oddity accuracy was greater during variable-interval components than during extinction components. Oddity response rate was not affected systematically by the alternating components. Changeovers between the oddity keys and the spatially isolated key were frequent during variable-interval components; responding occurred almost exclusively on the oddity keys during extinction components. This difference in performance during the two components was eliminated by arranging stimulus-correlated variable-interval reinforcement in the multiple schedule on the spatially isolated key: a stimulus was presented in the variable-interval components only when reinforcement became available, thereby reducing responding on this key to near-zero levels in both components while maintaining the variable-interval reinforcement. The effect of the multiple-schedule components on oddity accuracy was not altered, however, and thus apparently depended directly on concurrent reinforcement and not on differential sequential properties of concurrent responding during the two components.     

Alternative reinforcement effects on fixed-interval performance

Pigeons' key pecks were reinforced with food on a fixed-interval schedule. Food also was available at variable time periods either independently of responding or for not key pecking (a differential-reinforcement-of-other-behavior schedule). The latter condition arranged reinforcement following the first pause of t seconds after it became available according to a variable-time schedule. This schedule allowed separation of the effects of pause requirements ≤ five-seconds and reinforcement frequency. The time spent pausing increased as the duration of the pause required for reinforcement increased from 0 to 30 seconds and as the frequency of reinforcement for pausing increased from 0 to 2 reinforcers per minute. Key pecking was more evenly distributed within each fixed interval with shorter required pauses and with more frequent reinforcement for pausing. The results complement those obtained with other concurrent schedules in which the same operant response was reinforced in both components.     

Towards an empirical calculus of reinforcement value

Only one of two keys reinforces the subject with food. This key can assume one of two colors, each associated with a different fixed-ratio schedule for obtaining reinforcement. The function of the second key is to permit the animal to switch from the long schedule to the short schedule. If the difference between the ratio schedules is large enough, a preference for the shorter schedule is demonstrable. A quantitative index of preference is obtained as follows: each time the animal switches to the shorter schedule, the number of pecks required to produce the next switch is increased. As the “ante” on the switching key increases, the effective difference between the two ratio schedules decreases. After each food reinforcement, when the bird is exposed to the choice-situation, it takes longer before the bird switches again. This is used to “titrate” the bird's preference. If it does not switch within x sec, the progressively increasing ratio schedule of the switching key is decreased. A specific value, in terms of a rather specific number of responses the bird settles at on the choice key, is obtained. This equilibrium is employed as a dependent variable. Several variables of which it is a function are explored.     

Concurrent performances: synthesizing rate constancies by manipulating contingencies for a single response

An earlier experiment scheduled variable-interval reinforcement for pigeons' pecks on one key, and variable-interval reinforcement alternating with extinction, in a multiple schedule, for pecks on a second key. During the second key's extinction component, first-key pecking was relatively slow and continuous, rarely interrupted by second-key pecking; during the variable-interval component, first-key pecking was frequently interrupted by second-key pecking. When changeover delays operated, so that reinforced pecks on one key could not follow closely upon changeovers from the other key, rapid first-key pecking between interruptions compensated sufficiently for the time lost in second-key pecking that the overall rate of first-key pecking remained roughly constant across the alternating multiple-schedule components. The present experiments duplicated, on a single key, the temporal pattern of first-key pecking generated in the earlier experiments: components of continuous key availability were alternated with components of interrupted key availability. Approximately constant overall rates of responding were observed with a single-key equivalent of a changeover delay scheduled after interruptions and with manipulations of the on-off durations of the interruption cycle. Rate constancies in the original concurrent situation presumably depended on analogous contingencies that operated upon the concurrent responses, rather than on any constant “reserve” of responses.