Contrast and undermatching as a function of reinforcer duration and quality during multiple schedules

Eight pigeons pecked keys under multiple variable-interval two-minute variable-interval two-minute schedules. In Experiment 1, the reinforcers were 2, 4, or 8 seconds access to a food magazine. In Experiments 2 and 3, the reinforcers were grains that had been determined to be most-, moderately-, or non-preferred. Both positive and negative behavioral contrast occurred when the reinforcers in one component were held constant and the duration or type of reinforcer obtained in the other component varied. Undermatching occurred when the relative rate of responding during a component was plotted as a function of the relative duration of the reinforcers in that component.     

Stimulus duration as a measure of stimulus generalization

Four pigeons in the line-positive group were trained with a vertical line on a green background that signalled intermittent reinforcement while a plain green field signalled extinction. Four pigeons in the line-negative group were trained with the opposite discrimination. Response to a control key terminated any trial and initiated the next trial. The birds also used the control key during generalization tests to control the durations of trials in which various line orientations were presented. These durations were summed to provide generalization gradients of stimulus duration that were positive or negative in accordance with the trained discriminations. In Experiment 2, birds from the line-positive group were tested with a procedure in which the control key was not available on some trials. This provided an independent assessment of response rates to the test stimuli. These rates were used to predict the stimulus durations obtained when the control key was available. The findings supported a general model for the prediction of response distributions among concurrent stimuli from rates observed with single stimuli.     

Choice with probabilistic reinforcement: effects of delay and conditioned reinforcers.

Two experiments measured pigeons' choices between probabilistic reinforcers and certain but delayed reinforcers. In Experiment 1, a peck on a red key led to a 5-s delay and then a possible reinforcer (with a probability of .2). A peck on a green key led to a certain reinforcer after an adjusting delay. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In all conditions, red houselights were present during the 5-s delay on reinforced trials with the probabilistic alternative, but the houselight colors on nonreinforced trials differed across conditions. Subjects showed a stronger preference for the probabilistic alternative when the houselights were a different color (white or blue) during the delay on nonreinforced trials than when they were red on both reinforced and nonreinforced trials. These results supported the hypothesis that the value or effectiveness of a probabilistic reinforcer is inversely related to the cumulative time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. Experiment 2 tested some quantitative versions of this hypothesis by varying the delay for the probabilistic alternative (either 0 s or 2 s) and the probability of reinforcement (from .1 to 1.0). The results were best described by an equation that took into account both the cumulative durations of stimuli associated with the probabilistic reinforcer and the variability in these durations from one reinforcer to the next.     

Conditioned reinforcement and discrimination in second-order schedules

Pigeons were exposed to multiple second-order schedules in which responding on the “main key” was reinforced according to either a variable-interval or fixed-interval schedule by production of a brief stimulus on the “brief-stimulus key”. A response was required to the brief stimulus during its fourth (final) presentation to produce food; responses to the earlier brief stimuli indicated the extent to which the final brief stimulus was discriminated from preceding ones. Main-key response rates were higher in early components of paired brief-stimulus schedules, in which each brief stimulus was the same as that paired with reinforcement, than in comparable unpaired brief-stimulus or tandem schedules. Poor discrimination occurred between paired brief stimuli (Experiment I). When chain stimuli on the main key induced a discrimination between the first two and second two brief stimuli, the response-rate enhancement in the paired brief-stimulus schedule persisted (Experiment II). Rate enhancement diminished when the initial link of the chain included the first three components (Experiment IV). Eliminating the contingency between responding and brief-stimulus production also diminished rate enhancement (Experiment III). The results show that the discriminative and conditioned reinforcing effects of food-paired brief stimuli may be selectively manipulated and suggest that the reinforcing effects are modulated by other reinforcers in the situation.     

Theories of probabilistic reinforcement.

In three experiments, pigeons chose between two alternatives that differed in the probability of reinforcement and the delay to reinforcement. A peck at a red key led to a delay of 5 s and then a possible reinforcer. A peck at a green key led to an adjusting delay and then a certain reinforcer. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In Experiments 1 and 2, the intertrial interval was varied across conditions, and these variations had no systematic effects on choice. In Experiment 3, the stimuli that followed a choice of the red key differed across conditions. In some conditions, a red houselight was presented for 5 s after each choice of the red key. In other conditions, the red houselight was present on reinforced trials but not on nonreinforced trials. Subjects exhibited greater preference for the red key in the latter case. The results were used to evaluate four different theories of probabilistic reinforcement. The results were most consistent with the view that the value or effectiveness of a probabilistic reinforcer is determined by the total time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. According to this view, probabilistic reinforcers are analogous to reinforcers that are delivered after variable delays.     

Concurrent schedules: Interaction of reinforcer frequency and reinforcer duration

Six pigeons were trained on concurrent variable-interval schedules with unequal reinforcer durations for the two responses. The schedules arranged on the two keys were kept equal while they were varied in absolute size. As the overall reinforcer rate was increased, both response-allocation and time-allocation measures of choice showed a trend toward indifference, and measures of sensitivity to reinforcer-duration ratios significantly decreased. Recent reports have shown that the generalized matching law cannot describe the changes in behavior allocation under constant delay-, duration-, or rate-ratios when changes are made in the absolute levels of each of these variables. The present results complement these findings by demonstrating that the concatenated generalized matching law cannot describe the interactions of two reinforcer variables on behavior allocation.     

Choice, relative reinforcer duration, and the changeover ratio

Relative reinforcer duration was varied in concurrent schedules with a fixed-ratio four changeover requirement. The schedule in effect after each reinforcer was randomly chosen. For all three pigeons, relative response rates overmatched relative reinforcer durations. Time allocation was less extreme and, on the average, matched relative reinforcer duration. In a subsequent manipulation, the level of preference was shown to depend on the size of the changeover requirement. These results are similar to those from related unequal reinforcement-frequency procedures.     

Conditioned reinforcement value and choice.

The delay-reduction hypothesis of conditioned reinforcement states that the reinforcing value of a food-associated stimulus is determined by the delay to primary reinforcement signaled by the onset of the stimulus relative to the average delay to primary reinforcement in the conditioning situation. In contrast, most contemporary models of conditioned reinforcement strength posit that the reinforcing strength of a stimulus is some simple function only of the delay to primary reinforcement in the presence of stimulus. The delay-reduction hypothesis diverges from other conditioned reinforcement models in that it predicts that a fixed-duration food-paired stimulus will have different reinforcing values depending on the frequency of its presentation. In Experiment 1, pigeons' key pecks were reinforced according to concurrent-chains schedules with variable-interval 10-second and variable-interval 20-second terminal-link schedules. The initial-link schedule preceding the shorter terminal link was always variable-interval 60 seconds, and the initial-link schedule requirement preceding the longer terminal link was varied between 1 second and 60 seconds across conditions. In Experiment 2, the initial-link schedule preceding the longer of two terminal links was varied for each of three groups of pigeons. The terminal links of the concurrent chains for the three groups were variable-interval 10 seconds and 20 seconds, variable-interval 10 seconds and 30 seconds, and variable-interval 30 seconds and 50 seconds. In both experiments, preference for the shorter terminal link was either a bitonic function or an inverse function of the initial-link schedule preceding the longer terminal-link schedule. Consistent with the predictions of the delay-reduction hypothesis, the relative values of the terminal-link stimuli changed as a function of the overall frequency of primary reinforcement. Vaughan's (1985) melioration model, which was shown to be formally similar to Squires and Fantino's (1971) delay-reduction model, can be modified so as to predict these results without changing its underlying assumptions.     

Sample-duration effects on pigeons' delayed matching as a function of predictability of duration

Three experiments assessed the impact of sample duration on pigeons' delayed matching as a function of whether or not the samples themselves signaled how long they would remain on. When duration was uncorrelated with the sample appearing on each matching trial, the typical effect of duration was observed: Choice accuracy was higher with long (15-s) than with short (5-s) durations. By contrast, this difference either disappeared or reversed when the 5- and 15-s durations were correlated with the sample stimuli. Sample duration itself cued comparison choice by some birds in the latter (predictable) condition when duration was also correlated with the reinforced choice alternatives. However, even when duration could not provide a cue for choice, pigeons matched predictably short-duration samples as accurately as, or more accurately than, predictably long-duration samples. Moreover, this result was observed independently of whether the contextual conditions of the retention interval were the same as, or different from, those of the intertrial interval. These results strongly support the view that conditional stimulus control by the samples is partly a function of their conditioned reinforcing properties, as determined by the relative reduction in overall delay to reinforcement that they signal.     

Behavioral contrast for key pecking as a function of component duration when only one component varies

Pigeons pecked keys for food reinforcers delivered by multiple variable-interval 2-min variable-interval 2-min schedules. Positive behavioral contrast was created by changing one component to extinction; negative contrast was achieved by changing one component to a variable-interval 15-s schedule. The duration of each component was varied independently of the other from 5 to 960 s. The size of positive contrast was greatest when the extinction component was 30 or 60 s long. It did not change significantly with changes in the duration of the variable-interval 2-min component. The absolute size of negative contrast decreased with increases in the duration of the variable-interval 2-min component. It did not change significantly with changes in the duration of the variable-interval 15-s component. These results show that the size of contrast is determined primarily by the duration of the component that provides the less favorable conditions of reinforcement. These results are not predicted by current theories.     

Contingency and stimulus change in chained schedules of reinforcement

Higher rates of pecking were maintained by pigeons in the middle component of three-component chained fixed-interval schedules than in that component of corresponding multiple schedules (two extinction components followed by a fixed-interval component). This rate difference did not occur in equivalent tandem and mixed schedules, in which a single stimulus was correlated with the three components. The higher rates in components of chained schedules demonstrate a reinforcing effect of the stimulus correlated with the next component; the acquired functions of this stimulus make the vocabulary of conditioned reinforcement appropriate. Problems in defining conditioned reinforcement arise not from difficulties in demonstrating reinforcing effects but from disagreements about which experimental operations allow such reinforcing effects to be called conditioned.     

Effects of varying stimulus disparity and the reinforcer ratio in concurrent-schedule and signal-detection procedures.

The present study measured the effects of stimulus and reinforcer variations on pigeons' behavior in two different choice procedures. Two intensities of white light were presented as the stimuli on the main key in a switching-key concurrent schedule and as the sample stimuli in a signal-detection procedure. Under both procedures, the scheduled rate of reinforcement was varied across conditions to produce various ratios of obtained reinforcement. These ratios were obtained for seven pairs of light intensities. In the concurrent schedules, the effects of reinforcer-ratio variations were positively correlated with the physical disparity between the two light intensities. In the signal-detection procedure, changes in the reinforcer ratio produced greater effects on performance when stimulus disparity was very low or very high compared to those found at intermediate levels of stimulus disparity. This discrepancy creates a dilemma for existing behavioral models of signal-detection performance.     

Contrast and reallocation of extraneous reinforcers as a function of component duration and baseline rate of reinforcement

Four pigeons responded on multiple schedules arranged on a “main” key in a two-key experimental chamber. A constant schedule component was alternated with another component that was varied over conditions. On an extra response key, conjoint schedules of reinforcement that operated in both components were arranged concurrently with the multiple schedule on the main key. On the main key, changes in reinforcement rate in the varied component were inversely related to changes in response rates in the constant component (behavioral contrast). On the extra key, some reinforcers were reallocated between components, depending on the schedules in effect on the main key in the varied component. In the varied component, the obtained rates of reinforcement on the extra key were inversely related to main-key reinforcement rate. In the constant component, extra-key reinforcer rates were positively related to main-key reinforcer rates obtained in the varied component, and were not a function of response rates on the extra key. In two comparisons, the rate at which components alternated and the value of the main-key schedule in the constant component were varied. Consistent with earlier work, long components reduced the extent of contrast. Reductions in contrast as a function of component duration were accompanied by similar reductions in the extent of reinforcer reallocation on the extra key. In the second comparison, lowering the rate of reinforcement in the constant component increased the rate at which extra-key reinforcers were obtained, reduced the extent of reinforcer reallocation, and reduced contrast. Overall, the results are consistent with the suggestion that some contrast effects are due to the changes in extraneous reinforcement during the constant component, and that manipulations of component duration, and manipulations of the rate of reinforcement in the constant component, affect contrast because they influence the extent of extraneous reinforcer real-location.     

Responding of pigeons under variable-interval schedules of signaled-delayed reinforcement: effects of delay-signal duration.

Two experiments with pigeons examined the relation of the duration of a signal for delay ("delay signal") to rates of key pecking. The first employed a multiple schedule comprised of two components with equal variable-interval 60-s schedules of 27-s delayed food reinforcement. In one component, a short (0.5-s) delay signal, presented immediately following the key peck that began the delay, was increased in duration across phases; in the second component the delay signal initially was equal to the length of the programmed delay (27 s) and was decreased across phases. Response rates prior to delays were an increasing function of delay-signal duration. As the delay signal was decreased in duration, response rates were generally higher than those obtained under identical delay-signal durations as the signal was increased in duration. In Experiment 2 a single variable-interval 60-s schedule of 27-s delayed reinforcement was used. Delay-signal durations were again increased gradually across phases. As in Experiment 1, response rates increased as the delay-signal duration was increased. Following the phase during which the signal lasted the entire delay, shorter delay-signal-duration conditions were introduced abruptly, rather than gradually as in Experiment 1, to determine whether the gradual shortening of the delay signal accounted for the differences observed in response rates under identical delay-signal conditions in Experiment 1. Response rates obtained during the second exposures to the conditions with shorter signals were higher than those observed under identical conditions as the signal duration was increased, as in Experiment 1. In both experiments, rates and patterns of responding during delays varied greatly across subjects and were not systematically related to delay-signal durations. The effects of the delay signal may be related to the signal's role as a discriminative stimulus for adventitiously reinforced intradelay behavior, or the delay signal may have served as a conditioned reinforcer by virtue of the temporal relation between it and presentation of food.     

Effects on preference of reinforcement delay, number of reinforcers, and terminal-link duration

Three experiments used concurrent-chains procedures to examine the effects of reinforcement delay, number of reinforcers, and terminal-link duration on preference. In Condition 30 of Experiment 1, food was delivered after 30 seconds in each 150-second terminal link, with four additional food deliveries occurring at 30-second intervals in one of the links. In Condition 5, food was delivered after 5 seconds in each 25-second terminal link, and the four additional reinforcers were delivered at 5-second intervals. Preferences for the multiple-food chain were greater in Condition 30. In Experiment 2, the terminal link(s) providing only one reinforcer terminated immediately after delivery of the reinforcer. Preferences for the multiple-food chain were smaller than in Experiment 1. In Condition 5 of Experiment 3, food was delivered after 5, 75, 100, 125, and 150 seconds in one 150-second link and after 5 seconds in the other. Condition 50 differed only in that the first (or only) reinforcer in each link was delivered after 50 seconds instead of after 5 seconds. Preferences for the multiple-food chain were greater in Condition 50. Results of Experiments 1 and 2 do not correspond to results obtained by Moore (1979).     

Effect of varying the duration of grain presentation on automaintenance

In a series of three experiments the effects of variation in grain duration on automaintenance were evaluated. In the first experiment, key illumination was followed by grain only when pigeons did not peck the key. Each subject was exposed to 2-, 4-, and 8-second feeder durations in blocks of 10 sessions. Subjects pecked on a high percentage of trials at all feeder durations. The mean peck latency was shorter in the 8-second condition than in the two other conditions in five of six subjects. The conditional probability of pecking given successive keylight-grain pairings did not increase as the number of pairings increased. The second experiment was identical to the first, except that key pecking had no scheduled consequence. Under these conditions, all three subjects showed substantial responding. The recorded measures showed no systematic relationship to feeder duration in this study. In the third experiment, two different stimuli were followed by feeder presentations of either identical (2- or 8-second) or different (2- and 8-second) durations within each session. Subjects tended to respond sooner and with a higher overall rate in the presence of the stimulus associated with the longer feeder duration only when different feeder durations were presented within the same session. This result was confirmed by direct observation of the pigeons. The results of these experiments suggest that the effects of varying grain duration may be small, compared to the effects of varying other variables. The results also suggest that the location as well as the frequency of pecking may be an important measure in the analysis of factors controlling the pigeon's key peck.     

Rapid acquisition of preference in concurrent chains when alternatives differ on multiple dimensions of reinforcement

Pigeons responded in a concurrent-chains procedure in which terminal-link reinforcer variables were changed unpredictably across sessions. In Experiment 1, the terminal-link schedules were fixed-interval (FI) 8 s and FI 16 s, and the reinforcer magnitudes were 2 s and 4 s. In Experiment 2 the probability of reinforcement (100% or 50%) was varied with immediacy and magnitude. Multiple-regression analyses showed that pigeons' initial-link response allocation was determined by current-session reinforcer variables, similar to previous studies which have varied only immediacy (Grace, Bragason, & McLean, 2003). Sensitivity coefficients were positive and statistically significant for all reinforcer variables in both experiments. Analyses of responding within individual sessions showed that final levels of preference for dominated sessions, in which all reinforcer variables favored the same terminal link, were more extreme than for tradeoff sessions in which at least one reinforcer variable favored each alternative. This result implies that response allocation was determined by multiple reinforcer variables within individual sessions, consistent with the concatenated matching law. However, in Experiment 2, there was a nonlinear (sigmoidal) relationship between response allocation and relative value, which suggests the possibility that reinforcer variables may interact during acquisition, contrary to the matching law.     

Control of behavior by an establishing stimulus

Seventeen pigeons were exposed to a three-key discrete-trial procedure in which a peck on the lit center key produced food if, and only if, the left keylight was lit. The center key was illuminated by a peck on the lit right key. Of interest was whether subjects pecked the right key before or after the response-independent onset of the left keylight. Pecks on the right key after left-keylight onset suggest control of behavior by the left keylight—an establishing stimulus. In three experiments, the strength of center-keylight onset as conditioned reinforcer for a response on the right key was manipulated by altering the size of the reduction in time to food delivery correlated with its onset. Control of pigeons' key pecks by onset of the left keylight occurred on more trials per session when the center keylight was a relatively weak conditioned reinforcer and on fewer trials per session when the center keylight was a relatively strong condtioned reinforcer. Differences across conditions in the degree of control by onset of the establishing stimulus were greatest when changes in conditioned reinforcer strength occurred relatively frequently and were signaled. The results provide evidence of the function of an establishing stimulus.     

Conditioned reinforcement and choice with delayed and uncertain primary reinforcers.

In an adjusting-delay choice procedure, pigeons could peck on either a red key or a green key. A peck on the red key always led to a delay associated with red houselights and then food. The delay was adjusted over trials to estimate an indifference point--a delay at which the two keys were chosen about equally often. In some conditions, a peck on the green key led to food on all trials after delays of either 10 s or 30 s, and green houselights were lit during the delays. In other conditions, food was presented on only half of the green-key trials. If the green houselights continued to occur on both reinforcement and nonreinforcement trials, preference for the green key always decreased. Preference for the green key also decreased if half of the trials had 30-s houselights followed by food and the other half had no green houselights and no food. However, preference for the green key actually increased if half of the trials had 10-s green houselights followed by food and the other half had no green houselights followed by no food. The latter condition therefore demonstrated a case in which preference for an alternative increased when food was removed from half of the trials. The results suggest that the red and green houselights served as conditioned reinforcers. A hyperbolic decay model (Mazur, 1989) provided good predictions for all conditions by assuming that the strength of a conditioned reinforcer is inversely related to the total time spent in its presence before food is delivered.