Choice and delay of reinforcement: Effects of terminal-link stimulus and response conditions

In two experiments, pigeons were exposed to concurrent-chains schedules in which a single initial-link variable-interval schedule led to access to terminal links composed of fixed-interval or fixed-delay schedules. In Experiment 1, an 8-s (or 16-s) delay to reinforcement was associated with the standard key, while reinforcer delay values associated with the experimental key were varied from 4 to 32 s. The results of Experiment 1 showed undermatching of response ratios to delay ratios with terminal-link fixed-delay schedules, whereas in some pigeons matching or overmatching was evident with the fixed-interval schedules. In Experiment 2, one pair of reinforcer delay values, either 8 versus 16 s or 16 versus 32 s, was used. In the first condition of Experiment 2, different delays were associated with different keylight stimuli (cued condition). In the second condition, different terminal-link delays were associated with the same stimulus, either a blackout (uncued-blackout condition) or a white key (uncued-white condition). To examine the role of responses emitted during delays, the keys were retracted during a delay (key-absent condition) in the third condition and responses were required by a fixed-interval schedule in the fourth condition. Experiment 2 demonstrated that the choice proportions for the shorter delay were more extreme in the cued condition than in the uncued-blackout condition, and that the response requirement imposed by the fixed-interval schedules did not affect choice of the shorter delay, nor did the key-absent and key-present conditions. These results indicate that the keylight-stimulus conditions affected preference for the shorter of two delays and that the findings obtained in Experiment 1 depended mainly on the keylight-stimulus conditions of the terminal links (i.e., the conditioned reinforcing value of the terminal-link stimuli).     

Contiguity and conditioned reinforcement in probabilistic choice

In a baseline condition, pigeons chose between an alternative that always provided food following a 30-s delay (100% reinforcement) and an alternative that provided food half of the time and blackout half of the time following 30-s delays (50% reinforcement). The different outcomes were signaled by different-colored keylights. On average, each alternative was chosen approximately equally often, replicating the finding of suboptimal choice in probabilistic reinforcement procedures. The efficacy of the delay stimuli (keylights) as conditioned reinforcers was assessed in other conditions by interposing a 5-s gap (keylights darkened) between the choice response and one or more of the delay stimuli. The strength of conditioned reinforcement was measured by the decrease in choice of an alternative when the alternative contained a gap. Preference for the 50% alternative decreased in conditions in which the gap preceded either all delay stimuli, both delay stimuli for the 50% alternative, or the food stimulus for the 50% alternative, but preference was not consistently affected in conditions in which the gap preceded only the 100% delay stimulus or the blackout stimulus for the 50% alternative. These results support the notion that conditioned reinforcement underlies the finding of suboptimal preference in probabilistic reinforcement procedures, and that the signal for food on the 50% reinforcement alternative functions as a stronger conditioned reinforcer than the signal for food on the 100% reinforcement alternative. In addition, the results fail to provide evidence that the signal for blackout functions as a conditioned punisher.     

Effects of reinforcement rate and delay on the acquisition of lever pressing by rats.

The acquisition of lever pressing by naive rats, in the absence of shaping, was studied as a function of different rates and unsignaled delays of reinforcement. Groups of 3 rats were each exposed to tandem schedules that differed in either the first or the second component. First-component schedules were either continuous reinforcement or random-interval 15, 30, 60 or 120 s; second-component schedules were fixed-time 0, 1, 3, 6, 12, or 24 s. Rate of responding was low under continuous immediate reinforcement and higher under random-interval 15 s. Random interval 30-s and 60-s schedules produced lower rates that were similar to each other. Random-interval 120 s controlled the lowest rate in the immediate-reinforcement condition. Adding a constant 12-s delay to each of the first-component schedule parameters controlled lower response rates that did not vary systematically with reinforcement rate. The continuous and random-interval 60-s schedules of immediate reinforcement controlled higher global and first-component response rates than did the same schedules combined with longer delays, and first-component rates showed some graded effects of delay duration. In addition, the same schedules controlled higher second-component response rates in combination with a 1-s delay than in combination with longer delays. These results were related to those from previous studies on acquisition with delayed reinforcement as well as to those from similar reinforcement procedures used during steady-state responding.     

Molecular contingencies: reinforcement probability

Pigeons obtained food by responding in a discrete-trials two-choice probability-learning experiment involving temporal stimuli. A given response alternative, a left- or right-key peck, had 11 associated reinforcement probabilities within each session. Reinforcement probability for a choice was an increasing or a decreasing function of the time interval immediately preceding the choice. The 11 equiprobable temporal stimuli ranged from 1 to 11 sec in 1-sec classes. Preference tended to deviate from probability matching in the direction of maximizing; i.e., the percentage of choices of the preferred response alternative tended to exceed the probability of reinforcement for that alternative. This result was qualitatively consistent with probability-learning experiments using visual stimuli. The result is consistent with a molecular analysis of operant behavior and poses a difficulty for molar theories holding that local variations in reinforcement probability may safely be disregarded in the analysis of behavior maintained by operant paradigms.     

Temporal integration in duration and number discrimination

Temporal integration in duration and number discrimination by rats was investigated with the use of a psychophysical choice procedure. A response on one lever ("short" response) following a 1-s white-noise signal was followed by food reinforcement, and a response on the other lever ("long" response) following a 2-s white-noise signal was also followed by food reinforcement. Either response following a signal of one of five intermediate durations was unreinforced. This led to a psychophysical function in which the probability of a long response was related to signal duration in an ogival manner. On 2 test days, a white-noise signal with 5, 6, 7, 8, or 10 segments of either 0.5-s on and 0.5-s off or 1-s on and 1-s off was presented, and a choice response following these signals was unreinforced. The probability of a long response was the same function of a segmented signal and a continuous signal if each segment was considered equivalent to 200 ms. A quantitative fit of a scalar estimation theory suggested that the latencies to initiate temporal integration and to terminate the process are both about 200 ms, and that the same internal accumulation process can be used for counting and timing.     

The schedule dependency of the signaled reinforcement effect

In Experiment 1, rats leverpressed for food reinforcement on either a variable ratio (VR) 30 schedule or a variable interval (VI) 15-s schedule. One group in each condition received a signal filling a 500-ms delay of reinforcement. This treatment enhanced rates on the VR schedule, and attenuated rates on the VI schedule, relative to the rate seen in an unsignaled control condition. In Experiment 2 there was no delay of reinforcement and the signal and food were presented simultaneously. Attenuated rates of responding were observed on VI schedules with a range of mean interval values (15 to 300 s). Experiment 3 used a range of VR schedules (10 to 150) with simultaneous presentations of signal and food. A signal-induced enhancement of response rate was found at all VR values. In Experiment 4, a signal elevated response rates on a tandem VI VR schedule, but depressed rates on a tandem VR VI schedule, compared to control conditions receiving unsignaled delayed reinforcement. These results are taken to show that the effect of a signal accompanying reinforcement depends upon the nature of the behavior that is reinforced during exposure to a given schedule.     

Competitive fixed-interval performance in humans

Two persons responded in the same session in separate cubicles, but under a single schedule of reinforcement. Each time reinforcement was programmed, only the first response to occur, that is, the response of only one of the subjects, was reinforced. “Competitive” behavior that developed under these conditions was examined in three experiments. In Experiment 1 subjects responded under fixed-interval (FI) 30-s, 60-s, and 90-s schedules of reinforcement. Under the competition condition, relative to baseline conditions, the response rates were higher and the pattern was “break-and-run.” In Experiment 2, subjects were exposed first to a conventional FI schedule and then to an FI competition schedule. Next, they were trained to respond under either a differential-reinforcement-of-low-rate (DRL) or fixed-ratio (FR) schedule, and finally, the initial FI competition condition was reinstated. In this second exposure to the FI competition procedure, DRL subjects responded at lower rates than were emitted during the initial exposure to that condition and FR subjects responded at higher rates. For all subjects, however, responding gradually returned to the break-and-run pattern that had occurred during the first FI competition condition. Experiment 3 assessed potential variables contributing to the effects of the competitive FI contingencies during Experiments 1 and 2. Subjects were exposed to FI schedules where (a) probability of reinforcement at completion of each fixed interval was varied, or (b) a limited hold was in effect for reinforcement. Only under the limited hold was responding similar to that observed in previous experiments.     

Reinforcement Probability and Delay as Determinants of Human Impulsiveness

In behavior theory, “impulsiveness” refers to the choice of an immediate, small reinforcer over a delayed, large reinforcer. Such behavior generally is attributed to a reduction in the value of the large reinforcer as a function of the duration of delay. In contrast, social learning theorists have suggested that human impulsiveness can result from a lowered “expectancy” (subjective probability) of reinforcement. Effects of probability and delay were assessed by asking adults to make repeated choices between reinforcement schedules in which the reinforcers were slides of entertainment figures. An immediate, 5-s reinforcer was consistently chosen over an immediate, 40-s reinforcer if the probability of receiving the large reinforcer had previously been low (.20), implying that impulsiveness can occur without time-based discounting. However, reinforcement delay was also influential: Choice between a certain, small reinforcer and an uncertain, large reinforcer varied according to which reinforcer was immediate and which delayed.

     

Choice with probabilistic reinforcement: effects of delay and conditioned reinforcers.

Two experiments measured pigeons' choices between probabilistic reinforcers and certain but delayed reinforcers. In Experiment 1, a peck on a red key led to a 5-s delay and then a possible reinforcer (with a probability of .2). A peck on a green key led to a certain reinforcer after an adjusting delay. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In all conditions, red houselights were present during the 5-s delay on reinforced trials with the probabilistic alternative, but the houselight colors on nonreinforced trials differed across conditions. Subjects showed a stronger preference for the probabilistic alternative when the houselights were a different color (white or blue) during the delay on nonreinforced trials than when they were red on both reinforced and nonreinforced trials. These results supported the hypothesis that the value or effectiveness of a probabilistic reinforcer is inversely related to the cumulative time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. Experiment 2 tested some quantitative versions of this hypothesis by varying the delay for the probabilistic alternative (either 0 s or 2 s) and the probability of reinforcement (from .1 to 1.0). The results were best described by an equation that took into account both the cumulative durations of stimuli associated with the probabilistic reinforcer and the variability in these durations from one reinforcer to the next.     

Changes in functional response units with briefly delayed reinforcement

In two experiments, key-peck responding of pigeons was compared under variable-interval schedules that arranged immediate reinforcement and ones that arranged unsignaled delays of reinforcement. Responses during the nominal unsignaled delay periods had no effect on the reinforcer presentations. In Experiment 1, the unsignaled delays were studied using variable-interval schedules as baselines. Relative to the immediate reinforcement condition, 0.5-s unsignaled delays decreased the duration of the reinforced interresponse times and increased the overall frequency of short (<0.5-s) interresponse times. Longer, 5.0-s unsignaled delays increased the duration of the reinforced interresponse times and decreased the overall frequency of the short interresponse times. In Experiment 2, similar effects to those of Experiment 1 were obtained when the 0.5-s unsignaled delays were imposed upon a baseline schedule that explicitly arranged reinforcement of short interresponse times and therefore already generated a large number of short interresponse times. The results support earlier suggestions that the unsignaled 0.5-s delays change the functional response unit from a single key peck to a multiple key-peck unit. These findings are discussed in terms of the mechanisms by which contingencies control response structure in the absence of specific structural requirements.     

Sample-duration effects on pigeons' delayed matching as a function of predictability of duration

Three experiments assessed the impact of sample duration on pigeons' delayed matching as a function of whether or not the samples themselves signaled how long they would remain on. When duration was uncorrelated with the sample appearing on each matching trial, the typical effect of duration was observed: Choice accuracy was higher with long (15-s) than with short (5-s) durations. By contrast, this difference either disappeared or reversed when the 5- and 15-s durations were correlated with the sample stimuli. Sample duration itself cued comparison choice by some birds in the latter (predictable) condition when duration was also correlated with the reinforced choice alternatives. However, even when duration could not provide a cue for choice, pigeons matched predictably short-duration samples as accurately as, or more accurately than, predictably long-duration samples. Moreover, this result was observed independently of whether the contextual conditions of the retention interval were the same as, or different from, those of the intertrial interval. These results strongly support the view that conditional stimulus control by the samples is partly a function of their conditioned reinforcing properties, as determined by the relative reduction in overall delay to reinforcement that they signal.     

Theories of probabilistic reinforcement.

In three experiments, pigeons chose between two alternatives that differed in the probability of reinforcement and the delay to reinforcement. A peck at a red key led to a delay of 5 s and then a possible reinforcer. A peck at a green key led to an adjusting delay and then a certain reinforcer. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In Experiments 1 and 2, the intertrial interval was varied across conditions, and these variations had no systematic effects on choice. In Experiment 3, the stimuli that followed a choice of the red key differed across conditions. In some conditions, a red houselight was presented for 5 s after each choice of the red key. In other conditions, the red houselight was present on reinforced trials but not on nonreinforced trials. Subjects exhibited greater preference for the red key in the latter case. The results were used to evaluate four different theories of probabilistic reinforcement. The results were most consistent with the view that the value or effectiveness of a probabilistic reinforcer is determined by the total time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. According to this view, probabilistic reinforcers are analogous to reinforcers that are delivered after variable delays.     

Pigeons may not remember the stimuli that reinforced their recent behavior

In two experiments the conditioned reinforcing and delayed discriminative stimulus functions of stimuli that signal delays to reinforcement were studied. Pigeons' pecks to a center key produced delayed-matching-to-sample trials according to a variable-interval 60-s (or 30-s in 1 pigeon) schedule (Experiment 1) or a multiple variable-interval 20-s variable-interval 120-s schedule (Experiment 2). The trials consisted of a 2-s illumination of one of two sample key colors followed by delays ranging across phases from 0.1 to 27.0 s followed in turn by the presentation of matching and nonmatching comparison stimuli on the side keys. Pecks to the key color that matched the sample were reinforced with 4-s access to grain. Under some conditions of Experiment 1, pecks to nonmatching comparison stimuli produced a 4-s blackout and the start of the next interval. Under other conditions of Experiment 1 and each condition of Experiment 2, pecks to nonmatching stimuli had no effect and trials ended only when pigeons pecked the other, matching stimulus and received food. The functions relating pretrial response rates to delays differed markedly from those relating matching-to-sample accuracy to delays. Specifically, response rates remained relatively high until the longest delays (15.0 to 27.0 s) were arranged, at which point they fell to low levels. Matching accuracy was high at short delays, but fell to chance at delays between 3.0 and 9.0 s. In Experiment 2, both matching accuracy and response rates remained high over a wider range of delays in the variable-interval 120-s component relative to the variable-interval 20-s component. The difference in matching accuracy between the components was not due to an increased tendency in the variable-interval 20-s component toward proactive interference following short intervals. Thus, under these experimental conditions the conditioned reinforcing and the delayed discriminative functions of the sample stimulus depended on the same variables (delay and variable-interval value), but were nevertheless dissociated.     

Delayed reinforcement and delayed choice in symbolic matching to sample: Effects on stimulus discriminability

Six pigeons were trained to peck a red side key when the brighter of two white lights (S₁) had been presented on the center key, and to peck a green side key when the dimmer of two white lights (S₂) had been presented on the center key. Equal frequencies of reinforcers were provided for the two types of correct choice. Incorrect choices, red side-key pecks following S₂ presentations and green side-key pecks following S₁ presentations, resulted in blackout. With 0-s delay between choice and reinforcement, the delay between sample presentation and choice was varied from 0 to 20 s. Then, with 0-s delay between sample presentation and choice, the delay between choice and reinforcement was varied from 0 to 20 s. Both types of delay resulted in decreased discriminability (defined in terms of a signal-detection analysis) of the center-key stimuli, but delayed choice had more effect on discriminability than did delayed reinforcement. These data are consistent with the view that the two kinds of delay operate differently. The effect of a sample-choice delay may result from a degradation of the conditional discriminative stimuli during the delay; the effect of a choice-reinforcer delay may result from a decrement in control by differential reinforcement.     

Decision rules and signal detectability in a reinforcement-density discrimination

Two probabilistic schedules of reinforcement, one richer in reinforcement, the other leaner, were overlapping stimuli to be discriminated in a choice situation. One of two schedules was in effect for 12 seconds. Then, during a 6-second choice period, the first left-key peck was reinforced if the richer schedule had been in effect, and the first right-key peck was reinforced if the leaner schedule had been in effect. The two schedule stimuli may be viewed as two binomial distributions of the number of reinforcement opportunities. Each schedule yielded different frequencies of 16 substimuli. Each substimulus had a particular type of outcome pattern for the 12 seconds during which a schedule was in effect, and consisted of four consecutive light-cued 3-second T-cycles, each having 0 or 1 reinforced center-key pecks. Substimuli therefore contained 0 to 4 reinforcers. On any 3-second cycle, the first center-key peck darkened that key and was reinforced with probability .75 or .25 in the richer or leaner schedules, respectively. In terms of the theory of signal detection, detectability neared the maximum possible d′ for all four pigeons. Left-key peck probability increased when number of reinforcers in a substimulus increased, when these occurred closer to choice, or when pellets were larger for correct left-key pecks than for correct right-key pecks. Averaged over different temporal patterns of reinforcement in a substimulus, substimuli with the same number of reinforcers produced choice probabilities that matched relative expected payoff rather than maximized one alternative.     

A comparison of delays and ratio requirements in self-control choice.

In a discrete-trial procedure, pigeons could choose between 2-s and 6-s access to grain by making a single key peck. In Phase 1, the pigeons obtained both reinforcers by responding on fixed-ratio schedules. In Phase 2, they received both reinforcers after simple delays, arranged by fixed-time schedules, during which no responses were required. In Phase 3, the 2-s reinforcer was available through a fixed-time schedule and the 6-s reinforcer was available through a fixed-ratio schedule. In all conditions, the size of the delay or ratio leading to the 6-s reinforcer was systematically increased or decreased several times each session, permitting estimation of an "indifference point," the schedule size at which a subject chose each alternative equally often. By varying the size of the schedule for the 2-s reinforcer across conditions, several such indifference points were obtained from both fixed-time conditions and fixed-ratio conditions. The resulting "indifference curves" from fixed-time conditions and from fixed-ratio conditions were similar in shape, and they suggested that a hyperbolic equation describes the relation between ratio size and reinforcement value as well as the relation between reinforcer delay and its reinforcement value. The results from Phase 3 showed that subjects chose fixed-time schedules over fixed-ratio schedules that generated the same average times between a choice response and reinforcement.     

Reinforcement uncertainty enhances preference for choice in humans

Under concurrent‐chains schedules of reinforcement, participants often prefer situations that allow selection among alternatives (free choice) to situations that do not (forced choice). The present experiment examined the effects of reinforcement probability on choice preferences. Preferences for free versus forced choice were measured under a condition in which participants' choices were always reinforced (reinforcement probability of 1.0) and a condition in which outcomes were uncertain (reinforcement probability of 0.5). Forty‐four college students participated and preferences were examined under a concurrent‐chains schedule of reinforcement. Participants preferred free choice under uncertain reinforcement, but a bias toward free choice was not observed when reinforcement was certain. These results align with previous findings of preference for free choice under conditions of uncertainty, but suggest that preference may be dependent upon probabilistic reinforcement contingencies in the terminal links of the concurrent‐chains arrangement. Thus, reinforcement probability is an important variable to consider when conducting similar studies on the value of choice.     

Preference for less segmented fixed-time components in concurrent-chain schedules of reinforcement

A concurrent-chain procedure was used to examine choice between segmented and less segmented response-independent schedules of reinforcement. A pair of independent, concurrent variable-interval 60-s schedules were presented in the initial link, along with a 1.5-s changeover delay. A chained fixed-interval fixed-time and its corresponding tandem schedule constituted the terminal links. The length of the fixed-interval schedule in the terminal link was varied between 5 s and 30 s while that of the fixed-time schedule was kept at 5 s over conditions. The first components of both terminal-link schedules were accompanied by the same stimulus. Except in the baseline condition, the onset of the second component of the terminal-link chained schedule was accompanied by either a localized (key color) or a nonlocalized (dark houselight) stimulus change. Stimulus conditions were constant during the terminal-link tandem schedule. With three exceptions, pigeons demonstrated a slight preference for the tandem over the chained schedule in the terminal link. Furthermore, this preference varied inversely with the length of the first component. In general, these results are consistent with previous studies that reported an adverse effect on choice by segmenting an interval schedule into two or more components, but they are inconsistent with studies that reported preference for signaled over unsignaled delay of reinforcement.     

Effects of variations in local reinforcement rate on local response rate in variable interval schedules

Rats trained to lever press for sucrose were exposed to variable-interval schedules in which (i) the probability of reinforcement in each unit of time was a constant, (ii) the probability was high in the first ten seconds after reinforcement and low thereafter, (iii) the probability was low for ten seconds and high thereafter, (iv) the probability increased with time since reinforcement, or (v) the probability was initially zero and then increased with time since reinforcement. All schedules generated similar overall reinforcement rates. A peak in local response rate occurred several seconds after reinforcement under those schedules where reinforcement rate at this time was moderate or high ([i], [ii], and [iv]). Later in the inter-reinforcement interval, local response rate was roughly constant under those schedules with a constant local reinforcement rate ([i], [ii], and [iii]), but increased steadily when local reinforcement rate increased with time since reinforcement ([iv] and [v]). Postreinforcement pauses occurred on all schedules, but were much longer when local reinforcement rate was very low in the ten seconds after reinforcement ([iii]). The interresponse time distribution was highly correlated with the distribution of reinforced interresponse times, and the distribution of postreinforcement pauses was highly correlated with the distribution of reinforced postreinforcement pauses on some schedules. However, there was no direct evidence that these correlations resulted from selective reinforcement of classes of interresponse times and pauses.