Conditioned reinforcement value and choice.

The delay-reduction hypothesis of conditioned reinforcement states that the reinforcing value of a food-associated stimulus is determined by the delay to primary reinforcement signaled by the onset of the stimulus relative to the average delay to primary reinforcement in the conditioning situation. In contrast, most contemporary models of conditioned reinforcement strength posit that the reinforcing strength of a stimulus is some simple function only of the delay to primary reinforcement in the presence of stimulus. The delay-reduction hypothesis diverges from other conditioned reinforcement models in that it predicts that a fixed-duration food-paired stimulus will have different reinforcing values depending on the frequency of its presentation. In Experiment 1, pigeons' key pecks were reinforced according to concurrent-chains schedules with variable-interval 10-second and variable-interval 20-second terminal-link schedules. The initial-link schedule preceding the shorter terminal link was always variable-interval 60 seconds, and the initial-link schedule requirement preceding the longer terminal link was varied between 1 second and 60 seconds across conditions. In Experiment 2, the initial-link schedule preceding the longer of two terminal links was varied for each of three groups of pigeons. The terminal links of the concurrent chains for the three groups were variable-interval 10 seconds and 20 seconds, variable-interval 10 seconds and 30 seconds, and variable-interval 30 seconds and 50 seconds. In both experiments, preference for the shorter terminal link was either a bitonic function or an inverse function of the initial-link schedule preceding the longer terminal-link schedule. Consistent with the predictions of the delay-reduction hypothesis, the relative values of the terminal-link stimuli changed as a function of the overall frequency of primary reinforcement. Vaughan's (1985) melioration model, which was shown to be formally similar to Squires and Fantino's (1971) delay-reduction model, can be modified so as to predict these results without changing its underlying assumptions.     

Choice with uncertain outcomes: conditioned reinforcement effects.

Pigeons responded on concurrent chains with equal initial- and terminal-link durations. In all conditions, the terminal links of one chain ended reliably in reinforcement; the terminal links on the alternative chain ended in either food or blackout. In Experiment 1, the terminal-link stimuli were correlated with (signaled) the outcome, and the durations of the initial and terminal links were varied across conditions. Preference did not vary systematically across conditions. In Experiment 2, terminal-link durations were varied under different stimulus conditions. The initial links were variable-interval 80-s schedules. Preference for the reliable alternative was generally higher in unsignaled than in signaled conditions. Preference increased with terminal-link durations only in the unsignaled conditions. There were no consistent differences between conditions with and without a common signal for reinforcement on the two chains. In the first series of conditions in Experiment 3, a single response was required in the initial links, and the stimulus conditions during 50-s terminal links were varied. Preference for the reliable outcome approached 1.0 in unsignaled conditions and was considerably lower (below .50 for 3 of 5 subjects) in signaled conditions. In a final series of signaled conditions with relatively long terminal links, preference varied with duration of the initial links. The results extend previous findings and are discussed in terms of the delay reduction signaled by terminal-link stimuli.     

Value of conditioned reinforcers as a function of temporal context

In two experiments, pigeons were trained on a multiple-chain schedule, in which the initial link for one chain was a variable-interval (VI) 100 s schedule and for the other chain a VI 10 s schedule. The terminal links were both fixed-time 30 s schedules signaled by differently colored stimuli. Following training, the pigeons had their preference for the terminal-link stimuli tested either by presenting these stimuli in concurrent probes or by presenting these stimuli as reinforcement for completing novel initial links. In Experiment 1, pigeons significantly preferred the terminal-link stimulus that followed the long initial link in three of the five conditions. This preference was observed across all three testing procedures (concurrent chains, concurrent chains probes, and concurrent probes). Experiment 2 was a replication of this effect in one of the conditions from Experiment 1. The results demonstrate that temporal context does impact the value of a conditioned reinforcer in a manner consistent with delay-reduction theory, and inconsistent with other choice theories, such as the contextual choice model and scalar expectancy theory.     

Conditioned reinforcement by conditional discriminative stimuli.

A concurrent-chains schedule was used to examine how a delay to conditional discriminative stimuli affects conditioned reinforcement strength. Pigeons' key-peck responses in the initial link produced either of two terminal links according to independent variable-interval 30-s schedules. Each terminal link involved an identical successive conditional discrimination and was segmented into three links: a delay interval (green), a color conditional discriminative stimulus (blue or red), and a line conditional discriminative stimulus (vertical or horizontal lines). Food delivery occurred 45 s after entering the terminal link with a probability of .5, but its conditional probability (1.0 or 0) depended on the combination of the color and the line stimuli. One of the color stimuli occurred independently of further responding, 5 s after entry into the right terminal link, but it occurred 35 s after entry into the left terminal link. One of the line stimuli occurred independently of responding 40 s after entry into either terminal link, synchronized with the offset of the color stimulus. The initial-link relative response rate for the right was consistently higher in comparison with a control condition in which the color stimuli occurred 20 s after entry into either terminal link. The preference for the short delay to the color conditional discriminative stimuli suggests the possibility of conditioned reinforcement by information about the relation between the line conditional discriminative stimuli and the outcomes.     

Effects on choice of reinforcement delay and conditioned reinforcement

Pigeons chose between fixed-interval schedules of different durations presented in the terminal links of concurrent-chains schedules. The pair of schedules was always in the ratio of 2:1, but the absolute duration of the fixed intervals varied. In one set of conditions, the different terminal-link schedules were associated with different keylight stimuli (cued conditions). In a second set of conditions, the different terminal-link schedules were associated with the same stimulus (uncued conditions). Results from the cued conditions replicated previous findings that preference for the shorter fixed-interval schedule increased with fixed-interval duration. Preferences in the uncued conditions were lower than in the corresponding cued conditions but also increased with fixed-interval length. In addition, the degree of control under the uncued conditions was correlated with the extent to which the schedule during the terminal link was discriminated immediately upon entry into the terminal link. The pattern of results in both conditions was inconsistent with the notion that choice behavior matches relative immediacy of reinforcement. Reanalysis of previous evidence for matching (Chung and Herrnstein, 1967) showed that matching in fact did not occur, as the preferences of their subjects for the shorter of two delays also increased with the absolute size of the delays.     

Contiguity and conditioned reinforcement in probabilistic choice

In a baseline condition, pigeons chose between an alternative that always provided food following a 30-s delay (100% reinforcement) and an alternative that provided food half of the time and blackout half of the time following 30-s delays (50% reinforcement). The different outcomes were signaled by different-colored keylights. On average, each alternative was chosen approximately equally often, replicating the finding of suboptimal choice in probabilistic reinforcement procedures. The efficacy of the delay stimuli (keylights) as conditioned reinforcers was assessed in other conditions by interposing a 5-s gap (keylights darkened) between the choice response and one or more of the delay stimuli. The strength of conditioned reinforcement was measured by the decrease in choice of an alternative when the alternative contained a gap. Preference for the 50% alternative decreased in conditions in which the gap preceded either all delay stimuli, both delay stimuli for the 50% alternative, or the food stimulus for the 50% alternative, but preference was not consistently affected in conditions in which the gap preceded only the 100% delay stimulus or the blackout stimulus for the 50% alternative. These results support the notion that conditioned reinforcement underlies the finding of suboptimal preference in probabilistic reinforcement procedures, and that the signal for food on the 50% reinforcement alternative functions as a stronger conditioned reinforcer than the signal for food on the 100% reinforcement alternative. In addition, the results fail to provide evidence that the signal for blackout functions as a conditioned punisher.     

Choice and conditioned reinforcement.

A potential weakness of one formulation of delay-reduction theory is its failure to include a term for rate of conditioned reinforcement, that is, the rate at which the terminal-link stimuli occur in concurrent-chains schedules. The present studies assessed whether or not rate of conditioned reinforcement has an independent effect upon choice. Pigeons responded on either modified concurrent-chains schedules or on comparable concurrent-tandem schedules. The initial link was shortened on only one of two concurrent-chains schedules and on only one of two corresponding concurrent-tandem schedules. This manipulation increased rate of conditioned reinforcement sharply in the chain but not in the tandem schedule. According to a formulation of delay-reduction theory, when the outcomes chosen (the terminal links) are equal, as in Experiment 1, choice should depend only on rate of primary reinforcement; thus, choice should be equivalent for the tandem and chain schedules despite a large difference in rate of conditioned reinforcement. When the outcomes chosen are unequal, however, as in Experiment 2, choice should depend upon both rate of primary reinforcement and relative signaled delay reduction; thus, larger preferences should occur in the chain than in the tandem schedules. These predictions were confirmed, suggesting that increasing the rate of conditioned reinforcement on concurrent-chains schedules may have no independent effect on choice.     

Effects of delayed conditioned reinforcement in chain schedules.

The contingency between responding and stimulus change on a chain variable-interval 33-s, variable-interval 33-s, variable-interval 33-s schedule was weakened by interposing 3-s delays between either the first and second or the second and third links. No stimulus change signaled the delay interval and responses could occur during it, so the obtained delays were often shorter than the scheduled delay. When the delay occurred after the initial link, initial-link response rates decreased by an average of 77% with no systematic change in response rates in the second or third links. Response rates in the second link decreased an average of 59% when the delay followed that link, again with little effect on response rates in the first or third links. Because the effect of delaying stimulus change was comparable to the effect of delaying primary reinforcement in a simple variable-interval schedule, and the effect of the unsignaled delay was specific to the link in which the delay occurred, the results provide strong evidence for the concept of conditioned reinforcement.     

Preference for less segmented fixed-time components in concurrent-chain schedules of reinforcement

A concurrent-chain procedure was used to examine choice between segmented and less segmented response-independent schedules of reinforcement. A pair of independent, concurrent variable-interval 60-s schedules were presented in the initial link, along with a 1.5-s changeover delay. A chained fixed-interval fixed-time and its corresponding tandem schedule constituted the terminal links. The length of the fixed-interval schedule in the terminal link was varied between 5 s and 30 s while that of the fixed-time schedule was kept at 5 s over conditions. The first components of both terminal-link schedules were accompanied by the same stimulus. Except in the baseline condition, the onset of the second component of the terminal-link chained schedule was accompanied by either a localized (key color) or a nonlocalized (dark houselight) stimulus change. Stimulus conditions were constant during the terminal-link tandem schedule. With three exceptions, pigeons demonstrated a slight preference for the tandem over the chained schedule in the terminal link. Furthermore, this preference varied inversely with the length of the first component. In general, these results are consistent with previous studies that reported an adverse effect on choice by segmenting an interval schedule into two or more components, but they are inconsistent with studies that reported preference for signaled over unsignaled delay of reinforcement.     

SOME EFFECTS OF PROCEDURAL VARIATIONS ON CHOICE RESPONDING IN CONCURRENT CHAINS

The present research used pigeons in a three‐key operant chamber and varied procedural features pertaining to both initial and terminal links of concurrent chains. The initial links randomly alternated on the side keys during a session, while the terminal links always appeared on the center key. Both equal and unequal initial‐link schedules were employed, with either differential or nondifferential terminal‐link stimuli across conditions. The research was designed to neutralize initial‐ and terminal‐link spatial cues in order to gain a clearer understanding of the roles of conditioned reinforcement and delayed primary reinforcement in choice. With both equal and unequal initial links and with differential terminal‐link stimuli, all pigeons reliably preferred the chain with the shorter terminal link. However, with equal initial links and nondifferential stimuli, all pigeons were indifferent. With unequal initial links and nondifferential stimuli, some pigeons were also indifferent, while others actually reversed and preferred the chain with the shorter initial link, even though it was followed by the longer terminal link. The decrease if not reversal of the previous preferences implies that preferences in concurrent chains are a function of the conditioned reinforcement afforded by terminal‐link stimuli, rather than delayed primary reinforcement.     

Conditioned reinforcement dynamics in three-link chained schedules.

In two experiments rats were trained on three-link concurrent-chains schedules of reinforcement. In Experiment 1, additional entries to one terminal link were added during one of the middle links to a baseline schedule that was otherwise equal for the two chains, and, depending on the condition, these additional terminal-link presentations ended either in food or in no food. When food occurred, preference was always in favor of the chain with the additional terminal-link presentations (which also entailed a higher rate of reinforcement). When no food occurred at the end of the additional terminal links, the outcome depended on the nature of the stimuli associated with these additional terminal links. When stimuli different from the reinforced baseline terminal links were used for the no-food terminal links, preference was against the choice alternative that led to the extra periods of extinction. When the same stimulus was used for the two kinds of terminal links, preference was near indifference, that is, significantly greater than when different stimuli were used. In Experiment 2, rats learned repeated reversals of a simultaneous discrimination under a three-link concurrent-chains schedule, in which the food or no-food choice outcomes were delayed until the end of the chain. Different conditions were defined by the point in the chain at which differential stimuli occurred. When the middle and terminal links provided no differential stimuli, discrimination was acquired more slowly than when differential stimuli occurred in both links. When differential stimuli occurred in the middle but not the terminal links, acquisition rates were intermediate. Both experiments together show that the effects of stimuli in a chain schedule are due partly to the time to food correlated with the stimuli and partly to the time to the next conditioned reinforcer in the sequence.     

Concurrent-chains schedules as a method to study choice between alcohol-associated conditioned reinforcers

An extensive body of research using concurrent-chains schedules of reinforcement has shown that choice for one of two differentially valued food-associated stimuli is dependent upon the overall temporal context in which those stimuli are embedded. The present experiments examined whether the concurrent chains procedure was useful for the study of behavior maintained by alcohol and alcohol-associated stimuli. In Experiment 1, rats responded on concurrent-chains schedules with equal variable-interval (VI) 10-s schedules in the initial links. Across conditions, fixed-interval schedules in the terminal links were varied to yield 1∶1, 9∶1, and 1∶9 ratios of alcohol delivery. Initial-link response rates reflected changes in terminal-link schedules, with greater relative responding in the rich terminal link. In Experiment 2, terminal-link schedules remained constant with a 9∶1 ratio of alcohol delivery rates while the length of two equal duration initial-link schedules was varied. Preference for the rich terminal link was less extreme when initial links were longer (i.e., the initial-link effect), as has been previously reported with food reinforcers. This result suggests that the conditioned reinforcing value of an alcohol-associated stimulus depends on the temporal context in which it is embedded. The concurrent-chains procedure and quantitative models of concurrent chains performance may provide a useful framework within which to study how contextual variables modulate preference for drug-associated conditioned reinforcers.     

Effects of primary reinforcement on pigeons' initial-link responding under a concurrent chains schedule with nondifferntial terminal links

The effect of primary reinforcement on initial-link responding under concurrent-chains schedules with nondifferential terminal links was assessed in 12 pigeons. The iniitial and terminal links were variable-interval schedules (always the same for both alternatives). The positions (left or right key) of the initial-link stimuli (red or green) were randomized while the correlation between color and food amount remained constant within each condition. The terminal-link stimuli were always presented on the center key. Except in two control groups and conditions, the terminal-link stimuli were the same color (nondifferential, blue or yellow). Over six conditions, the differences in food amont and the durations of the initial- and terminal-link schedules were manipulated. In 57 of 60 cases, birds generated choice proportions above .50 in favor of the initial-link stimlus that was correlated with the larger reinforcer. There was some indication that preference increased with shortened terminal-link durations. Because the terminal-link stimuli were nondifferential, differential responding in the initial links cannot be explained easily by conditioned reinforcement represented by the terminal-link stimuli. Thus, primiary reinforcement has a direct effect on initial-link responding in concurrent-chains schedules.     

The effect of conditioned reinforcement rate on choice: a review

We review the nature of conditioned reinforcement, including evidence that conditioned reinforcers maintain choice behavior in concurrent schedules and that they elevate responding in the terminal links of concurrent‐chains schedules. A question has resurfaced recently: Do theories of choice in concurrent‐chains schedules need to include a term reflecting greater preference for higher rates of conditioned reinforcement? The review of several studies addressing this point suggests that such a term is inappropriate. Elevated rates of conditioned reinforcement (and responding) in the terminal links of concurrent‐chains schedules do not lead to greater preference in the initial link leading to the higher rate of conditioned reinforcement. If anything, the opposite preference is likely to occur. This result is not surprising, since the additional putative conditioned reinforcers in the terminal link are not correlated with a reduction in time to primary reinforcement nor with an increase in value.     

Choice for conditioned reinforcers in the signaled absence of primary reinforcement

Pigeons responded in a multiple schedule in which concurrent schedules of brief-stimulus presentation alternated with a component in which food was available (concurrent-chains component). In the initial links of the concurrent-chains component subjects chose either of two stimuli each correlated with the terminal link of one chain. The terminal links involved either variable-interval 30-second or variable-interval 60-second schedules. In the brief-stimulus component subjects chose between 0.5-second presentations of the terminal-link stimuli from the concurrent-chains component. Responding was generally maintained in the brief-stimulus component in two subjects for more than 300 sessions, suggesting that brief stimuli were conditioned reinforcers. During the brief-stimulus component, in 17 of 21 cases for which a minimal number of responses occurred, choice proportions above 0.55 were obtained for the brief-stimulus presentations correlated with the higher rate of primary reinforcement in the concurrent-chains component. These results support the suggestion that choice in conventional concurrent-chains procedures is partially controlled by production of the terminal-link stimuli.     

Suboptimal choice in a percentage-reinforcement procedure: effects of signal condition and terminal-link length.

Pigeons' choice between reliable (100%) and unreliable (50%) reinforcement was studied using a concurrent-chains procedure. Initial links were fixed-ratio 1 schedules, and terminal links were equal fixed-time schedules. The duration of the terminal links was varied across conditions. The terminal link on the reliable side always ended in food; the terminal link on the unreliable side ended with food 50% of the time and otherwise with blackout. Different stimuli present during the 50% terminal links signaled food or blackout outcomes under signaled conditions but were uncorrelated with outcomes under unsignaled conditions. In signaled conditions, most pigeons displayed a nearly exclusive preference for the 100% alternative when terminal links were short (5 or 10 s), but with terminal links of 30 s or longer, preference for the 100% alternative was sharply reduced (often to below .5). In unsignaled conditions, most pigeons showed extreme preference for the 100% alternative with either short (5 s) or longer (30 s) terminal links. Thus, pigeons' choice between reliable and unreliable reinforcement is influenced by both the signal conditions on the unreliable alternative and the duration of the terminal-link delay. With a long delay and signaled outcomes, many pigeons display a suboptimal tendency to choose the unreliable side.     

Disruption of responding maintained by conditioned reinforcement: alterations in response-conditioned-reinforcer relations

An observing procedure was used to investigate the effects of alterations in response-conditioned-reinforcer relations on observing. Pigeons responded to produce schedule-correlated stimuli paired with the availability of food or extinction. The contingency between observing responses and conditioned reinforcement was altered in three experiments. In Experiment 1, after a contingency was established in baseline between the observing response and conditioned reinforcement, it was removed and the schedule-correlated stimuli were presented independently of responding according to a variable-time schedule. The variable-time schedule was constructed such that the rate of stimulus presentations was yoked from baseline. The removal of the observing contingency reliably reduced rates of observing. In Experiment 2, resetting delays to conditioned reinforcement were imposed between observing responses and the schedule-correlated stimuli they produced. Delay values of 0, 0.5, 1, 5, and 10 s were examined. Rates of observing varied inversely as a function of delay value. In Experiment 3, signaled and unsignaled resetting delays between observing responses and schedule-correlated stimuli were compared. Baseline rates of observing were decreased less by signaled delays than by unsignaled delays. Disruptions in response-conditioned-reinforcer relations produce similar behavioral effects to those found with primary reinforcement.     

Choice and delay of reinforcement: Effects of terminal-link stimulus and response conditions

In two experiments, pigeons were exposed to concurrent-chains schedules in which a single initial-link variable-interval schedule led to access to terminal links composed of fixed-interval or fixed-delay schedules. In Experiment 1, an 8-s (or 16-s) delay to reinforcement was associated with the standard key, while reinforcer delay values associated with the experimental key were varied from 4 to 32 s. The results of Experiment 1 showed undermatching of response ratios to delay ratios with terminal-link fixed-delay schedules, whereas in some pigeons matching or overmatching was evident with the fixed-interval schedules. In Experiment 2, one pair of reinforcer delay values, either 8 versus 16 s or 16 versus 32 s, was used. In the first condition of Experiment 2, different delays were associated with different keylight stimuli (cued condition). In the second condition, different terminal-link delays were associated with the same stimulus, either a blackout (uncued-blackout condition) or a white key (uncued-white condition). To examine the role of responses emitted during delays, the keys were retracted during a delay (key-absent condition) in the third condition and responses were required by a fixed-interval schedule in the fourth condition. Experiment 2 demonstrated that the choice proportions for the shorter delay were more extreme in the cued condition than in the uncued-blackout condition, and that the response requirement imposed by the fixed-interval schedules did not affect choice of the shorter delay, nor did the key-absent and key-present conditions. These results indicate that the keylight-stimulus conditions affected preference for the shorter of two delays and that the findings obtained in Experiment 1 depended mainly on the keylight-stimulus conditions of the terminal links (i.e., the conditioned reinforcing value of the terminal-link stimuli).     

Choice and rate of reinforcement

Pigeons' responses in the presence of two concurrently available (initial-link) stimuli produced one of two different (terminal-link) stimuli. The rate of reinforcement in the presence of one terminal-link stimulus was three times that of the other. Three different pairs of identical but independent variable-interval schedules controlled entry into the terminal links. When the intermediate pair was in effect, the pigeons distributed their (choice) responses in the presence of the concurrently available stimuli of the initial links in the same proportion as reinforcements were distributed in the mutually exclusive terminal links. This finding was consistent with those of earlier studies. When either the pair of larger or smaller variable-interval schedules was in effect, however, proportions of choice responses did not match proportions of reinforcements. In addition, matching was not obtained when entry into the terminal links was controlled by unequal variable-interval schedules. A formulation consistent with extant data states that choice behavior is dependent upon the amount of reduction in the expected time to primary reinforcement, as signified by entry into one terminal link, relative to the amount of reduction in expected time to reinforcement signified by entry into the other terminal link.     

Suboptimal choice and initial‐link requirement

Pigeons (n = 14) were trained in a concurrent‐chains suboptimal choice procedure that tested the effect of an increased ratio requirement in the initial links. Fixed‐ratio 1 and 25 conditions were manipulated within subjects in a counterbalanced order. In all conditions, distinct terminal‐link stimuli on a suboptimal alternative signaled either primary reinforcement (20% of the time) or extinction (80% of the time). On an optimal alternative, two distinct terminal‐link stimuli each signaled a 50% chance of primary reinforcement. Preference for the suboptimal alternative was significantly attenuated, and in some birds completely reversed, by the larger response requirement irrespective of condition order. This larger response requirement also generated a notable increase in between‐subject variability. A measure of cumulative choice responding is introduced to mitigate the problems associated with traditional session averages. Ordinal predictions of some current theories of suboptimal choice are also considered in light of the results.