A molecular analysis of choice on concurrent-chains schedules.

Six pigeons responded on concurrent-chains schedules with either independent or interdependent equal variable-interval schedules in the initial links and unequal variable-interval schedules, always in a 2:1 ratio, in the terminal links. Relative response rates in the initial links increased across conditions as initial-link duration was shortened and decreased across conditions as terminal-link duration was shortened, replicating previous findings. Responses in the initial links were recorded in 5-s bins, and local or molecular relative response rates were calculated in order to ascertain how relative response rate varied as a function of time since the onset of the initial links. Two distinct molecular patterns were found. With interdependent initial links, relative response rates for the preferred key were elevated for the first 10 or 20 s of the initial links and then declined to an asymptotic value. With independent initial links, a negative recency effect was found similar to that reported by Killeen (1970). These two molecular patterns were related to the different momentary reinforcement probabilities resulting from independent and interdependent scheduling.     

Conditioned reinforcement value and choice.

The delay-reduction hypothesis of conditioned reinforcement states that the reinforcing value of a food-associated stimulus is determined by the delay to primary reinforcement signaled by the onset of the stimulus relative to the average delay to primary reinforcement in the conditioning situation. In contrast, most contemporary models of conditioned reinforcement strength posit that the reinforcing strength of a stimulus is some simple function only of the delay to primary reinforcement in the presence of stimulus. The delay-reduction hypothesis diverges from other conditioned reinforcement models in that it predicts that a fixed-duration food-paired stimulus will have different reinforcing values depending on the frequency of its presentation. In Experiment 1, pigeons' key pecks were reinforced according to concurrent-chains schedules with variable-interval 10-second and variable-interval 20-second terminal-link schedules. The initial-link schedule preceding the shorter terminal link was always variable-interval 60 seconds, and the initial-link schedule requirement preceding the longer terminal link was varied between 1 second and 60 seconds across conditions. In Experiment 2, the initial-link schedule preceding the longer of two terminal links was varied for each of three groups of pigeons. The terminal links of the concurrent chains for the three groups were variable-interval 10 seconds and 20 seconds, variable-interval 10 seconds and 30 seconds, and variable-interval 30 seconds and 50 seconds. In both experiments, preference for the shorter terminal link was either a bitonic function or an inverse function of the initial-link schedule preceding the longer terminal-link schedule. Consistent with the predictions of the delay-reduction hypothesis, the relative values of the terminal-link stimuli changed as a function of the overall frequency of primary reinforcement. Vaughan's (1985) melioration model, which was shown to be formally similar to Squires and Fantino's (1971) delay-reduction model, can be modified so as to predict these results without changing its underlying assumptions.     

Context effects on choice.

Four pigeons responded on a concurrent-chains schedule in four experiments that examined whether the effectiveness of a stimulus as a conditioned reinforcer is best described by a global approach, as measured by the average interreinforcement interval, or by a local contextual approach, as measured by the onset of the stimulus preceding the conditioned reinforcer. The interreinforcement interval was manipulated by the inclusion of an intertrial interval, which increased the overall time to reinforcement but did not change the local contingencies on a given trial A global analysis predicted choice for the richer alternative to decrease with the inclusion of an intertrial interval, whereas a local analysis predicted no change in preference. Experiment 1 examined sensitivity to intertrial intervals when each was signaled by the same houselight that operated throughout the session. In Experiment 2, the intertrial interval always was signaled by the stimulus correlated with the richer terminal link. In Experiment 3, the intertrial interval was signaled by the keylights correlated with the initial links and two novel houselights. Experiment 4 provided free food pseudorandomly during the intertrial interval. In all experiments, subjects' preferences were consistent with a local analysis of choice in concurrent chains. These results are discussed in terms of delay-reduction theory, which traditionally has failed to distinguish global and local contexts.     

Choice with uncertain outcomes: conditioned reinforcement effects.

Pigeons responded on concurrent chains with equal initial- and terminal-link durations. In all conditions, the terminal links of one chain ended reliably in reinforcement; the terminal links on the alternative chain ended in either food or blackout. In Experiment 1, the terminal-link stimuli were correlated with (signaled) the outcome, and the durations of the initial and terminal links were varied across conditions. Preference did not vary systematically across conditions. In Experiment 2, terminal-link durations were varied under different stimulus conditions. The initial links were variable-interval 80-s schedules. Preference for the reliable alternative was generally higher in unsignaled than in signaled conditions. Preference increased with terminal-link durations only in the unsignaled conditions. There were no consistent differences between conditions with and without a common signal for reinforcement on the two chains. In the first series of conditions in Experiment 3, a single response was required in the initial links, and the stimulus conditions during 50-s terminal links were varied. Preference for the reliable outcome approached 1.0 in unsignaled conditions and was considerably lower (below .50 for 3 of 5 subjects) in signaled conditions. In a final series of signaled conditions with relatively long terminal links, preference varied with duration of the initial links. The results extend previous findings and are discussed in terms of the delay reduction signaled by terminal-link stimuli.     

Suboptimal choice in a percentage-reinforcement procedure: effects of signal condition and terminal-link length.

Pigeons' choice between reliable (100%) and unreliable (50%) reinforcement was studied using a concurrent-chains procedure. Initial links were fixed-ratio 1 schedules, and terminal links were equal fixed-time schedules. The duration of the terminal links was varied across conditions. The terminal link on the reliable side always ended in food; the terminal link on the unreliable side ended with food 50% of the time and otherwise with blackout. Different stimuli present during the 50% terminal links signaled food or blackout outcomes under signaled conditions but were uncorrelated with outcomes under unsignaled conditions. In signaled conditions, most pigeons displayed a nearly exclusive preference for the 100% alternative when terminal links were short (5 or 10 s), but with terminal links of 30 s or longer, preference for the 100% alternative was sharply reduced (often to below .5). In unsignaled conditions, most pigeons showed extreme preference for the 100% alternative with either short (5 s) or longer (30 s) terminal links. Thus, pigeons' choice between reliable and unreliable reinforcement is influenced by both the signal conditions on the unreliable alternative and the duration of the terminal-link delay. With a long delay and signaled outcomes, many pigeons display a suboptimal tendency to choose the unreliable side.     

Effects of reinforcement context on choice

Two experiments investigated the effects of successive reinforcement contexts on choice. In the first, concurrent variable-interval schedules of primary reinforcement operated during the initial links of concurrent chains. The rate of this reinforcement arranged by the concurrent schedules was decreased across conditions: When it was higher than the terminal-link rate, preference for the higher frequency initial-link schedule increased relative to baseline. (During baseline, a standard concurrent-schedule procedure was in effect). When the initial-link reinforcement rate was lower than the terminal-link rate, preference converged toward indifference. In the second experiment, a chain schedule was available on a third key while a concurrent schedule was in effect on the side keys. When the terminal link of the chain schedule was produced, the side keys became inoperative. Availability of the chain schedule did not affect choice between the concurrent schedules. These results show that only when successive reinforcement contexts are produced by choice responding do those successive contexts affect choice in concurrent schedules.     

A quantitative analysis of sensitivity to the conditioned reinforcing value of terminal-link stimuli in a concurrent-chains schedule.

Pigeons were exposed to a concurrent-chains schedule in which a single variable-interval 30-s schedule was used in the initial links and fixed-time schedules were used in the terminal links. Three types of keylight conditions were used in the terminal links. In the first condition, different delays were associated with different keylight stimuli (cued condition). In the second condition, different delays were associated with the same stimulus, either a blackout (uncued blackout condition) or a white key (uncued white condition). Paired values of terminal-link fixed-time schedules differed by a constant ratio of 3:1, while the absolute value of delays was varied from 3 s to 54 s. The results showed that choice proportions for the shorter of two delays increased when the absolute size of the delays was increased for all keylight conditions. Further, the choice proportions for the shorter delay increased from the uncued blackout condition, to the uncued white condition, to the cued condition. A modified version of Fantino's (1969) delay-reduction model (expressed as a function relating the response ratio to the delay-reduction ratio) can be applied to these data by showing that sensitivity to delay reduction increased from the uncued blackout condition, to the uncued white condition, to the cued condition. Thus, the present study demonstrated that a modified version of the delay-reduction model can be used to assess quantitative differences in the terminal-link keylight condition in terms of sensitivity to delay reduction (i.e., the conditioned reinforcing value of the terminal-link keylight stimuli).     

Control of behavior by an establishing stimulus

Seventeen pigeons were exposed to a three-key discrete-trial procedure in which a peck on the lit center key produced food if, and only if, the left keylight was lit. The center key was illuminated by a peck on the lit right key. Of interest was whether subjects pecked the right key before or after the response-independent onset of the left keylight. Pecks on the right key after left-keylight onset suggest control of behavior by the left keylight—an establishing stimulus. In three experiments, the strength of center-keylight onset as conditioned reinforcer for a response on the right key was manipulated by altering the size of the reduction in time to food delivery correlated with its onset. Control of pigeons' key pecks by onset of the left keylight occurred on more trials per session when the center keylight was a relatively weak conditioned reinforcer and on fewer trials per session when the center keylight was a relatively strong condtioned reinforcer. Differences across conditions in the degree of control by onset of the establishing stimulus were greatest when changes in conditioned reinforcer strength occurred relatively frequently and were signaled. The results provide evidence of the function of an establishing stimulus.     

Sample-duration effects on pigeons' delayed matching as a function of predictability of duration

Three experiments assessed the impact of sample duration on pigeons' delayed matching as a function of whether or not the samples themselves signaled how long they would remain on. When duration was uncorrelated with the sample appearing on each matching trial, the typical effect of duration was observed: Choice accuracy was higher with long (15-s) than with short (5-s) durations. By contrast, this difference either disappeared or reversed when the 5- and 15-s durations were correlated with the sample stimuli. Sample duration itself cued comparison choice by some birds in the latter (predictable) condition when duration was also correlated with the reinforced choice alternatives. However, even when duration could not provide a cue for choice, pigeons matched predictably short-duration samples as accurately as, or more accurately than, predictably long-duration samples. Moreover, this result was observed independently of whether the contextual conditions of the retention interval were the same as, or different from, those of the intertrial interval. These results strongly support the view that conditional stimulus control by the samples is partly a function of their conditioned reinforcing properties, as determined by the relative reduction in overall delay to reinforcement that they signal.     

SOME EFFECTS OF PROCEDURAL VARIATIONS ON CHOICE RESPONDING IN CONCURRENT CHAINS

The present research used pigeons in a three‐key operant chamber and varied procedural features pertaining to both initial and terminal links of concurrent chains. The initial links randomly alternated on the side keys during a session, while the terminal links always appeared on the center key. Both equal and unequal initial‐link schedules were employed, with either differential or nondifferential terminal‐link stimuli across conditions. The research was designed to neutralize initial‐ and terminal‐link spatial cues in order to gain a clearer understanding of the roles of conditioned reinforcement and delayed primary reinforcement in choice. With both equal and unequal initial links and with differential terminal‐link stimuli, all pigeons reliably preferred the chain with the shorter terminal link. However, with equal initial links and nondifferential stimuli, all pigeons were indifferent. With unequal initial links and nondifferential stimuli, some pigeons were also indifferent, while others actually reversed and preferred the chain with the shorter initial link, even though it was followed by the longer terminal link. The decrease if not reversal of the previous preferences implies that preferences in concurrent chains are a function of the conditioned reinforcement afforded by terminal‐link stimuli, rather than delayed primary reinforcement.     

Dual effects on choice of conditioned reinforcement frequency and conditioned reinforcement value

Pigeons were presented with a concurrent‐chains schedule in which the total time to primary reinforcement was equated for the two alternatives (VI 30 s VI 60 s vs. VI 60 s VI 30 s). In one set of conditions, the terminal links were signaled by the same stimulus, and in another set of conditions they were signaled by different stimuli. Choice was in favor of the shorter terminal link when the terminal links were differentially signaled but in favor of the shorter initial link (and longer terminal link) when the terminal links shared the same stimulus. Preference reversed regularly with reversals of the stimulus condition and was unrelated to the discrimination between the two terminal links during the nondifferential stimulus condition. The present results suggest that the relative value of the terminal‐link stimuli and the relative rate of conditioned reinforcer presentation are important influences on choice behavior, and that models of conditioned reinforcement need to include both factors.     

Rapid acquisition in concurrent chains: evidence for a decision model

Pigeons' choice in concurrent chains can adapt to rapidly changing contingencies. Grace, Bragason, and McLean (2003) found that relative initial-link response rate was sensitive to the immediacy ratio in the current session when one of the terminal-link fixed-interval schedules was changed daily according to a pseudorandom binary sequence (e.g., Schofield & Davison, 1997). The present experiment tested whether the degree of variation in delays across sessions had any effect on acquisition rate in Grace et al.'s (2003) rapid-acquisition procedure. In one condition ("minimal variation"), the left terminal link was always fixed-interval 8 s and the right terminal link was either fixed-interval 4 s or fixed-interval 16 s. In the other condition ("maximal variation"), a unique pair of fixed-interval values was used in each session. Responding was sensitive to the current-session immediacy ratio in both conditions, but across subjects there was no systematic difference in sensitivity. These results challenge the view that initial-link responding in the rapid-acquisition procedure is determined by changes in the learned value of the terminal-link stimuli, and suggests instead that a process resembling categorical discrimination may control performance. A decision model based on the assumption that delays are categorized as short or long relative to the history of delays provided a good account of the data and shows promise in being able to explain other choice phenomena.     

Contiguity and conditioned reinforcement in probabilistic choice

In a baseline condition, pigeons chose between an alternative that always provided food following a 30-s delay (100% reinforcement) and an alternative that provided food half of the time and blackout half of the time following 30-s delays (50% reinforcement). The different outcomes were signaled by different-colored keylights. On average, each alternative was chosen approximately equally often, replicating the finding of suboptimal choice in probabilistic reinforcement procedures. The efficacy of the delay stimuli (keylights) as conditioned reinforcers was assessed in other conditions by interposing a 5-s gap (keylights darkened) between the choice response and one or more of the delay stimuli. The strength of conditioned reinforcement was measured by the decrease in choice of an alternative when the alternative contained a gap. Preference for the 50% alternative decreased in conditions in which the gap preceded either all delay stimuli, both delay stimuli for the 50% alternative, or the food stimulus for the 50% alternative, but preference was not consistently affected in conditions in which the gap preceded only the 100% delay stimulus or the blackout stimulus for the 50% alternative. These results support the notion that conditioned reinforcement underlies the finding of suboptimal preference in probabilistic reinforcement procedures, and that the signal for food on the 50% reinforcement alternative functions as a stronger conditioned reinforcer than the signal for food on the 100% reinforcement alternative. In addition, the results fail to provide evidence that the signal for blackout functions as a conditioned punisher.     

Preferences for and against stimuli paired with food.

Pigeons were presented with a concurrent-chains schedule in which terminal-link entries were assigned to two response keys on a percentage basis. The terminal links were fixed delays that sometimes ended with food and sometimes did not. In most conditions, 80% of the terminal links were assigned to one key, but a smaller percentage of the terminal links ended with food for this key, so the number of food reinforcers delivered by the two alternatives was equal. When the same terminal-link stimuli (orange houselights) were used for both alternatives, the pigeons showed a preference for whichever alternative delivered more frequent terminal links. When different terminal-link stimuli (green vs. red houselights) were used for the two alternatives, the pigeons showed a preference for whichever alternative delivered fewer terminal links when terminal-link durations were long, and no systematic preferences when terminal-link durations were short. This pattern of results was consistent with the predictions of Grace's (1994) contextual choice model. Preference for the alternative that delivered more frequent terminal links was usually stronger in the first few sessions of a condition than at the end of a condition, suggesting that the conditioned reinforcing effect of the additional terminal-link presentation was, in part, transitory.     

A decision model for steady-state choice in concurrent chains

Grace and McLean (2006) proposed a decision model for acquisition of choice in concurrent chains which assumes that after reinforcement in a terminal link, subjects make a discrimination whether the preceding reinforcer delay was short or long relative to a criterion. Their model was subsequently extended by Christensen and Grace (2008, 2009a, 2009b) to include effects of initial- and terminal-link duration on choice. We show that an expression for steady-state responding can be derived from the decision model, which enables a model for choice that provides an account of archival data that is equal or superior to the contextual choice model (Grace, 1994) and hyperbolic value-added model (Mazur, 2001) in terms of goodness of fit, parsimony, and parameter invariance. The success of the steady-state decision model validates the strategy of understanding acquisition phenomena as a bridge toward explaining choice at the molar level.     

Preference for conditioned reinforcement.

Pigeons were presented with a concurrent-chains schedule in which both choice alternatives led to the same terminal-link stimulus, which was followed by food. Superimposed on the food-reinforced presentations of the terminal-link stimulus was a second schedule of presentations of the same stimulus that were followed by no food. The absolute number of these no-food stimulus presentations was held constant while their relative frequency assigned to one or the other choice alternative was systematically varied. Preference for a given choice alternative tracked the relative frequency of these stimulus presentations, thus demonstrating that they served as reinforcers. These results resolve conflicts in the literature regarding the effect of conditioned reinforcement on choice.     

The effects of delayed reinforcement on free-operant responding

In previous studies of delayed reinforcement, response rate has been found to vary inversely with the response-reinforcer interval. However, in all of these studies the independent variable, response-reinforcer time, was confounded with the number of reinforcers presented in a fixed period of time (reinforcer frequency). In the present study, the frequency of available reinforcers was held constant, while temporal separation between response and reinforcer was independently manipulated. A repeating time cycle, T, was divided into two alternating time periods, t^D and t^Δ. The first response in t^D was reinforced at the end of the prevailing T cycle and extinction prevailed in t^Δ. Two placements for t^D were defined, an early t^D placement in which t^D precedes t^Δ and a late t^D placement in which t^D follows t^Δ. The duration of the early and late t^D was systematically decreased from 30 seconds (i.e., t^D = T) to 0.1 second. Manipulation of t^D placement and duration controlled the temporal separation between response and reinforcement, but it did not affect the frequency of programmed reinforcers, which was 1/T. The results show that early and late t^D placements of equal duration have similar overall effects upon response rate, reinforcer frequency, responses per reinforcer, and obtained response-reinforcer temporal separation. A stepwise regression analysis using log response rate as the dependent variable showed that the obtained delay was a significant first-step variable for six of eight subjects, with obtained reinforcer frequency significant for the remaining two subjects.     

Nonstable concurrent choice in pigeons

Six pigeons were trained on concurrent variable-interval schedules in which the arranged reinforcer ratios changed from session to session according to a 31-step pseudorandom binary sequence. This procedure allows a quantitative analysis of the degree to which performance in an experimental session is affected by conditions in previous sessions. Two experiments were carried out. In each, the size of the reinforcer ratios arranged between the two concurrent schedules was varied between 31-step conditions. In Experiment 1, the concurrent schedules were arranged independently, and in Experiment 2 they were arranged nonindependently. An extended form of the generalized matching law described the relative contribution of past and present events to present-session behavior. Total performance in sessions was mostly determined by the reinforcer ratio in that session and partially by reinforcers that had been obtained in previous sessions. However, the initial exposure to the random sequence produced a lower sensitivity to current-session reinforcers but no difference in overall sensitivity to reinforcement. There was no evidence that the size of the reinforcer ratios available on the concurrent schedules affected either overall sensitivity to reinforcement or the sensitivity to reinforcement in the current session. There was also no evidence of any different performance between independent and nonindependent scheduling. Because of these invariances, this experiment validates the use of the pseudorandom sequence for the fast determination of sensitivity to reinforcement.     

Response-dependent prechoice effects on foraging-related choice

Three experiments explored the influence of prechoice events on pigeons' preference. In two of three studies, a fixed-interval 200-second prechoice period preceded the initial links of a concurrent chain in which outcomes differed either (a) in terms of the delay to food or (b) in terms of amount of food and delay to food. In Experiment 3, the prechoice period preceded the initial links that provided a choice between a small single food presentation and two identical, more delayed food presentations. In all three cases, obtained choice proportions did not vary as a function of prechoice duration. These results suggest that a local-contextual view adequately describes the foraging context; they also have implications for the appropriate formulation of the delay-reduction theory of conditioned reinforcement and rate-maximizing views of optimal foraging theory.