Conditioned reinforcement value and choice.

The delay-reduction hypothesis of conditioned reinforcement states that the reinforcing value of a food-associated stimulus is determined by the delay to primary reinforcement signaled by the onset of the stimulus relative to the average delay to primary reinforcement in the conditioning situation. In contrast, most contemporary models of conditioned reinforcement strength posit that the reinforcing strength of a stimulus is some simple function only of the delay to primary reinforcement in the presence of stimulus. The delay-reduction hypothesis diverges from other conditioned reinforcement models in that it predicts that a fixed-duration food-paired stimulus will have different reinforcing values depending on the frequency of its presentation. In Experiment 1, pigeons' key pecks were reinforced according to concurrent-chains schedules with variable-interval 10-second and variable-interval 20-second terminal-link schedules. The initial-link schedule preceding the shorter terminal link was always variable-interval 60 seconds, and the initial-link schedule requirement preceding the longer terminal link was varied between 1 second and 60 seconds across conditions. In Experiment 2, the initial-link schedule preceding the longer of two terminal links was varied for each of three groups of pigeons. The terminal links of the concurrent chains for the three groups were variable-interval 10 seconds and 20 seconds, variable-interval 10 seconds and 30 seconds, and variable-interval 30 seconds and 50 seconds. In both experiments, preference for the shorter terminal link was either a bitonic function or an inverse function of the initial-link schedule preceding the longer terminal-link schedule. Consistent with the predictions of the delay-reduction hypothesis, the relative values of the terminal-link stimuli changed as a function of the overall frequency of primary reinforcement. Vaughan's (1985) melioration model, which was shown to be formally similar to Squires and Fantino's (1971) delay-reduction model, can be modified so as to predict these results without changing its underlying assumptions.     

Context effects on choice.

Four pigeons responded on a concurrent-chains schedule in four experiments that examined whether the effectiveness of a stimulus as a conditioned reinforcer is best described by a global approach, as measured by the average interreinforcement interval, or by a local contextual approach, as measured by the onset of the stimulus preceding the conditioned reinforcer. The interreinforcement interval was manipulated by the inclusion of an intertrial interval, which increased the overall time to reinforcement but did not change the local contingencies on a given trial A global analysis predicted choice for the richer alternative to decrease with the inclusion of an intertrial interval, whereas a local analysis predicted no change in preference. Experiment 1 examined sensitivity to intertrial intervals when each was signaled by the same houselight that operated throughout the session. In Experiment 2, the intertrial interval always was signaled by the stimulus correlated with the richer terminal link. In Experiment 3, the intertrial interval was signaled by the keylights correlated with the initial links and two novel houselights. Experiment 4 provided free food pseudorandomly during the intertrial interval. In all experiments, subjects' preferences were consistent with a local analysis of choice in concurrent chains. These results are discussed in terms of delay-reduction theory, which traditionally has failed to distinguish global and local contexts.     

Conditioned reinforcement and choice with delayed and uncertain primary reinforcers.

In an adjusting-delay choice procedure, pigeons could peck on either a red key or a green key. A peck on the red key always led to a delay associated with red houselights and then food. The delay was adjusted over trials to estimate an indifference point--a delay at which the two keys were chosen about equally often. In some conditions, a peck on the green key led to food on all trials after delays of either 10 s or 30 s, and green houselights were lit during the delays. In other conditions, food was presented on only half of the green-key trials. If the green houselights continued to occur on both reinforcement and nonreinforcement trials, preference for the green key always decreased. Preference for the green key also decreased if half of the trials had 30-s houselights followed by food and the other half had no green houselights and no food. However, preference for the green key actually increased if half of the trials had 10-s green houselights followed by food and the other half had no green houselights followed by no food. The latter condition therefore demonstrated a case in which preference for an alternative increased when food was removed from half of the trials. The results suggest that the red and green houselights served as conditioned reinforcers. A hyperbolic decay model (Mazur, 1989) provided good predictions for all conditions by assuming that the strength of a conditioned reinforcer is inversely related to the total time spent in its presence before food is delivered.     

Suboptimal choice in a percentage-reinforcement procedure: effects of signal condition and terminal-link length.

Pigeons' choice between reliable (100%) and unreliable (50%) reinforcement was studied using a concurrent-chains procedure. Initial links were fixed-ratio 1 schedules, and terminal links were equal fixed-time schedules. The duration of the terminal links was varied across conditions. The terminal link on the reliable side always ended in food; the terminal link on the unreliable side ended with food 50% of the time and otherwise with blackout. Different stimuli present during the 50% terminal links signaled food or blackout outcomes under signaled conditions but were uncorrelated with outcomes under unsignaled conditions. In signaled conditions, most pigeons displayed a nearly exclusive preference for the 100% alternative when terminal links were short (5 or 10 s), but with terminal links of 30 s or longer, preference for the 100% alternative was sharply reduced (often to below .5). In unsignaled conditions, most pigeons showed extreme preference for the 100% alternative with either short (5 s) or longer (30 s) terminal links. Thus, pigeons' choice between reliable and unreliable reinforcement is influenced by both the signal conditions on the unreliable alternative and the duration of the terminal-link delay. With a long delay and signaled outcomes, many pigeons display a suboptimal tendency to choose the unreliable side.     

Response-dependent prechoice effects on foraging-related choice

Three experiments explored the influence of prechoice events on pigeons' preference. In two of three studies, a fixed-interval 200-second prechoice period preceded the initial links of a concurrent chain in which outcomes differed either (a) in terms of the delay to food or (b) in terms of amount of food and delay to food. In Experiment 3, the prechoice period preceded the initial links that provided a choice between a small single food presentation and two identical, more delayed food presentations. In all three cases, obtained choice proportions did not vary as a function of prechoice duration. These results suggest that a local-contextual view adequately describes the foraging context; they also have implications for the appropriate formulation of the delay-reduction theory of conditioned reinforcement and rate-maximizing views of optimal foraging theory.     

Choice with uncertain outcomes: conditioned reinforcement effects.

Pigeons responded on concurrent chains with equal initial- and terminal-link durations. In all conditions, the terminal links of one chain ended reliably in reinforcement; the terminal links on the alternative chain ended in either food or blackout. In Experiment 1, the terminal-link stimuli were correlated with (signaled) the outcome, and the durations of the initial and terminal links were varied across conditions. Preference did not vary systematically across conditions. In Experiment 2, terminal-link durations were varied under different stimulus conditions. The initial links were variable-interval 80-s schedules. Preference for the reliable alternative was generally higher in unsignaled than in signaled conditions. Preference increased with terminal-link durations only in the unsignaled conditions. There were no consistent differences between conditions with and without a common signal for reinforcement on the two chains. In the first series of conditions in Experiment 3, a single response was required in the initial links, and the stimulus conditions during 50-s terminal links were varied. Preference for the reliable outcome approached 1.0 in unsignaled conditions and was considerably lower (below .50 for 3 of 5 subjects) in signaled conditions. In a final series of signaled conditions with relatively long terminal links, preference varied with duration of the initial links. The results extend previous findings and are discussed in terms of the delay reduction signaled by terminal-link stimuli.     

Choice and number of reinforcers

Pigeons were exposed to the concurrent-chains procedure in two experiments designed to investigate the effects of unequal numbers of reinforcers on choice. In Experiment 1, the pigeons were indifferent between long and short durations of access to variable-interval schedules of equal reinforcement density, but preferred a short high-density terminal link over a longer, lower density terminal link, even though in both sets of comparisons there were many more reinforcers per cycle in the longer terminal link. In Experiment 2, the pigeons preferred five reinforcers, the first of which was available after 30 sec, over a single reinforcer available at 30 sec, but only when the local interval between successive reinforcers was short. The pigeons were indifferent when this local interval was sufficiently long. The pigeons' behavior appeared to be under the control of local terminal-link variables, such as the intervals to the first reinforcer and between successive reinforcers, and was not well described in terms of transformed delays of reinforcement or reductions in average delay to reinforcement.     

Choice with delayed and probabilistic reinforcers: effects of variability, time between trials, and conditioned reinforcers.

In a discrete-trials procedure with pigeons, a response on a green key led to a 4-s delay (during which green houselights were lit) and then a reinforcer might or might not be delivered. A response on a red key led to a delay of adjustable duration (during which red houselights were lit) and then a certain reinforcer. The delay was adjusted so as to estimate an indifference point--a duration for which the two alternatives were equally preferred. Once the green key was chosen, a subject had to continue to respond on the green key until a reinforcer was delivered. Each response on the green key, plus the 4-s delay that followed every response, was called one "link" of the green-key schedule. Subjects showed much greater preference for the green key when the number of links before reinforcement was variable (averaging four) than when it was fixed (always exactly four). These findings are consistent with the view that probabilistic reinforcers are analogous to reinforcers delivered after variable delays. When successive links were separated by 4-s or 8-s "interlink intervals" with white houselights, preference for the probabilistic alternative decreased somewhat for 2 subjects but was unaffected for the other 2 subjects. When the interlink intervals had the same green houselights that were present during the 4-s delays, preference for the green key decreased substantially for all subjects. These results provided mixed support for the view that preference for a probabilistic reinforcer is inversely related to the duration of conditioned reinforcers that precede the delivery of food.     

Effects of reinforcement context on choice

Two experiments investigated the effects of successive reinforcement contexts on choice. In the first, concurrent variable-interval schedules of primary reinforcement operated during the initial links of concurrent chains. The rate of this reinforcement arranged by the concurrent schedules was decreased across conditions: When it was higher than the terminal-link rate, preference for the higher frequency initial-link schedule increased relative to baseline. (During baseline, a standard concurrent-schedule procedure was in effect). When the initial-link reinforcement rate was lower than the terminal-link rate, preference converged toward indifference. In the second experiment, a chain schedule was available on a third key while a concurrent schedule was in effect on the side keys. When the terminal link of the chain schedule was produced, the side keys became inoperative. Availability of the chain schedule did not affect choice between the concurrent schedules. These results show that only when successive reinforcement contexts are produced by choice responding do those successive contexts affect choice in concurrent schedules.     

Devaluation of stimuli contingent on choice: evidence for conditioned reinforcement

Pigeons were presented a concurrent-chains schedule of reinforcement that had terminal links of equal duration. The initial links of the schedule were periodically interrupted by 15-s periods during which an extinction schedule was in effect. The extinction periods were presented on either a response-contingent or a noncontingent basis. Relative response rate for the left alternative decreased when the extinction periods were accompanied by the left terminal-link stimulus. Relative response rate for the right alternative decreased when the extinction periods were accompanied by the right terminal-link stimulus. Relative response rate varied inversely with the frequency of presentation of the extinction periods but was unaffected by presence versus absence of the response contingency in the schedule of extinction-period presentation. Furthermore, relative response rate was unaffected by presentation of extinction periods accompanied by a novel stimulus. When the extinction periods were presented after reinforcement in the left terminal link instead of as interruptions of the initial links, relative response rate for the left alternative was reduced if the postreinforcement extinction period was accompanied by the terminal-link stimulus for the left chain and reduced less if the extinction period was accompanied by the terminal-link stimulus for the right chain. The results demonstrate that the correlation between the terminal-link stimulus and extinction influenced the relative response rate in the initial link.     

Choice and multiple reinforcers

Pigeons chose between equivalent two-component mixed and multiple terminal-link schedules of reinforcement in the concurrent-chains procedure. The pigeons preferred the multiple schedule over the mixed when the components of the compound schedules were differentiated in terms of density of reinforcement, but the pigeons were indifferent when the components were differentiated in terms of number of reinforcers per cycle. Taken together, these results indicate that a local variable, the interval to the first reinforcer, but not a molar variable, the number of reinforcers, was sufficient to differentiate the components and thereby evoke preference.     

Effects on preference of reinforcement delay, number of reinforcers, and terminal-link duration

Three experiments used concurrent-chains procedures to examine the effects of reinforcement delay, number of reinforcers, and terminal-link duration on preference. In Condition 30 of Experiment 1, food was delivered after 30 seconds in each 150-second terminal link, with four additional food deliveries occurring at 30-second intervals in one of the links. In Condition 5, food was delivered after 5 seconds in each 25-second terminal link, and the four additional reinforcers were delivered at 5-second intervals. Preferences for the multiple-food chain were greater in Condition 30. In Experiment 2, the terminal link(s) providing only one reinforcer terminated immediately after delivery of the reinforcer. Preferences for the multiple-food chain were smaller than in Experiment 1. In Condition 5 of Experiment 3, food was delivered after 5, 75, 100, 125, and 150 seconds in one 150-second link and after 5 seconds in the other. Condition 50 differed only in that the first (or only) reinforcer in each link was delivered after 50 seconds instead of after 5 seconds. Preferences for the multiple-food chain were greater in Condition 50. Results of Experiments 1 and 2 do not correspond to results obtained by Moore (1979).     

Choice between single and multiple delayed reinforcers.

Pigeons chose between alternatives that differed in the number of reinforcers and in the delay to each reinforcer. A peck on a red key produced the same consequences on every trial within a condition, but between conditions the number of reinforcers varied from one to three and the reinforcer delays varied between 5 s and 30 s. A peck on a green key produced a delay of adjustable duration and then a single reinforcer. The green-key delay was increased or decreased many times per session, depending on a subject's previous choices, which permitted estimation of an indifference point, or a delay at which a subject chose each alternative about equally often. The indifference points decreased systematically with more red-key reinforcers and with shorter red-key delays. The results did not support the suggestion of Moore (1979) that multiple delayed reinforcers have no effect on preference unless they are closely grouped. The results were well described in quantitative detail by a simple model stating that each of a series of reinforcers increases preference, but that a reinforcer's effect is inversely related to its delay. The success of this model, which considers only delay of reinforcement, suggested that the overall rate of reinforcement for each alternative had no effect on choice between those alternatives.     

Choice and conditioned reinforcement.

A potential weakness of one formulation of delay-reduction theory is its failure to include a term for rate of conditioned reinforcement, that is, the rate at which the terminal-link stimuli occur in concurrent-chains schedules. The present studies assessed whether or not rate of conditioned reinforcement has an independent effect upon choice. Pigeons responded on either modified concurrent-chains schedules or on comparable concurrent-tandem schedules. The initial link was shortened on only one of two concurrent-chains schedules and on only one of two corresponding concurrent-tandem schedules. This manipulation increased rate of conditioned reinforcement sharply in the chain but not in the tandem schedule. According to a formulation of delay-reduction theory, when the outcomes chosen (the terminal links) are equal, as in Experiment 1, choice should depend only on rate of primary reinforcement; thus, choice should be equivalent for the tandem and chain schedules despite a large difference in rate of conditioned reinforcement. When the outcomes chosen are unequal, however, as in Experiment 2, choice should depend upon both rate of primary reinforcement and relative signaled delay reduction; thus, larger preferences should occur in the chain than in the tandem schedules. These predictions were confirmed, suggesting that increasing the rate of conditioned reinforcement on concurrent-chains schedules may have no independent effect on choice.     

Preference for conditioned reinforcement.

Pigeons were presented with a concurrent-chains schedule in which both choice alternatives led to the same terminal-link stimulus, which was followed by food. Superimposed on the food-reinforced presentations of the terminal-link stimulus was a second schedule of presentations of the same stimulus that were followed by no food. The absolute number of these no-food stimulus presentations was held constant while their relative frequency assigned to one or the other choice alternative was systematically varied. Preference for a given choice alternative tracked the relative frequency of these stimulus presentations, thus demonstrating that they served as reinforcers. These results resolve conflicts in the literature regarding the effect of conditioned reinforcement on choice.     

The repeated insertion model for rankings: Missing link between two subset choice models

Several probabilistic models for subset choice have been proposed in the literature, for example, to explain approval voting data. We show that Marley et al.'s latent scale model is subsumed by Falmagne and Regenwetter's size-independent model, in the sense that every choice probability distribution generated by the former can also be explained by the latter. Our proof relies on the construction of a probabilistic ranking model which we label the repeated insertion model. This model is a special case of Marden's orthogonal contrast model class and, in turn, includes the classical Mallows -model as a special case. We explore its basic properties as well as its relationship to Fligner and Verducci's multistage ranking model.The authors are grateful to the National Science Foundation for grants SES98-18756 to Regenwetter and Peke, and SBR97-30076 to Regenwetter. This collaborative research was carried out in the context of the conference Random Utility 2000 held at Duke University and sponsored by NSF, the Fuqua School of Business and the Center for International Business Education and Research. We thank the editor and four referees for helpful suggestions and we are grateful to Prof. J. I. Marden for providing useful information on contrast models. We thank Moon-Ho Ho for programming and running the data analyses.     

Choice for conditioned reinforcers in the signaled absence of primary reinforcement

Pigeons responded in a multiple schedule in which concurrent schedules of brief-stimulus presentation alternated with a component in which food was available (concurrent-chains component). In the initial links of the concurrent-chains component subjects chose either of two stimuli each correlated with the terminal link of one chain. The terminal links involved either variable-interval 30-second or variable-interval 60-second schedules. In the brief-stimulus component subjects chose between 0.5-second presentations of the terminal-link stimuli from the concurrent-chains component. Responding was generally maintained in the brief-stimulus component in two subjects for more than 300 sessions, suggesting that brief stimuli were conditioned reinforcers. During the brief-stimulus component, in 17 of 21 cases for which a minimal number of responses occurred, choice proportions above 0.55 were obtained for the brief-stimulus presentations correlated with the higher rate of primary reinforcement in the concurrent-chains component. These results support the suggestion that choice in conventional concurrent-chains procedures is partially controlled by production of the terminal-link stimuli.