Token reinforcement, choice, and self-control in pigeons.

Pigeons were exposed to self-control procedures that involved illumination of light-emitting diodes (LEDs) as a form of token reinforcement. In a discrete-trials arrangement, subjects chose between one and three LEDs; each LED was exchangeable for 2-s access to food during distinct posttrial exchange periods. In Experiment 1, subjects generally preferred the immediate presentation of a single LED over the delayed presentation of three LEDs, but differences in the delay to the exchange period between the two options prevented a clear assessment of the relative influence of LED delay and exchange-period delay as determinants of choice. In Experiment 2, in which delays to the exchange period from either alternative were equal in most conditions, all subjects preferred the delayed three LEDs more often than in Experiment-1. In Experiment 3, subjects preferred the option that resulted in a greater amount of food more often if the choices also produced LEDs than if they did not. In Experiment 4, preference for the delayed three LEDs was obtained when delays to the exchange period were equal, but reversed in favor of an immediate single LED when the latter choice also resulted in quicker access to exchange periods. The overall pattern of results suggests that (a) delay to the exchange period is a more critical determinant of choice than is delay to token presentation; (b) tokens may function as conditioned reinforcers, although their discriminative properties may be responsible for the self-control that occurs under token reinforcer arrangements; and (c) previously reported differences in the self-control choices of humans and pigeons may have resulted at least in part from the procedural conventions of using token reinforcers with human subjects and food reinforcers with pigeon subjects.     

Choice and reinforcement delay

Previous studies of choice between two delayed reinforcers have indicated that the relative immediacy of the reinforcer is a major determinant of the relative frequency of responding. Parallel studies of choice between two interresponse times have found exceptions to this generality. The present study looked at the choice by pigeons between two delays, one of which was always four times longer than the other, but whose absolute durations were varied across conditions. The results indicated that choice is not uniquely determined by the relative immediacy of reinforcement, but that absolute delays are also involved. Models for concurrent chained schedules appear to be more applicable to the present data than the matching relation; however, these too failed to predict choice for long delays.     

Self-control in pigeons under the Mischel paradigm

Walter Mischel studied self-control in preschool children in the following manner: if the child waited for an interval to end, he or she received the more preferred of two reinforcers; if the child responded to terminate the interval by ringing a bell, the less preferred reinforcer was given. We used an analogous procedure to study self-control in pigeons: if the bird waited for a trial to end, it received the more preferred reinforcer; if the bird terminated the trial by pecking a key, the less preferred reinforcer was given. We explored the effects on self-control of a number of variables analogous to those studied by Mischel and co-workers, e.g., presence versus absence of reinforcers, of alternative responses, and of stimuli during the wait interval; prior experience of the subjects; and test paradigm. The results obtained with pigeons paralleled the results obtained by Mischel with human children.     

Multivariate stochastic processes compatible with “aspect” models of similarity and choice

Various recent works have developed feature or aspect models of similarity and preference. These models are more concerned with the fine detail of the judgment process than were prior models, but nevertheless they have not in general developed an underlying stochastic process compatible with the assumed structure. In this paper, we show that a particular class of multivariate stochastic processes, namely those associated with the Marshall-Olkin multivariate exponential distribution, generates several of these models. In particular, such stochastic processes (appropriately interpreted) yield Tversky's elimination by aspects model, Edgell and Geisler's (normal) additive random aspects model, and Shepard and Arabie's additive cluster model.This work was supported by Natural Science and Engineering Research Council of Canada Grant A8124 to A.A.J. Marley.     

Preference for mixed versus constant delays of reinforcement: Effect of probability of the short, mixed delay

Preference for mixed versus constant delays of reinforcement was studied with a concurrent-chain procedure. Lever pressing by rats in concurrently available variable-interval 60-second initial links occasionally produced mutually exclusive terminal-link reinforcement delays. A constant delay of reinforcement (either 15 seconds or 30 seconds) composed one terminal link and mixed delays (.2 second and twice the value of the constant delay) were arranged in the other terminal link. The proportion of .2-second delays in the mixed-delay terminal link took on values of 0, .1, .25, .5, .75, .9, and 1.0 over experimental conditions. Based on relative rates of responding in the initial links, preference for the mixed delays was a negatively accelerated function of the proportion of short, mixed delays. Three of five rats preferred the mixed delays to the constant delays when the proportion of short, mixed delays was .1 or higher, and all five rats preferred the mixed delays when the proportion of short, mixed delays was .25 or higher. Neither Squires and Fantino's (1971) delay-reduction model of choice nor a model based on the harmonic mean reinforcement delay provided a close estimate of choice proportions over the range of short-delay proportions studied. The delay-reduction model underestimated choice for the mixed delays at low and intermediate proportions of short delays, and the harmonic-mean-delay model overestimated choice for the mixed delays at intermediate and high proportions of short delays.     

Choice as a function of local versus molar reinforcement contingencies.

Rats were trained on a discrete-trial probability learning task. In Experiment 1, the molar reinforcement probabilities for the two response alternatives were equal, and the local contingencies of reinforcement differentially reinforced a win-stay, lose-shift response pattern. The win-stay portion was learned substantially more easily and appeared from the outset of training, suggesting that its occurrence did not depend upon discrimination of the local contingencies but rather only upon simple strengthening effects of individual reinforcements. Control by both types of local contingencies decreased with increases in the intertrial interval, although some control remained with intertrial intervals as long as 30 s. In Experiment 2, the local contingencies always favored win-shift and lose-shift response patterns but were asymmetrical for the two responses, causing the molar reinforcement rates for the two responses to differ. Some learning of the alternation pattern occurred with short intertrial intervals, although win-stay behavior occurred for some subjects. The local reinforcement contingencies were discriminated poorly with longer intertrial intervals. In the absence of control by the local contingencies, choice proportion was determined by the molar contingencies, as indicated by high exponent values for the generalized matching law with long intertrial intervals, and lower values with short intertrial intervals. The results show that when molar contingencies of reinforcement and local contingencies are in opposition, both may have independent roles. Control by molar contingencies cannot generally be explained by local contingencies.     

Subjective probability and delay.

Human subjects indicated their preference between a hypothetical $1,000 reward available with various probabilities or delays and a certain reward of variable amount available immediately. The function relating the amount of the certain-immediate reward subjectively equivalent to the delayed $1,000 reward had the same general shape (hyperbolic) as the function found by Mazur (1987) to describe pigeons' delay discounting. The function relating the certain-immediate amount of money subjectively equivalent to the probabilistic $1,000 reward was also hyperbolic, provided that the stated probability was transformed to odds against winning. In a second experiment, when human subjects chose between a delayed $1,000 reward and a probabilistic $1,000 reward, delay was proportional to the same odds-against transformation of the probability to which it was subjectively equivalent.     

Choice, changeover, and travel: A quantitative model

Six pigeons were trained on concurrent variable-interval schedules in which responding on fixed-interval schedules was required to give access to the alternate schedule. Responding on the concurrent schedules was not allowed, after changing over had commenced, until the changeover schedule had been completed. In Parts 1 to 3 of the experiment, the changeover fixed-interval schedules were equal and were 0 s, 10 s, and 20 s, respectively. In each part, the relative frequency of reinforcement obtained on the concurrent schedules was varied over at least five conditions. In Part 4, the concurrent schedules were equal, and one changeover fixed-interval schedule was twice the other. Under these conditions, the absolute sizes of the changeover schedules were varied. Increasing the changeover requirement from 0 s to 10 s (Parts 1 and 2) resulted in increases in the sensitivity of behavior allocation to reinforcers obtained, but no further increase was obtained when the changeover schedules were increased to 20 s (Part 3). In Part 4, performance was biased towards the concurrent schedule that took less time to enter. These results are consistent with a subtractive punishment model of travel in which the degree of punishment is measured by the number of reinforcers apparently lost from a schedule when the subject changes to that schedule. Absolute times spent on the main keys could be accurately described by a previous model of changeover performance.     

Choice, changeover, and travel

Since foraging in nature can be viewed as instrumental behavior, choice between sources of food, known as “patches,” can be viewed as choice between instrumental response alternatives. Whereas the travel required to change alternatives deters changeover in nature, the changeover delay (COD) usually deters changeover in the laboratory. In this experiment, pigeons were exposed to laboratory choice situations, concurrent variable-interval schedules, that were standard except for the introduction of a travel requirement for changeover. As the travel requirement increased, rate of changeover decreased and preference for a favored alternative strengthened. When the travel requirement was small, the relations between choice and relative reinforcement revealed the usual tendencies toward matching and undermatching. When the travel requirement was large, strong overmatching occurred. These results, together with those from experiments in which changeover was deterred by punishment or a fixed-ratio requirement, deviate from the matching law, even when a correction is made for cost of changeover. If one accepted an argument that the COD is analogous to travel, the results suggest that the norm in choice relations would be overmatching. This overmatching, however, might only be the sign of an underlying strategy approximating optimization.     

Choice and multiple reinforcers

Pigeons chose between equivalent two-component mixed and multiple terminal-link schedules of reinforcement in the concurrent-chains procedure. The pigeons preferred the multiple schedule over the mixed when the components of the compound schedules were differentiated in terms of density of reinforcement, but the pigeons were indifferent when the components were differentiated in terms of number of reinforcers per cycle. Taken together, these results indicate that a local variable, the interval to the first reinforcer, but not a molar variable, the number of reinforcers, was sufficient to differentiate the components and thereby evoke preference.