Molar versus local reinforcement probability as determinants of stimulus value.

During one component of a multiple schedule, pigeons were trained on a discrete-trial concurrent variable-interval variable-interval schedule in which one alternative had a high scheduled rate of reinforcement and the other a low scheduled rate of reinforcement. When the choice proportion between the alternatives matched their respective relative reinforcement frequencies, the obtained probabilities of reinforcement (reinforcer per peck) were approximately equal. In alternate components of the multiple schedule, a single response alternative was presented with an intermediate scheduled rate of reinforcement. During probe trials, each alternative of the concurrent schedule was paired with the constant alternative. The stimulus correlated with the high reinforcement rate was preferred over that with the intermediate rate, whereas the stimulus correlated with the intermediate rate of reinforcement was preferred over that correlated with the low rate of reinforcement. Preference on probe tests was thus determined by the scheduled rate of reinforcement. Other subjects were presented all three alternatives individually, but with a distribution of trial frequency and reinforcement probability similar to that produced by the choice patterns of the original subjects. Here, preferences on probe tests were determined by the obtained probabilities of reinforcement. Comparison of the two sets of results indicates that the availability of a choice alternative, even when not responded to, affects the preference for that alternative. The results imply that models of choice that invoke only obtained probability of reinforcement as the controlling variable (e.g., melioration) are inadequate.     

Residence time in concurrent foraging with fixed times to prey arrival

Five pigeons were trained in a concurrent foraging procedure in which reinforcers were occasionally available after fixed times in two discriminated patches. In Part 1 of the experiment, the fixed times summed to 10 s, and were individually varied between 1 and 9 s over five conditions, with the probability of a reinforcer being delivered at the fixed times always .5. In Part 2, both fixed times were 5 s, and the probabilities of food delivery were varied over conditions, always summing to 1.0. In Parts 3 and 4, one fixed time was kept constant (Part 3, 3 s; Part 4, 7 s) while the other fixed time was varied from 1 s to 15 s. Median residence times in both patches increased with increases in the food-arrival times in either patch, but increased considerably more strongly in the patch in which the arrival time was increased. However, when arrival times were very different in the two patches, residence time in the longer arrival-time patch often decreased. Patch residence also increased with increasing probability of reinforcement, but again tended to fall when one probability was much larger than the other. A detailed analysis of residence times showed that these comprised two distributions, one around a shorter mode that remained constant with changes in arrival times, and one around a longer mode that monotonically increased with increasing arrival time. The frequency of shorter residence times appeared to be controlled by the probability of, and arrival time of, reinforcers in the alternative patch. The frequency of longer residence times was controlled directly by the arrival time of reinforcers in a patch, but not by the probability of reinforcers in a patch. The environmental variables that control both staying in a patch and exiting from a patch need to be understood in the study both of timing processes and of foraging.     

Melioration revisited: a systematic replication of Vaughan (1981)

Organisms that behave so as to forfeit a relatively higher overall rate of reinforcement in favor of a relatively lower rate are said to engage in suboptimal choice. Suboptimal choice has been linked with maladaptive behavior in humans. Melioration theory offers one explanatory framework for suboptimal choice. Melioration theory suggests behavior is controlled by differences in local reinforcer rates between alternatives. Vaughan (1981) arranged two experimental conditions in which maximizing the overall rate of reinforcement required behavior that was compatible, or incompatible, with melioration. Vaughan found pigeons allocated more time to a locally richer alternative even when doing so resulted in suboptimal choice. However, Vaughan did not show whether these effects could systematically reverse and did not provide within‐session data to show that choice across short time spans remains under the control of differences in local reinforcer rates. The present study used pigeons to replicate and extend Vaughan's findings. We investigated shifts in overall‐ and within‐session choice across repeated conditions, according to arranged local contingencies. Behavior systematically followed changes in local contingencies for most pigeons. Within‐session data suggests that, providing differences in local reinforcer rates are discriminated, pigeons will allocate more time to a locally richer alternative, even if this leads to suboptimal choice. These findings facilitate the more confident use of similar procedures that investigate how melioration contributes to suboptimal choice.     

Choice behavior in transition: development of preference for the higher probability of reinforcement.

Ten acquisition curves were obtained from each of 4 pigeons in a two-choice discrete-trial procedure. In each of these 10 conditions, the two response keys initially had equal probabilities of reinforcement, and subjects' choice responses were about equally divided between the two keys. Then the reinforcement probabilities were changed so that one key had a higher probability of reinforcement (the left key in half of the conditions and the right key in the other half), and in nearly every case the subjects developed a preference for this key. The rate of acquisition of preference for this key was faster when the ratio of the two reinforcement probabilities was higher. For instance, acquisition of preference was faster in conditions with reinforcement probabilities of .12 and .02 than in conditions with reinforcement probabilities of .40 and .30, even though the pairs of probabilities differed by .10 in both cases. These results were used to evaluate the predictions of some theories of transitional behavior in choice situations. A trial-by-trial analysis of individual responses and reinforcers suggested that reinforcement had both short-term and long-term effects on choice. The short-term effect was an increased probability of returning to the same key on the one or two trials following a reinforcer. The long-term effect was a gradual increase in the proportion of responses on the key with the higher probability of reinforcement, an increase that usually continued for several hundred trials.     

Preference between variable-ratio and fixed-ratio schedules: local and extended relations.

Although it has repeatedly been demonstrated that pigeons, as well as other species, will often choose a variable schedule of reinforcement over an equivalent (or even richer) fixed schedule, the exact nature of that controlling relation has yet to be fully assessed. In this study pigeons were given repeated choices between concurrently available fixed-ratio and variable-ratio schedules. The fixed-ratio requirement (30 responses) was constant throughout the experiment, whereas the distribution of individual ratios making up the variable-ratio schedule changed across phases: The smallest and largest of these components were varied gradually, with the mean variable-ratio requirement constant at 60 responses. The birds' choices of the variable-ratio schedule tracked the size of the smallest variable-ratio component. A minimum variable-ratio component at or near 1 produced strong preference for the variable-ratio schedule, whereas increases in the minimum variable-ratio component resulted in reduced preference for the variable-ratio schedule. The birds' behavior was qualitatively consistent with Mazur's (1984) hyperbolic model of delayed reinforcement and could be described as approximate maximizing with respect to reinforcement value.     

Determinants of contrast in the signal-key procedure: Evidence against additivity theory

Two experiments are reported that challenge the interpretation of previous results with the signal-key procedure, in which the discriminative stimuli are located on a response key different from the key associated with the operant response requirement. Experiment 1 replicated the procedure of Keller (1974), and found that contrast effects on the operant key occurred reliably for only one of four subjects. High rates to the signal key initially occurred for only one subject, but modifications of the procedure produced substantial rates to the signal key for all subjects. In all cases, however, signal-key behavior was greatly reduced by the addition of a changeover delay which prevented reinforcement within 2 seconds of the last peck to the signal key, suggesting that signal-key pecking was maintained primarily by adventitious reinforcement. Experiment 2 modified the signal-key procedure by using three response keys, so that the discriminative stimuli on the signal key controlled different responses during all phases of training. With this modification, reliable contrast effects on the operant key occurred for all subjects, suggesting that the failure to find contrast in previous studies has been due to the confounding of changes in the discrimination requirements with changes in relative rate of reinforcement. The results challenge the additivity theory of contrast, and suggest that “elicited” behavior plays a minor role, if any, in the determination of contrast effects in multiple schedules.     

Concurrent-schedule performance in transition: changeover delays and signaled reinforcer ratios

Six pigeons were trained in experimental sessions that arranged six or seven components with various concurrent-schedule reinforcer ratios associated with each. The order of the components was determined randomly without replacement. Components lasted until the pigeons had received 10 reinforcers, and were separated by 10-s blackout periods. The component reinforcer ratios arranged in most conditions were 27:1, 9:1, 3:1, 1:1, 1:3, 1:9 and 1:27; in others, there were only six components, three of 27:1 and three of 1:27. In some conditions, each reinforcement ratio was signaled by a different red-yellow flash frequency, with the frequency perfectly correlated with the reinforcer ratio. Additionally, a changeover delay was arranged in some conditions, and no changeover delay in others. When component reinforcer ratios were signaled, sensitivity to reinforcement values increased from around 0.40 before the first reinforcer in a component to around 0.80 before the 10th reinforcer. When reinforcer ratios were not signaled, sensitivities typically increased from zero to around 0.40. Sensitivity to reinforcement was around 0.20 lower in no-changeover-delay conditions than in changeover-delay conditions, but increased in the former after exposure to changeover delays. Local analyses showed that preference was extreme towards the reinforced alternative for the first 25 s after reinforcement in changeover-delay conditions regardless of whether components were signaled or not. In no-changeover-delay conditions, preference following reinforcers was either absent, or, following exposure to changeover delays, small. Reinforcers have both local and long-term effects on preference. The former, but not the latter, is strongly affected by the presence of a changeover delay. Stimulus control may be more closely associated with longer-term, more molar, reinforcer effects.     

Local preference in concurrent schedules: the effects of reinforcer sequences

We investigated the effects that sequences of reinforcers obtained from the same response key have on local preference in concurrent variable-interval schedules with pigeons as subjects. With an overall reinforcer rate of one every 27 s, on average, reinforcers were scheduled dependently, and the probability that a reinforcer would be arranged on the same alternative as the previous reinforcer was manipulated. Throughout the experiment, the overall reinforcer ratio was 1:1, but across conditions we varied the average lengths of same-key reinforcer sequences by varying this conditional probability from 0 to 1. Thus, in some conditions, reinforcer locations changed frequently, whereas in others there tended to be very long sequences of same-key reinforcers. Although there was a general tendency to stay at the just-reinforced alternative, this tendency was considerably decreased in conditions where same-key reinforcer sequences were short. Some effects of reinforcers are at least partly to be accounted for by their signaling subsequent reinforcer locations.     

Choice in a variable environment: every reinforcer counts

Six pigeons were trained in sessions composed of seven components, each arranged with a different concurrent-schedule reinforcer ratio. These components occurred in an irregular order with equal frequency, separated by 10-s blackouts. No signals differentiated the different reinforcer ratios. Conditions lasted 50 sessions, and data were collected from the last 35 sessions. In Part 1, the arranged overall reinforcer rate was 2.22 reinforcers per minute. Over conditions, number of reinforcers per component was varied from 4 to 12. In Part 2, the overall reinforcer rate was six per minute, with both 4 and 12 reinforcers per component. Within components, log response-allocation ratios adjusted rapidly as more reinforcers were delivered in the component, and the slope of the choice relation (sensitivity) leveled off at moderately high levels after only about eight reinforcers. When the carryover from previous components was taken into account, the number of reinforcers in the components appeared to have no systematic effect on the speed at which behavior changed after a component started. Consequently, sensitivity values at each reinforcer delivery were superimposable. However, adjustment to changing reinforcer ratios was faster, and reached greater sensitivity values, when overall reinforcer rate was higher. Within a component, each successive reinforcer from the same alternative ("confirming") had a smaller effect than the one before, but single reinforcers from the other alternative ("disconfirming") always had a large effect. Choice in the prior component carried over into the next component, and its effects could be discerned even after five or six reinforcement and nonreinforcement is suggested.     

Inverse relations between preference and contrast

Pigeons were trained on a multiple schedule in which two target components with identical reinforcement schedules were followed by either the same-valued schedule or by extinction. Response rate increased in both target components but was higher in the target component followed by extinction, replicating previous findings of positive anticipatory contrast. A similar design was used to study negative contrast, in that the two target components were followed either by the same-valued schedule or by a higher valued schedule. Negative contrast occurred equally, on average, in both target components, thus failing to demonstrate negative contrast that is specifically anticipatory in nature. When the stimuli correlated with the two target components were paired in choice tests, the pattern of preference was in the opposite direction. For the positive contrast procedure, no significant preference between the two target stimuli was evident. But for the negative contrast procedure, preference favored the stimulus followed by the higher valued schedule. The results demonstrate a functional dissociation between positive and negative contrast in relation to stimulus value. More generally, the results demonstrate an inverse relation between response rate and preference and challenge existing accounts of contrast in terms of the concept of relative value.     

SOME EFFECTS OF PROCEDURAL VARIATIONS ON CHOICE RESPONDING IN CONCURRENT CHAINS

The present research used pigeons in a three‐key operant chamber and varied procedural features pertaining to both initial and terminal links of concurrent chains. The initial links randomly alternated on the side keys during a session, while the terminal links always appeared on the center key. Both equal and unequal initial‐link schedules were employed, with either differential or nondifferential terminal‐link stimuli across conditions. The research was designed to neutralize initial‐ and terminal‐link spatial cues in order to gain a clearer understanding of the roles of conditioned reinforcement and delayed primary reinforcement in choice. With both equal and unequal initial links and with differential terminal‐link stimuli, all pigeons reliably preferred the chain with the shorter terminal link. However, with equal initial links and nondifferential stimuli, all pigeons were indifferent. With unequal initial links and nondifferential stimuli, some pigeons were also indifferent, while others actually reversed and preferred the chain with the shorter initial link, even though it was followed by the longer terminal link. The decrease if not reversal of the previous preferences implies that preferences in concurrent chains are a function of the conditioned reinforcement afforded by terminal‐link stimuli, rather than delayed primary reinforcement.     

Biasing the pacemaker in the behavioral theory of timing

In the behavioral theory of timing, pacemaker rate is determined by overall rate of reinforcement. A two-alternative free-operant psychophysical procedure was employed to investigate whether pacemaker period was also sensitive to the differential rate of reinforcement. Responding on a left key during the first 25 s and on a right key during the second 25 s of a 50-s trial was reinforced at variable intervals, and variable-interval schedule values during the two halves of the trials were varied systematically. Responding on the right key during the first 25 s and on the left key during the second 25 s was not reinforced. Estimates of pacemaker period were derived from fits of a function predicted by the behavioral theory of timing to right-key response proportions in consecutive 5-s bins of the 50-s trial. Estimates of pacemaker period were shortest when the differential reinforcer rate most strongly favored right-key responses, and were longest when the differential reinforcer rate most strongly favored left-key responses. The results were consistent with the conclusion that pacemaker rate is influenced by relative reinforcer rate.     

Choice in situations of time-based diminishing returns: immediate versus delayed consequences of action.

Pigeons chose between two schedules of food presentation, a fixed-interval schedule and a progressive-interval schedule that began at 0 s and increased by 20 s with each food delivery provided by that schedule. Choosing one schedule disabled the alternate schedule and stimuli until the requirements of the chosen schedule were satisfied, at which point both schedules were again made available. Fixed-interval duration remained constant within individual sessions but varied across conditions. Under reset conditions, completing the fixed-interval schedule not only produced food but also reset the progressive interval to its minimum. Blocks of sessions under the reset procedure were interspersed with sessions under a no-reset procedure, in which the progressive schedule value increased independent of fixed-interval choices. Median points of switching from the progressive to the fixed schedule varied systematically with fixed-interval value, and were consistently lower during reset than during no-reset conditions. Under the latter, each subject's choices of the progressive-interval schedule persisted beyond the point at which its requirements equaled those of the fixed-interval schedule at all but the highest fixed-interval value. Under the reset procedure, switching occurred at or prior to that equality point. These results qualitatively confirm molar analyses of schedule preference and some versions of optimality theory, but they are more adequately characterized by a model of schedule preference based on the cumulated values of multiple reinforcers, weighted in inverse proportion to the delay between the choice and each successive reinforcer.     

The behavioral theory of timing: Reinforcer rate determines pacemaker rate

In the behavioral theory of timing, pulses from a hypothetical Poisson pacemaker produce transitions between states that are correlated with adjunctive behavior. The adjunctive behavior serves as a discriminative stimulus for temporal discriminations. The present experiments tested the assumption that the average interpulse time of the pacemaker is proportional to interreinforcer interval. Responses on a left key were reinforced at variable intervals for the first 25 s since the beginning of a 50-s trial, and right-key responses were reinforced at variable intervals during the second 25 s. Psychometric functions relating proportion of right-key responses to time since trial onset, in 5-s intervals across the 50-s trial, were sigmoidal in form. Average interpulse times derived by fitting quantitative predictions from the behavioral theory of timing to obtained psychometric functions decreased when the interreinforcer interval was decreased and increased when the interreinforcer interval was increased, as predicted by the theory. In a second experiment, average interpulse times estimated from trials without reinforcement followed global changes in interreinforcer interval, as predicted by the theory. Changes in temporal discrimination as a function of interreinforcer interval were therefore not influenced by the discrimination of reinforcer occurrence. The present data support the assumption of the behavioral theory of timing that interpulse time is determined by interreinforcer interval.     

Choice with delayed and probabilistic reinforcers: effects of variability, time between trials, and conditioned reinforcers.

In a discrete-trials procedure with pigeons, a response on a green key led to a 4-s delay (during which green houselights were lit) and then a reinforcer might or might not be delivered. A response on a red key led to a delay of adjustable duration (during which red houselights were lit) and then a certain reinforcer. The delay was adjusted so as to estimate an indifference point--a duration for which the two alternatives were equally preferred. Once the green key was chosen, a subject had to continue to respond on the green key until a reinforcer was delivered. Each response on the green key, plus the 4-s delay that followed every response, was called one "link" of the green-key schedule. Subjects showed much greater preference for the green key when the number of links before reinforcement was variable (averaging four) than when it was fixed (always exactly four). These findings are consistent with the view that probabilistic reinforcers are analogous to reinforcers delivered after variable delays. When successive links were separated by 4-s or 8-s "interlink intervals" with white houselights, preference for the probabilistic alternative decreased somewhat for 2 subjects but was unaffected for the other 2 subjects. When the interlink intervals had the same green houselights that were present during the 4-s delays, preference for the green key decreased substantially for all subjects. These results provided mixed support for the view that preference for a probabilistic reinforcer is inversely related to the duration of conditioned reinforcers that precede the delivery of food.     

Detecting a nonevent: Delayed presence-versus-absence discrimination in pigeons

Eight pigeons were trained on a delayed presence-versus-absence discrimination paradigm in which a sample stimulus was presented on some trials but not on others. If a sample was presented, then a response to one choice key produced food. If no sample was presented, a response to the other choice key produced food. The basic finding was that performance remained constant and well above 50% correct on no-sample trials as the retention interval increased, whereas performance dropped precipitously (to below 50% correct) on sample trials. In the second phase of the experiment, all of the trials were no-sample trials, and reinforcers were delivered probabilistically for one group of pigeons and according to time-based schedules for the other group. The exact reinforcement probabilities used in Phase 2 were those calculated to be in effect on no-sample trials in Phase 1 (according to a discrete-state model of performance). Subjects did not show exclusive preference for the richer alternative on no-sample trials in the first phase, but those in the probabilistic group developed near-exclusive preference for the richer alternative during the second phase. These data are inconsistent with the predictions of the discrete-state model, but are easily accommodated by an account based on signal detection theory, which also can be applied effectively to discrimination of event duration and the “subjective shortening” effect.     

Pigeons' choices in situations of diminishing returns: fixed- versus progressive-ratio schedules.

In two different discrete-trial procedures, pigeons were faced with choices between fixed-ratio and progressive-ratio schedules. The latter schedules entail diminishing returns, a feature analogous to foraging situations in the wild. In the first condition (no reset), subjects chose between a progressive-ratio schedule that increased in increments of 20 throughout a session and a fixed-ratio schedule that was constant across blocks of sessions. The size of the fixed ratio was varied parametrically through an ascending and then a descending series. In the reset condition, the same fixed-ratio values were used, but each selection (and completion) of the fixed ratio reset the progressive-ratio schedule back to its minimal value. In the no-reset procedure, the pigeons tended to cease selecting the progressive ratio when it equaled or slightly exceeded the fixed-ratio value, whereas in reset, they chose the fixed ratio well in advance of that equality point. These results indicate sensitivity to molar as well as to molecular reinforcement rates, and those molar relationships are similar to predictions based on the marginal value theorem of optimal foraging theory (e.g., Charnov, 1976). However, although previous results with monkeys (Hineline & Sodetz, 1987) appeared to minimize responses per reinforcement, the present results corresponded more closely to predictions based on sums-of-reciprocals of distance from point of choice to each of the next four reinforcers. Results obtained by Hodos and Trumbule (1967) with chimpanzees in a similar procedure were intermediate between these two relationships. Variability of choices, as well as median choice points, differed between the reset and no-reset conditions.(ABSTRACT TRUNCATED AT 250 WORDS)     

Influence of temporal context on value in the multiple-chains and successive-encounters procedures

This set of studies explored the influence of temporal context across multiple-chain and multiple-successive-encounters procedures. Following training with different temporal contexts, the value of stimuli sharing similar reinforcement schedules was assessed by presenting these stimuli in concurrent probes. The results for the multiple-chain schedule indicate that temporal context does impact the value of a conditioned reinforcer consistent with delay-reduction theory, such that a stimulus signaling a greater reduction in delay until reinforcement has greater value. Further, nonreinforced stimuli that are concurrently presented with the preferred terminal link also have greater value, consistent with value transfer. The effects of context on value for conditions with the multiple-successive-encounters procedure, however, appear to depend on whether the search schedule or alternate handling schedule was manipulated, as well as on whether the tested stimuli were the rich or lean schedules in their components. Overall, the results help delineate the conditions under which temporal context affects conditioned-reinforcement value (acting as a learning variable) and the conditions under which it does not (acting as a performance variable), an issue of relevance to theories of choice.     

Assessing preference for reinforcers using demand curves, work-rate functions, and expansion paths

Much research has focused on the effects of environmental variability on foraging decisions. However, the general pattern of preference for variability in delay to reward and aversion to variability in amount of reward remains unexplained a either a mechanistic or a functional level. Starlings' preferences between a fixed and a variable option were studied in two treatments, A and D. The fixed option was the same in both treatments (20-s fixed-interval delay, five units food). In Treatment A the variable option gave two equiprobable amounts of food (20-s delay, three or seven units) and in D it gave two equiprobable delays to food (2.5-s or 60.5-s delays, five units). In both treatments the programmed ratio [amount/(intertrial interval+latency+delay)] in the fixed option equaled the arithmetic mean of the two possible ratios in the variable option (ITI = 40 s, latency = 1 s). The variable option was strongly preferred in Treatment D and was weakly avoided in Treatment A. These results are discussed in the light of two theoretical models, a form of constrained rate maximization and a version of scalar expectancy theory. The latter accommodates more of the data and is based on independently verifiable assumptions, including Weber's law.