DYNAMICS OF CHOICE: RELATIVE RATE AND AMOUNT AFFECT LOCAL PREFERENCE AT THREE DIFFERENT TIME SCALES

To examine extended control over local choice, the present study investigated preference in transition as food‐rate ratio provided by two levers changed across seven components within daily sessions, and food‐amount ratio changed across phases. Phase 1 arranged a food‐amount ratio of 4:1 (i.e., the left lever delivered four pellets and the right lever one pellet); Phase 2 reversed the food‐amount ratio to 1:4, and in Phase 3 the food‐amount ratio was 3:2. At a relatively extended time scale, preference was described well by a linear relation between log response ratio and log rate ratio (the generalized matching law). A small amount of carryover occurred from one rate ratio to the next but disappeared after four food deliveries. Estimates of sensitivity to food‐amount ratio were around 1.0 and were independent of rate ratio. Analysis across food deliveries within rate‐ratio components showed that the effect of a small amount was diminished by the presence of a large amount—that is, when a larger amount was present in the situation (three or four pellets), the value of a small amount (one or two pellets) became paltry. More local analysis of visits to the levers between food deliveries showed that postfood visits following a large amount were disproportionately longer than following a small amount. Continuing food deliveries from the same source tended to make visits less dependent on relative amount, but a discontinuation (i.e., food from the other lever) reinstated dependence on relative amount. Analysis at a still smaller time scale revealed preference pulses following food deliveries that confirmed the tendency toward dependence on absolute amount with continuing deliveries, and toward dependence on relative amount following discontinuations. A mathematical model based on a linear‐operator equation accounts for many of the results. The larger and longer preference following a switch to a larger amount is consistent with the idea that local preference depends on relatively extended variables even on short time scales.     

STIMULUS EFFECTS ON LOCAL PREFERENCE: STIMULUS—RESPONSE CONTINGENCIES,STIMULUS—FOOD PAIRING,AND STIMULUS—FOOD CORRELATION

Four pigeons were trained in a procedure in which concurrent‐schedule food ratios changed unpredictably across seven unsignaled components after 10 food deliveries. Additional green‐key stimulus presentations also occurred on the two alternatives, sometimes in the same ratio as the component food ratio, and sometimes in the inverse ratio. In eight experimental conditions, we varied the contingencies surrounding these additional stimuli: In two conditions, stimulus onset and offset were noncontingent; in another two, stimulus onset was noncontingent, and offset was response contingent. In four conditions, both stimulus onset and offset were contingent, and in two of these conditions the stimulus was simultaneously paired with food delivery. Sensitivity to component food ratios was significantly higher when stimulus onset was response contingent compared to when it was noncontingent. Choice changes following food delivery were similar in all eight conditions. Choice changes following stimuli were smaller than those following food, and directionally were completely determined by the food‐ratio:stimulus‐ratio correlation, not by the stimulus contingency nor by whether the stimulus was paired with food or not. These results support the idea that conditional reinforcers may best be viewed as signals for next‐food location rather than as stimuli that have acquired hedonic value, at least when the signals are differential with respect to future conditions.     

Control by past and present stimuli depends on the discriminated reinforcer differential

The extent to which a stimulus exerts control over behavior depends largely on its informativeness. However, when reinforcers have discriminative properties, they often exert less control over behavior than do other less reliable stimuli such as elapsed time. We investigated why less reliable cues in the present often overshadow stimulus control by more reliable cues presented in the recent past, by manipulating the reliability and duration of stimulus presentations. Five pigeons worked on a modified concurrent schedule in which the location of the response that produced the last reinforcer was a discriminative stimulus for the likely time and location of the next reinforcer. In some conditions, either the location of the previous reinforcer, or the location of the next reinforcer, was signaled by a red key light. This stimulus was either Brief, occurring for 10 s starting a fixed time after the most recent reinforcer, or Extended, being present at all times between food deliveries. Brief and Extended stimuli that signaled the same information had a similar effect on choice when they were present, but control by Brief stimuli weakened as time since stimulus offset elapsed. Control was divided among stimuli in the present and recent past according to the apparent reliability of the information signaled about the next reinforcer. More reliable stimuli in the present degraded, but did not erase, control by less reliable stimuli presented in the recent past. Thus, we conclude that less reliable stimuli in the present control behavior to a greater degree than do more reliable stimuli in the recent past because these more reliable stimuli are forgotten, and hence their relation to the likely availability of food cannot be discriminated.     

THE BASIS OF SUPERSTITIOUS BEHAVIOR: CHANCE CONTINGENCY,STIMULUS SUBSTITUTION,OR APPETITIVE BEHAVIOR?

This research examined three explanations for the "superstitious" behavior of pigeons under frequent fixed-time delivery of food: accidental response-reward contingency, stimulus substitution, and elicited species-typical appetitive behavior. The behavior observed in these studies consisted of occasional postfood locomotion away from the food hopper, and a predominant pattern of activity directed toward the hopper wall (wall-directed behavior), including approaching, stepping side to side, scratching with the feet, bumping with the breast, pendulum movements of the extended neck, and head bobbing, though not pecking. The consistency of these behavior patterns argued against explanation by accidental response contingencies, and the complexity of behavior was incompatible with the classic stimulus-substitution account. These studies also showed that: (1) response contingencies and prior stimulus experience can modify wall-directed behavior, but within definable limits; (2) pecking sometimes can be obtained in birds of specific strains, and by providing extended training; (3) placing the hopper in the floor at the center of a large chamber replaces wall-directed behavior with circling in a manner that resembles ground foraging for food. We conclude that superstitious behavior under periodic delivery of food probably develops from components of species-typical patterns of appetitive behavior related to feeding. These patterns are elicited by a combination of frequent food presentations and the supporting stimuli present in the environment.     

Local effects of delayed food

Five pigeons were trained on a procedure in which seven concurrent variable-interval schedules arranged seven different food-rate ratios in random sequence in each session. Each of these components lasted for 10 response-produced food deliveries, and components were separated by 10-s blackouts. We varied delays to food (signaled by blackout) between the two response alternatives in an experiment with three phases: In Phase 1, the delay on one alternative was 0 s, and the other was varied between 0 and 8 s; in Phase 2, both delays were equal and were varied from 0 to 4 s; in Phase 3, the two delays summed to 8 s, and each was varied from 1 to 7 s. The results showed that increasing delay affected local choice, measured by a pulse in preference, in the same way as decreasing magnitude, but we found also that increasing the delay at the other alternative increased local preference. This result casts doubt on the traditional view that a reinforcer strengthens a response depending only on the reinforcer's value discounted by any response-reinforcer delay. The results suggest that food guides, rather than strengthens, behavior.     

DO CONDITIONAL REINFORCERS COUNT?

Six pigeons were trained on a procedure in which seven components arranged different food-delivery ratios on concurrent variable-interval schedules each session. The components were unsignaled, lasted for 10 food deliveries, and occurred in random order with a 60-s blackout between components. The schedules were arranged using a switching-key procedure in which two responses on a center key changed the schedules and associated stimuli on two side keys. In Experiment 1, over five conditions, an increasing proportion of food deliveries accompanied by a magazine light was replaced with the presentation of the magazine light only. Local analyses of preference showed preference pulses toward the alternative that had just produced either a food-plus-magazine-light or magazine-light-only presentation, but pulses after food deliveries were always greater than those after magazine lights. Increasing proportions of magazine lights did not change the size of preference pulses after food or magazine-light presentations. Experiment 2 investigated the effects of correlations between food ratios and magazine-light ratios: In Condition 6, magazine-light ratios in components were inversely correlated (-1.0) with food ratios, and in Condition 7, magazine-light ratios were uncorrelated with food ratios. In Conditions 8 and 9, pecks also produced occasional 2.5-s flashes of a green keylight. In Condition 8, food and magazine-light ratios were correlated 1.0 whereas food and green-key ratios were correlated -1.0. In Condition 9, food and green-key ratios were correlated 1.0 whereas food and magazine-light ratios were correlated -1.0. Preference pulses toward alternatives after magazine lights and green keys depended on the correlation between these event ratios and the food ratios: If the ratios were correlated +1.0, positive preference pulses resulted; if the correlation was -1.0, preference pulses were negative. These results suggest that the Law of Effect has more to do with events signaling consequences than with strengthening responses.     

Key pecking under response-independent food presentation after long simple and compound stimuli

Sixteen pigeons were trained to peck a key using a response-independent (auto-shaping) procedure of food presentation. The 4-sec grain presentations were independent of responding but a keylight stimulus preceded each, with a 4-min interval between the grain presentation and the next stimulus. Subjects were divided into four groups, with two durations of the keylight (30 or 120 sec) and either one or four successive colors on the response key preceding food delivery. In Phase 2, the birds were continued with the same keylight duration but were presented the alternative number of key colors. All pigeons pecked the key during the stimulus. Birds in the two groups with the 30-sec stimulus duration began to respond significantly sooner than birds with the 120-sec duration. There were no significant differences in rate of pecking between groups by the last five days of Phase 1. In Phase 1, the pigeons exposed to the four stimulus components showed an increase in rate of pecking over the four components as grain presentation approached. The pigeons with one stimulus component did not exhibit this regularity. Analogous conditions in Phase 2 had similar results except for one group. The implications of the occurrence of key pecking due to response-independent food delivery for multiple and chained schedules were pointed out.     

Net Amount of Food Affects Autoshaped Response Rate, Response Latency, and Gape Amplitude in Pigeons

The contribution of net amount of food to conditioned response strength and topography was assessed in four pigeons under autoshaping contingencies. In each session, under one baseline phase and three replication phases, three trial types were presented sequentially: One conditioned stimulus (keylight) signaled one small pellet, another signaled one large pellet, and a third signaled seven small pellets which were weight-matched to one large pellet. Five dependent variables were response rates and latencies based on the occurrence of both keyswitch closures and gapes (beak openings) and gape amplitudes. In result, net amount of food, not pellet diameter or number, affected all dependent variables. Notably, gape amplitudes elicited by the seven-small pellet keylights were larger than the gape amplutides elicited by the one-small pellet keylights even though the gape amplitudes elicited by both unconditioned stimuli (one or seven pellets) were equally small. This mismatch between conditioned and unconditioned responses is incompatible with stimulus substitution accounts but is compatible with an associative strength account. Furthermore, the changes in the dependent variables were most likely determined by Pavlovian and not by inadvertent operant contingencies. The findings demonstrate that an analysis of classical conditioning benefits from the inclusion of topographical measures.     

Discriminated timeout avoidance in pigeons: the roles of added stimuli

Two experiments examined pigeons' postponement of a signaled extinction period, or timeout (TO), from an ongoing schedule of response-dependent food delivery. A concurrent-operant procedure was used in which responses on one (food) key produced food according to a variable-interval schedule and responses on a second (postponement) key delayed the next scheduled TO according to a response-TO (R-TO) interval. A series of response-independent stimulus changes on the food key temporally partitioned the R-TO into three equal segments (S1, S2, and S3). Postponement responses, in addition to postponing TO, also reinstated S1, the stimulus correlated with the greatest temporal distance from TO. In Experiment 1, the R-TO interval was manipulated systematically across blocks of sessions (conditions) at a given ratio of R-TO:TO duration. This R-TO:TO ratio was manipulated across blocks of conditions (phases). Postponement response rates varied inversely with R-TO interval in each phase. Changes in the R-TO:TO ratio did not produce consistent differences except at the 1:10 ratio for some pigeons, where it disrupted postponement responding in some conditions. Most of the postponement responses occurred in the presence of S2 and S3, the stimuli most proximal to TO, whereas most of the food-key responses occurred in S1. In Experiment 2, the R-TO contingencies were systematically manipulated in the presence of the time-correlated stimuli. In one set of conditions, the R-TO contingencies were made either ineffective or less effective in the presence of one or more stimuli. Postponement responses typically shifted to stimuli in the presence of which responses were relatively more effective. Postponement responses decreased markedly when the added stimuli were removed, and then recovered when the stimuli were reinstated. Results from both experiments indicate that the added stimuli in a discriminated TO-avoidance procedure serve predominately discriminative functions, delineating periods during which behavior is maximally effective. The results parallel those obtained in shock-avoidance procedures, providing further evidence that TO functions as an aversive stimulus.     

Limits to preference and the sensitivity of choice to rate and amount of food

Studies of choice holding food‐amount ratio constant while varying food‐rate ratio within sessions showed that local changes in preference depend on relative amount of food. The present study investigated whether sensitivity of choice to food‐rate ratio and sensitivity to food‐amount ratio are independent of one another when food‐rate ratios are varied across sessions and food‐amount ratios are varied within sessions. Food deliveries for rats’ presses on the left and right levers were scheduled according to three different food‐rate ratios of 1:1, 9:1, and 1:9; each food‐rate ratio lasted for 106 sessions and was arranged independently of seven food‐amount ratios (7:1, 6:2, 5:3, 4:4, 3:5, 2:6, and 1:7 food pellets) occurring within sessions in random sequence. Each amount ratio lasted for 10 food deliveries and was separated from another by a 60‐s blackout. Sensitivity to rate ratio was high (1.0) across food deliveries. Sensitivity to amount ratio was low when food rates were equal across alternatives, but was high when rate ratio and amount ratio opposed one another. When rate ratio and amount ratio went in the same direction, choice ratio reached an elevenfold limit which reduced sensitivity to approximately zero. We conclude that three factors affect sensitivity to amount: (1) the limit to preference, (2) the equal effect on preference of amounts greater than four pellets, and (3) the absence of differential effects of switches in amount in the equal‐rates (1:1) condition. Taken together, these findings indicate that rate and amount only sometimes combine independently as additive variables to determine preference when amount ratios vary frequently within sessions.     

Behavioral effects of delayed timeouts from reinforcement

Timeouts are sometimes used in applied settings to reduce target responses, and in some circumstances delays are unavoidably imposed between the onset of a timeout and the offset of the response that produces it. The present study examined the effects of signaled and unsignaled timeouts in rats exposed to concurrent fixed‐ratio 1 fixed‐ratio 1 schedules of food delivery, where each response on one lever, the location of which changed across conditions, produced both food and a delayed 10‐s timeout. Delays of 0 to 38 s were examined. Delayed timeouts often, but not always, substantially reduced the number of responses emitted on the lever that produced timeouts relative to the number emitted on the lever that did not produce timeouts. In general, greater sensitivity was observed to delayed timeouts when they were signaled. These results demonstrate that delayed timeouts, like other delayed consequences, can affect behavior, albeit less strongly than immediate consequences.     

Matching and conditioned reinforcement rate

Attempts to examine the effects of variations in relative conditioned reinforcement rate on choice have been confounded by changes in rates of primary reinforcement or changes in the value of the conditioned reinforcer. To avoid these problems, this experiment used concurrent observing responses to examine sensitivity of choice to relative conditioned reinforcement rate. In the absence of observing responses, unsignaled periods of food delivery on a variable-interval 90-s schedule alternated with extinction on a center key (i.e., a mixed schedule was in effect). Two concurrently available observing responses produced 15-s access to a stimulus differentially associated with the schedule of food delivery (S+). The relative rate of S+ deliveries arranged by independent variable-interval schedules for the two observing responses varied across conditions. The relation between the ratio of observing responses and the ratio of S+ deliveries was well described by the generalized matching law, despite the absence of changes in the rate of food delivery. In addition, the value of the S+ deliveries likely remained constant across conditions because the ratio of S+ to mixed schedule food deliveries remained constant. Assuming that S+ deliveries serve as conditioned reinforcers, these findings are consistent with the functional similarity between primary and conditioned reinforcers suggested by general choice theories based on the concatenated matching law (e.g., contextual choice and hyperbolic value-added models). These findings are inconsistent with delay reduction theory, which has no terms for the effects of rate of conditioned reinforcement in the absence of changes in rate of primary reinforcement.     

Stimulus control depends on the subjective value of the outcome

Stimuli that provide information about likely future reinforcers tend to shift behavior, provided a reliable relation between the stimulus and the reinforcer can be discriminated. Stimuli that are apparently more reliable exert greater control over behavior. We asked how the subjective value (measured in terms of preference) of reinforcers associated with stimuli influences stimulus control. Five pigeons worked on a concurrent chains procedure in which half of all trials ended in a smaller reinforcer sooner, and the other half in a larger reinforcer later. In Signaled trials, the color and flash duration on the keys in the initial link signaled the outcome of the trial. In Conflicting probe trials, the color and the flash duration signaled conflicting information about the outcome of the trial. Choice in Signaled trials shifted toward the signaled outcome, but was never exclusive. In Conflicting probe trials, control was divided idiosyncratically between the 2 stimulus dimensions, but still favored the outcome with the higher subjective value. Thus, stimulus control depends not only on the perceived reliability of stimuli, but also on the subjective value of the outcome.     

Response persistence under variable-time schedules following immediate and unsignalled delayed reinforcement

Key pecking of three pigeons was maintained in separate components of a multiple schedule by either immediate reinforcement (i.e., tandem variable-time fixed-interval schedule) or unsignalled delayed reinforcement (i.e., tandem variable-interval fixed-time schedule). The relative rate of food delivery was equal across components, and this absolute rate differed across conditions. Immediate reinforcement always generated higher response rates than did unsignalled delayed reinforcement. Then, variable-time schedules of food delivery replaced the contingencies just described such that food was delivered at the same rate but independently of responding. In most cases, response rates decreased to near-zero levels. In addition, response persistence was not systematically different between multiple-schedule components across pigeons. The implications of the results for the concepts of response strength and the response-reinforcer relation are noted.     

SCHEDULE-INDUCED DRINKING: ELICITATION,ANTICIPATION, OR BEHAVIORAL INTERACTION?

We carried out five experiments with rats on fixed-time schedules in order to define the relation between drinking and individual food-pellet presentations. In Experiment 1, unsignaled extra food occurred at the end of occasional fixed intervals, and we compared subsequent drinking patterns with drinking before the extra food presentation. In Experiment 2 we presented signaled and unsignaled extra food and measured elicited and anticipatory drinking patterns. In Experiment 3, we observed the persistence of modified drinking patterns when several consecutive intervals ended with extra pellets. In Experiments 4 and 5, we varied the magnitude of food delivery across (rather than within) sessions to replicate published findings. Results show that schedule-induced drinking is neither elicited by food presentations nor induced by stimuli associated with a high food rate. All subjects seemed to follow a simple rule: during any stimulus signaling an increase in the local probability of food delivery within a session, engage in food-related behavior to the exclusion of drinking. Schedule-induced drinking appears to be the result of dynamic interactions among food-related behavior, drinking, and other motivated behavior, rather than a direct effect of the contingencies of food reinforcement.     

Choosing schedules of signaled appetitive events over schedules of unsignaled ones

Two experiments were completed allowing albino rats to choose between signaled and unsignaled reward conditions. These experiments examined the effects on preference of (1) response dependent versus response-independent reward and, (2) food pellets versus chocolate milk as the reward. All subjects preferred the signaled condition over the unsignaled condition, whether exposed to response-dependent, or to response-independent delivery of rewards. Preference was controlled most effectively by presenting both the signal itself and the correlated stimulus identifying the signaled condition. The signal presented alone (Extinction 3) controlled preference more effectively than did the stimulus correlated with the signaled condition (Extinction 2). The second experiment showed that the quality of the reinforcer (pellets and chocolate milk) did not affect preference for signaled reward since all subjects preferred the signaled condition at levels comparable to those observed in Experiment 1, with food pellets. These results, along with others, argue against species differences, response-dependency, and reinforcer quality as variables affecting the direction of preference.     

Frequency and value both matter in the suboptimal choice procedure

Pigeons chose between two options on a concurrent‐chains task with a single response requirement in the initial link. The suboptimal option ended with food 20% of the time whereas the optimal option ended with food 80% of the time. During a Sig‐Both condition, terminal‐link stimuli on both options signaled whether or not food would occur. During a Sig‐Sub condition, terminal‐link stimuli on the suboptimal option provided differential signals, but stimuli on the optimal option did not differentially signal the food and no food outcomes. Initial‐link choices revealed a clear preference for the optimal option in the Sig‐Both condition, but preference shifted toward suboptimality in the Sig‐Sub condition. These findings show that pigeon suboptimal choice is not singularly driven by signal value, as has been suggested, but also by reinforcer frequency.     

Contingent Stimuli Signal Subsequent Reinforcer Ratios

Conditioned reinforcer effects may be due to the stimulus' discriminative rather than its strengthening properties. While this was demonstrated in a frequently‐changing choice procedure, a single attempt to replicate in a relatively static choice environment failed. We contend that this was because the information provided by the stimuli was nonredundant in the frequently‐changing preparation, and redundant in the steady‐state arrangement. In the present experiments, 6 pigeons worked in a steady‐state concurrent schedule procedure with nonredundant informative stimuli (red keylight illuminations). When a response‐contingent red keylight signaled that the next food delivery was more likely on one of the two alternatives, postkeylight choice responding was reliably for that alternative. This effect was enhanced after a history of extended informative red keylight presentation (Experiment 2). These results lend support to recent characterizations of conditioned reinforcer effects as reflective of a discriminative, rather than a reinforcing, property of the stimulus.     

Suboptimal choice in a percentage-reinforcement procedure: effects of signal condition and terminal-link length.

Pigeons' choice between reliable (100%) and unreliable (50%) reinforcement was studied using a concurrent-chains procedure. Initial links were fixed-ratio 1 schedules, and terminal links were equal fixed-time schedules. The duration of the terminal links was varied across conditions. The terminal link on the reliable side always ended in food; the terminal link on the unreliable side ended with food 50% of the time and otherwise with blackout. Different stimuli present during the 50% terminal links signaled food or blackout outcomes under signaled conditions but were uncorrelated with outcomes under unsignaled conditions. In signaled conditions, most pigeons displayed a nearly exclusive preference for the 100% alternative when terminal links were short (5 or 10 s), but with terminal links of 30 s or longer, preference for the 100% alternative was sharply reduced (often to below .5). In unsignaled conditions, most pigeons showed extreme preference for the 100% alternative with either short (5 s) or longer (30 s) terminal links. Thus, pigeons' choice between reliable and unreliable reinforcement is influenced by both the signal conditions on the unreliable alternative and the duration of the terminal-link delay. With a long delay and signaled outcomes, many pigeons display a suboptimal tendency to choose the unreliable side.     

Dynamic effects of food magnitude on interim-terminal interaction

We tested the assumption of a facilitatory relation between periodic food presentation and schedule-induced drinking by examination of (a) elicited drinking, (b) drinking in anticipation of food delivery, and (c) possible indirect effects of food delivery on drinking. We exposed rats to a fixed-time 60-second schedule in which interfood intervals ended in either one or four food pellets with equal probability. In Phases 1 and 3, a stimulus signaled the magnitude of upcoming food presentation. In Phase 2, the stimulus was eliminated. Changes in drinking and “head-in-feeder” distributions within interfood intervals demonstrated that head-in-feeder was controlled directly by food presentation, but drinking was not. Head-in-feeder increased and drinking was reduced when large meals began or ended an interval. In Phases 4 to 6, meal size was manipulated across sessions yielding a positive relation between meal size and schedule-induced drinking. We conclude: (1) Schedule-induced drinking is determined by distributions of food-related behavior and results from indirect effects of food delivery; and (2) the amount of schedule-induced drinking and the form of the drinking distributions in this experiment can be accurately explained by two assumptions: (a) Food presentation facilitates food-related behavior through elicitation and anticipation; and (b) food-related behavior and drinking are reciprocally, linearly related.