The Δ–∑ hypothesis: How contrast and reinforcement rate combine to generate suboptimal choice

When given a choice between two alternatives, each offering food after the same delay with different but signaled probabilities, pigeons often prefer the low probability alternative. This preference is surprising because pigeons fail to maximize the rate of food intake; they exhibit a suboptimal preference. We advance a new explanation, the Δ–∑ hypothesis, in which the difference in probability of reinforcement within terminal links (Δ) and the overall reinforcement probability rate of each alternative (∑) are the key variables responsible for such suboptimal preference. We tested the Δ–∑ hypothesis in two experiments. In Experiment 1, we manipulated the Δs while maintaining constant all other parameters of the task, in particular the ∑s. We predicted a preference for the alternative with the larger Δ. In Experiment 2, we examined the effect of the overall reinforcement probabilities, the ∑s, while maintaining constant all other parameters of the task, in particular the Δs. We predicted a preference for the larger ∑. The results of both experiments support the Δ–∑ hypothesis.     

Pigeons (Columba livia) associate time intervals with symbols in a touch screen task: evidence for ordinality but not summation

The present experiments examined whether pigeons can sum symbols that are associated with various temporal consequences in a touch screen apparatus. Pigeons were trained to discriminate between two visual symbols that were associated with 0, 1, 2, 3, 4, or 5 s either of delay to 4 s of hopper access (delay group) or duration of hopper access (reward group). In Experiment 1, the pigeons in both groups learned to select the symbol associated with the more favorable outcome, and they successfully transferred this discrimination to novel symbol pairs. However, when tested with 2 pairs of symbols associated with different summed durations, they responded on the basis of a simple response rule rather than the sum of the symbol pair. In Experiment 2, the reward group was presented with four symbols at once and was allowed to successively choose one symbol at a time. All pigeons chose the symbols in order from largest to smallest. This indicates that pigeons formed an ordered representation of symbols associated with different time intervals, even though they did not sum the symbols.     

The differentiation of response numerosities in the pigeon

Two experiments examined how pigeons differentiate response patterns along the dimension of number. In Experiment 1, 5 pigeons received food after pecking the left key at least N times and then switching to the right key (Mechner's Fixed Consecutive Number schedule). Parameter N varied across conditions from 4 to 32. Results showed that run length on the left key followed a normal distribution whose mean and standard deviation increased linearly with N; the coefficient of variation approached a constant value (the scalar property). In Experiment 2, 4 pigeons received food with probability p for pecking the left key exactly four times and then switching. If that did not happen, the pigeons still could receive food by returning to the left key and pecking it for a total of at least 16 times and then switching. Parameter p varied across conditions from 1.0 to .25. Results showed that when p= 1.0 or p=.5, pigeons learned two response numerosities within the same condition. When p=.25, each pigeon adapted to the schedule differently. Two of them emitted first runs well described by a mixture of two normal distributions, one with mean close to 4 and the other with mean close to 16 pecks. A mathematical model for the differentiation of response numerosity in Fixed Consecutive Number schedules is proposed.     

TO WAIT OR TO RESPOND?

Emitting a certain response and waiting for a specified time without making that response had the same consequence. In Experiment 1, food-deprived pigeons were as likely to wait as to respond only if waiting provided food at a much higher frequency than did pecking. In Experiment 2, the consequence for humans was a brief light flash and tone. People were not biased for responding over waiting. Instead, their choices suggested crude payoff maximization. In Experiment 3, pigeons again obtained food, but they were not food deprived and could eat freely at each opportunity. Their behavior was more like that of the humans of Experiment 2 than that of food-deprived pigeons given small quantities of food at each feeding opportunity. The three experiments together showed that biases for responding over waiting were neither inherent characteristics of species nor inevitable outcomes of particular schedules. Choice between active search and waiting depended on ecological–motivational factors even when species and schedules were held constant.     

Contrast and undermatching as a function of reinforcer duration and quality during multiple schedules

Eight pigeons pecked keys under multiple variable-interval two-minute variable-interval two-minute schedules. In Experiment 1, the reinforcers were 2, 4, or 8 seconds access to a food magazine. In Experiments 2 and 3, the reinforcers were grains that had been determined to be most-, moderately-, or non-preferred. Both positive and negative behavioral contrast occurred when the reinforcers in one component were held constant and the duration or type of reinforcer obtained in the other component varied. Undermatching occurred when the relative rate of responding during a component was plotted as a function of the relative duration of the reinforcers in that component.     

Income maximizing on concurrent ratio-interval schedules of reinforcement

Three experiments examined the effect of food availability on pigeons' choice behavior under concurrent schedules of reinforcement. In Experiment 1, 3 pigeons earned their daily food ration by choosing, in 30-min sessions, between concurrent variable-ratio 30 variable-interval 40-s schedules. Food presentations during both schedules lasted 2 or 12 s, depending upon the condition. Relative variable-ratio response rate was inversely related to hopper duration. In Experiment 2, 4 pigeons received their daily feeding by responding on the same schedule pair as in Experiment 1 (with 4-s food presentations) in sessions that varied in length from 10 to 30 min, depending on the condition. The length of a vertical slit projected on a response key increased with time so that “passage of time” might be more easily discriminable. As session duration decreased, relative variable-ratio response rate increased. In Experiment 3, 4 pigeons chose between two variable-interval 40-s schedules. One schedule operated without regard to the schedule selected, whereas the other operated only when the subject responded in its presence (dependent). Although these schedules had the same feedback function, preference for the dependent variable interval increased as session duration decreased from 30 to 10 min. The preference changes in these studies reveal the operation of an income-maximizing process in choice.     

Variable-ratio versus variable-interval schedules: response rate, resistance to change, and preference

Two experiments asked whether resistance to change depended on variable-ratio as opposed to variable-interval contingencies of reinforcement and the different response rates they establish. In Experiment 1, pigeons were trained on multiple random-ratio random-interval schedules with equated reinforcer rates. Baseline response rates were disrupted by intercomponent food, extinction, and prefeeding. Resistance to change relative to baseline was greater in the interval component, and the difference was correlated with the extent to which baseline response rates were higher in the ratio component. In Experiment 2, pigeons were trained on multiple variable-ratio variable-interval schedules in one half of each session and on concurrent chains in the other half in which the terminal links corresponded to the multiple-schedule components. The schedules were varied over six conditions, including two with equated reinforcer rates. In concurrent chains, preference strongly overmatched the ratio of obtained reinforcer rates. In multiple schedules, relative resistance to response-independent food during intercomponent intervals, extinction, and intercomponent food plus extinction depended on the ratio of obtained reinforcer rates but was less sensitive than was preference. When reinforcer rates were similar, both preference and relative resistance were greater for the variable-interval schedule, and the differences were correlated with the extent to which baseline response rates were higher on the variable-ratio schedule, confirming the results of Experiment 1. These results demonstrate that resistance to change and preference depend in part on response rate as well as obtained reinforcer rate, and challenge the independence of resistance to change and preference with respect to response rate proposed by behavioral momentum theory.     

Time preferences in long-tailed macaques (Macaca fascicularis) and humans (Homo sapiens)

Rosati et al. (Curr Biol 17(19):1663–1668, 2007) found in a self-control test in which choice was between a smaller, immediately delivered food and a larger, delayed food, that chimpanzees preferred the larger reward (self-control); humans, however, preferred the smaller reward (impulsivity). They attributed their results to a species difference in self-control. In Experiment 1, monkeys (long-tailed macaques) were exposed to a self-control task in two conditions: where the food was hidden under differently colored bowls and where it was visible. When these two conditions were compared, choice shifted from greater preference for the impulsive alternative in the hidden condition to greater preference for the self-control alternative in the visible condition. Additionally, in both conditions, preference shifted from self-control to impulsivity over sessions. These results were explained in terms of the reversed-contingency effect (a propensity to reach for more over less when rewards are visible) and not to a capacity for self-control. In Experiment 2, humans that demonstrated preference for more over less in choice preferred the impulsive alternative when choice to either alternative was followed by the same intertrial interval—a preference that accelerates trial rates relative to preference of the self-control alternative. When trial rates were equated so that neither choice accelerated session’s end, humans demonstrated self-control. These results suggest that Rosati et al.’s demonstration of impulsivity in humans was due to participants’ desire to minimize session time.     

Effects of reinforcer magnitude on responding under differential-reinforcement-of-low-rate schedules of rats and pigeons

Experiment I investigated the effects of reinforcer magnitude on differential-reinforcement-of-low-rate (DRL) schedule performance in three phases. In Phase 1, two groups of rats (n = 6 and 5) responded under a DRI. 72-s schedule with reinforcer magnitudes of either 30 or 300 microl of water. After acquisition, the water amounts were reversed for each rat. In Phase 2, the effects of the same reinforcer magnitudes on DRL 18-s schedule performance were examined across conditions. In Phase 3, each rat responded unider a DR1. 18-s schedule in which the water amotnts alternated between 30 and 300 microl daily. Throughout each phase of Experiment 1, the larger reinforcer magnitude resulted in higher response rates and lower reinforcement rates. The peak of the interresponse-time distributions was at a lower value tinder the larger reinforcer magnitude. In Experiment 2, 3 pigeons responded under a DRL 20-s schedule in which reinforcer magnitude (1-s or 6-s access to grain) varied iron session to session. Higher response rates and lower reinforcement rates occurred tinder the longer hopper duration. These results demonstrate that larger reinforcer magnitudes engender less efficient DRL schedule performance in both rats and pigeons, and when reinforcer magnitude was held constant between sessions or was varied daily. The present results are consistent with previous research demonstrating a decrease in efficiency as a function of increased reinforcer magnituide tinder procedures that require a period of time without a specified response. These findings also support the claim that DRI. schedule performance is not governed solely by a timing process.     

Human symbolic matching-to-sample performance: Effects of reinforcer and sample-stimulus probabilities

Four experiments, each with 6 human subjects, varied the distribution of reinforcers for correct responses and the probability of sample-stimulus presentation in symbolic matching-to-sample procedures. Experiment 1 held the sample-stimulus probability constant and varied the ratio of reinforcers obtained for correct responses on the two alternatives across conditions. There was a positive relation between measures of response bias and the ratio of reinforcers. Experiment 2 held the ratio of reinforcers constant and varied the sample-stimulus probability across conditions. Unlike previous studies that used pigeons as subjects, there was a negative relation between bias and the ratio of sample-stimulus presentations. In Experiment 3, the sample-stimulus probability and the reinforcer ratio covaried across conditions. Response bias did not vary systematically across conditions. In Experiments 1 to 3, correct responses were reinforced intermittently. Experiment 4 used the same procedure as Experiment 3, but all correct responses now produced some scheduled consequence. There was a positive relation between response bias and the ratio of reinforcers. The results suggest that human performance in these tasks was controlled by both the relative frequency of reinforced responses and the relative frequency of nonreinforced responses.     

Use of kind information for object individuation in young domestic chicks

In this paper, we studied the ability of newborn chicks to use kind information (sortal objects) provided by social and food attractors to determine the number of distinct objects present in an event (object individuation). Newly hatched chicks were reared with five imprinting objects and were fed mealworms. Chicks’ spontaneous tendency to approach the larger group of items was exploited. At test, on day 2 post-hatching, chicks observed two events in which objects, differing in kind, were each hidden behind one of two identical screens. Approaching either screen was considered a preferential choice. In Experiment 1, chicks presented with two social versus two food attractors did not exhibit any preference. In contrast, in Experiment 2, when chicks saw two different attractors (one social and one food) hidden behind a screen and one attractor hidden twice (i.e. moved back and forth two times) behind the other screen, they spontaneously approached the two different attractors rather than the single one seen twice. An explanation based on the preference for the more varied set was ruled out in Experiment 3: chicks did not preferentially choose between two different versus two identical objects when both groups were simultaneously presented. Results suggest for the first time that a non-human species uses kind information for individuating objects in a cross-basic-level contrast (i.e. food and social items) with minimal experience. As social and food stimuli differ in property as well as in kind information, the alternative explanation accounting for use of property information alone is also discussed.     

The decay of short-term implicit memory: Unpacking lag

McKone (1995) reported a short-lived repetition priming effect, superimposed on long-lived priming. This short-term implicit memory survived a few intervening items and several seconds for words but decayed precipitously for nonwords, producing a lag 3 lexicality interaction. Here, mechanisms of decay are studied by disconfounding the time delay and interference components of lag. In Experiment 1, time delay was varied while number of intervening items was held constant, and vice versa. In Experiment 2, priming was plotted as a function of time delay, with and without intervening items. Using a lexical decision task, both experiments found independent contributions of time and interference to the decay of short-term priming. Further, Experiment 2 attributed the lag 3 lexicality interaction to a particular sensitivity of nonword traces to interference. An illustration of how these effects might arise in the word recognition system is provided.     

Pigeons (Columba livia) learn to link numerosities with symbols

After responding to each element in varying, successive numerosity displays, pigeons (Columba livia) had to choose, out of an array of symbols, the symbol designated to correspond to the preceding number of elements. After extensive training, 5 pigeons responded with significant accuracy to the numerosities 1 to 4, and 2 pigeons to the numerosities 1 to 5. Several tests showed that feedback tones accompanying element pecks, the familiarity of element configurations, and the shape of the elements were not crucial to this performance. One test, however, indicated that the number of pecks issued to the elements was important for numerosities above 2. An additional test confirmed that the birds chose the symbol that corresponded to a particular numerosity rather than the positions that the symbols had held during training.     

Commitment, choice and self-control

When offered a choice (Choice Y) between a small immediate reward (2-sec exposure to grain) and a large reward (4-sec exposure to grain) delayed by 4 sec, pigeons invariably preferred the small, immediate reward. However, when offered a choice (Choice X) between a delay of T seconds followed by Choice Y and a delay of T seconds followed by restriction to the large delayed reward only, the pigeon's choice depended on T. When T was small, the pigeons chose the alternative leading to Choice Y (and then chose the small, immediate reward). When T was large, the pigeons chose the alternative leading to the large delayed reward only. The reversal of preference as T increases is predicted by several recent models for choice between various amounts and delays of reward. The preference for the large delayed alternative with long durations of T parallels everyday instances of advance commitment to a given course of action. Such commitment may be seen as a prototype for self-control.     

Waiting by mistake: Symbolic representation of rewards modulates intertemporal choice in capuchin monkeys,preschool children and adult humans

In the Delay choice task subjects choose between a smaller immediate option and a larger delayed option. This paradigm, also known as intertemporal choice task, is frequently used to assess delay tolerance, interpreting a preference for the larger delayed option as willingness to wait. However, in the Delay choice task subjects face a dilemma between two preferred responses: “go for more” (i.e., selecting the larger, but delayed, option) vs. “go for sooner” (i.e., selecting the immediate, but smaller, option). When the options consist of visible food amounts, at least some of the choices of the larger delayed option might be due to a failure to inhibit a prepotent response towards the larger option rather than to a sustained delay tolerance. To disentangle this issue, we tested 10 capuchin monkeys, 101 preschool children, and 88 adult humans in a Delay choice task with food, low-symbolic tokens (objects that can be exchanged with food and have a one-to-one correspondence with food items), and high-symbolic tokens (objects that can be exchanged with food and have a one-to-many correspondence with food items). This allows evaluating how different methods of representing rewards modulate the relative contribution of the “go for more” and “go for sooner” responses. Consistently with the idea that choices for the delayed option are sometimes due to a failure at inhibiting the prepotent response for the larger quantity, we expected high-symbolic tokens to decrease the salience of the larger option, thus reducing “go for more” responses. In fact, previous findings have shown that inhibiting prepotent responses for quantity is easier when the problem is framed in a symbolic context. Overall, opting for the larger delayed option in the visible-food version of the Delay choice task seems to partially result from an impulsive preference for quantity, rather than from a sustained delay tolerance. In capuchins and children high-symbolic stimuli decreased the individual’s preference for the larger reward by distancing from its appetitive features. Conversely, the sophisticated symbolic skills of adult humans prevented the distancing effect of high-symbolic stimuli in this population, although this result may be due to methodological differences between adult humans and the other two populations under study. Our data extend the knowledge concerning the influence of symbols on both human and non-human primate behavior and add a new element to the interpretation of the Delay choice task. Since high-symbolic stimuli decrease the individual’s preference for the larger reward by eliminating those choices due to prepotent responses towards the larger quantity, they allow to better discriminate responses based on genuine delay aversion. Thus, these findings invite greater caution in interpreting the results obtained with the visible-food version of the Delay choice task, which may overestimate delay tolerance.     

Capuchin monkeys (Cebus apella) are sensitive to others’ reward: an experimental analysis of food-choice for conspecifics

The issue whether non-human primates have other-regarding preference and/or inequity aversion has been under debate. We investigated whether tufted capuchin monkeys are sensitive to others’ reward in various experimental food sharing settings. Two monkeys faced each other. The operator monkey chose one of two food containers placed between the participants, each containing a food item for him/herself and another for the recipient. The recipient passively received either high- or low-value food depending on the operator’s choice, whereas the operator obtained the same food regardless of his/her choice. The recipients were either the highest- or lowest-ranking member of the group, and the operators were middle-ranking. In Experiment 1, the operators chose the high-value food for the subordinate recipient more frequently than when there was no recipient, whereas they were indifferent in their choice for the dominant. This differentiated behavior could have been because the dominant recipient frequently ate the low-value food. In Experiment 2, we increased the difference in the value of the two food items so that both recipients would reject the low-value food. The results were the same as in Experiment 1. In Experiment 3, we placed an opaque screen in front of the recipient to examine effects of visual contact between the participants. The operators’ food choice generally shifted toward providing the low-value food for the recipient. These results suggest that capuchins are clearly sensitive to others’ reward and that they show other-regarding preference or a form of inequity aversion depending upon the recipients and the presence of visual contact.     

“We will work for you” – Social influence may suppress individual food preferences in a communicative situation in dogs

The level of motivation (i.e. incentive power) is thought to be one of the most important factors affecting performance and learning in various tasks. We investigated whether reward quality has an effect on the performance of family dogs in a two-way object choice test in which they can find the hidden food by relying on distal momentary human pointing cues. In three experiments we varied (1) the type of food reward according to the subjects’ own preference; (2) the quality of the reward offered at the same time in the indicated and not-indicated locations; and (3) the order of the high or low quality rewards in consecutive sessions. In Experiment 1, we first tested whether dogs prefer one kind of reward over another. Then one group was tested with the ‘preferred’ food as reward in the indicated bowl, while dogs in the other group received the ‘non-preferred’ food as reward. We did not find any difference between the performance and choice latencies of the two groups. In Experiment 2 for the first group, the indicated bowl contained a piece of carrot and the not-indicated bowl was empty. In the second group the indicated bowl contained carrot, but the not-indicated bowl contained sausage. According to a preliminary preference test, most dogs prefer sausage over carrot invariably. After 20 trials, the two groups performed surprisingly similarly. There was no difference found between groups in the number of correct choices, incorrect choices and non-choices. However, the comparison between the first and last five trials revealed that subjects who found sausage when they chose the not-indicated bowl (did not follow the pointing) chose the non-indicated bowl significantly more often toward the end of their test session. In Experiment 3, each dog received two sessions with 12 pointing trials in each. For the first session, one group was rewarded with sausage and the other with carrot upon choosing the indicated bowl. In the second session, the indicated bowl contained dry dog food for both groups. We found that correct choices and response latencies did not change over two sessions in the ‘sausage’ group. In the ‘carrot’ group, the dogs chose faster in the second session, but their performance did not improve; in fact, they chose the not-indicated bowl more often than the indicated bowl. As a conclusion, we can say that reward quality had some effect on dogs’ choice behavior in these experiments. The drop in their performance was not drastic, taking into account the general refusal to eat one of the ‘rewards’ (carrot) during the preference tests and also during the test trials. It seems that incentive contrast may play a relatively minor role in dog-human social interactions. Appropriate reward quality can be very important in asocial problem solving tasks, but, when interacting with humans, following human signals may override the effect of changed incentive power.     

Effects of intertrial reinforcers on self-control choice.

In three experiments, pigeons chose between a small amount of food delivered after a short delay and a larger amount delivered after a longer delay. A discrete-trial adjusting-delay procedure was used to estimate indifference points--pairs of delay-amount combinations that were chosen about equally often. In Experiment 1, when additional reinforcers were available during intertrial intervals on a variable-interval schedule, preference for the smaller, more immediate reinforcer increased. Experiment 2 found that this shift in preference occurred partly because the variable-interval schedule started sooner after the smaller, more immediate reinforcer, but there was still a small shift in preference when the durations and temporal locations of the variable-interval schedules were identical for both alternatives. Experiment 3 found greater increases in preference for the smaller, more immediate reinforcer with a variable-interval 15-s schedule than with a variable-interval 90-s schedule. The results were generally consistent with a model that states that the impact of any event that follows a choice response declines according to a hyperbolic function with increasing time since the moment of choice.     

Determinants of pigeons' choices in token-based self-control procedures

Four pigeons were exposed to a token-based self-control procedure with stimulus lights serving as token reinforcers. Smaller-reinforcer choices produced one token immediately; larger-reinforcer choices produced three tokens following a delay. Each token could be exchanged for 2-s access to food during a signaled exchange period each trial. The main variables of interest were the exchange delays (delays from the choice to the exchange stimulus) and the food delays (also timed from the choice), which were varied separately and together across blocks of sessions. When exchange delays and food delays were shorter following smaller-reinforcer choices, strong preference for the smaller reinforcer was observed. When exchange delays and food delays were equal for both options, strong preference for the larger reinforcer was observed. When food delays were equal for both options but exchange delays were shorter for smaller-reinforcer choices, preference for the larger reinforcer generally was less extreme than under conditions in which both exchange and food delays were equal. When exchange delays were equal for both options but food delays were shorter for smaller-reinforcer choices, preference for the smaller reinforcer generally was less extreme than under conditions in which both exchange and food delays favored smaller-reinforcer choices. On the whole, the results were consistent with prior research on token-based self-control procedures in showing that choices are governed by reinforcer immediacy when exchange and food delays are unequal and by reinforcer amount when exchange and food delays are equal. Further, by decoupling the exchange delays from food delays, the results tentatively support a role for the exchange stimulus as a conditioned reinforcer.     

Arousal, changeover responses, and preference in concurrent schedules

Pigeons were trained on multiple schedules that provided concurrent reinforcement in each of two components. In Experiment 1, one component consisted of a variable-interval (VI) 40-s schedule presented with a VI 20-s schedule, and the other a VI 40-s schedule presented with a VI 80-s schedule. After extended training, probe tests measured preference between the stimuli associated with the two 40-s schedules. Probe tests replicated the results of Belke (1992) that showed preference for the 40-s schedule that had been paired with the 80-s schedule. In a second condition, the overall reinforcer rate provided by the two components was equated by adding a signaled VI schedule to the component with the lower reinforcer rate. Probe results were unchanged. In Experiment 2, pigeons were trained on alternating concurrent VI 30-s VI 60-s schedules. One schedule provided 2-s access to food and the other provided 6-s access. The larger reinforcer magnitude produced higher response rates and was preferred on probe trials. Rate of changeover responding, however, did not differ as a function of reinforcer magnitude. The present results demonstrate that preference on probe trials is not a simple reflection of the pattern of changeover behavior established during training.