On the measurement of reinforcement frequency in the study of preference

In a two-link, concurrent-chain schedule, pigeons' pecks on each key during the initial link occasionally produced a terminal link, during which only that key was operative. Responses in the terminal link were reinforced with food on either fixed-interval or variable-interval schedules. In one experiment, relative amount of responding in the initial link equaled the relative harmonic rate of reinforcement in the terminal links. In a second experiment, the selection of interreinforcement intervals in variable-interval schedules in the terminal links was such that rates of reinforcement based on the harmonic or on the arithmetic means of the interreinforcement intervals predicted opposite preferences in the initial links. The observed preference was consistent with that predicted by the harmonic rather than by the arithmetic rates of reinforcement.     

The psychological distance to reward

Pigeons' responses in the presence of two concurrently available (initial-link) stimuli produced entry into one of two different and mutually exclusive terminal link stimuli according to identical but independent variable-interval schedules. In one experiment, a two-component chained fixed-interval schedule produced food in one terminal link while a simple fixed-interval schedule produced food in the other terminal link. When the interreinforcement intervals were equal in the two terminal links (i.e., the simple fixed-interval was twice the size of each of the components in the chained schedule) pigeons preferred the simple fixed-interval as measured by their relative rates of responding in the concurrently available initial links. This preference increased as the duration of the terminal links increased. The preference could be reversed by making the simple fixed-interval schedule sufficiently longer than the chained schedule. In the second experiment, the terminal links consisted of two- vs three-component chained fixed-intervals, again with equal interreinforcement intervals. Pigeons preferred the two-component chain to the three-component chain, although these results were less consistent and less dramatic than those in the first experiment. Again, preference increased as the duration of the terminal links increased. The results show that an organism's choice for a schedule will be substantially lowered by the chaining operation even when the interreinforcement interval remains constant.     

Preference for mixed-interval versus fixed-interval schedules

Pigeons were trained on a two-link concurrent chain schedule in which responses on two keys were reinforced according to independent variable-interval schedules by the production of a change in key color. Further responses on the key on which the stimulus change had been produced gave a single food reinforcement and a return to concurrent variable-interval conditions. On one key the terminal link was a two-valued mixed-interval schedule, while on the other, the terminal link was a fixed-interval schedule. When the mixed-interval values were kept constant and the fixed-interval values varied, relative response rates in the initial concurrent links matched relative reinforcement rates in the terminal links when these were computed from cubic transformations of the reciprocals of the intervals comprising the terminal link schedules.     

Choice and transformed interreinforcement intervals

Pigeons chose between two aperiodic, time-based schedules of reinforcement. The arithmetic mean interreinforcement interval of the first schedule was short, but the harmonic mean was long, whereas the arithmetic mean interreinforcement interval of the second schedule was long, but the harmonic mean was short. The pigeons preferred the schedule with the shorter harmonic mean in a concurrent-chains procedure when a terminal link ended after the first scheduled reinforcer had been gained on a terminal-link entry, but reversed their preferences, such that they preferred the schedule with the shorter arithmetic mean, when the terminal links ended after a fixed duration of exposure to the schedule. Moreover, the pigeons preferred the schedule with the shorter arithmetic mean in a two-key concurrent variable-interval variable-interval procedure, as well as in a concurrent variable-time variable-time, changeover-key procedure. The data suggest that an aggregate property of a schedule may not yield valid information about the responding that schedule will maintain as a choice alternative.     

Choice for periodic schedules of reinforcement

Pigeons' responses in the presence of two concurrently available (initial-link) stimuli produced one of two different (terminal-link) stimuli according to identical but independent variable-interval schedules. Responding in the mutually exclusive terminal links was reinforced with food according to fixed-ratio schedules for six pigeons and according to fixed-interval schedules for two pigeons. None of the pigeons matched the proportion of (choice) responses in the initial links to the proportion of the rates of reinforcement obtained during the terminal links. Instead, as the values of each of the terminal-link schedules were increased by a constant amount, the choice proportions for the stimulus associated with the smaller of the two values increased, even though the relative rates of reinforcement during the terminal links decreased. These results are incompatible with those from previous studies with aperiodic (variable-interval or variable-ratio) schedules. The present results do suggest, however, that in transforming aperiodic schedules into their periodic equivalents, it may be necessary to consider the size of the smallest interreinforcement interval comprising the terminal-link schedules.     

Preference for fixed-interval schedules: an alternative model

Pigeons were trained under concurrent chain schedules in which the initial links were equal aperiodic schedules and the terminal links were fixed-interval schedules. Choice proportions in the initial links were measured in 26 experimental conditions. The data showed the inadequacy of previous models of concurrent chain performance. A new model was suggested in which choice is a joint function of terminal-link times, overall reinforcement rates, and terminal-link entries. This model accounted for 94% of the variance in the present data and for substantial percentages of the variance in previously reported data. The model simplifies to matching between response ratios and obtained reinforcement rate ratios for simple concurrent schedule performance.     

Choice between two-component chained and tandem schedules

Pigeons were trained on a two-key choice procedure in which a pair of equal and concurrently available variable-interval schedules (initial links) arranged entry into one or the other of two mutually exclusive schedules (terminal links) that ended in primary reinforcement. The terminal links were two-component chained or tandem schedules. Responses during the initial links were distributed equally on the two keys whenever the terminal links were associated with the same sets of interreinforcement intervals. Whether or not the terminal-link interreinforcement intervals were the same on the two keys, initial-link responding was affected by neither the presence nor relative durations of differentially signalled components within a terminal-link schedule. The simplest interpretation of these results is that initial-link responding is maintained directly by delayed primary reinforcement, rather than conditioned reinforcement afforded by the stimuli correlated with the terminal-link schedule components. This finding suggests that aspects of chained schedule performance usually attributed to conditioned reinforcement might best be reinterpreted in terms of delayed primary reinforcement and various discriminative functions served by the component-correlated stimuli.     

Resistance to change produced by access to fixed-delay versus variable-delay terminal links

Pigeons' responding was reinforced on a multiple schedule consisting of two two-link chain schedules presented in regular alternation. Responding in initial links (always variable-interval 60-s) produced a key-color change and access to a terminal link. The terminal link for one chain provided food after a fixed delay (fixed-interval or fixed-time); the terminal link for the other provided food after a variable delay (variable-interval or variable-time). The average duration of the terminal-link schedules was varied across conditions, but in every condition the arithmetic mean of the variable-delay terminal-link schedule was equal to the duration of the fixed delay. Response rates were higher in the initial links of the chains with the variable-delay terminal links. Response-decreasing operations (satiation, extinction) were used after performances reached asymptote. Response rates maintained by access to variable-delay terminal links tended to be more resistant to change than were rates maintained by access to fixed-delay terminal links. These results are consistent with the preference for variable- over fixed-interval terminal links observed with concurrent-chains schedules, suggesting (1) that immediacy of reinforcement influences the conditioned reinforcing potency of access to a terminal link and (2) that choice in concurrent chains and resistance of responding to change may be manifestations of the same effect of reinforcement.     

Effects of reinforcement context on choice

Two experiments investigated the effects of successive reinforcement contexts on choice. In the first, concurrent variable-interval schedules of primary reinforcement operated during the initial links of concurrent chains. The rate of this reinforcement arranged by the concurrent schedules was decreased across conditions: When it was higher than the terminal-link rate, preference for the higher frequency initial-link schedule increased relative to baseline. (During baseline, a standard concurrent-schedule procedure was in effect). When the initial-link reinforcement rate was lower than the terminal-link rate, preference converged toward indifference. In the second experiment, a chain schedule was available on a third key while a concurrent schedule was in effect on the side keys. When the terminal link of the chain schedule was produced, the side keys became inoperative. Availability of the chain schedule did not affect choice between the concurrent schedules. These results show that only when successive reinforcement contexts are produced by choice responding do those successive contexts affect choice in concurrent schedules.     

Context effects on choice.

Four pigeons responded on a concurrent-chains schedule in four experiments that examined whether the effectiveness of a stimulus as a conditioned reinforcer is best described by a global approach, as measured by the average interreinforcement interval, or by a local contextual approach, as measured by the onset of the stimulus preceding the conditioned reinforcer. The interreinforcement interval was manipulated by the inclusion of an intertrial interval, which increased the overall time to reinforcement but did not change the local contingencies on a given trial A global analysis predicted choice for the richer alternative to decrease with the inclusion of an intertrial interval, whereas a local analysis predicted no change in preference. Experiment 1 examined sensitivity to intertrial intervals when each was signaled by the same houselight that operated throughout the session. In Experiment 2, the intertrial interval always was signaled by the stimulus correlated with the richer terminal link. In Experiment 3, the intertrial interval was signaled by the keylights correlated with the initial links and two novel houselights. Experiment 4 provided free food pseudorandomly during the intertrial interval. In all experiments, subjects' preferences were consistent with a local analysis of choice in concurrent chains. These results are discussed in terms of delay-reduction theory, which traditionally has failed to distinguish global and local contexts.     

SOME EFFECTS OF PROCEDURAL VARIATIONS ON CHOICE RESPONDING IN CONCURRENT CHAINS

The present research used pigeons in a three‐key operant chamber and varied procedural features pertaining to both initial and terminal links of concurrent chains. The initial links randomly alternated on the side keys during a session, while the terminal links always appeared on the center key. Both equal and unequal initial‐link schedules were employed, with either differential or nondifferential terminal‐link stimuli across conditions. The research was designed to neutralize initial‐ and terminal‐link spatial cues in order to gain a clearer understanding of the roles of conditioned reinforcement and delayed primary reinforcement in choice. With both equal and unequal initial links and with differential terminal‐link stimuli, all pigeons reliably preferred the chain with the shorter terminal link. However, with equal initial links and nondifferential stimuli, all pigeons were indifferent. With unequal initial links and nondifferential stimuli, some pigeons were also indifferent, while others actually reversed and preferred the chain with the shorter initial link, even though it was followed by the longer terminal link. The decrease if not reversal of the previous preferences implies that preferences in concurrent chains are a function of the conditioned reinforcement afforded by terminal‐link stimuli, rather than delayed primary reinforcement.     

Preference for less segmented fixed-time components in concurrent-chain schedules of reinforcement

A concurrent-chain procedure was used to examine choice between segmented and less segmented response-independent schedules of reinforcement. A pair of independent, concurrent variable-interval 60-s schedules were presented in the initial link, along with a 1.5-s changeover delay. A chained fixed-interval fixed-time and its corresponding tandem schedule constituted the terminal links. The length of the fixed-interval schedule in the terminal link was varied between 5 s and 30 s while that of the fixed-time schedule was kept at 5 s over conditions. The first components of both terminal-link schedules were accompanied by the same stimulus. Except in the baseline condition, the onset of the second component of the terminal-link chained schedule was accompanied by either a localized (key color) or a nonlocalized (dark houselight) stimulus change. Stimulus conditions were constant during the terminal-link tandem schedule. With three exceptions, pigeons demonstrated a slight preference for the tandem over the chained schedule in the terminal link. Furthermore, this preference varied inversely with the length of the first component. In general, these results are consistent with previous studies that reported an adverse effect on choice by segmenting an interval schedule into two or more components, but they are inconsistent with studies that reported preference for signaled over unsignaled delay of reinforcement.     

Preference for simple interval schedules of reinforcement in concurrent chains: Effects of segmentation ratio

A concurrent-chains procedure was used to examine pigeons' preferences between segmented and unsegmented terminal-link schedules of reinforcement. During the initial link, a pair of independent, concurrent variable-interval 60-s schedules was in effect. In the terminal link, reinforcement was provided by a chain fixed-interval fixed-interval schedule on one key and by a simple fixed-interval schedule with an equal interreinforcement interval in the other. The relative duration between the first and second components (segmentation ratio) in the terminal-link chained schedule was systematically varied while the terminal-link duration was kept constant at either 15 s or 30 s in two sets of conditions. With few exceptions, the simple schedule was preferred to the chained schedule. Furthermore, this preference was inversely related to the size of the segmentation ratio in the segmented schedule. When the segmentation ratio was smaller than 1:1, preference was more extreme for a 30-s condition than for a 15-s condition. However, preference decreased more rapidly in conditions with the longer terminal-link duration when the ratio increased. Taken together, these results were consistent with previous findings concerning the effect of the terminal-link duration on choice between segmented and unsegmented schedules. In addition, the data suggested that segmentation ratio in a segmented schedule constitutes another potent factor influencing preference for the unsegmented schedule.     

Preference for unsegmented interreinforcement intervals in concurrent chains

Five pigeons were trained under concurrent-chain schedules in which a pair of independent, concurrent variable-interval 60-s schedules were presented in the initial link and either both variable-interval or both fixed-interval schedules were presented in the terminal link. Except for the baseline, one of the terminal-link schedules was always a two-component chained schedule and the other was either a simple or a tandem schedule of equal mean interreinforcement interval. The values of the fixed-interval schedules were either 15 s or 60 s; that of the variable-interval schedules was always 60 s. A 1.5-s changeover delay operated during the initial link in some conditions. The pigeons preferred a simple or a tandem schedule to a chain. For the fixed-interval schedules, this preference was greater when the fixed interval was 60 s than when it was 15 s. For the variable-interval schedules, the preferences were less pronounced and occurred only when the changeover delay was in effect. For a given type of schedule and interreinforcement interval, similar preferences were obtained whether the nonchained schedule was a tandem or simple schedule. The changeover delay generally inflated preference and lowered the changeover rate, especially when the terminal-link schedules were either short (15 s) or aperiodic (variable-interval). The results were consistent with the notion that segmenting the interreinforcement interval of a schedule into a chain lowers the preference for it.     

Inverse relation between choice and local response rate with a schedule of response-produced blackouts

Pigeons were exposed to a two-key concurrent chains schedule in which identical frequencies and distributions of food presentations generated different response rates in the terminal links. An inverse relation between local rate of response in the terminal links and relative frequency of response in the initial links was observed. The high response rate was produced in one terminal link by a second-order schedule in which responding produced brief blackouts of the response key. Responding under the same schedule in the other terminal link did not produce blackouts. Under initial training and after spatial reversal of the terminal-link schedules, two of three pigeons had lower relative frequencies of response in the initial member of the chain with the higher terminal link response rate. The third pigeon showed no change in preference at reversal.     

Choice with initial and terminal link reinforcement: an alternative self-control paradigm

Self-control is demonstrated when a less desirable immediate outcome is chosen to ensure a substantially better future. In a novel animal analogue of this situation, primary reinforcement was delivered in both the initial and terminal links of a concurrent chain schedule. Rats made initial link choices between equal amounts of ethanol-free or ethanol-containing milk. Choosing the ethanol-free reinforcer resulted in delivery of the larger terminal link reinforcer and was thus analogous to self-control. Self-control decreased as the delay between initial and terminal links increased. The results have implications for human choice situations where decisions are made between two immediately available reinforcement alternatives each associated with a different delayed outcome.     

Value of conditioned reinforcers as a function of temporal context

In two experiments, pigeons were trained on a multiple-chain schedule, in which the initial link for one chain was a variable-interval (VI) 100 s schedule and for the other chain a VI 10 s schedule. The terminal links were both fixed-time 30 s schedules signaled by differently colored stimuli. Following training, the pigeons had their preference for the terminal-link stimuli tested either by presenting these stimuli in concurrent probes or by presenting these stimuli as reinforcement for completing novel initial links. In Experiment 1, pigeons significantly preferred the terminal-link stimulus that followed the long initial link in three of the five conditions. This preference was observed across all three testing procedures (concurrent chains, concurrent chains probes, and concurrent probes). Experiment 2 was a replication of this effect in one of the conditions from Experiment 1. The results demonstrate that temporal context does impact the value of a conditioned reinforcer in a manner consistent with delay-reduction theory, and inconsistent with other choice theories, such as the contextual choice model and scalar expectancy theory.     

Preference for fixed-interval terminal links in a three-key concurrent chain schedule

Pigeons were trained on three-key concurrent chain schedules in which the initial links were variable-interval schedules and the terminal links were fixed-interval schedules. In the first experiment, the initial links were all equal and the terminal-link schedule on one key only was varied. In the second part of the experiment, the terminal-link schedules were all fixed, but different, and the initial-link schedule on one key was varied. Relative response rates in the initial links did not match either the relative arranged, nor the relative obtained, terminal-link reinforcement rates. The relations between independent and dependent variables in three-key concurrent chains were similar to, but not identical with, those found in two-key chains comprising the same schedule types.     

Dual effects on choice of conditioned reinforcement frequency and conditioned reinforcement value

Pigeons were presented with a concurrent‐chains schedule in which the total time to primary reinforcement was equated for the two alternatives (VI 30 s VI 60 s vs. VI 60 s VI 30 s). In one set of conditions, the terminal links were signaled by the same stimulus, and in another set of conditions they were signaled by different stimuli. Choice was in favor of the shorter terminal link when the terminal links were differentially signaled but in favor of the shorter initial link (and longer terminal link) when the terminal links shared the same stimulus. Preference reversed regularly with reversals of the stimulus condition and was unrelated to the discrimination between the two terminal links during the nondifferential stimulus condition. The present results suggest that the relative value of the terminal‐link stimuli and the relative rate of conditioned reinforcer presentation are important influences on choice behavior, and that models of conditioned reinforcement need to include both factors.