Violations of stochastic transitivity on concurrent chains: Implications for theories of choice

The concurrent-chains procedure has been used to measure how choice depends on various aspects of reinforcement, such as its delay and its magnitude. Navarick and Fantino (1972, 1974, 1975) have found that choice in this procedure can violate the condition of stochastic transitivity that is required if a unidimensional scale for reinforcements is to be possible. It is shown in this paper that two simple unidimensional models of choice on concurrent chains can produce violations of stochastic transitivity. It is argued that such violations may result from the complex contingencies of the concurrent-chains procedure.     

A contextual model of concurrent-chains choice

An extension of the generalized matching law incorporating context effects on terminal-link sensitivity is proposed as a quantitative model of behavior under concurrent chains. The contextual choice model makes many of the same qualitative predictions as the delay-reduction hypothesis, and assumes that the crucial contextual variable in concurrent chains is the ratio of average times spent, per reinforcement, in the terminal and initial links; this ratio controls differential effectiveness of terminal-link stimuli as conditioned reinforcers. Ninety-two concurrent-chains data sets from 19 published studies were fitted to the model. Averaged across all studies, the model accounted for 90% of the variance in pigeons' relative initial-link responding. The model therefore demonstrates that a matching law analysis of concurrent chains—the assumption that relative initial-link responding equals relative terminal-link value—remains quantitatively viable. Because the model reduces to the generalized matching law when terminal-link duration is zero, it provides a quantitative integration of concurrent schedules and concurrent chains.     

Effects of reinforcement context on choice

Two experiments investigated the effects of successive reinforcement contexts on choice. In the first, concurrent variable-interval schedules of primary reinforcement operated during the initial links of concurrent chains. The rate of this reinforcement arranged by the concurrent schedules was decreased across conditions: When it was higher than the terminal-link rate, preference for the higher frequency initial-link schedule increased relative to baseline. (During baseline, a standard concurrent-schedule procedure was in effect). When the initial-link reinforcement rate was lower than the terminal-link rate, preference converged toward indifference. In the second experiment, a chain schedule was available on a third key while a concurrent schedule was in effect on the side keys. When the terminal link of the chain schedule was produced, the side keys became inoperative. Availability of the chain schedule did not affect choice between the concurrent schedules. These results show that only when successive reinforcement contexts are produced by choice responding do those successive contexts affect choice in concurrent schedules.     

Assessing the value of choice in a token system

Responding of 4 children was assessed under conditions in which (a) no programmed contingencies were arranged for target behavior, (b) responding produced tokens that could be exchanged for a single highly preferred edible item, and (c) responding produced a token that could be exchanged for a variety of preferred edible items. After assessing the effects of these contingencies, the preferences of 3 participants were assessed using a concurrent-chains schedule. Preference for the opportunity to choose from the same or qualitatively different edible items varied across participants, and findings were generally consistent with those of Tiger, Hanley, and Hernandez (2006).     

Effects of a history of differential reinforcement on preference for choice

The effects of a history of differential reinforcement for selecting a free-choice versus a restricted-choice stimulus arrangement on the subsequent responding of 7 undergraduates in a computer-based game of chance were examined using a concurrent-chains arrangement and a multiple-baseline-across-participants design. In the free-choice arrangement, participants selected three numbers, in any order, from an array of eight numbers presented on the computer screen. In the restricted-choice arrangement, participants selected the order of three numbers preselected from the array of eight by a computer program. In initial sessions, all participants demonstrated no consistent preference or preference for restricted choice. Differential reinforcement of free-choice selections resulted in increased preference for free choice immediately and in subsequent sessions in the absence of programmed differential outcomes. For 5 participants, changes in preference for choice were both robust and lasting, suggesting that a history of differential reinforcement for choice may affect preference for choice.     

Devaluation of stimuli contingent on choice: evidence for conditioned reinforcement

Pigeons were presented a concurrent-chains schedule of reinforcement that had terminal links of equal duration. The initial links of the schedule were periodically interrupted by 15-s periods during which an extinction schedule was in effect. The extinction periods were presented on either a response-contingent or a noncontingent basis. Relative response rate for the left alternative decreased when the extinction periods were accompanied by the left terminal-link stimulus. Relative response rate for the right alternative decreased when the extinction periods were accompanied by the right terminal-link stimulus. Relative response rate varied inversely with the frequency of presentation of the extinction periods but was unaffected by presence versus absence of the response contingency in the schedule of extinction-period presentation. Furthermore, relative response rate was unaffected by presentation of extinction periods accompanied by a novel stimulus. When the extinction periods were presented after reinforcement in the left terminal link instead of as interruptions of the initial links, relative response rate for the left alternative was reduced if the postreinforcement extinction period was accompanied by the terminal-link stimulus for the left chain and reduced less if the extinction period was accompanied by the terminal-link stimulus for the right chain. The results demonstrate that the correlation between the terminal-link stimulus and extinction influenced the relative response rate in the initial link.     

Suboptimal choice and initial‐link requirement

Pigeons (n = 14) were trained in a concurrent‐chains suboptimal choice procedure that tested the effect of an increased ratio requirement in the initial links. Fixed‐ratio 1 and 25 conditions were manipulated within subjects in a counterbalanced order. In all conditions, distinct terminal‐link stimuli on a suboptimal alternative signaled either primary reinforcement (20% of the time) or extinction (80% of the time). On an optimal alternative, two distinct terminal‐link stimuli each signaled a 50% chance of primary reinforcement. Preference for the suboptimal alternative was significantly attenuated, and in some birds completely reversed, by the larger response requirement irrespective of condition order. This larger response requirement also generated a notable increase in between‐subject variability. A measure of cumulative choice responding is introduced to mitigate the problems associated with traditional session averages. Ordinal predictions of some current theories of suboptimal choice are also considered in light of the results.     

Unification of models for choice between delayed reinforcers.

Two models for choice between delayed reinforcers, Fantino's delay-reduction theory and Killeen's incentive theory, are reviewed. Incentive theory is amended to incorporate the effects of arousal on alternate types of behavior that might block the reinforcement of the target behavior. This amended version is shown to differ from the delay-reduction theory in a term that is an exponential in incentive theory and a difference in delay-reduction theory. A power series approximation to the exponential generates a model that is formally identical with delay-reduction theory. Correlations between delay-reduction theory and the amended incentive theory show excellent congruence over a range of experimental conditions. Although the assumptions that gave rise to delay-reduction theory and incentive theory remain different and testable, the models deriving from the theories are unlikely to be discriminable by parametric experimental tests. This congruence of the models is recognized by naming the common model the delayed reinforcement model, which is then compared with other models of choice such as Killeen and Fetterman's (1988) behavioral theory of timing, Mazur's (1984) equivalence rule, and Vaughan's (1985) melioration theory.     

Dual effects on choice of conditioned reinforcement frequency and conditioned reinforcement value

Pigeons were presented with a concurrent‐chains schedule in which the total time to primary reinforcement was equated for the two alternatives (VI 30 s VI 60 s vs. VI 60 s VI 30 s). In one set of conditions, the terminal links were signaled by the same stimulus, and in another set of conditions they were signaled by different stimuli. Choice was in favor of the shorter terminal link when the terminal links were differentially signaled but in favor of the shorter initial link (and longer terminal link) when the terminal links shared the same stimulus. Preference reversed regularly with reversals of the stimulus condition and was unrelated to the discrimination between the two terminal links during the nondifferential stimulus condition. The present results suggest that the relative value of the terminal‐link stimuli and the relative rate of conditioned reinforcer presentation are important influences on choice behavior, and that models of conditioned reinforcement need to include both factors.     

Reinforcement uncertainty enhances preference for choice in humans

Under concurrent‐chains schedules of reinforcement, participants often prefer situations that allow selection among alternatives (free choice) to situations that do not (forced choice). The present experiment examined the effects of reinforcement probability on choice preferences. Preferences for free versus forced choice were measured under a condition in which participants' choices were always reinforced (reinforcement probability of 1.0) and a condition in which outcomes were uncertain (reinforcement probability of 0.5). Forty‐four college students participated and preferences were examined under a concurrent‐chains schedule of reinforcement. Participants preferred free choice under uncertain reinforcement, but a bias toward free choice was not observed when reinforcement was certain. These results align with previous findings of preference for free choice under conditions of uncertainty, but suggest that preference may be dependent upon probabilistic reinforcement contingencies in the terminal links of the concurrent‐chains arrangement. Thus, reinforcement probability is an important variable to consider when conducting similar studies on the value of choice.     

The effect of conditioned reinforcement rate on choice: a review

We review the nature of conditioned reinforcement, including evidence that conditioned reinforcers maintain choice behavior in concurrent schedules and that they elevate responding in the terminal links of concurrent‐chains schedules. A question has resurfaced recently: Do theories of choice in concurrent‐chains schedules need to include a term reflecting greater preference for higher rates of conditioned reinforcement? The review of several studies addressing this point suggests that such a term is inappropriate. Elevated rates of conditioned reinforcement (and responding) in the terminal links of concurrent‐chains schedules do not lead to greater preference in the initial link leading to the higher rate of conditioned reinforcement. If anything, the opposite preference is likely to occur. This result is not surprising, since the additional putative conditioned reinforcers in the terminal link are not correlated with a reduction in time to primary reinforcement nor with an increase in value.     

Suboptimal choice in a percentage-reinforcement procedure: effects of signal condition and terminal-link length.

Pigeons' choice between reliable (100%) and unreliable (50%) reinforcement was studied using a concurrent-chains procedure. Initial links were fixed-ratio 1 schedules, and terminal links were equal fixed-time schedules. The duration of the terminal links was varied across conditions. The terminal link on the reliable side always ended in food; the terminal link on the unreliable side ended with food 50% of the time and otherwise with blackout. Different stimuli present during the 50% terminal links signaled food or blackout outcomes under signaled conditions but were uncorrelated with outcomes under unsignaled conditions. In signaled conditions, most pigeons displayed a nearly exclusive preference for the 100% alternative when terminal links were short (5 or 10 s), but with terminal links of 30 s or longer, preference for the 100% alternative was sharply reduced (often to below .5). In unsignaled conditions, most pigeons showed extreme preference for the 100% alternative with either short (5 s) or longer (30 s) terminal links. Thus, pigeons' choice between reliable and unreliable reinforcement is influenced by both the signal conditions on the unreliable alternative and the duration of the terminal-link delay. With a long delay and signaled outcomes, many pigeons display a suboptimal tendency to choose the unreliable side.     

Context effects on choice.

Four pigeons responded on a concurrent-chains schedule in four experiments that examined whether the effectiveness of a stimulus as a conditioned reinforcer is best described by a global approach, as measured by the average interreinforcement interval, or by a local contextual approach, as measured by the onset of the stimulus preceding the conditioned reinforcer. The interreinforcement interval was manipulated by the inclusion of an intertrial interval, which increased the overall time to reinforcement but did not change the local contingencies on a given trial A global analysis predicted choice for the richer alternative to decrease with the inclusion of an intertrial interval, whereas a local analysis predicted no change in preference. Experiment 1 examined sensitivity to intertrial intervals when each was signaled by the same houselight that operated throughout the session. In Experiment 2, the intertrial interval always was signaled by the stimulus correlated with the richer terminal link. In Experiment 3, the intertrial interval was signaled by the keylights correlated with the initial links and two novel houselights. Experiment 4 provided free food pseudorandomly during the intertrial interval. In all experiments, subjects' preferences were consistent with a local analysis of choice in concurrent chains. These results are discussed in terms of delay-reduction theory, which traditionally has failed to distinguish global and local contexts.     

Transfer tests of stimulus value in concurrent chains

We report two experiments that use transfer tests to investigate whether in concurrent chains the value of a terminal-link stimulus is affected by the alternate terminal link. In Experiment 1, two groups of pigeons were trained on multiple concurrent-chains schedules in which switching between the schedules was via pecking a changeover key. For one group, the terminal links were fixed-interval 8 s versus fixed-interval 16 s in one component and fixed-interval 16 s versus fixed-interval 32 s in the other component. For a second group, the terminal links were variable-interval 10 s versus variable-interval 20 s in one component and variable-interval 20 s versus variable-interval 40 s in the other. After sufficient baseline training had been given so that performances had stabilized, transfer tests were conducted in which the two chains with equal terminal-link schedules were presented together as a new concurrent pair. For 6 of the 7 subjects, initial-link responding changed fairly rapidly during the test in the manner predicted if the values of the terminal links were equal. In Experiment 2, pigeons were trained on multiple concurrent chains using a two-key procedure, and the terminal links were the same variable-interval schedules as in Experiment 1. After baseline training, transfer tests were conducted that assessed (a) the relative reinforcing strength of the terminal-link stimuli in a novel initial-link situation and (b) the relative ability of those stimuli to evoke responding. The data from the reinforcing strength test were consistent with those from Experiment 1, but those from the evocation strength test were not. Although this discrepancy shows that responding in transfer tests is not solely a function of stimulus value, the results from both experiments suggest, overall, that value is determined by the stimulus—reinforcer relation independently of the alternative terminal link.     

Conditioned reinforcement value and choice.

The delay-reduction hypothesis of conditioned reinforcement states that the reinforcing value of a food-associated stimulus is determined by the delay to primary reinforcement signaled by the onset of the stimulus relative to the average delay to primary reinforcement in the conditioning situation. In contrast, most contemporary models of conditioned reinforcement strength posit that the reinforcing strength of a stimulus is some simple function only of the delay to primary reinforcement in the presence of stimulus. The delay-reduction hypothesis diverges from other conditioned reinforcement models in that it predicts that a fixed-duration food-paired stimulus will have different reinforcing values depending on the frequency of its presentation. In Experiment 1, pigeons' key pecks were reinforced according to concurrent-chains schedules with variable-interval 10-second and variable-interval 20-second terminal-link schedules. The initial-link schedule preceding the shorter terminal link was always variable-interval 60 seconds, and the initial-link schedule requirement preceding the longer terminal link was varied between 1 second and 60 seconds across conditions. In Experiment 2, the initial-link schedule preceding the longer of two terminal links was varied for each of three groups of pigeons. The terminal links of the concurrent chains for the three groups were variable-interval 10 seconds and 20 seconds, variable-interval 10 seconds and 30 seconds, and variable-interval 30 seconds and 50 seconds. In both experiments, preference for the shorter terminal link was either a bitonic function or an inverse function of the initial-link schedule preceding the longer terminal-link schedule. Consistent with the predictions of the delay-reduction hypothesis, the relative values of the terminal-link stimuli changed as a function of the overall frequency of primary reinforcement. Vaughan's (1985) melioration model, which was shown to be formally similar to Squires and Fantino's (1971) delay-reduction model, can be modified so as to predict these results without changing its underlying assumptions.     

A decision model for steady-state choice in concurrent chains

Grace and McLean (2006) proposed a decision model for acquisition of choice in concurrent chains which assumes that after reinforcement in a terminal link, subjects make a discrimination whether the preceding reinforcer delay was short or long relative to a criterion. Their model was subsequently extended by Christensen and Grace (2008, 2009a, 2009b) to include effects of initial- and terminal-link duration on choice. We show that an expression for steady-state responding can be derived from the decision model, which enables a model for choice that provides an account of archival data that is equal or superior to the contextual choice model (Grace, 1994) and hyperbolic value-added model (Mazur, 2001) in terms of goodness of fit, parsimony, and parameter invariance. The success of the steady-state decision model validates the strategy of understanding acquisition phenomena as a bridge toward explaining choice at the molar level.     

Preference for mixed versus constant delays of reinforcement: Effect of probability of the short, mixed delay

Preference for mixed versus constant delays of reinforcement was studied with a concurrent-chain procedure. Lever pressing by rats in concurrently available variable-interval 60-second initial links occasionally produced mutually exclusive terminal-link reinforcement delays. A constant delay of reinforcement (either 15 seconds or 30 seconds) composed one terminal link and mixed delays (.2 second and twice the value of the constant delay) were arranged in the other terminal link. The proportion of .2-second delays in the mixed-delay terminal link took on values of 0, .1, .25, .5, .75, .9, and 1.0 over experimental conditions. Based on relative rates of responding in the initial links, preference for the mixed delays was a negatively accelerated function of the proportion of short, mixed delays. Three of five rats preferred the mixed delays to the constant delays when the proportion of short, mixed delays was .1 or higher, and all five rats preferred the mixed delays when the proportion of short, mixed delays was .25 or higher. Neither Squires and Fantino's (1971) delay-reduction model of choice nor a model based on the harmonic mean reinforcement delay provided a close estimate of choice proportions over the range of short-delay proportions studied. The delay-reduction model underestimated choice for the mixed delays at low and intermediate proportions of short delays, and the harmonic-mean-delay model overestimated choice for the mixed delays at intermediate and high proportions of short delays.     

Preferences for and against stimuli paired with food.

Pigeons were presented with a concurrent-chains schedule in which terminal-link entries were assigned to two response keys on a percentage basis. The terminal links were fixed delays that sometimes ended with food and sometimes did not. In most conditions, 80% of the terminal links were assigned to one key, but a smaller percentage of the terminal links ended with food for this key, so the number of food reinforcers delivered by the two alternatives was equal. When the same terminal-link stimuli (orange houselights) were used for both alternatives, the pigeons showed a preference for whichever alternative delivered more frequent terminal links. When different terminal-link stimuli (green vs. red houselights) were used for the two alternatives, the pigeons showed a preference for whichever alternative delivered fewer terminal links when terminal-link durations were long, and no systematic preferences when terminal-link durations were short. This pattern of results was consistent with the predictions of Grace's (1994) contextual choice model. Preference for the alternative that delivered more frequent terminal links was usually stronger in the first few sessions of a condition than at the end of a condition, suggesting that the conditioned reinforcing effect of the additional terminal-link presentation was, in part, transitory.