Choosing among multiple alternatives: Relative and overall reinforcer rates

Choice behavior among two alternatives has been widely researched, but fewer studies have examined the effect of multiple (more than two) alternatives on choice. Two experiments investigated whether changing the overall reinforcer rate affected preference among three and four concurrently scheduled alternatives. Experiment 1 trained six pigeons on concurrent schedules with three alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 9:3:1 with the configuration counterbalanced across pigeons. The overall rate of reinforcement was varied across conditions. Preference between the pair of keys arranging the 9:3 reinforcer ratio was less extreme than the pair arranging the 3:1 reinforcer ratio regardless of overall reinforcer rate. This difference was attributable to the richer alternative receiving fewer responses per reinforcer than the other alternatives. Experiment 2 trained pigeons on concurrent schedules with four alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 8:4:2:1, and the overall reinforcer rate was varied. Next, two of the alternatives were put into extinction and the random interval duration was changed from 60 s to 5 s. The ratio of absolute response rates was independent of interval length across all conditions. In both experiments, an analysis of sequences of visits following each reinforcer showed that the pigeons typically made their first response to the richer alternative irrespective of which alternative was just reinforced. Performance on these three‐ and four‐alternative concurrent schedules is not easily extrapolated from corresponding research using two‐alternative concurrent schedules.     

Resistance to extinction following variable-interval reinforcement: reinforcer rate and amount

Rats obtained food-pellet reinforcers by nose poking a lighted key. Experiment 1 examined resistance to extinction following single-schedule training with different variable-interval schedules, ranging from a mean interval of 16 min to 0.25 min. That is, for each schedule, the rats received 20 consecutive daily baseline sessions and then a session of extinction (i.e., no reinforcers). Resistance to extinction (decline in response rate relative to baseline) was negatively related to the rate of reinforcers obtained during baseline, a relation analogous to the partial-reinforcement-extinction effect. A positive relation between these variables emerged, however, when the unit of extinction was taken as the mean interreinforcer interval that had been in effect during training (i.e., as an omitted reinforcer during extinction). In a second experiment, rats received blocks of training sessions, all with the same variable-interval schedule but with a reinforcer of four pellets for some blocks and one pellet for others. Resistance to extinction was greater following training with the larger (four pellets) than with the smaller (one pellet) reinforcer. Taken together, these results support the principle that greater reinforcement during training (e.g., higher rate or larger amount) engenders greater resistance to extinction even when the different conditions of reinforcement are varied between blocks of sessions.     

Behavioral momentum and resistance to extinction across repeated extinction tests

The present experiments assessed whether resistance to extinction of pigeons' key pecking decreased across repeated extinction tests. An additional impetus for this research was to determine how the quantitative framework provided by behavioral momentum theory might be used to describe any such changes across tests. Pigeons pecked keys in two‐component multiple schedules (one component associated with a higher reinforcer rate and the other with a lower rate) in which baseline and extinction conditions alternated. In Experiment 1, baseline and extinction conditions alternated every session, and, in Experiment 2, these conditions lasted for 10 and 7 sessions, respectively. Resistance to extinction decreased across successive extinction conditions in both experiments. Fits of the behavioral‐momentum based model of extinction to the data returned uncertain results in Experiment 1 but implicated both generalization decrement and response–reinforcer contingency termination as the possible mechanisms responsible for behavior change in Experiment 2. Thus, these data suggest that experimental manipulations that affect discrimination of changes in reinforcement contingencies may influence resistance to extinction by modulating the disruptive impacts of removing reinforcers from the experimental context and of suspending response–reinforcer contingencies.     

Sensitivity to relative reinforcer rate in concurrent schedules: independence from relative and absolute reinforcer duration

Twelve pigeons responded on two keys under concurrent variable-interval (VI) schedules. Over several series of conditions, relative and absolute magnitudes of reinforcement were varied. Within each series, relative rate of reinforcement was varied and sensitivity of behavior ratios to reinforcer-rate ratios was assessed. When responding at both alternatives was maintained by equal-sized small reinforcers, sensitivity to variation in reinforcer-rate ratios was the same as when large reinforcers were used. This result was observed when the overall rate of reinforcement was constant over conditions, and also in another series of concurrent schedules in which one schedule was kept constant at VI ached 120 s. Similarly, reinforcer magnitude did not affect the rate at which response allocation approached asymptote within a condition. When reinforcer magnitudes differred between the two responses and reinforcer-rate ratios were varied, sensitivity of behavior allocation was unaffected although response bias favored the schedule that arranged the larger reinforcers. Analysis of absolute response rates ratio sensitivity to reinforcement occurrred on the two keys showed that this invariance of response despite changes in reinforcement interaction that were observed in absolute response rates on the constant VI 120-s schedule. Response rate on the constant VI 120-s schedule was inversely related to reinforcer rate on the varied key and the strength of this relation depended on the relative magnitude of reinforcers arranged on varied key. Independence of sensitivity to reinforcer-rate ratios from relative and absolute reinforcer magnitude is consistent with the relativity and independence assumtions of the matching law.     

Contrast effects in response rate and accuracy of delayed matching to sample

Behavioural contrast is an inverse relation between the response rate in one component of a multiple schedule and the reinforcer rate in an alternated component. To explore possible contrast effects in accuracy as well as response rate, four pigeons were trained in multiple schedules where key pecking produced delayed matching-to-sample trials on a variable-interval schedule. Reinforcer probability for correct matches was constant at .3 in one component, and the conditions of reinforcement were varied in the second component. In Experiment 1, the varied component arranged the same contingencies as the constant component but with reinforcer probabilities of .9 or .1 across conditions. In the varied component, both response rate and accuracy of delayed matching were directly related to reinforcer probability; in the constant component, however, contrast effects on response rate were weak, and there was no evidence of contrast in accuracy of matching. In Experiment 2, the varied component was either variable interval with immediate food reinforcement or extinction. Reliable contrast effects were obtained in both response rate and in accuracy of matching in the constant component, and their magnitudes were correlated within and between subjects. The results of Experiment 2 join previous findings of covariation in the effects of reinforcement on free-operant responding and accuracy of discrimination.     

Response–reinforcer dependency and resistance to change

The effects of the response–reinforcer dependency on resistance to change were studied in three experiments with rats. In Experiment 1, lever pressing produced reinforcers at similar rates after variable interreinforcer intervals in each component of a two‐component multiple schedule. Across conditions, in the fixed component, all reinforcers were response‐dependent; in the alternative component, the percentage of response‐dependent reinforcers was 100, 50 (i.e., 50% response‐dependent and 50% response‐independent) or 10% (i.e., 10% response‐dependent and 90% response‐independent). Resistance to extinction was greater in the alternative than in the fixed component when the dependency in the former was 10%, but was similar between components when this dependency was 100 or 50%. In Experiment 2, a three‐component multiple schedule was used. The dependency was 100% in one component and 10% in the other two. The 10% components differed on how reinforcers were programmed. In one component, as in Experiment 1, a reinforcer had to be collected before the scheduling of other response‐dependent or independent reinforcers. In the other component, response‐dependent and ‐independent reinforcers were programmed by superimposing a variable‐time schedule on an independent variable‐interval schedule. Regardless of the procedure used to program the dependency, resistance to extinction was greater in the 10% components than in the 100% component. These results were replicated in Experiment 3 in which, instead of extinction, VT schedules replaced the baseline schedules in each multiple‐schedule component during the test. We argue that the relative change in dependency from Baseline to Test, which is greater when baseline dependencies are high rather than low, could account for the differential resistance to change in the present experiments. The inconsistencies in results across the present and previous experiments suggest that the effects of dependency on resistance to change are not well understood. Additional systematic analyses are important to further understand the effects of the response–reinforcer relation on resistance to change and to the development of a more comprehensive theory of behavioral persistence.     

Contrast and reallocation of extraneous reinforcers between multiple-schedule components

Four pigeons responded in components of multiple schedules in which two responses were available and reinforced with food. Pecks on the left key (“main” key) were reinforced at a constant rate in one component and at a rate that varied over conditions in the other component. When reinforcer rate was varied, behavioral contrast occurred in the constant component. On the right key (“extra” key), five variable-interval schedules and one variable-ratio schedule, presented conjointly, arranged reinforcers for responses in all conditions. These conjoint schedules were common to both multiple-schedule components—rather than unique to particular components—and reinforcers from these schedules could therefore be arranged in one component and obtained during the other component. In this way, the additional reinforcers were analogous to the “extraneous” reinforcers thought to maintain behavior other than pecking in conventional multiple schedules. Response rate on the extra key did not change systematically over conditions in the constant component, and in the varied component extra responding was inversely related to main-key reinforcement. All subjects obtained more extra-key reinforcers in whichever component arranged fewer main-key reinforcers. Consistent with the theory that reallocation of extraneous reinforcers may cause behavioral contrast, absolute reinforcer rate for the extra key in the constant component was low in conditions that produced positive contrast on the main key and high in those that produced negative contrast. Also consistent with this theory, behavioral contrast was reduced in two conditions that canceled extra-key reinforcers that had been arranged but not obtained at the end of components. Thus, a constraint on reallocation markedly reduced the extent of contrast.     

Reinforcer-ratio variation and its effects on rate of adaptation

Six pigeons were trained in sessions that consisted of six or seven concurrent-schedule components, each of which could have a different reinforcer ratio arranged in it. The components were unsignaled and occurred in a random order separated by 10-s blackouts. The overall reinforcer rate arranged in each component was 2.22 reinforcers per minute. In Experiment 1, the range of reinforcer ratios in the seven components was varied from a condition in which the ratios were always 1:1, to a condition in which the ratios varied between concurrent variable-interval 27 s extinction (EXT) and concurrent extinction variable-interval 27 s (ratios of 1:EXT, 9:1, 3:1, 1:1, 1:3, 1:9, EXT:1). In Experiment 2, the range of reinforcer ratios was always 27:1 to 1:27, and the presence and absence of the intermediate reinforcer ratios used in Experiment 1 (9:1, 3:1, 1:1, 1:3, 1:9) were investigated. Log response-allocation ratios in components changed rapidly with increasing numbers of reinforcers in components, and Experiment 1 showed that sensitivity to reinforcement was usually higher when the range of reinforcer ratios was greater. When the range of reinforcer ratios was kept constant in Experiment 2, the presence or absence of less extreme reinforcer ratios had no clear effect on sensitivity. At a local level, individual reinforcers had predictable quantitative effects on response ratios: Successive same-alternative reinforcers in a component had rapidly diminishing effects in both experiments. Reinforcers obtained on the opposite alternative to one or more prior reinforcers always had large effects on preference, and these changes were greater when the range of reinforcer ratios was greater. The effects of such reinforcers in changing preference were enhanced, and produced clear preference reversals, when intermediate reinforcer ratios were absent in Experiment 2. Two processes, one local to reinforcers and one with a longer time course, may be necessary to account for these results.     

Arousal, changeover responses, and preference in concurrent schedules

Pigeons were trained on multiple schedules that provided concurrent reinforcement in each of two components. In Experiment 1, one component consisted of a variable-interval (VI) 40-s schedule presented with a VI 20-s schedule, and the other a VI 40-s schedule presented with a VI 80-s schedule. After extended training, probe tests measured preference between the stimuli associated with the two 40-s schedules. Probe tests replicated the results of Belke (1992) that showed preference for the 40-s schedule that had been paired with the 80-s schedule. In a second condition, the overall reinforcer rate provided by the two components was equated by adding a signaled VI schedule to the component with the lower reinforcer rate. Probe results were unchanged. In Experiment 2, pigeons were trained on alternating concurrent VI 30-s VI 60-s schedules. One schedule provided 2-s access to food and the other provided 6-s access. The larger reinforcer magnitude produced higher response rates and was preferred on probe trials. Rate of changeover responding, however, did not differ as a function of reinforcer magnitude. The present results demonstrate that preference on probe trials is not a simple reflection of the pattern of changeover behavior established during training.     

Sensitivity to reinforcer duration in a self-control procedure

In a concurrent-chains procedure, pigeons' responses on left and right keys were followed by reinforcers of different durations at different delays following the choice responses. Three pairs of reinforcer delays were arranged in each session, and reinforcer durations were varied over conditions. In Experiment 1 reinforcer delays were unequal, and in Experiment 2 reinforcer delays were equal. In Experiment 1 preference reversal was demonstrated in that an immediate short reinforcer was chosen more frequently than a longer reinforcer delayed 6 s from the choice, whereas the longer reinforcer was chosen more frequently when delays to both reinforcers were lengthened. In both experiments, choice responding was more sensitive to variations in reinforcer duration at overall longer reinforcer delays than at overall shorter reinforcer delays, independently of whether fixed-interval or variable-interval schedules were arranged in the choice phase. We concluded that preference reversal results from a change in sensitivity of choice responding to ratios of reinforcer duration as the delays to both reinforcers are lengthened.     

Additional free reinforcers increase persistence of problem behavior in a clinical context: A partial replication of laboratory findings

Behavioral momentum theory is a quantitative framework used to characterize the persistence of behavior during response disruptors as a function of baseline stimulus–reinforcer relations. Results of several investigations have shown that alternative reinforcement can increase the resistance to change of a target response during extinction. In the present study, concomitant variable‐interval fixed‐time schedules of reinforcement for problem behavior were employed to simulate naturalistic situations involving the superimposition of response‐independent reinforcers on a baseline schedule of reinforcement for problem behavior, as in the common use of noncontingent reinforcement treatments. Resistance to change of problem behavior was assessed during postsession periods of extinction by comparing response rates in extinction following sessions with and without additional reinforcer deliveries arranged by fixed‐time schedules. For 2 out of 3 participants, problem behavior tended to be more resistant to extinction following periods in which additional fixed‐time reinforcers were delivered. These results are discussed in terms of potential effects of noncontingent reinforcement on problem behavior when the intervention is discontinued or implemented without good treatment integrity.     

Effects of signaled and unsignaled alternative reinforcement on persistence and relapse in children and pigeons

Three experiments explored the impact of different reinforcer rates for alternative behavior (DRA) on the suppression and post‐DRA relapse of target behavior, and the persistence of alternative behavior. All experiments arranged baseline, intervention with extinction of target behavior concurrently with DRA, and post‐treatment tests of resurgence or reinstatement, in two‐ or three‐component multiple schedules. Experiment 1, with pigeons, arranged high or low baseline reinforcer rates; both rich and lean DRA schedules reduced target behavior to low levels. When DRA was discontinued, the magnitude of relapse depended on both baseline reinforcer rate and the rate of DRA. Experiment 2, with children exhibiting problem behaviors, arranged an intermediate baseline reinforcer rate and rich or lean signaled DRA. During treatment, both rich and lean DRA rapidly reduced problem behavior to low levels, but post‐treatment relapse was generally greater in the DRA‐rich than the DRA‐lean component. Experiment 3, with pigeons, repeated the low‐baseline condition of Experiment 1 with signaled DRA as in Experiment 2. Target behavior decreased to intermediate levels in both DRA‐rich and DRA‐lean components. Relapse, when it occurred, was directly related to DRA reinforcer rate as in Experiment 2. The post‐treatment persistence of alternative behavior was greater in the DRA‐rich component in Experiment 1, whereas it was the same or greater in the signaled‐DRA‐lean component in Experiments 2 and 3. Thus, infrequent signaled DRA may be optimal for effective clinical treatment.     

Choice in a variable environment: effects of blackout duration and extinction between components

Pigeons were trained in a procedure in which sessions included seven four- or 10-reinforcer components, each providing a different reinforcer ratio that ranged from 27:1 to 1:27. The components were arranged in random order, and no signals differentiated the component reinforcer ratios. Each condition lasted 50 sessions, and the data from the last 35 sessions were analyzed. Previous results using 10-s blackouts between components showed some carryover of preference from one component to the next, and this effect was investigated in Experiment 1 by varying blackout duration from 1 s to 120 s. The amount of carryover decreased monotonically as the blackout duration was lengthened. Preference also decreased between reinforcers within components, suggesting that preference change during blackout might follow the same function as preference change between reinforcers. Experiment 2 was designed to measure preference change between components more directly and to relate this to preference change during blackout. In two conditions a 60-s blackout occurred between components, and in two other conditions a 60-s period of unsignaled extinction occurred between components. Preference during the extinction period progressively fell toward indifference, and the level of preference following extinction was much the same as that following blackout. Although these results are consistent with Davison and Baum's (2000) theory of the effects of reinforcers on local preference, other findings suggest that theory is incomplete: After a sequence of reinforcers from one alternative, some residual preference remained after 60 s of extinction or blackout, indicating the possibility of an additional longer term accumulation of reinforcer effects than originally suggested.     

Nonstable concurrent choice in pigeons

Six pigeons were trained on concurrent variable-interval schedules in which the arranged reinforcer ratios changed from session to session according to a 31-step pseudorandom binary sequence. This procedure allows a quantitative analysis of the degree to which performance in an experimental session is affected by conditions in previous sessions. Two experiments were carried out. In each, the size of the reinforcer ratios arranged between the two concurrent schedules was varied between 31-step conditions. In Experiment 1, the concurrent schedules were arranged independently, and in Experiment 2 they were arranged nonindependently. An extended form of the generalized matching law described the relative contribution of past and present events to present-session behavior. Total performance in sessions was mostly determined by the reinforcer ratio in that session and partially by reinforcers that had been obtained in previous sessions. However, the initial exposure to the random sequence produced a lower sensitivity to current-session reinforcers but no difference in overall sensitivity to reinforcement. There was no evidence that the size of the reinforcer ratios available on the concurrent schedules affected either overall sensitivity to reinforcement or the sensitivity to reinforcement in the current session. There was also no evidence of any different performance between independent and nonindependent scheduling. Because of these invariances, this experiment validates the use of the pseudorandom sequence for the fast determination of sensitivity to reinforcement.     

Does the context of reinforcement affect resistance to change?

Eight pigeons were trained on multiple schedules of reinforcement where pairs of components alternated in blocks on different keys to define 2 local contexts. On 1 key, components arranged 160 and 40 reinforcers/hr; on the other, components arranged 40 and 10 reinforcers/hr. Response rates in the 40/hr component were higher in the latter pair. Within pairs, resistance to prefeeding and resistance to extinction were generally greater in the richer component. The two 40/hr components did not differ in resistance to prefeeding, but the 40/hr component that alternated with 10/hr was more resistant to extinction. This discrepancy was interpreted by an algebraic model relating response strength to component reinforcer rate, including generalization decrement. According to this model, strength is independent of context, consistent with research on schedule preference.     

Early extinction effects following intermittent reinforcement: Little evidence of extinction bursts

The occurrence of extinction bursts—transient increases in response rate in excess of those observed in baseline during the period immediately following discontinuation of reinforcement of a response—was examined. In Experiment 1, key pecking of pigeons was reinforced according to a multiple schedule in which a variable-ratio schedule alternated with an interval schedule in which the reinforcers were yoked to the preceding variable-ratio component. In Experiments 2 and 3, rats were screened such that the lever-press response rates of different rats maintained by variable-interval schedules were either relatively high or relatively low. Following these baseline conditions, in Experiments 1 and 2 responding was extinguished by eliminating the food reinforcer and in Experiment 3 by removing the response–reinforcer dependency. Responses immediately following extinction implementation were examined. Response increases relative to baseline during the first 20 min of a 324.75-min extinction session (Experiment 1) or during the first 30-min extinction session (Experiments 2 and 3) were rare and unsystematic. The results (a) reinforce earlier meta-analyses concluding that extinction bursts may be a less ubiquitous early effect of extinction than has been suggested and (b) invite further experimentation to establish their generality as a function of preceding reinforcement conditions.     

Variable-ratio versus variable-interval schedules: response rate, resistance to change, and preference

Two experiments asked whether resistance to change depended on variable-ratio as opposed to variable-interval contingencies of reinforcement and the different response rates they establish. In Experiment 1, pigeons were trained on multiple random-ratio random-interval schedules with equated reinforcer rates. Baseline response rates were disrupted by intercomponent food, extinction, and prefeeding. Resistance to change relative to baseline was greater in the interval component, and the difference was correlated with the extent to which baseline response rates were higher in the ratio component. In Experiment 2, pigeons were trained on multiple variable-ratio variable-interval schedules in one half of each session and on concurrent chains in the other half in which the terminal links corresponded to the multiple-schedule components. The schedules were varied over six conditions, including two with equated reinforcer rates. In concurrent chains, preference strongly overmatched the ratio of obtained reinforcer rates. In multiple schedules, relative resistance to response-independent food during intercomponent intervals, extinction, and intercomponent food plus extinction depended on the ratio of obtained reinforcer rates but was less sensitive than was preference. When reinforcer rates were similar, both preference and relative resistance were greater for the variable-interval schedule, and the differences were correlated with the extent to which baseline response rates were higher on the variable-ratio schedule, confirming the results of Experiment 1. These results demonstrate that resistance to change and preference depend in part on response rate as well as obtained reinforcer rate, and challenge the independence of resistance to change and preference with respect to response rate proposed by behavioral momentum theory.     

Every reinforcer counts: reinforcer magnitude and local preference

Six pigeons were trained on concurrent variable-interval schedules. Sessions consisted of seven components, each lasting 10 reinforcers, with the conditions of reinforcement differing between components. The component sequence was randomly selected without replacement. In Experiment 1, the concurrent-schedule reinforcer ratios in components were all equal to 1.0, but across components reinforcer-magnitude ratios varied from 1:7 through 7:1. Three different overall reinforcer rates were arranged across conditions. In Experiment 2, the reinforcer-rate ratios varied across components from 27:1 to 1:27, and the reinforcer-magnitude ratios for each alternative were changed across conditions from 1:7 to 7:1. The results of Experiment 1 replicated the results for changing reinforcer-rate ratios across components reported by Davison and Baum (2000, 2002): Sensitivity to reinforcer-magnitude ratios increased with increasing numbers of reinforcers in components. Sensitivity to magnitude ratio, however, fell short of sensitivity to reinforcer-rate ratio. The degree of carryover from component to component depended on the reinforcer rate. Larger reinforcers produced larger and longer postreinforcer preference pulses than did smaller reinforcers. Similar results were found in Experiment 2, except that sensitivity to reinforcer magnitude was considerably higher and was greater for magnitudes that differed more from one another. Visit durations following reinforcers measured either as number of responses emitted or time spent responding before a changeover were longer following larger than following smaller reinforcers, and were longer following sequences of same reinforcers than following other sequences. The results add to the growing body of research that informs model building at local levels.     

Resistance to extinction versus extinction as discrimination

The hypothesis that response strength might be measured by persistence of responding in the face of extinction was discredited in the 1960s because experiments showed that responding persists longer following intermittent reinforcers than following continuous reinforcers. Instead, researchers proposed that the longer persistence following intermittent reinforcers arises because intermittent reinforcement more closely resembles extinction—a discrimination theory. Attention to resistance to extinction revived because one observation seemed to support the persistence hypothesis: Following training on a multiple schedule with unequal components, responding usually persisted longer in the formerly richer component than in the formerly lean component. This observation represents an anomaly, however, because results with single schedules and concurrent schedules contradict it. We suggest that the difference in results arises because the multiple-schedule procedure, while including extensive training on stimulus discrimination, includes no training on discrimination between food available and food unavailable, whereas comparable single- and concurrent-schedule procedures include such training with repeated extinction. In Experiment 1, we replicated the original result, and in Experiment 2 showed that when the multiple-schedule procedure includes training on food/no-food discrimination, extinction following multiple schedules contradicts behavioral momentum theory and agrees with the discrimination theory and research with single and concurrent schedules.     

A theory of attending, remembering, and reinforcement in delayed matching to sample

A theory of attending and reinforcement in conditional discriminations is extended to working memory in delayed matching to sample by adding terms for disruption of attending during the retention interval. Like its predecessor, the theory assumes that reinforcers and disruptors affect the independent probabilities of attending to sample and comparison stimuli in the same way as the rate of overt free-operant responding as suggested by Nevin and Grace, and that attending is translated into discriminative performance by the model of Davison and Nevin. The theory accounts for the effects of sample-stimulus discriminability and retention-interval disruption on the levels and slopes of forgetting functions, and for the diverse relations between accuracy and sensitivity to reinforcement reported in the literature. It also accounts for the effects of reinforcer probability in multiple schedules on the levels and resistance to change of forgetting functions; for the effects of reinforcer probabilities signaled within delayed-matching trials; and for the effects of reinforcer delay, sample duration, and intertrial-interval duration. The model accounts for some data that have been problematic for previous theories, and makes testably different predictions of the effects of reinforcer probabilities and disruptors on forgetting functions in multiple schedules and signaled trials.