Variable-ratio versus variable-interval schedules: response rate, resistance to change, and preference

Two experiments asked whether resistance to change depended on variable-ratio as opposed to variable-interval contingencies of reinforcement and the different response rates they establish. In Experiment 1, pigeons were trained on multiple random-ratio random-interval schedules with equated reinforcer rates. Baseline response rates were disrupted by intercomponent food, extinction, and prefeeding. Resistance to change relative to baseline was greater in the interval component, and the difference was correlated with the extent to which baseline response rates were higher in the ratio component. In Experiment 2, pigeons were trained on multiple variable-ratio variable-interval schedules in one half of each session and on concurrent chains in the other half in which the terminal links corresponded to the multiple-schedule components. The schedules were varied over six conditions, including two with equated reinforcer rates. In concurrent chains, preference strongly overmatched the ratio of obtained reinforcer rates. In multiple schedules, relative resistance to response-independent food during intercomponent intervals, extinction, and intercomponent food plus extinction depended on the ratio of obtained reinforcer rates but was less sensitive than was preference. When reinforcer rates were similar, both preference and relative resistance were greater for the variable-interval schedule, and the differences were correlated with the extent to which baseline response rates were higher on the variable-ratio schedule, confirming the results of Experiment 1. These results demonstrate that resistance to change and preference depend in part on response rate as well as obtained reinforcer rate, and challenge the independence of resistance to change and preference with respect to response rate proposed by behavioral momentum theory.     

Preference,resistance to change,and the cumulative decision model

According to behavioral momentum theory (Nevin & Grace, 2000a), preference in concurrent chains and resistance to change in multiple schedules are independent measures of a common construct representing reinforcement history. Here I review the original studies on preference and resistance to change in which reinforcement variables were manipulated parametrically, conducted by Nevin, Grace and colleagues between 1997 and 2002, as well as more recent research. The cumulative decision model proposed by Grace and colleagues for concurrent chains is shown to provide a good account of both preference and resistance to change, and is able to predict the increased sensitivity to reinforcer rate and magnitude observed with constant‐duration components. Residuals from fits of the cumulative decision model to preference and resistance to change data were positively correlated, supporting the prediction of behavioral momentum theory. Although some questions remain, the learning process assumed by the cumulative decision model, in which outcomes are compared against a criterion that represents the average outcome value in the current context, may provide a plausible model for the acquisition of differential resistance to change.     

Preference and resistance to change with constant- and variable-duration terminal links: independence of reinforcement rate and magnitude

Pigeons responded in a three-component multiple concurrent-chains procedure in which the variable-interval reinforcement schedules were the same across components but magnitudes differed across components. The terminal links were arranged either as a variable delay followed by presentation of a reinforcer ("variable duration") or as a fixed period of access to the schedule during which a variable number of reinforcers could be earned ("constant duration"). Relative reinforcement rate was varied parametrically across both types of conditions. After baseline training in each condition, resistance to change of terminal-link responding was assessed by delivering food during the initial links according to a variable-time schedule. Both preference and resistance to change were more sensitive to reinforcement-rate differences in the constant-duration conditions. Sensitivities of preference and resistance to change to relative reinforcement rate did not change depending on relative reinforcement magnitude. Taken together, these results confirm and extend those of prior studies, and suggest that reinforcement rate and magnitude combine additively to determine preference and resistance to change. A single structural relation linking preference and resistance to change describes all the data from this and several related studies.     

Resistance to change and preference for variable versus fixed response sequences

In Experiment 1, 4 pigeons were trained on a multiple chain schedule in which the initial link was a variable-interval (VI) 20-s schedule signalled by a red or green center key, and terminal links required four responses made to the left (L) and/or right (R) keys. In the REPEAT component, signalled by red keylights, only LRLR terminal-link response sequences were reinforced, while in the VARY component, signalled by green keylights, terminal-link response sequences were reinforced if they satisfied a variability criterion. The reinforcer rate for both components was equated by adjusting the reinforcer probability for correct REPEAT sequences across sessions. Results showed that initial- and terminal-link responding in the VARY component was generally more resistant to prefeeding, extinction, and response-independent food than responding in the REPEAT component. In Experiment 2, the REPEAT and VARY contingencies were arranged as terminal links of a concurrent chain and the relative reinforcer rate was manipulated across conditions. For all pigeons, initial-link response allocation was biased toward the alternative associated with the VARY terminal link. These results replicate previous reports that operant variation is more resistant to change than operant repetition (Doughty & Lattal, 2001), and show that variation is preferred to repetition with reinforcer-related variables controlled. Behavioral momentum theory (Nevin & Grace, 2000) predicts the covariation of preference and resistance to change in Experiments 1 and 2, but does not explain why these aspects of behavior should depend on contingencies that require repetition or variation.     

On the relation between preference and resistance to change

Nevin (1979) noted that preference in concurrent chains and resistance to change in multiple schedules were correlated, in that both measures were affected similarly by variations in parameters of reinforcement such as rate, immediacy, and magnitude. To investigate the relationship between preference and resistance to change directly, we used a within-session procedure that arranged concurrent chains in one half of the session and a multiple schedule in the other half. The same variable-interval schedules served as terminal links in concurrent chains and as the components of the multiple schedule, and were signaled by the same stimuli. After performances had stabilized, responding in the multiple schedule was disrupted by delivering response-independent reinforcement during the blackout periods between components. Both preference in concurrent chains and relative resistance to change of multiple-schedule responding were well described as power functions of relative reinforcement rate, as predicted by current quantitative models (Grace, 1994; Nevin, 1992b). In addition, unsystematic variation in preference and resistance to change was positively correlated, which suggests that preference and resistance to change are independent measures of a single construct. That construct could be described as the learning that occurs regarding the prevailing conditions of reinforcement in a distinctive stimulus situation.     

Choice and amount of reinforcement in rats

Rats were exposed to concurrent-chains schedules in which the terminal links were equal, fixed-interval (FI) schedules terminating in one or a varying number of food pellets. In most rats, choice proportions for the larger reinforcer increased with increases in reinforcer amount (e.g., from one to five food pellets). When log response ratios were plotted against log reinforcer amount ratios, the results indicated that the effects of reinforcer amount depended on the length of fixed-interval terminal links, by showing that rats undermatched their response ratios to reinforcer amount ratios with the shorter terminal links (FI 5 s, Experiment 1), whereas they overmatched with the longer terminal links (FI 20 s, Experiment 2). These results demonstrated that the manipulation of FI terminal-link schedules affected the sensitivity of choice to reinforcer amount, and are consistent with the previous findings that choice proportions for the larger of two reinforcers (one vs three food pellets) increased with increases in the length of FI terminal-link schedules.     

Comparing preference and resistance to change in constant- and variable-duration schedule components

Two experiments explored preference and resistance to change in concurrent chains in which the terminal links were variable-interval schedules that ended either after a single reinforcer had been delivered (variable duration) or after a fixed period of access to the schedule (constant duration). In Experiment 1, pigeons' preference between the same pair of terminal links overmatched relative reinforcement rate when the terminal links were of constant duration, but not when they were of variable duration. Responding during the richer terminal link decreased less, relative to baseline, when response-independent food was presented during the initial links according to a variable-time schedule. In Experiment 2, all subjects consistently preferred a terminal link that consisted of 20-s access to a variable-interval 20-s schedule over a terminal link that ended after one reinforcer had been delivered by the same schedule. Results of resistance-to-change tests corresponded to preference, as responding during the constant-duration terminal link decreased less, relative to baseline, when disrupted by both response-independent food during the initial links and prefeeding. Overall, these data extend the general covariation of preference and resistance to change seen in previous studies. However, they suggest that reinforcement numerosity, including variability in the number of reinforcers per terminal-link entry, may sometimes affect preference and resistance to change in ways that are difficult to explain in terms of current models.     

Response strength in multiple periodic and aperiodic schedules

Responding in multiple periodic and aperiodic schedules of equal mean reinforcement rate was examined during extinction, satiation, and in the presence of various free-food schedules. In Experiments I and II, pigeons were trained on multiple variable-interval–fixed-interval schedules. Decreases in the rate of responding due to extinction, satiation, or food schedules were approximately equal regardless of the temporal pattern of reinforcer presentation. In Experiment III, pigeons responded on a two-component multiple schedule in which each component was a two-member homogeneous response chain terminating in a fixed-interval schedule during one component and in a variable-interval schedule during the other. The length of both terminal links was varied over a series of conditions. Initial-link responding in the fixed-interval component was reduced more by increasing terminal-link length than was initial-link responding in the variable-interval component. However, no differences in resistance to satiation and extinction were obtained across the fixed and variable components. If the relative decrease in responding produced by satiation and extinction is used as an index of the “value” of the conditions maintaining responding, then these data suggest that fixed and variable schedules of equal mean length are equally valued. This conclusion, however, is not consistent with findings of preference for variable over fixed schedules obtained in studies using concurrent-chain procedures.     

Resistance To Change As A Function Of Stimulus-reinforcer And Location-reinforcer Contingencies

Pigeons responded on two keys in each component of a multiple concurrent schedule. In one series of conditions the distribution of reinforcers between keys within one component was varied so as to produce changes in ratios of reinforcer totals for key locations when summed across components. In a second series, reinforcer allocation between components was varied so as to produce changes in ratios of reinforcer totals for components, summed across key locations. In each condition, resistance to change was assessed by presenting response-independent reinforcers during intercomponent blackouts and (for the first series) by extinction of responding on both keys in both components. Resistance to change for response totals within a component was always greater for the component with the larger total reinforcer rate. However, resistance to change for response totals at a key location was not a positive function of total reinforcement for pecking that key; indeed, relative resistance to extinction for the two locations showed a weak negative relation to ratios of reinforcer totals for key location. These results confirm the determination of resistance to change by stimulus—reinforcer contingencies.     

Choice between single and multiple reinforcers in concurrent-chains schedules

Pigeons responded on concurrent-chains schedules with equal variable-interval schedules as initial links. One terminal link delivered a single reinforcer after a fixed delay, and the other terminal link delivered either three or five reinforcers, each preceded by a fixed delay. Some conditions included a postreinforcer delay after the single reinforcer to equate the total durations of the two terminal links, but other conditions did not include such a postreinforcer delay. With short initial links, preference for the single-reinforcer alternative decreased when a postreinforcer delay was present, but with long initial links, the postreinforcer delays had no significant effect on preference. In conditions with a postreinforcer delay, preference for the single-reinforcer alternative frequently switched from above 50% to below 50% as the initial links were lengthened. This pattern of results was consistent with delay-reduction theory (Squires & Fantino, 1971), but not with the contextual-choice model (Grace, 1994) or the hyperbolic value-added model (Mazur, 2001) as they have usually been applied. However, the hyperbolic value-added model could account for the results if its calculations were expanded to include reinforcers delivered in later terminal links. The implications of these findings for models of concurrent-chains performance are discussed.     

Resistance to change produced by access to fixed-delay versus variable-delay terminal links

Pigeons' responding was reinforced on a multiple schedule consisting of two two-link chain schedules presented in regular alternation. Responding in initial links (always variable-interval 60-s) produced a key-color change and access to a terminal link. The terminal link for one chain provided food after a fixed delay (fixed-interval or fixed-time); the terminal link for the other provided food after a variable delay (variable-interval or variable-time). The average duration of the terminal-link schedules was varied across conditions, but in every condition the arithmetic mean of the variable-delay terminal-link schedule was equal to the duration of the fixed delay. Response rates were higher in the initial links of the chains with the variable-delay terminal links. Response-decreasing operations (satiation, extinction) were used after performances reached asymptote. Response rates maintained by access to variable-delay terminal links tended to be more resistant to change than were rates maintained by access to fixed-delay terminal links. These results are consistent with the preference for variable- over fixed-interval terminal links observed with concurrent-chains schedules, suggesting (1) that immediacy of reinforcement influences the conditioned reinforcing potency of access to a terminal link and (2) that choice in concurrent chains and resistance of responding to change may be manifestations of the same effect of reinforcement.     

Conditioned reinforcement dynamics in three-link chained schedules.

In two experiments rats were trained on three-link concurrent-chains schedules of reinforcement. In Experiment 1, additional entries to one terminal link were added during one of the middle links to a baseline schedule that was otherwise equal for the two chains, and, depending on the condition, these additional terminal-link presentations ended either in food or in no food. When food occurred, preference was always in favor of the chain with the additional terminal-link presentations (which also entailed a higher rate of reinforcement). When no food occurred at the end of the additional terminal links, the outcome depended on the nature of the stimuli associated with these additional terminal links. When stimuli different from the reinforced baseline terminal links were used for the no-food terminal links, preference was against the choice alternative that led to the extra periods of extinction. When the same stimulus was used for the two kinds of terminal links, preference was near indifference, that is, significantly greater than when different stimuli were used. In Experiment 2, rats learned repeated reversals of a simultaneous discrimination under a three-link concurrent-chains schedule, in which the food or no-food choice outcomes were delayed until the end of the chain. Different conditions were defined by the point in the chain at which differential stimuli occurred. When the middle and terminal links provided no differential stimuli, discrimination was acquired more slowly than when differential stimuli occurred in both links. When differential stimuli occurred in the middle but not the terminal links, acquisition rates were intermediate. Both experiments together show that the effects of stimuli in a chain schedule are due partly to the time to food correlated with the stimuli and partly to the time to the next conditioned reinforcer in the sequence.     

A theory of attending, remembering, and reinforcement in delayed matching to sample

A theory of attending and reinforcement in conditional discriminations is extended to working memory in delayed matching to sample by adding terms for disruption of attending during the retention interval. Like its predecessor, the theory assumes that reinforcers and disruptors affect the independent probabilities of attending to sample and comparison stimuli in the same way as the rate of overt free-operant responding as suggested by Nevin and Grace, and that attending is translated into discriminative performance by the model of Davison and Nevin. The theory accounts for the effects of sample-stimulus discriminability and retention-interval disruption on the levels and slopes of forgetting functions, and for the diverse relations between accuracy and sensitivity to reinforcement reported in the literature. It also accounts for the effects of reinforcer probability in multiple schedules on the levels and resistance to change of forgetting functions; for the effects of reinforcer probabilities signaled within delayed-matching trials; and for the effects of reinforcer delay, sample duration, and intertrial-interval duration. The model accounts for some data that have been problematic for previous theories, and makes testably different predictions of the effects of reinforcer probabilities and disruptors on forgetting functions in multiple schedules and signaled trials.     

Preference for and effects of variable-as opposed to fixed-reinforcer duration

Pigeons were trained on multiple schedules in which a fixed number of pecks produced either a fixed or a variable period of access to food, the average variable-duration reinforcement equalling the fixed. Pecking rates were generally higher during the variable-duration component. Subsequent performance on concurrent schedules revealed an initial preference for variable-duration reinforcement for all subjects; for most subjects, this preference was sustained. For one subject, the average variable duration was gradually reduced to half the fixed duration: continued preference for the variable component resulted in a loss of up to 30% of available reinforcement time. A return to multiple schedules with unequal pay-off shifted the preference to the greater fixed duration, and this preference was maintained even when the variable duration was again raised to equal the fixed duration. For the remaining subjects, the initial variable-duration preference on concurrent schedules was gradually replaced by a side preference. When the range of variable durations was varied, keeping the average variable duration equal to the fixed, the occasional longer reinforcers sustained a preference for variable-reinforcer durations for three of the four subjects.     

An integrative model for the study of behavioral momentum.

Behavioral momentum is the product of response rate and resistance to change. The data on relative resistance to change are summarized for pigeons responding on single-key two-component multiple schedules, in the initial links of two-key multiple chained schedules, and in equivalent components of two-key serial schedules. For single-key procedures, the ratio of resistance to change in two schedule components is shown to depend on the ratio of reinforcer rates obtained in the presence of the component stimuli. For two-key procedures, the ratio of resistance to change in equivalent components is shown to depend on the ratio of reinforcer rates correlated with key locations. A model based on stimulus-reinforcer contingencies that combines the reinforcer rates in schedule components summed over key locations and reinforcer rates correlated with key locations summed over components, each expressed relative to the session average reinforcer rate, gives a good account of the data. An extension of the relative law of effect for multiple schedules fails to provide a complete account of resistance to change, but both approaches are needed for a comprehensive understanding of behavioral momentum.     

Resistance to change of operant variation and repetition

A multiple chained schedule was used to compare the relative resistance to change of variable and fixed four-peck response sequences in pigeons. In one terminal link, a response sequence produced food only if it occurred infrequently relative to 15 other response sequences (vary). In the other terminal link, a single response sequence produced food (repeat). Identical variable-interval schedules operated in the initial links. During baseline, lower response rates generally occurred in the vary initial link, and similar response and reinforcement rates occurred in each terminal link. Resistance of responding to prefeeding and three rates of response-independent food delivered during the intercomponent intervals then was compared between components. During each disruption condition, initial- and terminal-link response rates generally were more resistant in the vary component than in the repeat component. During the response-independent food conditions, terminal-link response rates were more resistant than initial-link response rates in each component, but this did not occur during prefeeding. Variation (in vary) and repetition (in repeat) both decreased during the response-independent food conditions in the respective components, but with relatively greater disruption in repeat. These results extend earlier findings demonstrating that operant variation is more resistant to disruption than is operant repetition and suggest that theories of response strength, such as behavioral momentum theory, must consider factors other than reinforcement rate. The implications of the results for understanding operant response classes are discussed.     

Temporal context, preference, and resistance to change

According to behavioral momentum theory, preference and relative resistance to change in concurrent-chains schedules are correlated and reflect the relative conditioned value of discriminative stimuli. In the present study, we explore the generality of this relation by manipulating the temporal context within a concurrent-chains procedure through changes in the duration of the initial links. Consistent with previous findings, preference for a richer terminal link was less extreme with longer initial links across three experiments with pigeons. In Experiment 1, relative resistance to change and preference were related inversely when responding was disrupted with response-independent food presentations during initial links, replicating a previous finding with rats. However, more food was presented with longer initial links, confounding the disrupter and initial-link duration. In Experiment 2, presession feeding was used instead and eliminated the negative relation between relative resistance to change and preference, but relative resistance to change was not sensitive to relative terminal-link reinforcement rates. In Experiment 3, with more extreme relative terminal-link reinforcement rates, increasing initial-link duration similarly decreased preference and relative resistance to change for the richer terminal link. Thus, when conditions of disruption are equal and assessed under the appropriate reinforcement conditions, changes in temporal context impact relative resistance to change and preference similarly.     

Experience with dynamic reinforcement rates decreases resistance to extinction

The ability of organisms to detect reinforcer‐rate changes in choice preparations is positively related to two factors: the magnitude of the change in rate and the frequency with which rates change. Gallistel (2012) suggested similar rate‐detection processes are responsible for decreases in responding during operant extinction. Although effects of magnitude of change in reinforcer rate on resistance to extinction are well known (e.g., the partial‐reinforcement‐extinction effect), effects of frequency of changes in rate prior to extinction are unknown. Thus, the present experiments examined whether frequency of changes in baseline reinforcer rates impacts resistance to extinction. Pigeons pecked keys for variable‐interval food under conditions where reinforcer rates were stable and where they changed within and between sessions. Overall reinforcer rates between conditions were controlled. In Experiment 1, resistance to extinction was lower following exposure to dynamic reinforcement schedules than to static schedules. Experiment 2 showed that resistance to presession feeding, a disruptor that should not involve change‐detection processes, was unaffected by baseline‐schedule dynamics. These findings are consistent with the suggestion that change detection contributes to extinction. We discuss implications of change‐detection processes for extinction of simple and discriminated operant behavior and relate these processes to the behavioral‐momentum based approach to understanding extinction.     

DOES SENSITIVITY TO MAGNITUDE DEPEND ON THE TEMPORAL DISTRIBUTION OF REINFORCEMENT?

Our research addressed the question of whether sensitivity to relative reinforcer magnitude in concurrent chains depends on the distribution of reinforcer delays when the terminal-link schedules are equal. In Experiment 1, 12 pigeons responded in a two-component procedure. In both components, the initial links were concurrent variable-interval 40-s variable-interval 40-s, and the terminal links were both 20-s interval schedules in which responses were reinforced by either 4-s of grain in one, or 2-s of grain in the other. The only difference between the components was whether the terminal-link schedules were fixed interval or variable intervals. For all subjects, the relative rate of responding in the initial links for the terminal link that produced the 4-s reinforcer was greater when the terminal links were fixed-interval schedules than when they were variable-interval schedules. This result is contrary to the prediction of Grace's (1994) contextual choice model, but is consistent with both Mazur's (2001) hyperbolic value-added model and Killeen's (1985) incentive theory. In Experiment 2, 4 pigeons responded in a concurrent-chains procedure in which 4-s or 2-s reinforcers were provided independently of responding according to equal fixed-time or mixed-time schedules. Preference for the 4-s reinforcer increased as the variability of the intervals comprising the mixed-time schedules was decreased. Generalized-matching sensitivity of initial-link response allocation to relative reinforcer magnitude was proportional to the geometric mean of the terminal-link delays.