All Behavior is choice: Revisiting an evolutionary theory's account of behavior on single schedules

The axiomatic principle that all behavior is choice was incorporated into a revised implementation of an evolutionary theory's account of behavior on single schedules. According to this implementation, target responding occurs in the context of background responding and reinforcement. In Phase 1 of the research, the target responding of artificial organisms (AOs) animated by the revised theory was found to be well described by an exponentiated hyperbola, the parameters of which varied as a function of the background reinforcement rate. In Phase 2, the effect of reinforcer magnitude on the target behavior of the AOs was studied. As in Phase 1, the AOs' behavior was well described by an exponentiated hyperbola, the parameters of which varied with both the target reinforcer magnitude and the background reinforcement rate. Evidence from experiments with live organisms was found to be consistent with the Phase-1 predictions of the revised theory. The Phase-2 predictions have not been tested. The revised implementation of the theory can be used to study the effects of superimposing punishment on single-schedule responding, and it may lead to the discovery of a function that relates response rate to both the rate and magnitude of reinforcement on single schedules.     

Reinforcer magnitude (sucrose concentration) and the matching law theory of response strength

This experiment investigated the relationship between reinforcer magnitude (sucrose concentration) and response rate. The purpose was to evaluate the behavior of two parameters of an equation that predicts absolute response rate as a function of reinforcement rate and two free parameters. According to Herrnstein's (1970) theory of reinforced behavior, one parameter of this "response-strength equation" measures the efficacy of the reinforcer maintaining responding and the other parameter measures motoric components of response rate, such as response duration. Seven rats served as subjects. Experimental sessions consisted of a series of five different variable-interval schedules of reinforcement, each in effect for 5 minutes. Within each session, obtained reinforcement rates varied over more than a 30-fold range, from about 20 per hour to 700 per hour. The reinforcer was sucrose solution, and, between sessions, its concentration was varied from 0.0 to 0.64 molar (0 to 21.9%). For sucrose concentrations of 0.16 to 0.64 m, response rate was a negatively accelerated function of reinforcement rate. Increases in sucrose concentration increased response rates maintained by low but not high reinforcement rates. This pattern of changes corresponds to a change in the reinforcement-efficacy parameter of the response-strength equation. In contrast, the motor-performance parameter did not change as a function of sucrose concentration. These findings are inconsistent with the results of a similar study (Bradshaw, Szabadi, & Bevan, 1978) but support Herrnstein's theory of reinforced behavior.     

Sensitivity to relative reinforcer rate in concurrent schedules: independence from relative and absolute reinforcer duration

Twelve pigeons responded on two keys under concurrent variable-interval (VI) schedules. Over several series of conditions, relative and absolute magnitudes of reinforcement were varied. Within each series, relative rate of reinforcement was varied and sensitivity of behavior ratios to reinforcer-rate ratios was assessed. When responding at both alternatives was maintained by equal-sized small reinforcers, sensitivity to variation in reinforcer-rate ratios was the same as when large reinforcers were used. This result was observed when the overall rate of reinforcement was constant over conditions, and also in another series of concurrent schedules in which one schedule was kept constant at VI ached 120 s. Similarly, reinforcer magnitude did not affect the rate at which response allocation approached asymptote within a condition. When reinforcer magnitudes differred between the two responses and reinforcer-rate ratios were varied, sensitivity of behavior allocation was unaffected although response bias favored the schedule that arranged the larger reinforcers. Analysis of absolute response rates ratio sensitivity to reinforcement occurrred on the two keys showed that this invariance of response despite changes in reinforcement interaction that were observed in absolute response rates on the constant VI 120-s schedule. Response rate on the constant VI 120-s schedule was inversely related to reinforcer rate on the varied key and the strength of this relation depended on the relative magnitude of reinforcers arranged on varied key. Independence of sensitivity to reinforcer-rate ratios from relative and absolute reinforcer magnitude is consistent with the relativity and independence assumtions of the matching law.     

Resistance to change and the law of effect

Three experiments using multiple schedules of reinforcement explored the implications of resistance-to-change findings for the response-reinforcer relation described by the law of effect, using both steady-state responding and responding recorded in the first few sessions of conditions. In Experiment 1, when response-independent reinforcement was increased during a third component, response rate in Components 1 and 2 decreased. This response-rate reduction was proportionately greater in a component in which reinforcer magnitude was small (2-s access to wheat) than in the component in which it was large (6-s access to wheat). However, when reinforcer rates in the two components were varied together in Experiments 2 and 3, response-rate change was the same regardless of the magnitude of reinforcers used in the two components, so that sensitivity of response rates to reinforcer rates (Experiment 2) and of response-rate ratios to reinforcer-rate ratios (Experiment 3) was unaffected by the magnitude of the reinforcers. Therefore, the principles determining resistance to change, described by behavioral momentum theory, seem not to apply when the source of behavior change is the variation of reinforcement contingencies that maintain the behavior. The use of extinction as a manipulation to study resistance to change is questioned.     

Behavioral variability in an evolutionary theory of behavior dynamics

McDowell’s evolutionary theory of behavior dynamics (McDowell, 2004) instantiates populations of behaviors (abstractly represented by integers) that evolve under the selection pressure of the environment in the form of positive reinforcement. Each generation gives rise to the next via low‐level Darwinian processes of selection, recombination, and mutation. The emergent patterns can be analyzed and compared to those produced by biological organisms. The purpose of this project was to explore the effects of high mutation rates on behavioral variability in environments that arranged different reinforcer rates and magnitudes. Behavioral variability increased with the rate of mutation. High reinforcer rates and magnitudes reduced these effects; low reinforcer rates and magnitudes augmented them. These results are in agreement with live‐organism research on behavioral variability. Various combinations of mutation rates, reinforcer rates, and reinforcer magnitudes produced similar high‐level outcomes (equifinality). These findings suggest that the independent variables that describe an experimental condition interact; that is, they do not influence behavior independently. These conclusions have implications for the interpretation of high levels of variability, mathematical undermatching, and the matching theory. The last part of the discussion centers on a potential biological counterpart for the rate of mutation, namely spontaneous fluctuations in the brain's default mode network.     

Experience with dynamic reinforcement rates decreases resistance to extinction

The ability of organisms to detect reinforcer‐rate changes in choice preparations is positively related to two factors: the magnitude of the change in rate and the frequency with which rates change. Gallistel (2012) suggested similar rate‐detection processes are responsible for decreases in responding during operant extinction. Although effects of magnitude of change in reinforcer rate on resistance to extinction are well known (e.g., the partial‐reinforcement‐extinction effect), effects of frequency of changes in rate prior to extinction are unknown. Thus, the present experiments examined whether frequency of changes in baseline reinforcer rates impacts resistance to extinction. Pigeons pecked keys for variable‐interval food under conditions where reinforcer rates were stable and where they changed within and between sessions. Overall reinforcer rates between conditions were controlled. In Experiment 1, resistance to extinction was lower following exposure to dynamic reinforcement schedules than to static schedules. Experiment 2 showed that resistance to presession feeding, a disruptor that should not involve change‐detection processes, was unaffected by baseline‐schedule dynamics. These findings are consistent with the suggestion that change detection contributes to extinction. We discuss implications of change‐detection processes for extinction of simple and discriminated operant behavior and relate these processes to the behavioral‐momentum based approach to understanding extinction.     

Response strength in extreme multiple schedules

Four pigeons were trained in a series of two-component multiple schedules. Reinforcers were scheduled with random-interval schedules. The ratio of arranged reinforcer rates in the two components was varied over 4 log units, a much wider range than previously studied. When performance appeared stable, prefeeding tests were conducted to assess resistance to change. Contrary to the generalized matching law, logarithms of response ratios in the two components were not a linear function of log reinforcer ratios, implying a failure of parameter invariance. Over a 2 log unit range, the function appeared linear and indicated undermatching, but in conditions with more extreme reinforcer ratios, approximate matching was observed. A model suggested by McLean (1991), originally for local contrast, predicts these changes in sensitivity to reinforcer ratios somewhat better than models by Herrnstein (1970) and by Williams and Wixted (1986). Prefeeding tests of resistance to change were conducted at each reinforcer ratio, and relative resistance to change was also a nonlinear function of log reinforcer ratios, again contrary to conclusions from previous work. Instead, the function suggests that resistance to change in a component may be determined partly by the rate of reinforcement and partly by the ratio of reinforcers to responses.     

Unit price as a useful metric in analyzing effects of reinforcer magnitude.

In this paper, we applied the behavioral-economic concept of unit price to the study of reinforcer magnitude in an attempt to provide a consistent account of the effects of reinforcer magnitude on behavior. Recent research in the experimental analysis of behavior and in behavioral pharmacology suggests that reinforcer magnitude interacts with the schedule of reinforcement to determine response rate and total consumption. The utility of the unit-price concept thus stems from its ability to quantify this interaction as a cost-benefit ratio (i.e., unit price = characteristics of the schedule of reinforcement divided by magnitude of reinforcement). Research employing the unit-price concept has shown that as unit price increases, a positively decelerating function exists for consumption (i.e., a function with an increasingly negative slope, when plotted on log coordinates) and a bitonic function exists for response rate. Based on these findings, the present analysis applied the unit-price concept to those studies of reinforcer magnitude and drug self-administration that examined the effects of reinforcer magnitude on response rate using simple schedules of reinforcement (e.g., fixed-ratio schedule). This resulted in three findings: (a) Reinforcer-magnitude manipulations and schedule manipulations interact in a manner that can be quantified in terms of unit price as benefit and cost factors, respectively; (b) different reinforcer-magnitude manipulations are functionally interchangeable as benefit factors in the unit-price ratio; and (c) these conclusions appear warranted despite the differences in reinforcers (food or drug), species (dogs, monkeys, or rats), and schedules (interval or ratio), and despite the fact that these studies were not designed for a unit-price analysis. In methodological terms, these results provide further evidence that employing the unit-price concept is a parsimonious method for examining the effects of reinforcer magnitude. In theoretical terms, these results suggest that a single process may underlie the effect of combined reinforcer-magnitude and schedule manipulations.     

Concurrent variable-interval variable-ratio schedules can provide only weak evidence for matching

Herrnstein and Heyman (1979) showed that when pigeons' pecking is reinforced on concurrent variable-interval variable-ratio schedules, (1) their behavior ratios match the ratio of the schedules' reinforcer frequencies, and (2) there is more responding on the variable interval. Since maximizing the reinforcement rate would require responding more on the variable ratio, these results were presented as establishing the primacy of matching over maximizing. In the present report, different ratios of behavior were simulated on a computer to see how they would affect reinforcement rates on these concurrent schedules. Over a wide range of experimenter-specified choice ratios, matching obtained — a result suggesting that changes in choice allocation produced changes in reinforcer frequencies that correspond to the matching outcome. Matching also occurred at arbitrarily selected choice ratios when reinforcement rates were algebraically determined by each schedule's reinforcement-feedback function. Additionally, three birds were exposed to concurrent variable-interval variable-ratio schedules contingent on key pecking in which hopper durations were varied in some conditions to produce experimenter-specified choice ratios. Matching generally obtained between choice ratios and reinforcer-frequency ratios at these different choice ratios. By suggesting that reinforcer frequencies track choice on this procedure, instead of vice versa, this outcome questions whether matching-as-outcome was due to matching-as-process in the Herrnstein and Heyman study.     

Effects of reinforcement rate and reinforcer magnitude on choice behavior of humans

Two experiments with human subjects investigated the effects of rate of reinforcement and reinforcer magnitude upon choice. In Experiment 1, each of five subjects responded on four concurrent variable-interval schedules. In contrast to previous studies using non-human organisms, relative response rate did not closely match relative rate of reinforcement. Discrepancies ranged from 0.03 to 0.43 (mean equal to 0.19). Similar discrepancies were found between relative amount of time spent responding on each schedule and the corresponding relative rates of reinforcement. In Experiment 2, in which reinforcer magnitude was varied for each of five subjects, similar discrepancies ranging from 0.05 to 0.50 (mean equal to 0.21), were found between relative response rate and relative proportion of reinforcers received. In both experiments, changeover rates were lower on the long-interval concurrent schedules than on the short-interval ones. The results suggest that simple application of previous generalizations regarding the effects of reinforcement rate and reinforcer magnitude on choice for variable-interval schedules does not accurately describe human behavior in a simple laboratory situation.     

Concurrent reinforcement schedules for problem behavior and appropriate behavior: experimental applications of the matching law

This study evaluated how children who exhibited functionally equivalent problem and appropriate behavior allocate responding to experimentally arranged reinforcer rates. Relative reinforcer rates were arranged on concurrent variable-interval schedules and effects on relative response rates were interpreted using the generalized matching equation. Results showed that relative rates of responding approximated relative rates of reinforcement. Finally, interventions for problem behavior were evaluated and differential reinforcement of alternative behavior and extinction procedures were implemented to increase appropriate behavior and decrease problem behavior. Practical considerations for the application of the generalized matching equation specific to severe problem behavior are discussed, including difficulties associated with defining a reinforced response, and obtaining steady state responding in clinical settings.     

Effects of qualitatively different reinforcers on the parameters of the response-strength equation.

This experiment examined the relationship between two qualitatively different reinforcers and the parameters of a quantitative model of reinforced responding, referred to as the response-strength equation or the Herrnstein equation. A group of rats was first food deprived and later water deprived. An 11.5% sucrose solution served as the reinforcer in the food-deprivation condition, and water was the reinforcer in the water-deprivation condition. Each experimental session consisted of a series of seven variable-interval schedules, providing reinforcement rates that varied between 20 and 1,200 reinforcers per hour. The response rates increased in a negatively accelerating function in a manner consistent with the response-strength equation. This equation has two fitted parameters, k and Re. According to one theory, the k parameter is a measure of motor performance, and Re is indicative of the relative reinforcement efficacy of the background uncontrollable sources of reinforcement in relation to the experimentally arranged reinforcer. In this study, k did not change as a result of the different reinforcers, but Re was significantly larger in the sucrose-reinforcement condition. These results are consistent with the interpretation that k and Re measure two independent and experimentally distinguishable parameters and provide further evidence that absolute response rate is a function of relative reinforcement rate, as implied by the derivation of the response-strength equation based on the matching law.     

A COMPUTATIONAL THEORY OF SELECTION BY CONSEQUENCES APPLIED TO CONCURRENT SCHEDULES

Virtual organisms animated by a computational theory of selection by consequences responded on symmetrical and asymmetrical concurrent schedules of reinforcement. The theory instantiated Darwinian principles of selection, reproduction, and mutation such that a population of potential behaviors evolved under the selection pressure exerted by reinforcement from the environment. The virtual organisms' steady‐state behavior was well described by the power function matching equation, and the parameters of the equation behaved in ways that were consistent with findings from experiments with live organisms. Together with previous research on single‐alternative schedules (McDowell, 2004; McDowell & Caron, 2007) these results indicate that the equations of matching theory are emergent properties of the evolutionary dynamics of selection by consequences.     

Rat choice in rapidly changing concurrent schedules

In two experiments, experimentally naïve rats were trained in concurrent variable‐interval schedules in which the reinforcer ratios changed daily according to a pseudorandom binary sequence. In Experiment 1, relative response rates showed clear sensitivity to current‐session reinforcer ratios, but not to previous sessions' reinforcer ratios. Within sessions, sensitivity to the current session's reinforcement rates increased steadily, and by session end, response ratios approached matching to the current‐session reinforcer ratios. Across sessions, sensitivity to the current session's reinforcer ratio decreased with continued exposure to the pseudorandom binary sequence, contrary to expectations based on previous studies demonstrating learning sets. Using a second group of naïve rats, Experiment 2 replicated the main results from Experiment 1 and showed that although there were increases over sessions in both changeover rate and response rate during the changeover delay, neither could explain the accompanying reductions in sensitivity. We consider the role of reinforcement history, showing that our results can be simulated using two separate representations, one local and one nonlocal, but a more complex approach will be needed to bring together these results and other history effects such as learning sets and spontaneous recovery.     

Zebrafish choice behavior is sensitive to reinforcer rate,immediacy, and magnitude ratios

Behavioral flexibility has, in part, been defined by choice behavior changing as a function of changes in reinforcer payoffs. We examined whether the generalized matching law quantitatively described changes in choice behavior in zebrafish when relative reinforcer rates, delays/immediacy, and magnitudes changed between two alternatives across conditions. Choice was sensitive to each of the three reinforcer properties. Sensitivity estimates to changes in relative reinforcer rates were greater when 2 variable-interval schedules were arranged independently between alternatives (Experiment 1a) than when a single schedule pseudorandomly arranged reinforcers between alternatives (Experiment 1b). Sensitivity estimates for changes in relative reinforcer immediacy (Experiment 2) and magnitude (Experiment 3) were similar but lower than estimates for reinforcer rates. These differences in sensitivity estimates are consistent with studies examining other species, suggesting flexibility in zebrafish choice behavior in the face of changes in payoff as described by the generalized matching law.     

Discrete-trial choice in pigeons: Effects of reinforcer magnitude

The preference of pigeons for large reinforcers which occasionally followed a response versus small reinforcers which invariably followed a response was studied in a discrete-trial situation. Two differently colored keys were associated with the two reinforcement alternatives, and preference was measured as the proportion of choice trials on which the key associated with uncertain reinforcement was pecked. A combination of choice and guidance trials insured that received distributions of reinforcement equalled the scheduled distributions. For five of six subjects, preference for the uncertain reinforcer appeared to be a linear function of the magnitude of the certain reinforcer. In addition, there was greater preference for the response alternative associated with uncertain reinforcement than would be expected on the basis of net reinforcer value.     

Temporal constraint on choice: Sensitivity and bias in multiple schedules

In multiple schedules of reinforcement, ratios of responses in successive components are relatively insensitive to ratios of obtained reinforcers. An analysis is proposed that attributes changes in absolute response rates to concurrent interactions between programmed reinforcement and extraneous reinforcement in other components. The analysis predicts that ratios of responses in successive components vary with reinforcer ratios, qualified by a term describing the reinforcement context, that is, programmed and extraneous reinforcers. Two main predictions from the analysis were confirmed in an experiment in which pigeons' responses were reinforced in the components of a multiple schedule and analog extraneous reinforcement was scheduled for an alternative response in each component. Sensitivity of response and time ratios to reinforcer ratios in the multiple schedules varied as a function of the rate of extraneous reinforcers. Bias towards responding in one component of the multiple schedule varied as an inverse function of the ratios of extraneous reinforcer rate in the two components. The data from this and previous studies of multiple-concurrent performance were accurately predicted by our analysis and supported our contention that the allocation of behavior in multiple-schedule components depends on the relative values of concurrently-available reinforcers within each component.     

Three predictions of the economic concept of unit price in a choice context

Economic theory makes three predictions about consumption and response output in a choice situation: (a) When plotted on logarithmic coordinates, total consumption (i.e., summed across concurrent sources of reinforcement) should be a positively decelerating function, and total response output should be a bitonic function of unit price increases; (b) total consumption and response output should be determined by the value of the unit price ratio, independent of its cost and benefit components; and (c) when a reinforcer is available at the same unit price across all sources of reinforcement, consumption should be equal between these sources. These predictions were assessed in human cigarette smokers who earned cigarette puffs in a two-choice situation at a range of unit prices. In some sessions, smokers chose between different amounts of puffs, both available at identical unit prices. Individual subjects' data supported the first two predictions but failed to support the third. Instead, at low unit prices, the relatively larger reinforcer (and larger response requirement) was preferred, whereas at high unit prices, the smaller reinforcer (and smaller response requirement) was preferred. An expansion of unit price is proposed in which handling costs and the discounted value of reinforcers available according to ratio schedules are incorporated.     

Relative reinforcer rates and magnitudes do not control concurrent choice independently

One assumption of the matching approach to choice is that different independent variables control choice independently of each other. We tested this assumption for reinforcer rate and magnitude in an extensive parametric experiment. Five pigeons responded for food reinforcement on switching-key concurrent variable-interval variable-interval schedules. Across conditions, the ratios of reinforcer rates and of reinforcer magnitudes on the two alternatives were both manipulated. Control by each independent variable, as measured by generalized-matching sensitivity, changed significantly with the ratio of the other independent variable. Analyses taking the model-comparison approach, which weighs improvement in goodness-of-fit against increasing number of free parameters, were inconclusive. These analyses compared a model assuming constant sensitivity to magnitude across all reinforcer-rate ratios with two alternative models. One of those alternatives allowed sensitivity to magnitude to vary freely across reinforcer-rate ratios, and was less efficient than the common-sensitivity model for all pigeons, according to the Schwarz-Bayes information criterion. The second alternative model constrained sensitivity to magnitude to be equal for pairs of reinforcer-rate ratios that deviated from unity by proportionately equal amounts but in opposite directions. This model was more efficient than the common-magnitude-sensitivity model for 2 of the pigeons, but not for the other 3. An analysis of variance, carried out independently of the generalized-matching analysis, also showed a significant interaction between the effects of reinforcer rate and reinforcer magnitude on choice. On balance, these results suggest that the assumption of independence inherent in the matching approach cannot be maintained. Relative reinforcer rates and magnitudes do not control choice independently.