Behavioral variability in an evolutionary theory of behavior dynamics

McDowell’s evolutionary theory of behavior dynamics (McDowell, 2004) instantiates populations of behaviors (abstractly represented by integers) that evolve under the selection pressure of the environment in the form of positive reinforcement. Each generation gives rise to the next via low‐level Darwinian processes of selection, recombination, and mutation. The emergent patterns can be analyzed and compared to those produced by biological organisms. The purpose of this project was to explore the effects of high mutation rates on behavioral variability in environments that arranged different reinforcer rates and magnitudes. Behavioral variability increased with the rate of mutation. High reinforcer rates and magnitudes reduced these effects; low reinforcer rates and magnitudes augmented them. These results are in agreement with live‐organism research on behavioral variability. Various combinations of mutation rates, reinforcer rates, and reinforcer magnitudes produced similar high‐level outcomes (equifinality). These findings suggest that the independent variables that describe an experimental condition interact; that is, they do not influence behavior independently. These conclusions have implications for the interpretation of high levels of variability, mathematical undermatching, and the matching theory. The last part of the discussion centers on a potential biological counterpart for the rate of mutation, namely spontaneous fluctuations in the brain's default mode network.     

A COLLATERAL EFFECT OF REWARD PREDICTED BY MATCHING THEORY

Matching theory describes a process by which organisms distribute their behavior between two or more concurrent schedules of reinforcement (Herrnstein, 1961). In an attempt to determine the generality of matching theory to applied settings, 2 students receiving special education were provided with academic response alternatives. Using a combined simultaneous treatments design and reversal design, unequal ratio schedules of reinforcement were varied across two academic responses. Findings indicated that both subjects allocated higher rates of responses to the richer schedule of reinforcement, although only one responded exclusively to the richer schedule. The present results lend support to a postulation that positive reinforcement may have undesirable collateral effects that are predicted by matching theory (Balsam & Bondy, 1983).     

Theoretical implications of quantitative properties of interval timing and probability estimation in mouse and rat

Although theoretical discussions typically assume that positive and negative reinforcement differ, the literature contains little unambiguous evidence that they produce differential behavioral effects. To test whether the two types of consequences control behavior differently, we pitted money‐gain positive reinforcement and money‐loss‐avoidance negative reinforcement, scheduled through identically programmed variable‐cycle schedules, against each other in concurrent schedules. Contingencies of response‐produced feedback, normally different in positive and negative reinforcement, were made symmetrical. Steeper matching slopes were produced compared to a baseline consisting of all positive reinforcement. This free‐operant differential outcomes effect supports the notion that that stimulus‐presentation positive reinforcement and stimulus‐elimination negative reinforcement are functionally “different.” However, a control experiment showed that the feedback asymmetry of more traditional positive and negative reinforcement schedules also is sufficient to create a “difference” when the type of consequence is held constant. We offer these findings as a small step in meeting the very large challenge of moving negative reinforcement theory beyond decades of relative quiescence.     

Momentary maximizing in concurrent schedules with a minimum interchangeover interval

Eight pigeons were trained on concurrent variable-interval variable-interval schedules with a minimum interchangeover time programmed as a consequence of changeovers. In Experiment 1 the reinforcement schedules remained constant while the minimum interchangeover time varied from 0 to 200 s. Relative response rates and relative time deviated from relative reinforcement rates toward indifference with long minimum interchangeover times. In Experiment 2 different reinforcement ratios were scheduled in successive experimental conditions with the minimum interchangeover time constant at 0, 2, 10, or 120 s. The exponent of the generalized matching equation was close to 1.0 when the minimum interchangeover time was 0 s (the typical procedure for concurrent schedules without a changeover delay) and decreased as that duration was increased. The data support the momentary maximizing theory and contradict molar maximizing theories and the melioration theory.     

WITHIN‐SESSION TRANSITIONS IN CHOICE: A STRUCTURAL AND QUANTITATIVE ANALYSIS

The present study used within‐session transitions between two concurrent schedules to evaluate choice in transition. Eight female Long‐Evans rats were trained to respond under concurrent schedules of reinforcement during experimental sessions that lasted 22 hr. The generalized matching equation was used to model steady‐state behavior at the end of each session, while transitional behavior that emerged following the change in reinforcement schedules was modeled using a logistic equation. The generalized matching and logistic equations were appropriate models for behavior generated during single‐session transitions. A local analysis of behavior on the two response alternatives during acquisition was used to determine the source of preference as revealed in response ratios. The number of “low‐response” visits, those containing three to five responses, remained stable. Preference ratios largely reflected a sharp increase in the number of visits with long response bouts on the rich alternative and a decrease in the number of such visits to the leaner alternative.     

QUANTIFICATION OF ETHANOL'S ANTIPUNISHMENT EFFECT IN HUMANS USING THE GENERALIZED MATCHING EQUATION

Increases in rates of punished behavior by the administration of anxiolytic drugs (called antipunishment effects) are well established in animals but not humans. The present study examined antipunishment effects of ethanol in humans using a choice procedure. The behavior of 5 participants was placed under six concurrent variable‐interval schedules of monetary reinforcement. In three of the six concurrent schedules, punishment, in the form of monetary loss, was superimposed on one alternative. Data were analyzed according to the generalized matching equation which distinguishes between bias (allocation of behavior beyond what matching to relative reinforcer densities would predict) and sensitivity to reinforcement (how well behavior tracks relative reinforcer densities). In addition, participants completed a pencil‐tapping test. Under placebo punishment conditions, all participants demonstrated low response rates and a bias against the alternative associated with punishment, despite a resultant loss of available reinforcers. Bias against the punished alternative was dose‐dependently reduced in participants shown to be most sensitive to ethanol (0.6, 1.2, and 1.8 g/kg) in measures of overall responding and on the pencil‐tapping test. No ethanol‐induced change in bias was noted when punishment was not imposed. Sensitivity to reinforcement also decreased for participants shown to be sensitive to ethanol. In addition to extending antipunishment effects to humans, these results also show that antipunishment effects can be quantified via the matching equation.     

Optimal behavior and concurrent variable interval schedules

Behavior maintained with 2-component concurrent variable interval schedules of reinforcement (CONC VIVI) is described well by the matching law. Deviations from matching behavior have been handled by adding free parameters to the matching law equation. With CONC VIVI schedules there are infinitely many solutions to the matching law equation at each value of the procedural parameters. However, at each value of the procedural parameters, only one combination of durations of intervals spent in each VI component (dwell times) yields the combined maximum reinforcement rate. The equations that yield the optimal dwell times solution for CONC VIVI schedules are mathematically incompatible with the matching law. Optimal performance and matching coincide only when the parameter values of the two VI components are equal. It seems reasonable to use optimal behavior to assess performance in these schedules. Researchers have not compared optimal and empirical performances in CONC VIVI possibly because the equations for optimal dwell times (ODT) can be solved only numerically. We present a table of ODT for a wide range of VIs and changeover delays. We also derive a function m that can be used to compare matching data and the matching behavior predictions of optimization. We prove that 0.5<m<1.003502, and we describe some of the more nteresting properties of the function.     

Choice, matching, and human behavior: A review of the literature

This review concerns human performance on concurrent schedules of reinforcement. Studies indicate that humans match relative behavior to relative rate of reinforcement. Herrnstein's proportional matching equation describes human performance but most studies do not evaluate the equation at the individual level. Baum's generalized matching equation has received strong support with humans as subjects. This equation permits the investigation of sources of deviation from ideal matching and a few studies have suggested variables which control such deviations in humans. While problems with instructional control are raised, the overall findings support the matching law as a principle of human choice.     

The law of effect and avoidance: a quantitative relationship between response rate and shock-frequency reduction

Two experiments were conducted to investigate the quantitative relationship between response rate and reinforcement frequency in single and multiple variable-interval avoidance schedules. Responses cancelled delivery of shocks that were scheduled by variable-interval schedules. When shock-frequency reduction was taken as the measure of reinforcement, the relationship between response rate and reinforcement frequency on single variable-interval avoidance schedules was accurately described by Herrnstein's (1970) equation for responding on single variable-interval schedules of positive reinforcement. On multiple variable-interval avoidance schedules with brief components, asymptotic relative response rate matched relative shock-frequency reduction. The results suggest that many interactions between response rates and shock-frequency reduction in avoidance can be understood within the framework of the generalized matching relation, as applied by Herrnstein (1970) to positive reinforcement.     

Falsification of matching theory's account of single-alternative responding: Herrnstein's k varies with sucrose concentration

Eight rats pressed levers for varying concentrations of sucrose in water under eight variable-interval schedules that specified a wide range of reinforcement rate. Herrnstein's (1970) hyperbolic equation described the relation between reinforcement and responding well. Although the y asymptote, k, of the hyperbola appeared roughly constant over conditions that approximated conditions used by Heyman and Monaghan (1994), k varied when lower concentration solutions were included. Advances in matching theory that reflect asymmetries between response alternatives and insensitive responding were incorporated into Herrnstein's equation. After fitting the modified equation to the data, Herrnstein's k also increased. The results suggest that variation in k can be detected under a sufficiently wide range of reinforcer magnitudes, and they also suggest that matching theory's account of response strength is false. The results support qualitative predictions made by linear system theory.     

Concurrent fixed-interval variable-ratio schedules and the matching relation

Five rats responded under concurrent fixed-interval variable-ratio schedules of food reinforcement. Fixed-interval values ranged from 50-seconds to 300-seconds and variable-ratio values ranged from 30 to 360; a five-second changeover delay was in effect throughout the experiment. The relations between reinforcement ratios obtained from the two schedules and the ratios of responses and time spent on the schedules were described by Baum's (1974) generalized matching equation. All subjects undermatched both response and time ratios to reinforcement ratios, and all subjects displayed systematic bias in favor of the variable-ratio schedules. Response ratios undermatched reinforcement ratios less than did time ratios, but response ratios produced greater bias than did time ratios for every subject and for the group as a whole. Local rates of responding were generally higher on the variable-ratio than on the fixed-interval schedules. When responding was maintained by both schedules, a period of no responding on either schedule immediately after fixed-interval reinforcement typically was followed by high-rate responding on the variable-ratio schedule. At short fixed-interval values, when a changeover to the fixed-interval schedule was made, responding usually continued until fixed-interval reinforcement was obtained; at longer values, a changeover back to the variable-ratio schedule usually occurred when fixed-interval reinforcement was not forthcoming within a few seconds, and responding then alternated between the two schedules every few seconds until fixed-interval reinforcement finally was obtained.     

All Behavior is choice: Revisiting an evolutionary theory's account of behavior on single schedules

The axiomatic principle that all behavior is choice was incorporated into a revised implementation of an evolutionary theory's account of behavior on single schedules. According to this implementation, target responding occurs in the context of background responding and reinforcement. In Phase 1 of the research, the target responding of artificial organisms (AOs) animated by the revised theory was found to be well described by an exponentiated hyperbola, the parameters of which varied as a function of the background reinforcement rate. In Phase 2, the effect of reinforcer magnitude on the target behavior of the AOs was studied. As in Phase 1, the AOs' behavior was well described by an exponentiated hyperbola, the parameters of which varied with both the target reinforcer magnitude and the background reinforcement rate. Evidence from experiments with live organisms was found to be consistent with the Phase-1 predictions of the revised theory. The Phase-2 predictions have not been tested. The revised implementation of the theory can be used to study the effects of superimposing punishment on single-schedule responding, and it may lead to the discovery of a function that relates response rate to both the rate and magnitude of reinforcement on single schedules.     

Matching: its acquisition and generalization

Choice typically is studied by exposing organisms to concurrent variable-interval schedules in which not only responses controlled by stimuli on the key are acquired but also switching responses and likely other operants as well. In the present research, discriminated key-pecking responses in pigeons were first acquired using a multiple schedule that minimized the reinforcement of switching operants. Then, choice was assessed during concurrent-probe periods in which pairs of discriminative stimuli were presented concurrently. Upon initial exposure to concurrently presented stimuli, choice approximated exclusive preference for the alternative associated with the higher reinforcement frequency. Concurrent schedules were then implemented that gave increasingly greater opportunities for switching operants to be conditioned. As these operants were acquired, the relation of relative response frequency to relative reinforcement frequency converged toward a matching relation. An account of matching with concurrent schedules is proposed in which responding exclusively to the discriminative stimulus associated with the higher reinforcement frequency declines as the concurrent stimuli become more similar and other operants-notably switching-are acquired and generalize to stimuli from both alternatives. The concerted effect of these processes fosters an approximate matching relation in commonly used concurrent procedures.     

Choosing among natural stimuli

Pigeons worked on concurrent variable-interval, variable-interval schedules with the alternatives signaled by slides either containing trees or not. The schedules were designed to hold both overall and relative rates of reinforcement within narrowly constrained limits, and slides were quasi-randomly ordered each day. Responding to the two alternatives was well described by the generalized matching equation with substantial undermatching. Using an adaptation of the matching law, we estimated that the subjects were correctly classifying 82% to 95% of exemplars. The matching performance transferred to new exemplars of trees and nontrees with only slight generalization decrement. The pigeons appeared to be discriminating among exemplars even when the alternatives provided equal rates of reinforcement and the average relative performances were close to 50%.