Concurrent schedules: Effects of reinforcement rate and changeover delay on time allocation in a three-alternative procedure

Pigeons were exposed to a continuous choice procedure where three alternatives alternated in a fixed, recycling order (ABCABC, etc.). Responses were reinforced according to independent variable-interval schedules. For three birds, the reinforcement rate for responses on alternative C was varied. For three other birds, the duration of the changeover delay after the changeover to C was varied. For both groups, the reinforcement rates and changeover delay durations associated with A and B were constant throughout the experiment. The time proportion at A relative to B increased as a function of the reinforcement rate for responses on C and decreased as a function of the duration of the changeover delay during C. The results show that the proportion of time spent at a variable-interval alternative of a continuous choice procedure is not completely determined by the reinforcement rates provided by the alternatives. The results support the assumption that time allocation is governed by delayed reinforcement of changeover behaviour.     

Sensitivity to relative reinforcer rate in concurrent schedules: independence from relative and absolute reinforcer duration

Twelve pigeons responded on two keys under concurrent variable-interval (VI) schedules. Over several series of conditions, relative and absolute magnitudes of reinforcement were varied. Within each series, relative rate of reinforcement was varied and sensitivity of behavior ratios to reinforcer-rate ratios was assessed. When responding at both alternatives was maintained by equal-sized small reinforcers, sensitivity to variation in reinforcer-rate ratios was the same as when large reinforcers were used. This result was observed when the overall rate of reinforcement was constant over conditions, and also in another series of concurrent schedules in which one schedule was kept constant at VI ached 120 s. Similarly, reinforcer magnitude did not affect the rate at which response allocation approached asymptote within a condition. When reinforcer magnitudes differred between the two responses and reinforcer-rate ratios were varied, sensitivity of behavior allocation was unaffected although response bias favored the schedule that arranged the larger reinforcers. Analysis of absolute response rates ratio sensitivity to reinforcement occurrred on the two keys showed that this invariance of response despite changes in reinforcement interaction that were observed in absolute response rates on the constant VI 120-s schedule. Response rate on the constant VI 120-s schedule was inversely related to reinforcer rate on the varied key and the strength of this relation depended on the relative magnitude of reinforcers arranged on varied key. Independence of sensitivity to reinforcer-rate ratios from relative and absolute reinforcer magnitude is consistent with the relativity and independence assumtions of the matching law.     

Performances on ratio and interval schedules of reinforcement: Data and theory

Two differences between ratio and interval performance are well known: (a) Higher rates occur on ratio schedules, and (b) ratio schedules are unable to maintain responding at low rates of reinforcement (ratio “strain”). A third phenomenon, a downturn in response rate at the highest rates of reinforcement, is well documented for ratio schedules and is predicted for interval schedules. Pigeons were exposed to multiple variable-ratio variable-interval schedules in which the intervals generated in the variable-ratio component were programmed in the variable-interval component, thereby “yoking” or approximately matching reinforcement in the two components. The full range of ratio performances was studied, from strained to continuous reinforcement. In addition to the expected phenomena, a new phenomenon was observed: an upturn in variable-interval response rate in the midrange of rates of reinforcement that brought response rates on the two schedules to equality before the downturn at the highest rates of reinforcement. When the average response rate was corrected by eliminating pausing after reinforcement, the downturn in response rate vanished, leaving a strictly monotonic performance curve. This apparent functional independence of the postreinforcement pause and the qualitative shift in response implied by the upturn in variable-interval response rate suggest that theoretical accounts will require thinking of behavior as partitioned among at least three categories, and probably four: postreinforcement activity, other unprogrammed activity, ratio-typical operant behavior, and interval-typical operant behavior.     

The general matching law describes choice on concurrent variable-interval schedules of wheel-running reinforcement.

Six male Wistar rats were exposed to concurrent variable-interval schedules of wheel-running reinforcement. The reinforcer associated with each alternative was the opportunity to run for 15 s, and the duration of the changeover delay was 1 s. Results suggested that time allocation was more sensitive to relative reinforcement rate than was response allocation. For time allocation, the mean slopes and intercepts were 0.82 and 0.008, respectively. In contrast, for response allocation, mean slopes and intercepts were 0.60 and 0.03, respectively. Correction for low response rates and high rates of changing over, however, increased slopes for response allocation to about equal those for time allocation. The results of the present study suggest that the two-operant form of the matching law can be extended to wheel-running reinforcement. 'I'he effects of a low overall response rate, a short Changeover delay, and long postreinforcement pausing on the assessment of matching in the present study are discussed.     

A theory of attending and reinforcement in conditional discriminations

A model of conditional discrimination performance (Davison & Nevin, 1999) is combined with the notion that unmeasured attending to the sample and comparison stimuli, in the steady state and during disruption, depends on reinforcement in the same way as predicted for overt free-operant responding by behavioral momentum theory (Nevin & Grace, 2000). The rate of observing behavior, a measurable accompaniment of attending, is well described by an equation for steady-state responding derived from momentum theory, and the resistance to change of observing conforms to predictions of momentum theory, supporting a key assumption of the model. When probabilities of attending are less than 1.0, the model accounts for some aspects of conditional-discrimination performance that posed problems for the Davison-Nevin model: (a) the effects of differential reinforcement on the allocation of responses to the comparison stimuli and on accuracy in several matching-to-sample and signal-detection tasks where the differences between the stimuli or responses were varied across conditions, (b) the effects of overall reinforcer rate on the asymptotic level and resistance to change of both response rate and accuracy of matching to sample in multiple schedules, and (c) the effects of fixed-ratio reinforcement on accuracy. Some tests and extensions of the model are suggested, and the role of unmeasured events in behavior theory is considered.     

Designing interventions that include delayed reinforcement: implications of recent laboratory research

The search for robust and durable interventions in everyday situations typically involves the use of delayed reinforcers, sometimes delivered well after a target behavior occurs. Integrating the findings from laboratory research on delayed reinforcement can contribute to the design and analysis of those applied interventions. As illustrations, we examine articles from the Journal of the Experimental Analysis of Behavior that analyzed delayed reinforcement with respect to response allocation (A. M. Williams & Lattal, 1999), stimulus chaining (B. A. Williams, 1999), and self-control (Jackson & Hackenberg, 1996). These studies help to clarify the conditions under which delayed reinforcement (a) exercises control of behavior, (b) entails conditioned reinforcement, and (c) displaces the effects of immediate reinforcement. The research has applied implications, including the development of positive social behavior and teaching people to make adaptive choices. DESCRIPTORS: delayed reinforcement, response allocation, stimulus chains, self-control, integration of basic and applied research     

A comparison of game theory and learning theory

It is shown that Estes' formula for the asymptotic behavior of a subject under conditions of partial reinforcement can be derived from the assumption that the subject is behaving rationally in a certain game-theoretic sense and attempting to minimax his regret. This result illustrates the need for specifying the frame of reference or set of the subject when using the assumption of rationality to predict his behavior.     

Dynamics of time matching: Arousal makes better seem worse

Matching of time allocation across alternatives in proportion to relative reinforcement rates is a ubiquitous finding in the animal-learning literature on choice. The dynamics of the underlying mechanism, however, remain poorly understood. A recent finding by Belke (1992) profoundly challenges scalar expectancy theory (SET; Gibbon et al., 1988) and other accounts of matching in concurrent variable interval (VI) schedules. He studied concurrent probe tests of stimuli associated with equal VIs but trained in alternative concurrent pairs. In training, one was preferred and the other not. Unreinforced probes revealed a strong preference for the alternative preferred in training. An experiment is reported replicating this result and showing that it is not due to generalization of preference levels from training. When the probe is between the two preferred training stimuli, the richer schedule is unpreferred. A SET account of these results is presented which implicates two processes in time allocation: (1) the choice between alternatives based on memory for delays to reinforcement, and (2) the times at which such choices are made. The former process is sensitive to reinforcement scheduling; the latter is sensitive to arousal levels induced by overall reinforcement rates in training.     

The role of adventitious reinforcement during differential reinforcement of other behavior: A systematic replication

Differential reinforcement of other behavior (DRO) is commonly used to decrease problem behavior by presenting reinforcers contingent upon the absence of a target response. Although it is well demonstrated that DROs decrease response rates, the processes producing these decreases are not well understood. The present study systematically replicated previous research assessing whether adventitious reinforcement of alternative behavior contributes to the effectiveness of DRO. We presented university students with two options on a computer and reinforced target responding on a variable-ratio schedule. Next, we compared decreases in target-response rates and any increases in alternative responding during DRO schedules versus yoked variable-time schedules or extinction probes. DRO schedules resulted in the lowest target-response rate and highest alternative-response rate. These findings generally provide some support for the adventitious reinforcement of “other” behavior.     

Concurrent variable-interval variable-ratio schedules can provide only weak evidence for matching

Herrnstein and Heyman (1979) showed that when pigeons' pecking is reinforced on concurrent variable-interval variable-ratio schedules, (1) their behavior ratios match the ratio of the schedules' reinforcer frequencies, and (2) there is more responding on the variable interval. Since maximizing the reinforcement rate would require responding more on the variable ratio, these results were presented as establishing the primacy of matching over maximizing. In the present report, different ratios of behavior were simulated on a computer to see how they would affect reinforcement rates on these concurrent schedules. Over a wide range of experimenter-specified choice ratios, matching obtained — a result suggesting that changes in choice allocation produced changes in reinforcer frequencies that correspond to the matching outcome. Matching also occurred at arbitrarily selected choice ratios when reinforcement rates were algebraically determined by each schedule's reinforcement-feedback function. Additionally, three birds were exposed to concurrent variable-interval variable-ratio schedules contingent on key pecking in which hopper durations were varied in some conditions to produce experimenter-specified choice ratios. Matching generally obtained between choice ratios and reinforcer-frequency ratios at these different choice ratios. By suggesting that reinforcer frequencies track choice on this procedure, instead of vice versa, this outcome questions whether matching-as-outcome was due to matching-as-process in the Herrnstein and Heyman study.     

Response additivity: effects of superimposed free reinforcement on a variable-interval baseline

Three experiments examined the effects of superimposing free reinforcement (Free VI 30-sec) on behavior maintained by a response dependent mult VI 2-min VI 2-min schedule of reinforcement. Experiment I used pigeons as subjects, key pecking as the response, and colors of response key as the stimuli associated with the multiple-schedule components. When free reinforcement was added during only one component (Differential condition) a large and highly significant increase in response rate developed in this component. Adding free reinforcement during both components (Nondifferential condition) produced smaller and far less-consistent effects. An entirely different pattern of results was obtained in two subsequent experiments, where similar procedures and reinforcement conditions were used with rats as subjects and bar pressing as the response. In both Experiments II and III, response rates decreased to the stimulus associated with added free reinforcement in the Differential condition. These findings are interpreted as the result of interactions between behavior maintained by response-reinforcer contingencies and behavior maintained by stimulus-reinforcer contingencies. As such, they support the main assumption of an autoshaping theory of behavioral contrast, that additivity of responding generated by the two kinds of contingency can occur only in situations favorable to autoshaping.     

Temporal constraint on choice: Sensitivity and bias in multiple schedules

In multiple schedules of reinforcement, ratios of responses in successive components are relatively insensitive to ratios of obtained reinforcers. An analysis is proposed that attributes changes in absolute response rates to concurrent interactions between programmed reinforcement and extraneous reinforcement in other components. The analysis predicts that ratios of responses in successive components vary with reinforcer ratios, qualified by a term describing the reinforcement context, that is, programmed and extraneous reinforcers. Two main predictions from the analysis were confirmed in an experiment in which pigeons' responses were reinforced in the components of a multiple schedule and analog extraneous reinforcement was scheduled for an alternative response in each component. Sensitivity of response and time ratios to reinforcer ratios in the multiple schedules varied as a function of the rate of extraneous reinforcers. Bias towards responding in one component of the multiple schedule varied as an inverse function of the ratios of extraneous reinforcer rate in the two components. The data from this and previous studies of multiple-concurrent performance were accurately predicted by our analysis and supported our contention that the allocation of behavior in multiple-schedule components depends on the relative values of concurrently-available reinforcers within each component.     

Differential reinforcement of alternative behavior increases resistance to extinction: clinical demonstration, animal modeling, and clinical test of one solution

Basic research with pigeons on behavioral momentum suggests that differential reinforcement of alternative behavior (DRA) can increase the resistance of target behavior to change. This finding suggests that clinical applications of DRA may inadvertently increase the persistence of target behavior even as it decreases its frequency. We conducted three coordinated experiments to test whether DRA has persistence-strengthening effects on clinically significant target behavior and then tested the effectiveness of a possible solution to this problem in both a nonhuman and clinical study. Experiment 1 compared resistance to extinction following baseline rates of reinforcement versus higher DRA rates of reinforcement in a clinical study. Resistance to extinction was substantially greater following DRA. Experiment 2 tested a rat model of a possible solution to this problem. Training an alternative response in a context without reinforcement of the target response circumvented the persistence-strengthening effects of DRA. Experiment 3 translated the rat model into a novel clinical application of DRA. Training an alternative response with DRA in a separate context resulted in lower resistance to extinction than employing DRA in the context correlated with reinforcement of target behavior. The value of coordinated bidirectional translational research is discussed.     

QUANTIFICATION OF ETHANOL'S ANTIPUNISHMENT EFFECT IN HUMANS USING THE GENERALIZED MATCHING EQUATION

Increases in rates of punished behavior by the administration of anxiolytic drugs (called antipunishment effects) are well established in animals but not humans. The present study examined antipunishment effects of ethanol in humans using a choice procedure. The behavior of 5 participants was placed under six concurrent variable‐interval schedules of monetary reinforcement. In three of the six concurrent schedules, punishment, in the form of monetary loss, was superimposed on one alternative. Data were analyzed according to the generalized matching equation which distinguishes between bias (allocation of behavior beyond what matching to relative reinforcer densities would predict) and sensitivity to reinforcement (how well behavior tracks relative reinforcer densities). In addition, participants completed a pencil‐tapping test. Under placebo punishment conditions, all participants demonstrated low response rates and a bias against the alternative associated with punishment, despite a resultant loss of available reinforcers. Bias against the punished alternative was dose‐dependently reduced in participants shown to be most sensitive to ethanol (0.6, 1.2, and 1.8 g/kg) in measures of overall responding and on the pencil‐tapping test. No ethanol‐induced change in bias was noted when punishment was not imposed. Sensitivity to reinforcement also decreased for participants shown to be sensitive to ethanol. In addition to extending antipunishment effects to humans, these results also show that antipunishment effects can be quantified via the matching equation.     

An equation for behavioral contrast.

Pigeons were trained on a three-component multiple schedule in which the rates of reinforcement in the various components were systematically varied. Response rates were described by an equation that posits that the response-strengthening effects of reinforcement are inversely related to the context of reinforcement in which it occurs, and that the context is calculated as the weighted average of the various sources of reinforcement in the situation. The quality of fits was comparable to that found with previous quantitative analyses of concurrent schedules, especially for relative response rates, with over 90% of the variance accounted for in every case. As with previous research, reinforcements in the component that was to follow received greater weights in determining the context than did reinforcements in the preceding component.     

Independence of reinforcement delay and magnitude in concurrent chains

A three-component concurrent-chains procedure was used to investigate preference between terminal-link schedules that differed in delay and magnitude of reinforcement. Response and time allocation data were well described by a generalized matching model. Sensitivity to delay appeared to be lower when reinforcement magnitudes were unequal than when they were equal, but when obtained rather than programmed time spent responding in the initial links was used in the model, the difference vanished. The results support independence of delay and magnitude as separate dimensions of reinforcement value, as required by the matching law, and the assumption of the contextual choice model (Grace, 1994) that sensitivities to delay and magnitude are affected similarly by temporal context. Although there was statistical evidence for interaction between successive components, the effects were small and transient. The multiple-component concurrent-chains procedure should prove useful in future research on multidimensional preference, although it may be necessary to control obtained initial-link time more precisely.     

Behavioral stereotypy and the generalized matching equation

The development of behavioral stereotypy is a common result of exposure to both response-dependent and response-independent reinforcement procedures. The generalized matching equation and two dynamic versions of that equation, which take into account the time differential between reinforcements and their effect on behavior, predict this outcome of many procedures involving reinforcement. Following from the assumption that distinct response topographies, distinct response sequences, or orientations to distinct stimuli can be treated in the equations as distinct classes of behavior, the equations predict that-at least for matching and undermatching-the behavior class that is most biased relative to other behavior classes of the same type will tend to predominate to the exclusion or near exclusion of those behavior classes.