A response duration schedule: effects of training, extinction, and deprivation

Rats were trained to hold down a lever for at least 40 consecutive seconds. When the lever had been held down for 40 sec, white noise came on. Releasing the bar in the presence of the noise turned off the noise and operated a feeder that delivered a pellet of food. At the end of training, frequency distributions of response durations peaked at 40 to 41 sec. If as in training, holding down the lever produced white noise at the end of 40 sec, and release of the lever terminated the noise and operated the feeder, but no food delivery occurred, duration distributions and several other measures were initially not very different from when food was delivered. However, if during extinction white noise was never produced by lever holding, and feeder operation did not occur upon lever release, most responses were shorter than 1 sec in duration, some were much longer than 41 sec, and duration distributions did not peak at 40 to 41 sec. When reinforcement was reinstated after extinction, performance quickly returned to pre-extinction measures. Further sessions at different levels of deprivation produced only temporary disruptions in performance.     

Interreinforcement time, work time, and the postreinforcement pause

Six rats were trained with food deliveries contingent upon their pressing a lever and holding it down for fixed, cumulative durations. Hold requirements were varied from 7.5 seconds to 120 seconds. Lever holding was maintained reliably at hold requirements as long as 30 seconds to 105 seconds for different rats. At longer hold requirements, lever holding was erratic and tended to occur only early in sessions. At shorter and intermediate requirements, the patterns of lever holding resembled those of responding under fixed-ratio schedules for discrete responses, with breaks in responding immediately after reinforcement alternating with relatively continuous lever holding until the next reinforcement. At longer hold requirements, postpause lever holding frequently was interrupted with additional pauses. The duration of postreinforcement pauses increased linearly with the scheduled hold requirement. However, for five of six rats, the hold requirement, which represents the actual time spent lever holding per reinforcer, accounted for somewhat less variance in pause duration than did interreinforcement time.     

Involvement of the rat anterior cingulate cortex in control of instrumental responses guided by reward expectancy

The anterior cingulate cortex (ACC) plays a critical role in stimulus-reinforcement learning and reward-guided selection of actions. Here we conducted a series of experiments to further elucidate the role of the ACC in instrumental behavior involving effort-based decision-making and instrumental learning guided by reward-predictive stimuli. In Experiment 1, rats were trained on a cost-benefit T-maze task in which they could either choose to climb a barrier to obtain a high reward (four pellets) in one arm or a low reward (two pellets) in the other with no barrier present. In line with previous studies, our data reveal that rats with quinolinic acid lesions of the ACC selected the response involving less work and smaller reward. Experiment 2 demonstrates that breaking points of instrumental performance under a progressive ratio schedule were similar in sham-lesioned and ACC-lesioned rats. Thus, lesions of the ACC did not interfere with the effort a rat is willing to expend to obtain a specific reward in this test. In a subsequent task, we examined effort-based decision-making in a lever-press task where rats had the choice between pressing a lever to receive preferred food pellets under a progressive ratio schedule, or free feeding on a less preferred food, i.e. lab chow. Results show that sham- and ACC-lesioned animals had similar breaking points and ingested comparable amounts of less-preferred food. Together, the results of Experiment 1 and 2 suggest that the ACC plays a role in evaluating how much effort to expend for reward; however, the ACC is not necessary in all situations requiring an assessment of costs and benefits. In Experiment 3 we investigated learning and reversal learning of instrumental responses guided by reward predictive stimuli. A reaction time (RT) task demanding conditioned lever release was used in which the upcoming reward magnitude (five vs. one food pellet) was signalled in advance by discriminative visual stimuli. Results revealed that rats with ACC lesions were able to discriminate reward magnitude-predictive stimuli and to adapt instrumental behavior to reversed stimulus-reward magnitude contingencies. Thus, in a simple discrimination task as used here, the ACC appears not to be required to discriminate reward magnitude-predictive stimuli and to use the learned significance of the stimuli to guide instrumental behavior.     

Rats' lever-press durations as psychophysical judgments of time

Hungry rats received food following lever-press durations exceeding a minimum value, which ranged from 0 to 6.4 sec. When no intertrial intervals separated successive presses, modal press durations remained at very short values as the minimum value required for food was increased. This was particularly true immediately after a food presentation. When an 8-sec intertrial interval followed each lever release, modal press durations were always at or beyond the minimum value required for food, and outcome of the preceding press had no effect on press duration. Possible reasons for the effects of intertrial intervals included punishment of short presses, increased delay of reinforcement of short presses, and reduced density of reinforcement. In addition, functions relating discrete-trials lever-press duration to minimum duration required for food were found to be qualitatively and quantitatively similar to the power functions recently proposed by Catania (1970) for interresponse time and response latency. This similarity was taken as support for a general psychophysical law of temporal judgments.     

Short-term memory for time intervals

The effect of intertrial interval, preset interval, and retention interval on the performance of rats in a time estimation task was described. On each trial a signal was presented for a duration of 2 to 8 sec. Eighteen rats were trained to press one lever (the short response) if the signal was shorter than 4 sec, and another lever (the long response) if the signal was longer than 4 sec. When trials were massed (Experiment 1), the percentage long response was affected by the classification of the previous signal, but not by its actual duration. This suggests that the animals remembered the response made on the previous trial, but not the signal duration. If a response was not permitted on the previous trial (Experiment 2), the duration or classification of the previous signal had no effect on performance. This supports the conclusion from the first experiment and suggests that an animal can reset its internal clock in less than 2 sec. In Experiment 3, the difference limen of the psychophysical function increased with the duration of the retention interval, but the point of subjective equality did not change. This suggests that resetting of the internal clock occurs on a non-time dimension.     

Effects of deprivation and reinforcement-magnitude on response variability

Animals were trained to displace any one of five response keys in order to put themselves in a stimulus condition in which reinforcement could be obtained by depressing a response lever. Decreased deprivation and magnitude of reinforcement were found to increase the variability of the distribution of key responses. The relevance of these findings to other experiments in which deprivation, reinforcement magnitude, and intermittent reinforcement were studied is discussed.     

Successive negative contrast in one-way avoidance learning in rats

In three experiments, successive negative contrast was examined in one-way avoidance learning. Reward magnitude in first (pre-shift) and second (post-shift) phases was manipulated by time spent in the safe compartment. Experiment 1 demonstrated that when time in the danger compartment was held constant, a group shifted from a large reward--30 sec spent in the safe compartment--to a small reward--1 sec--showed poor performance and longer response latency than a group conditioned with the small reward in both phases. Experiment 2 replicated this effect with a less intense shock and also demonstrated that a group shifted from large to small reward performed more poorly than a group exposed to large reward--30 sec--in both phases. Finally, Experiment 3 showed that changes in intertrial interval, defined as total time spent in the safe compartment and the danger compartment before the onset of the warning signal, were not responsible for this contrast effect. These results suggest that time spent in a safe place can act as appetitive incentive during one-way avoidance learning.     

Effects of food priming on instrumental acquisition and performance

Four experiments investigated the effects of a priming food reinforcement, given 0 or 75 sec pretrial, on runway performance of rats. The studies differed in the use of between-versus-within-subject designs, and by using food or water as the goal reinforcer. In Experiment 1, using food as the goal reward, subjects primed with food 0 sec pretrial conditioned slower than subjects primed 75 sec pretrial. In Experiment 2, using water as the goal reward, subjects primed with food 0 sec pretrial conditioned faster. These differences were evident on both prefed trials and on nonprefed test trials. Experiments 3 and 4 showed an opposite pattern of results when within-subject comparisons of 0- and 75-sec pretrial intervals were used: Food priming immediately pretrial facilitated food-rewarded running but inhibited water-rewarded running. The results suggest prefeeding has differential effects on acquisition and performance of instrumental behavior, and also depending upon the similarity of the priming and goal reinforcers.     

Rapid acquisition of discrete-trial lever-press avoidance: effects of signal-shock interval

Acquisition of discrete-trial lever-press avoidance learning was studied in three experiments. Experiment I compared a new training procedure, which produces rates of lever-press avoidance learning comparable to those obtained in shuttle boxes, with a “conventional”, less efficient training procedure. A factorial design was used to compare continuous versus intermittent shock and a long-variable versus a short-fixed signal-shock interval. Learning was best in the groups trained with the long and variable interval and poorest in those trained with the short and fixed interval. Type of shock had no effect. Experiment II separated the effects of duration from those of variability of the signal-shock interval. Fixed and variable intervals of 10 and 60 sec were tested and duration was the only significant factor. Experiment III addressed the effect of the differential opportunity to avoid provided by long signal-shock intervals by varying this interval from 10 to 60 sec in 10-sec steps. Only the 10-sec group showed slow acquisition relative to the others. Analysis of avoidance response latencies showed that the distributions for all groups were positively skewed and that skewness increased with increasing duration of the signal-shock interval. At intervals longer than 20 sec, the animals made progressively less use of their increased opportunity to respond. The data do not support the opportunity-to-respond interpretation of the effects of duration of signal-shock interval and suggest that some type of inhibitory process may block lever-press avoidance learning at intervals as short as 10 sec. The significance of these findings for species-specific defense reaction and preparedness theories was emphasized.     

Deprivation and reward stimuli as compound stimuli

In Experiment 1 the experimental group was tested with a deprivation level and a reward magnitude which it had experienced previously but which it had not experienced in combination. This group was inferior in test performance to a group which had experienced the test deprivation-reward combination prior to test. These results were interpreted as indicating that deprivation stimuli and reward stimuli form a compound stimulus and training on the elements of the compound produces performance inferior to training directly on the compound. In Experiment 2, the decrement associated with two different shifts in deprivation and reward did not differ despite the different size change of total incentive involved in the two shifts. The results were interpreted as indicating that the deprivation-reward stimulus is not produced by a single underlying incentive mechanism.     

Abortive responding during punishment of bar holding

A continuous chain of homogeneous responding was established in rats by training animals to hold a lever down for 10 sec or longer before releasing it for food reinforcement. When criterion releases were subsequently punished, completed holding chains were greatly suppressed, aborted chains increased markedly, while the rate of chain initiations remained unchanged.     

Duration discrimination: effects of probability of stimulus presentation

Monkeys initiated a stimulus by pressing on the center of three levers and the stimulus terminated independently of behavior 60, 80, 90, or 100 sec later. Presses on the right lever were reinforced with food following the three briefer durations, and presses on the left lever, following the 100-sec duration. Incorrect responses produced a 10-sec timeout. Probability of presenting the 100-sec duration was manipulated in the range from 0.25 to 0.75, with the probabilities of the briefer durations remaining equal and summing to one minus the probability of the 100-sec duration. Percentage of responses on either side lever was functionally related to both the probability of presenting the 100-sec stimulus and to stimulus duration. An analysis of the data based on the theory of signal detection resulted in operating characteristics that were linear when plotted on normal-normal coordinates. The percentage of responses on either lever approximated the optimal values for maximizing reinforcement probability in each condition of the experiment.     

Reward Density And Variable-interval Schedule Performance In An Open Economy

There is no general agreement regarding the form of the relation between response rate and reinforcement rate when single schedules of reinforcement are studied in an open economy. The present study assessed the form of this relation using reward density, which incorporates both reinforcement rate and duration of access to food, as an independent variable. Reward density was manipulated with 4 pigeons by changing the value of the variable-interval schedule, the hopper duration, or both. The relations between response rate and reward density were sharply rising and hyperbolic in 3 of 4 pigeons, replicating results obtained by Catania and Reynolds (1968). Because eating efficiency was lower in conditions that provided longer hopper durations, programmed reward densities differed from obtained reward densities. When response rates were examined as a function of obtained reward densities, the same relations were demonstrated more strongly. In further clarifying the relation between response rate and reward density in an open economy, these results lend support to the conclusion that open and closed economies yield different behavioral effects.     

Visual Reinforcement Signals Interfere with the Effects of Reinforcer Magnitude Manipulations

Three experiments examined the effect of reinforcement magnitude on free-operant response rates. In Experiment 1, rats that received four food pellets responded faster than rats that received one pellet on a variable ratio 30 schedule. However, when the food hopper was illuminated during reinforcer delivery, there was no difference between the rates of response produced by the two magnitudes of reward. In Experiment 2, there was no difference in response rates emitted by rats receiving either one or four pellets of food as reward on a random interval (RI) 60-s schedule. In Experiment 3, rats responding on an RI 30-s schedule did so at a lower rate with four pellets as reinforcement than with one pellet. This effect was abolished by the illumination of the food hopper during reinforcement delivery. These results indicate that the influence of magnitude is obscured by manipulations which signal the delivery of reinforcement.     

Human performance on an analogue of an interval bisection task

Two experiments used normal adult human subjects in an analogue of a time interval bisection task frequently used with animals. All presented durations were defined by the time between two very brief clicks, and all durations were less than 1 sec, to avoid complications arising from chronometric counting. In Experiment 1 different groups of subjects received standard durations of either 0.2 and 0.8 or 0.1 and 0.9 sec and then classified a range of durations including these values in terms of their similarity to the standard short (0.2- or 0.1-sec) and long (0.8- or 0.9-sec) durations. The bisection point (defined as the duration classified as "long" on 50% of trials) was located at 0.43 sec in the 0.2-0.8 group, and at 0.46 sec in the 0.1-0.9 group. Experiment 2 replicated Experiment 1 using a within-subject procedure. The bisection point of both 0.2- and 0.8 sec and 0.1- and 0.9-sec durations was found to be 0.44 sec. Both experiments thus found the bisection point to be located at a duration just lower than the arithmetic mean of the standard short and long durations, rather than at the geometric mean, as in animal experiments. Some other performance measures, such as difference limen, and Weber ratio, were, however, of similar values to those found in bisection tasks with animals. A theoretical model assuming that humans bisect by taking the difference between a presented duration and the short and long standards, as well as having a bias to respond "long", fitted the data well. The model incorporated scalar representations of standard durations and thus illustrated a way in which the obtained results, although different from those found with animal subjects, could be reconciled with scalar timing theory.     

Run length, visit duration, and reinforcers per visit in concurrent performance

The contingencies in each alternative of concurrent procedures consist of reinforcement for staying and reinforcement for switching. For the stay contingency, behavior directed at one alternative earns and obtains reinforcers. For the switch contingency, behavior directed at one alternative earns reinforcers but behavior directed at the other alternative obtains them. In Experiment 1, responses on the main lever, in S1, incremented stay and switch schedules and obtained a stay reinforcer when it became available. Responses on the switch lever changed S1 to S2 and obtained switch reinforcers when available. In S2, neither responses on the main lever nor on the switch lever were reinforced, but a switch response changed S2 to S1. Run lengths and visit durations were a function of the ratio of the scheduled probabilities of reinforcement (staying/switching). From run lengths and visit durations, traditional concurrent performance was synthesized, and that synthesized performance was consistent with the generalized matching law. Experiment 2 replicated and extended this analysis to concurrent variable-interval schedules. The synthesized results challenge any theory of matching that requires a comparison among the alternatives.     

Instrumental performance on negative schedules

Three experiments examined the performance of rats pressing a lever for food reinforcement on a schedule in which high rates of response resulted in lowered rates of reinforcement (i.e. a schedule with a negative component). In Experiment 1, rats responded on a variable interval (VI) schedule with a conjoint component such that every 30 responses a reinforcement programmed by the VI schedule was cancelled. These subjects generally emitted a lower response rate than rats responding on a VI schedule yoked to the former subjects with respect to the delivery of reinforcement, although response rate differences were sometimes not large. Similar response-rate effects were obtained in Experiment 2 using a within-subject yoking procedure. In Experiment 3, reinforced interresponse times were matched on negative and VI schedules yoked in terms of reinforcement rate, and the response rate emitted in these conditions were similar. These results give support to theories of instrumental conditioning that stress the strengthening and shaping properties of reinforcement.     

Time limits for completing fixed ratios. III. Stimulus variables

Pigeons received food only if they took longer than a specified time to begin and complete a fixed ratio. In Experiment 1, ratios with shorter durations had no stimulus consequence; in Experiment 2, these ratios ended with a stimulus change. In both studies, the mean time to complete the ratio exceeded requirements of less than 30 sec, approximately matched requirements of 30 sec, and fell progressively short of matching thereafter. Variability increased together with the means. The various effects resembled those of temporal differentiation experiments involving single responses. Although both number of ratios and time separating successive food presentations increased along with ratio duration, control experiments showed that differential reinforcement of duration, rather than either form or reinforcer intermittency, accounted for the performance. Experiment 2 also studied the effects of adding a stimulus that signalled when the required time had elapsed. The stimulus produced durations that matched even the most stringent requirements. This precision was not maintained when the stimulus was removed. Temporal differentiation schedules seem to have similar effects regardless of the response class and temporal property involved.     

Effects of reinforcement rate and reinforcer magnitude on choice behavior of humans

Two experiments with human subjects investigated the effects of rate of reinforcement and reinforcer magnitude upon choice. In Experiment 1, each of five subjects responded on four concurrent variable-interval schedules. In contrast to previous studies using non-human organisms, relative response rate did not closely match relative rate of reinforcement. Discrepancies ranged from 0.03 to 0.43 (mean equal to 0.19). Similar discrepancies were found between relative amount of time spent responding on each schedule and the corresponding relative rates of reinforcement. In Experiment 2, in which reinforcer magnitude was varied for each of five subjects, similar discrepancies ranging from 0.05 to 0.50 (mean equal to 0.21), were found between relative response rate and relative proportion of reinforcers received. In both experiments, changeover rates were lower on the long-interval concurrent schedules than on the short-interval ones. The results suggest that simple application of previous generalizations regarding the effects of reinforcement rate and reinforcer magnitude on choice for variable-interval schedules does not accurately describe human behavior in a simple laboratory situation.     

Discrimination of temporal relations by pigeons

In four experiments, pigeons were tested on a duration comparison task involving the successive presentation of two visual stimuli that varied in duration from trial to trial. Following presentation of the durations, two choice keys were lit, and reinforcement for choices was based on the temporal relation between duration of the pair. In Experiment 1, the range of durations was varied over conditions. Responding changed as an orderly function of the ratio of the two durations. There was a decrease in discrimination accuracy as average duration increased over condition but no difference in accuracy between shorter and longer problems within a duration range. There was no systematic response bias over conditions for all problems within a range, but there was a bias to report the second duration longer than the first for "long" problems within a range. In Experiment 2, the pigeons were transferred from a task involving spatially differentiated choices to one involving hue-differentiated choices. Performance was similar to that of the spatial procedure of Experiment 1. Additional analyses revealed that although information provided by a single duration of the pair was sometimes predictive of the temporal relation between pair members, responding was also based on the relation and comparison of both durations. In Experiment 3, the pigeons were exposed to a single duration range that included many durations from the four ranges of Experiment 1. Discrimination accuracy was comparable in the fourth and longest category. Manipulation of absolute reinforcement rate in Experiment 4 resulted in no chang in discrimination accuracy, suggesting that the decline in accuracy over conditions of Experiment 1 could not be attributed to decreases in reinforcement rate that accompanied lengthier durations. The results are discussed in terms of theories of animal timing, with Staddon's (1983, 1984) temporal perspective model providing the most systematic account of all aspects of performance.