Choice between response rates

Three pigeons were required to peck a single key at a higher and a lower rate, corresponding to two classes of shorter and longer concurrently reinforced interresponse times. Food reinforcers arranged by a single variable-interval schedule were randomly allocated to the two reinforced interresponse times. The absolute durations of reinforced interresponse times were varied while the total reinforcements per hour was held constant and the relative duration, i.e., the relative reciprocal, of the shorter reinforcer class was held constant at 0.70. Preference for the higher rate of responding, as measured by the relative frequency of responses terminating interresponse times in the shorter reinforced class, depended on the absolute reinforced response rates. Preference for the higher reinforced rate increased from a level of near-indifference (0.50) at high reinforced response rates, through the matching level (0.70) at intermediate reinforced response rates, to a virtually exclusive preference (>0.90) at low reinforced response rates. These results resemble corresponding preference functions obtained with two-key concurrent-chains schedules and thereby provide another sense in which it may be said that interresponse-time distributions from interval schedules estimate preference functions for the component response rates corresponding to different classes of reinforced interresponse times.     

Changes in functional response units with briefly delayed reinforcement

In two experiments, key-peck responding of pigeons was compared under variable-interval schedules that arranged immediate reinforcement and ones that arranged unsignaled delays of reinforcement. Responses during the nominal unsignaled delay periods had no effect on the reinforcer presentations. In Experiment 1, the unsignaled delays were studied using variable-interval schedules as baselines. Relative to the immediate reinforcement condition, 0.5-s unsignaled delays decreased the duration of the reinforced interresponse times and increased the overall frequency of short (<0.5-s) interresponse times. Longer, 5.0-s unsignaled delays increased the duration of the reinforced interresponse times and decreased the overall frequency of the short interresponse times. In Experiment 2, similar effects to those of Experiment 1 were obtained when the 0.5-s unsignaled delays were imposed upon a baseline schedule that explicitly arranged reinforcement of short interresponse times and therefore already generated a large number of short interresponse times. The results support earlier suggestions that the unsignaled 0.5-s delays change the functional response unit from a single key peck to a multiple key-peck unit. These findings are discussed in terms of the mechanisms by which contingencies control response structure in the absence of specific structural requirements.     

Choice and behavioral patterning

Ten pigeons pecked left and right keys in a discrete-trials experiment in which access to food was contingent upon changeovers to the right key after particular runs of left-key pecks. In each of three sets of conditions, two run lengths were reinforced according to a concurrent variable-interval schedule: reinforcement followed runs of either 1 or 2, 1 or 4, or 2 or 4 left-key pecks preceding changeovers. The intertrial interval separating successive pecks was varied from .5 to 10.0 sec, and the relative frequency of reinforcement for the shorter of the two reinforced runs was varied from 0 to .75. The contingencies established local behavioral patterning that roughly approximated that required for reinforcement. For a fixed pair of reinforced run lengths, preference for the shorter of the two frequently increased as the intertrial interval increased and therefore as the minimum temporal durations of both reinforced runs increased. Preference for the shorter of the two also increased as its corresponding relative frequency of reinforcement increased. Both of these effects on preference were qualitatively similar to corresponding effects in previous research with two different kinds of reinforced behavioral patterns, interresponse times and interchangeover times. In all these experiments, analytical units were found in the temporal patterns of behavior, not in the behavior immediately contiguous with a reinforcer. It is suggested that a particular local temporal pattern of behavior is established to the extent to which it is repeatedly remembered when reinforcers are delivered, regardless of whether the delivery of a reinforcer is explicitly contingent upon that pattern.     

Decision rules and signal detectability in a reinforcement-density discrimination

Two probabilistic schedules of reinforcement, one richer in reinforcement, the other leaner, were overlapping stimuli to be discriminated in a choice situation. One of two schedules was in effect for 12 seconds. Then, during a 6-second choice period, the first left-key peck was reinforced if the richer schedule had been in effect, and the first right-key peck was reinforced if the leaner schedule had been in effect. The two schedule stimuli may be viewed as two binomial distributions of the number of reinforcement opportunities. Each schedule yielded different frequencies of 16 substimuli. Each substimulus had a particular type of outcome pattern for the 12 seconds during which a schedule was in effect, and consisted of four consecutive light-cued 3-second T-cycles, each having 0 or 1 reinforced center-key pecks. Substimuli therefore contained 0 to 4 reinforcers. On any 3-second cycle, the first center-key peck darkened that key and was reinforced with probability .75 or .25 in the richer or leaner schedules, respectively. In terms of the theory of signal detection, detectability neared the maximum possible d′ for all four pigeons. Left-key peck probability increased when number of reinforcers in a substimulus increased, when these occurred closer to choice, or when pellets were larger for correct left-key pecks than for correct right-key pecks. Averaged over different temporal patterns of reinforcement in a substimulus, substimuli with the same number of reinforcers produced choice probabilities that matched relative expected payoff rather than maximized one alternative.     

Preference as a function of absolute response durations

The durations of 2 responses, 2 categories of reinforced nondiscriminated interresponse times, were varied while their relative durations were held approximately constant, with the longer about 2 1/2 times longer than the shorter. Three pigeons pecked for food. Reinforcers for the shorter and longer responses were arranged by a concurrent variable-interval, variable-interval schedule. Preference for the shorter response increased when both were lengthened. These results, taken together with previous results for discriminated interresponse times, show that preference for the shorter of 2 responses depends on their absolute durations, whether they are discriminated or not and regardless of autoshaped key pecks that may occur in the discriminated case. Time-allocation-matching was not generally obtained. The results qualitatively agree with an associative learner, a computational processing model derived from a molecular analysis of behavior.     

Choice with probabilistic reinforcement: effects of delay and conditioned reinforcers.

Two experiments measured pigeons' choices between probabilistic reinforcers and certain but delayed reinforcers. In Experiment 1, a peck on a red key led to a 5-s delay and then a possible reinforcer (with a probability of .2). A peck on a green key led to a certain reinforcer after an adjusting delay. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In all conditions, red houselights were present during the 5-s delay on reinforced trials with the probabilistic alternative, but the houselight colors on nonreinforced trials differed across conditions. Subjects showed a stronger preference for the probabilistic alternative when the houselights were a different color (white or blue) during the delay on nonreinforced trials than when they were red on both reinforced and nonreinforced trials. These results supported the hypothesis that the value or effectiveness of a probabilistic reinforcer is inversely related to the cumulative time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. Experiment 2 tested some quantitative versions of this hypothesis by varying the delay for the probabilistic alternative (either 0 s or 2 s) and the probability of reinforcement (from .1 to 1.0). The results were best described by an equation that took into account both the cumulative durations of stimuli associated with the probabilistic reinforcer and the variability in these durations from one reinforcer to the next.     

A comparison of ratio and interval reinforcement schedules with comparable interreinforcement times

Pigeons were trained to peck keys on fixed-ratio and fixed-interval schedules of food reinforcement. Both schedules produced a pattern of behavior characterized as pause and run, but the relation of pausing to time between reinforcers differed for the two schedules even when mean time between reinforcers was the same. Pausing in the fixed ratio occupied less of the time between reinforcers for shorter interreinforcer times. For two of three birds, the relation was reversed at longer interreinforcer times. As an interreinforcer time elapsed, there was an increasing tendency to return to responding for the fixed interval, but a roughly constant tendency to return to responding for the fixed-ratio schedule. In Experiment 1 these observations were made for both single-reinforcement schedules and multiple schedules of fixed-ratio and fixed-interval reinforcement. In Experiment 2 the observations were extended to a comparison of fixed-ratio versus variable-interval reinforcement schedules, where the distribution of interreinforcement times in the variable interval approximated that for the fixed ratio.     

The local organization of behavior: dissociations between a pigeon's behavior and self-reports of that behavior

The purpose of the experiment was to study the relation between what an organism does in a setting that demands temporal patterning of behavior and what it reports it has done. More specifically, a pigeon produced two classes, shorter and longer, of temporal patterns of key pecks (interresponse times) on a center key. Occasionally, a symbolic matching-to-sample probe arranged on side keys asked the pigeon whether its most recent pattern was a shorter or longer one. The longer reinforced pattern was always three times as long as the shorter one and the two patterns were reinforced equally often. Absolute duration of reinforced patterns was varied. In some conditions, interresponse-time distributions on the center key were bimodal, indicating a clear behavioral adaptation to the contingency, yet a bird did not report very well by appropriate side-key responding what its most recent interresponse time had been. In other conditions, the interresponse-time distributions were less clearly bimodal, yet a bird reported more accurately its previous interresponse time as shorter or longer. Thus, there was a dissociation between how well behavior on the center key conformed to the schedule requirement and how well a bird reported what it was doing on the center key. In addition, as absolute duration of the reinforced patterns was increased, a bird categorized its most recent pattern less well even as its preference for the shorter pattern increased dramatically. These results were interpreted as an example of the phenomenon of dissociation between tacit knowledge and knowledge.     

SINGLE‐SAMPLE DISCRIMINATION OF DIFFERENT SCHEDULES' REINFORCED INTERRESPONSE TIMES

Food‐deprived rats in Experiment 1 responded to one of two tandem schedules that were, with equal probability, associated with a sample lever. The tandem schedules' initial links were different random‐interval schedules. Their values were adjusted to approximate equality in time to completing each tandem schedule's response requirements. The tandem schedules differed in their terminal links: One reinforced short interresponse times; the other reinforced long ones. Tandem‐schedule completion presented two comparison levers, one of which was associated with each tandem schedule. Pressing the lever associated with the sample‐lever tandem schedule produced a food pellet. Pressing the other produced a blackout. The difference between terminal‐link reinforced interresponse times varied across 10‐trial blocks within a session. Conditional‐discrimination accuracy increased with the size of the temporal difference between terminal‐link reinforced interresponse times. In Experiment 2, one tandem schedule was replaced by a random ratio, while the comparison schedule was either a tandem schedule that only reinforced long interresponse times or a random‐interval schedule. Again, conditional‐discrimination accuracy increased with the temporal difference between the two schedules' reinforced interresponse times. Most rats mastered the discrimination between random ratio and random interval, showing that the interresponse times reinforced by these schedules can serve to discriminate between these schedules.     

DISCRIMINATION OF VARIABLE SCHEDULES IS CONTROLLED BY INTERRESPONSE TIMES PROXIMAL TO REINFORCEMENT

In Experiment 1, food‐deprived rats responded to one of two schedules that were, with equal probability, associated with a sample lever. One schedule was always variable ratio, while the other schedule, depending on the trial within a session, was: (a) a variable‐interval schedule; (b) a tandem variable‐interval, differential‐reinforcement‐of‐low‐rate schedule; or (c) a tandem variable‐interval, differential‐reinforcement‐of‐high‐rate schedule. Completion of a sample‐lever schedule, which took approximately the same time regardless of schedule, presented two comparison levers, one associated with each sample‐lever schedule. Pressing the comparison lever associated with the schedule just presented produced food, while pressing the other produced a blackout. Conditional‐discrimination accuracy was related to the size of the difference in reinforced interresponse times and those that preceded it (predecessor interresponse times) between the variable‐ratio and other comparison schedules. In Experiment 2, control by predecessor interresponse times was accentuated by requiring rats to discriminate between a variable‐ratio schedule and a tandem schedule that required emission of a sequence of a long, then a short interresponse time in the tandem's terminal schedule. These discrimination data are compatible with the copyist model from Tanno and Silberberg (2012) in which response rates are determined by the succession of interresponse times between reinforcers weighted so that each interresponse time's role in rate determination diminishes exponentially as a function of its distance from reinforcement.     

The copyist model of response emission

The variety of different performances maintained by schedules of reinforcement complicates comprehensive model creation. The present account assumes the simpler goal of modeling the performances of only variable reinforcement schedules because they tend to maintain steady response rates over time. The model presented assumes that rate is determined by the mean of interresponse times (time between two responses) between successive reinforcers, averaged so that their contribution to that mean diminishes exponentially with distance from reinforcement. To respond, the model randomly selects an interresponse time from the last 300 of these mean interresponse times, the selection likelihood arranged so that the proportion of session time spent emitting each of these 300 interresponse times is the same. This interresponse time defines the mean of an exponential distribution from which one is randomly chosen for emission. The response rates obtained approximated those found on several variable schedules. Furthermore, the model reproduced three effects: (1) the variable ratio maintaining higher response rates than does the variable interval; (2) the finding for variable schedules that when the reinforcement rate varies from low to high, the response rate function has an ascending and then descending limb; and (3) matching on concurrent schedules. Because these results are due to an algorithm that reproduces reinforced interresponse times, responding to single and concurrent schedules is viewed as merely copying what was reinforced before.     

SUPPRESSIVE AND FACILITATIVE EFFECTS OF SHOCK INTENSITY AND INTERRESPONSE TIMES FOLLOWED BY SHOCK

Although response‐dependent shock often suppresses responding, response facilitation can occur. In two experiments, we examined the suppressive and facilitative effects of shock by manipulating shock intensity and the interresponse times that produced shock. Rats' lever presses were reinforced on a variable‐interval 40‐s schedule of food presentation. Shock followed either long or short interresponse times. Shock intensity was raised from 0.05 mA to 0.4 mA or 0.8 mA. Overall, shock contingent on long interresponse times punished long interresponse times and increased response rates. Shock contingent on short interresponse times punished short interresponse times and decreased response rates. In Experiment 1, raising the range of interresponse times that produced shock enhanced these effects. In Experiment 2, the effects of shock intensity depended on the interresponse times that produced shock. When long interresponse times produced shock, low intensities increased response rates. High intensities decreased response rates. When short interresponse times produced shock, high shock intensities punished short interresponse times and decreased response rates more than low intensities. The results may explain why punishment procedures occasionally facilitate responding and establish parameters for future studies of punishment.     

A quantitative interresponse-time analysis of DRL performance differentiates similar effects of the antidepressant desipramine and the novel anxiolytic gepirone.

We describe an interresponse-time analysis of performance on a differential-reinforcement-of-low-rate 72-s schedule. This analysis compares the obtained interresponse-time distribution of individual rats to a corresponding random interresponse-time distribution. The random interresponse-time distribution is a negative exponential probability function; it predicts the relative distribution of interresponse times if the rat emitted the same number of responses randomly (i.e., with a constant probability) with respect to time. The analysis provides quantitative measures of peak location and dispersion of the interresponse times toward random performance. In Experiment 1, an unexpected outcome of this analysis was that the rats would have obtained more reinforcers had they responded at the same rate but randomly. Based on the interresponse-time analysis in Experiment 1, it was shown that rats trained on the differential-reinforcement-of-low-rate 72-s schedule could increase the number of reinforcers obtained in two ways: first, by a coherent shift of the interresponse-time distribution toward longer durations and, second, by dispersal of the interresponse times toward a random interresponse-time distribution. Experiment 2 applied the analysis described in Experiment 1 to the effects of desipramine and gepirone. Both drugs decreased response rate and increased reinforcement rate, but their effects on the distribution of interresponse times were different. The increase in reinforcement rate observed with desipramine was accompanied by a coherent shift of the reinforcement rate observed with gepirone was accompanied by dispersal of the interresponse-time distribution toward the random negative exponential prediction.     

Theories of probabilistic reinforcement.

In three experiments, pigeons chose between two alternatives that differed in the probability of reinforcement and the delay to reinforcement. A peck at a red key led to a delay of 5 s and then a possible reinforcer. A peck at a green key led to an adjusting delay and then a certain reinforcer. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In Experiments 1 and 2, the intertrial interval was varied across conditions, and these variations had no systematic effects on choice. In Experiment 3, the stimuli that followed a choice of the red key differed across conditions. In some conditions, a red houselight was presented for 5 s after each choice of the red key. In other conditions, the red houselight was present on reinforced trials but not on nonreinforced trials. Subjects exhibited greater preference for the red key in the latter case. The results were used to evaluate four different theories of probabilistic reinforcement. The results were most consistent with the view that the value or effectiveness of a probabilistic reinforcer is determined by the total time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. According to this view, probabilistic reinforcers are analogous to reinforcers that are delivered after variable delays.     

Two-key concurrent paced variable-interval paced variable-interval schedules of reinforcement

Nine pigeons were used in two experiments in which a response was reinforced if a variable-interval schedule had assigned a reinforcement and if the response terminated an interresponse time within a certain interval, or class, of interresponse times. One such class was scheduled on one key, and a second class was scheduled on a second key. The procedure was, therefore, a two-key concurrent paced variable-interval paced variable-interval schedule. In Exp. I, the lengths of the two reinforced interresponse times were varied. The relative frequency of responding on a key approximately equalled the relative reciprocal of the length of the interresponse time reinforced on that key. In Exp. II, the relative frequency and relative magnitude of reinforcement were varied. The relative frequency of responding on the key for which the shorter interresponse time was reinforced was a monotonically increasing, negatively accelerated function of the relative frequency of reinforcement on that key. The relative frequency of responding depended on the relative magnitude of reinforcement in approximately the same way as it depended on the relative frequency of reinforcement. The relative frequency of responding on the key for which the shorter interresponse time was reinforced depended on the lengths of the two reinforced interresponse times and on the relative frequency and relative magnitude of reinforcement in the same way as the relative frequency of the shorter interresponse time depended on these variables in previous one-key concurrent schedules of reinforcement for two interresponse times.     

Response-rate differences in variable-interval and variable-ratio schedules: An old problem revisited

In Experiment 1, a variable-ratio 10 schedule became, successively, a variable-interval schedule with only the minimum interreinforcement intervals yoked to the variable ratio, or a variable-interval schedule with both interreinforcement intervals and reinforced interresponse times yoked to the variable ratio. Response rates in the variable-interval schedule with both interreinforcement interval and reinforced interresponse time yoking fell between the higher rates maintained by the variable-ratio schedule and the lower rates maintained by the variable-interval schedule with only interreinforcement interval yoking. In Experiment 2, a tandem variable-interval 15-s variable-ratio 5 schedule became a yoked tandem variable-ratio 5 variable-interval x-s schedule, and a tandem variable-interval 30-s variable-ratio 10 schedule became a yoked tandem variable-ratio 10 variable-interval x-s schedule. In the yoked tandem schedules, the minimum interreinforcement intervals in the variable-interval components were those that equated overall interreinforcement times in the two phases. Response rates did not decline in the yoked schedules even when the reinforced interresponse times became longer. Experiment 1 suggests that both reinforced interresponse times and response rate–reinforcement rate correlations determine response-rate differences in variable-ratio 10 and yoked variable-interval schedules in rats. Experiment 2 suggests a minimal role for the reinforced interresponse time in determining response rates on tandem variable-interval 30-s variable-ratio 10 and yoked tandem variable-ratio 10 variable-interval x-s schedules in rats.     

The influence of ultraviolet radiation on the pigeon's color discrimination

Two experiments demonstrated the pigeon's sensitivity to ultraviolet light. In Experiment I, pigeons' responses were reinforced on a multiple schedule with a variable-interval reinforcement schedule in one component and extinction in the other component. Response rates were quite different in the two components where the 520-nm stimuli signalling each component differed only in that one of them contained a 366-nm ultraviolet component. In Experiment II, pigeons were trained to peck one side key when two halves of a split field were of different wavelength and to peck another side key when they were of the same wavelength. Initially, field halves contained both "visible" and ultraviolet components of energy. Discrimination performance improved when the ultraviolet component was removed from one field half. It was argued that the critical change in the stimulus was a color change, rather than a brightness one, or a fluorescence of structures in the pigeon's eye.     

A molecular analysis of multiple schedule interactions: negative contrast

The present experiments investigated the relationship between changes in the relative reinforced interresponse-time distributions and the occurrence of positive and negative contrast in multiple variable-interval—variable-interval and multiple variable-interval—extinction schedules of reinforcement. Experiment I demonstrated that changes in the interresponse-time distributions were consistently correlated with response-rate changes referred to as positive and negative contrast. Corresponding changes in the reinforced interresponse-time distributions suggested that negative contrast resulted as an inductive effect of selectively reinforcing long interresponse times in the altered component at the moment the baseline schedule was reintroduced. Experiment II demonstrated that the magnitude of the negative-contrast effect could be significantly decreased if the altered component schedule was modified in order to prevent the reinforcement of these interresponse times during the first few sessions of baseline recovery. The results supported a proposal that interresponse time—reinforcer relations may act as amplifiers or attenuators of negative contrast.     

Interresponse-time analysis of stimulus control in multiple schedules

Interresponse-time distributions were recorded in two components of multiple variable-interval schedules that were varied over several conditions. Values of the exponent for power functions relating ratios of interresponse times emitted per opportunity to ratios of reinforcers obtained in the two components varied with interresponse-time class interval. The exponent (sensitivity to reinforcement) afforded a measure of stimulus control exerted by the discriminative stimuli. Exponents were near zero for short interresponse times, consistent with previous conclusions that responses following short interresponse times are controlled by response-produced or proprioceptive stimuli. Values of exponents increased with longer interresponse times, indicating strong control by exteroceptive stimuli over responses following interresponse times of approximately one second or longer.     

Signalled reinforcement in differential-reinforcement-of-low rate schedules

At several fixed and variable minimum reinforced interresponse times, a stimulus was added to differential-reinforcement-of-low-rate schedules to signal the availability or nonavailability of reinforcement. As the minimum reinforced interresponse time increased, the rate of unreinforced responding decreased. Changing from fixed to variable minimum interresponse time in the basic differential-reinforcement-of-low-rate schedule further decreased the rate of unreinforced responding. Both effects were to some degree reversible. For fixed minimum reinforced interresponse times of 30 sec or shorter, most unreinforced responses terminated interresponse times just short of that required for reinforcement. The minimum reinforced interresponse time and the number of short response latencies (≤0.5 sec) to the onset of the signal were negatively correlated. Both of these analyses suggested that at values of 30 sec or shorter, the subjects discriminated the availability of the reinforcer more on the basis of time than on the basis of presence or absence of the signal.