Context specificity of conditioned-reinforcement effects on discrimination acquisition

Pigeons were trained on a series of reversals of a simultaneous form discrimination in which the trial outcomes were separated from the choice responses by an 8-s delay interval. Different conditions were defined by the stimuli occurring during the two halves of the delay interval. Discrimination learning was greatly facilitated by having differential stimuli during the delay following correct versus incorrect choices. When the differential stimuli appeared only at the midpoint of the delay, some facilitation occurred relative to when no different stimuli occurred, but there was substantially less facilitation than when the differential stimuli occurred immediately contingent on choice. A reversed-stimulus condition, in which the stimulus at the onset of the delay following a correct choice was the same as that during the last segment of the delay following an incorrect choice, and the stimulus at the onset of the delay following an incorrect choice was the same as that preceding food during the last segment of the delay following a correct choice, also facilitated discrimination learning relative to the nondifferential stimulus conditions.     

Choice with delayed and probabilistic reinforcers: effects of variability, time between trials, and conditioned reinforcers.

In a discrete-trials procedure with pigeons, a response on a green key led to a 4-s delay (during which green houselights were lit) and then a reinforcer might or might not be delivered. A response on a red key led to a delay of adjustable duration (during which red houselights were lit) and then a certain reinforcer. The delay was adjusted so as to estimate an indifference point--a duration for which the two alternatives were equally preferred. Once the green key was chosen, a subject had to continue to respond on the green key until a reinforcer was delivered. Each response on the green key, plus the 4-s delay that followed every response, was called one "link" of the green-key schedule. Subjects showed much greater preference for the green key when the number of links before reinforcement was variable (averaging four) than when it was fixed (always exactly four). These findings are consistent with the view that probabilistic reinforcers are analogous to reinforcers delivered after variable delays. When successive links were separated by 4-s or 8-s "interlink intervals" with white houselights, preference for the probabilistic alternative decreased somewhat for 2 subjects but was unaffected for the other 2 subjects. When the interlink intervals had the same green houselights that were present during the 4-s delays, preference for the green key decreased substantially for all subjects. These results provided mixed support for the view that preference for a probabilistic reinforcer is inversely related to the duration of conditioned reinforcers that precede the delivery of food.     

Conditioned reinforcement and choice with delayed and uncertain primary reinforcers.

In an adjusting-delay choice procedure, pigeons could peck on either a red key or a green key. A peck on the red key always led to a delay associated with red houselights and then food. The delay was adjusted over trials to estimate an indifference point--a delay at which the two keys were chosen about equally often. In some conditions, a peck on the green key led to food on all trials after delays of either 10 s or 30 s, and green houselights were lit during the delays. In other conditions, food was presented on only half of the green-key trials. If the green houselights continued to occur on both reinforcement and nonreinforcement trials, preference for the green key always decreased. Preference for the green key also decreased if half of the trials had 30-s houselights followed by food and the other half had no green houselights and no food. However, preference for the green key actually increased if half of the trials had 10-s green houselights followed by food and the other half had no green houselights followed by no food. The latter condition therefore demonstrated a case in which preference for an alternative increased when food was removed from half of the trials. The results suggest that the red and green houselights served as conditioned reinforcers. A hyperbolic decay model (Mazur, 1989) provided good predictions for all conditions by assuming that the strength of a conditioned reinforcer is inversely related to the total time spent in its presence before food is delivered.     

Food-paired stimuli as conditioned reinforcers: effects of d-amphetamine.

Seven pigeons were studied in two experiments in which key pecks were reinforced under a second-order schedule wherein satisfaction of variable-interval schedule requirements produced food or a brief stimulus. In the second part of each session, responses produced only the brief stimulus according to a variable-interval schedule (food extinction). For the 4 pigeons in Experiment 1, the response key was red throughout the session. In separate phases, the brief stimulus was either paired with food, not paired with food, or not presented during extinction. d-Amphetamine (0.3 to 10.0 mg/kg) dose-dependently reduced food-maintained responding during the first part of the session and, at intermediate dosages, increased responding during the extinction portion of the session. The magnitude of these increases, however, did not consistently depend on whether the brief stimulus was paired, not paired, or not presented. It was also true that under nondrug conditions, response rates during extinction did not differ reliably depending on pairing operations for the brief stimulus. In Experiment 2, 3 different pigeons responded under a procedure wherein the key was red in the component with food presentations and blue in the extinction component (i.e., multiple schedule). Again, d-amphetamine produced dose-related decreases in responding during the first part of a session and increases in responding in the second part of the session. These increases, however, were related to the pairing operations; larger increases were observed when the brief stimulus was paired with food than when it was not or when it was not presented at all. Under nondrug conditions, the paired brief stimulus controlled higher response rates during extinction than did a nonpaired stimulus or no stimulus. These findings suggest that d-amphetamine can enhance the efficacy of conditioned reinforcers, and that this effect may be more robust if conditioned reinforcers occur in the context of a signaled period of extinction.     

Value transmission in discrimination learning involving stimulus chains.

Rats learned a series of reversals of a positional discrimination in which responses to one lever led to delayed food and responses to a second lever led to no food. Interpolated within the delays leading to the different outcomes were two-link stimulus chains. The pairing of each stimulus element with the delayed outcome of food or no food varied across reversals. Either stimulus element could have the same correlation with outcome as occurred on the preceding reversal or the opposite correlation as on the preceding reversal. New reversals were acquired more quickly when both stimulus elements had the same status as during the preceding reversal, and were acquired most slowly when both stimulus elements had the opposite status as that of the preceding reversal. The rate of learning was intermediate when only one of the stimulus elements had the same status as that during the preceding reversal. All of the data are compatible with an interpretation in terms of backward chaining of stimulus value.     

Choice with probabilistic reinforcement: effects of delay and conditioned reinforcers.

Two experiments measured pigeons' choices between probabilistic reinforcers and certain but delayed reinforcers. In Experiment 1, a peck on a red key led to a 5-s delay and then a possible reinforcer (with a probability of .2). A peck on a green key led to a certain reinforcer after an adjusting delay. This delay was adjusted over trials so as to estimate an indifference point, or a duration at which the two alternatives were chosen about equally often. In all conditions, red houselights were present during the 5-s delay on reinforced trials with the probabilistic alternative, but the houselight colors on nonreinforced trials differed across conditions. Subjects showed a stronger preference for the probabilistic alternative when the houselights were a different color (white or blue) during the delay on nonreinforced trials than when they were red on both reinforced and nonreinforced trials. These results supported the hypothesis that the value or effectiveness of a probabilistic reinforcer is inversely related to the cumulative time per reinforcer spent in the presence of stimuli associated with the probabilistic alternative. Experiment 2 tested some quantitative versions of this hypothesis by varying the delay for the probabilistic alternative (either 0 s or 2 s) and the probability of reinforcement (from .1 to 1.0). The results were best described by an equation that took into account both the cumulative durations of stimuli associated with the probabilistic reinforcer and the variability in these durations from one reinforcer to the next.     

Choice for conditioned reinforcers in the signaled absence of primary reinforcement

Pigeons responded in a multiple schedule in which concurrent schedules of brief-stimulus presentation alternated with a component in which food was available (concurrent-chains component). In the initial links of the concurrent-chains component subjects chose either of two stimuli each correlated with the terminal link of one chain. The terminal links involved either variable-interval 30-second or variable-interval 60-second schedules. In the brief-stimulus component subjects chose between 0.5-second presentations of the terminal-link stimuli from the concurrent-chains component. Responding was generally maintained in the brief-stimulus component in two subjects for more than 300 sessions, suggesting that brief stimuli were conditioned reinforcers. During the brief-stimulus component, in 17 of 21 cases for which a minimal number of responses occurred, choice proportions above 0.55 were obtained for the brief-stimulus presentations correlated with the higher rate of primary reinforcement in the concurrent-chains component. These results support the suggestion that choice in conventional concurrent-chains procedures is partially controlled by production of the terminal-link stimuli.     

Choice with certain and uncertain reinforcers in an adjusting-delay procedure.

A discrete-trials adjusting-delay procedure was used to investigate the conditions under which pigeons might show a preference for partial reinforcement over 100% reinforcement, an effect reported in a number of previous experiments. A peck on a red key always led to a delay with red houselights and then food. In each condition, the duration of the red-houselight delay was adjusted to estimate an indifference point. In 100% reinforcement conditions, a peck on a green key always led to a delay with green houselights and then food. In partial-reinforcement conditions, a peck on the green key led either to the green houselights and food or to white houselights and no food. In some phases of the experiment, statistically significant preference for partial reinforcement over 100% reinforcement was found, but this effect was observed in only about half of the pigeons. The effect was largely eliminated when variability in the delay stimulus colors was equated for 50% reinforcement conditions and 100% reinforcement conditions. Idiosyncratic preferences for certain colors or for stimulus variability may be at least partially responsible for the effect.     

Choice with uncertain outcomes: conditioned reinforcement effects.

Pigeons responded on concurrent chains with equal initial- and terminal-link durations. In all conditions, the terminal links of one chain ended reliably in reinforcement; the terminal links on the alternative chain ended in either food or blackout. In Experiment 1, the terminal-link stimuli were correlated with (signaled) the outcome, and the durations of the initial and terminal links were varied across conditions. Preference did not vary systematically across conditions. In Experiment 2, terminal-link durations were varied under different stimulus conditions. The initial links were variable-interval 80-s schedules. Preference for the reliable alternative was generally higher in unsignaled than in signaled conditions. Preference increased with terminal-link durations only in the unsignaled conditions. There were no consistent differences between conditions with and without a common signal for reinforcement on the two chains. In the first series of conditions in Experiment 3, a single response was required in the initial links, and the stimulus conditions during 50-s terminal links were varied. Preference for the reliable outcome approached 1.0 in unsignaled conditions and was considerably lower (below .50 for 3 of 5 subjects) in signaled conditions. In a final series of signaled conditions with relatively long terminal links, preference varied with duration of the initial links. The results extend previous findings and are discussed in terms of the delay reduction signaled by terminal-link stimuli.     

Within-session changes in responding during concurrent schedules with different reinforcers in the components.

Rats and pigeons responded on several concurrent schedules that provided different reinforcers in the two components (food and water for rats, Experiment 1; wheat and mixed grain for pigeons, Experiment 2). The rate of responding and the time spent responding on each component usually changed within the session. The within-session changes in response rates and time spent responding usually followed different patterns for the two components of a concurrent schedule. For most subjects, the bias and sensitivity to reinforcement parameters of the generalized matching law, as well as the percentage of the variance accounted for, decreased within the session. Negative sensitivity parameters were sometimes found late in the session for the concurrent food-water schedules. These results imply that within-session changes in responding could cause problems for assessing the validity of quantitative theories of concurrent-schedule responding when the components provide different reinforcers. They question changes in a general motivational state, such as arousal, as a complete explanation for within-session changes in responding. The results are compatible with satiation for, or sensitization-habituation to, the reinforcers as explanations.     

Effects of delayed conditioned reinforcement in chain schedules.

The contingency between responding and stimulus change on a chain variable-interval 33-s, variable-interval 33-s, variable-interval 33-s schedule was weakened by interposing 3-s delays between either the first and second or the second and third links. No stimulus change signaled the delay interval and responses could occur during it, so the obtained delays were often shorter than the scheduled delay. When the delay occurred after the initial link, initial-link response rates decreased by an average of 77% with no systematic change in response rates in the second or third links. Response rates in the second link decreased an average of 59% when the delay followed that link, again with little effect on response rates in the first or third links. Because the effect of delaying stimulus change was comparable to the effect of delaying primary reinforcement in a simple variable-interval schedule, and the effect of the unsignaled delay was specific to the link in which the delay occurred, the results provide strong evidence for the concept of conditioned reinforcement.     

Choice with delayed and probabilistic reinforcers: effects of prereinforcer and postreinforcer stimuli.

In Experiment 1, pigeons' pecks on a green key led to a 5-s delay with green houselights, and then food was delivered on 20% (or, in other conditions, 50%) of the trials. Pecks on a red key led to an adjusting delay with red houselights, and then food was delivered on every trial. The adjusting delay was used to estimate indifference points: delays at which the two alternatives were chosen about equally often. Varying the presence or absence of green houselights during the delays that preceded possible food deliveries had large effects on choice. In contrast, varying the presence of the green or red houselights in the intertrial intervals had no effects on choice. In Experiment 2, pecks on the green key led to delays of either 5 s or 30 s with green houselights, and then food was delivered on 20% of the trials. Varying the duration of the green houselights on nonreinforced trials had no effect on choice. The results suggest that the green houselights served as a conditioned reinforcer at some times but not at others, depending on whether or not there was a possibility that a primary reinforcer might be delivered. Given this interpretation of what constitutes a conditioned reinforcer, most of the results were consistent with the view that the strength of a conditioned reinforcer is inversely related to its duration.     

Choice between single and multiple delayed reinforcers.

Pigeons chose between alternatives that differed in the number of reinforcers and in the delay to each reinforcer. A peck on a red key produced the same consequences on every trial within a condition, but between conditions the number of reinforcers varied from one to three and the reinforcer delays varied between 5 s and 30 s. A peck on a green key produced a delay of adjustable duration and then a single reinforcer. The green-key delay was increased or decreased many times per session, depending on a subject's previous choices, which permitted estimation of an indifference point, or a delay at which a subject chose each alternative about equally often. The indifference points decreased systematically with more red-key reinforcers and with shorter red-key delays. The results did not support the suggestion of Moore (1979) that multiple delayed reinforcers have no effect on preference unless they are closely grouped. The results were well described in quantitative detail by a simple model stating that each of a series of reinforcers increases preference, but that a reinforcer's effect is inversely related to its delay. The success of this model, which considers only delay of reinforcement, suggested that the overall rate of reinforcement for each alternative had no effect on choice between those alternatives.     

Simultaneous auditory discrimination.

Stimuli in many visual stimulus control studies typically are presented simultaneously; in contrast the stimuli in auditory discrimination studies are presented successively. Many everyday auditory stimuli that control responding occur simultaneously. This suggests that simultaneous auditory discriminations should be readily acquired. The purpose of the present experiment was to train rats in a simultaneous auditory discrimination. The apparatus consisted of a cage with two response levers mounted on one wall and a speaker mounted adjacent to each lever. A feeder was mounted on the opposite wall. In a go-right/go-left procedure, two stimuli were presented on each trial, a wide-band noise burst through one speaker and a 2-kHz complex signal through the other. The stimuli alternated randomly from side to side across trials, and the stimulus correlated with reinforcement for presses varied across subjects. The rats acquired the discrimination in 400 to 700 trials, and no response position preference developed during acquisition. The ease with which the simultaneous discrimination was acquired suggests that procedures, such as matching to sample, that require simultaneous presentation of stimuli can be used with auditory stimuli in animals having poor vision.     

Patterns of responding within sessions.

Rates of responding changed systematically across sessions for rats pressing levers and keys and for pigeons pressing treadles and pecking keys. A bitonic function in which response rates increased and then decreased across sessions was the most common finding, although an increase in responding also occurred alone. The change in response rate was usually large. The function relating responding to time in session had the following general characteristics: It appeared early in training, and further experience moved and reduced its peak; it was flatter for longer sessions; and it was flatter, more symmetrical, and peaked later for lower than for higher rates of reinforcement. Factors related to reinforcement exerted more control over the location of the peak rate of responding and the steepness of the decline in response rates than did factors related to responding. These within-session changes in response rates have fundamental theoretical and methodological implications.     

Delay and number of food reinforcers: Effects on choice and latencies

Pigeons were given a choice between two identical-duration situations (terminal links of chain schedules). One terminal link of the choice pair provided two food deliveries, and the other provided five. The exact times of these food deliveries differed between the terminal links and were varied over conditions. A single response during the initial link gave immediate access to the corresponding terminal link. Forced trials, during which only one of the initial-link keys was lighted, were interspersed with choice trials during which both initial-link keys were lighted. Choice tended to favor whichever terminal link was correlated with the higher sum of the immediacies (i.e., the sum of the reciprocals of the delays to each of the reinforcers following the choice, with all delays measured from the choice). Latencies on forced trials and on choice trials also were related (negatively) to the sum of the immediacies. This correlation among response measures (choice and latencies) suggests that both measures are manifestations of the effect of conditioned reinforcement on response tendencies.     

Contiguity and conditioned reinforcement in probabilistic choice

In a baseline condition, pigeons chose between an alternative that always provided food following a 30-s delay (100% reinforcement) and an alternative that provided food half of the time and blackout half of the time following 30-s delays (50% reinforcement). The different outcomes were signaled by different-colored keylights. On average, each alternative was chosen approximately equally often, replicating the finding of suboptimal choice in probabilistic reinforcement procedures. The efficacy of the delay stimuli (keylights) as conditioned reinforcers was assessed in other conditions by interposing a 5-s gap (keylights darkened) between the choice response and one or more of the delay stimuli. The strength of conditioned reinforcement was measured by the decrease in choice of an alternative when the alternative contained a gap. Preference for the 50% alternative decreased in conditions in which the gap preceded either all delay stimuli, both delay stimuli for the 50% alternative, or the food stimulus for the 50% alternative, but preference was not consistently affected in conditions in which the gap preceded only the 100% delay stimulus or the blackout stimulus for the 50% alternative. These results support the notion that conditioned reinforcement underlies the finding of suboptimal preference in probabilistic reinforcement procedures, and that the signal for food on the 50% reinforcement alternative functions as a stronger conditioned reinforcer than the signal for food on the 100% reinforcement alternative. In addition, the results fail to provide evidence that the signal for blackout functions as a conditioned punisher.     

Rapid acquisition in concurrent chains: evidence for a decision model

Pigeons' choice in concurrent chains can adapt to rapidly changing contingencies. Grace, Bragason, and McLean (2003) found that relative initial-link response rate was sensitive to the immediacy ratio in the current session when one of the terminal-link fixed-interval schedules was changed daily according to a pseudorandom binary sequence (e.g., Schofield & Davison, 1997). The present experiment tested whether the degree of variation in delays across sessions had any effect on acquisition rate in Grace et al.'s (2003) rapid-acquisition procedure. In one condition ("minimal variation"), the left terminal link was always fixed-interval 8 s and the right terminal link was either fixed-interval 4 s or fixed-interval 16 s. In the other condition ("maximal variation"), a unique pair of fixed-interval values was used in each session. Responding was sensitive to the current-session immediacy ratio in both conditions, but across subjects there was no systematic difference in sensitivity. These results challenge the view that initial-link responding in the rapid-acquisition procedure is determined by changes in the learned value of the terminal-link stimuli, and suggests instead that a process resembling categorical discrimination may control performance. A decision model based on the assumption that delays are categorized as short or long relative to the history of delays provided a good account of the data and shows promise in being able to explain other choice phenomena.     

Drug discrimination using a conditioned taste-aversion paradigm in rhesus monkeys.

The development of drug discrimination was assessed in rhesus monkeys using the conditioned taste-aversion paradigm. Monkeys were initially trained to respond under a fixed-ratio 30-response schedule of food-pellet delivery to assess the rate-decreasing effects of alprazolam (0.03 to 3 mg/kg, i.m., 60 min presession). Alprazolam decreased responding at doses greater than 0.1 mg/kg. Discriminative stimulus effects of alprazolam were then assessed by giving 0.03 mg/kg before sessions in which 1.8 mEq/kg lithium chloride was given immediately after the session (alprazolam/lithium session). On intervening days, saline was given before and after the session (saline/saline session). Rates of responding decreased over successive alprazolam/lithium sessions and also during the saline/saline session that immediately followed an alprazolam/lithium session. During subsequent saline/saline sessions, rates of responding returned to levels near baseline rates within two to four sessions. The discriminative stimulus effects of alprazolam were then assessed by giving 0.1 mg/kg before sessions in which 1 mg/kg d-amphetamine was given immediately after the session (alprazolam/d-amphetamine session). Rates of responding decreased during subsequent alprazolam/d-amphetamine sessions in drug-experienced monkeys, but did not decrease during intervening saline/saline sessions. These findings demonstrate that drug stimuli associated with postsession drug injections can rapidly develop control over behavior and suggest that similar methods be explored in the assessment of drug discrimination.