Conditioned reinforcement in second-order schedules

Pigeons responded under a schedule in which food was presented only after a fixed number of fixed-interval components were completed. Two such second-order schedules were studied: under one, 30 consecutive 2-min fixed-interval components were required; under the other, 15 consecutive 4-min fixed-interval components were required. Under both schedules, when a 0.7-sec stimulus light was presented at completion of each fixed interval, positively accelerated responding developed in each component. When no stimulus change occurred at completion of each fixed interval, relatively low and constant rates of responding prevailed in each component; a similar result was obtained when a 0.7-sec stimulus change occurred at completion of each fixed interval except the one which terminated with primary reinforcement. The 0.7-sec stimulus correlated with food delivery was an effective conditioned reinforcer in maintaining patterns of responding in fixed-interval components despite low average frequencies of food reinforcement.     

Conditioned reinforcement and choice

In a series of three experiments, rats were exposed to successive schedule components arranged on two levers, in which lever pressing produced a light, and nose-key pressing produced water in 50% of the light periods. When one auditory signal was presented only during those light periods correlated with water on one lever, and a different signal was presented only during those light periods correlated with nonreinforcement on the other lever, the former lever was preferred in choice trials, and higher rates of responding were maintained on the former lever in nonchoice (forced) trials. Thus, the rats preferred a schedule component that included a conditioned reinforcer over one that did not, with the schedules of primary reinforcement and the information value of the signals equated. Preferences were maintained when one or the other of the auditory signals was deleted, but were not established in naive subjects when training began with either the positive or negative signal only. Discriminative control of nose-key pressing by the auditory signals was highly variable across subjects and was not correlated with choice.     

Conditioned reinforcement and discrimination performance

Pigeons were trained in a three-key chamber to peck one side key in the presence of a vertical line on the center key and to peck the other side key in the presence of a horizontal line. Correct choice responses were reinforced with food according to fixed- and variable-ratio, fixed-interval, and differential-reinforcement-of-long-latency schedules of reinforcement. For each schedule, the birds performed under each of two conditions: (1) each correct choice response produced a brief presentation of stimuli intermittently paired with food, then the next trial; (2) each correct choice response produced an intertrial interval only. For all schedules except one long latency schedule, response rates were higher under the condition of brief stimulus presentation than under the comparable control condition. Presentation of brief magazine stimuli increased choice accuracy. The amount of change in accuracy was correlated with the rate of food presentation. Performance under the schedules with highest food reinforcement rates showed no enhancement; performance under the schedules with the lowest reinforcement rates showed the greatest enhancement.     

Conditioned reinforcement and response strength

Stimuli associated with primary reinforcers appear themselves to acquire the capacity to strengthen behavior. This paper reviews research on the strengthening effects of conditioned reinforcers within the context of contemporary quantitative choice theories and behavioral momentum theory. Based partially on the finding that variations in parameters of conditioned reinforcement appear not to affect response strength as measured by resistance to change, long-standing assertions that conditioned reinforcers do not strengthen behavior in a reinforcement-like fashion are considered. A signposts or means-to-an-end account is explored and appears to provide a plausible alternative interpretation of the effects of stimuli associated with primary reinforcers. Related suggestions that primary reinforcers also might not have their effects via a strengthening process are explored and found to be worthy of serious consideration.     

Conditioned reinforcement versus time to reinforcement in chain schedules.

Pigeons were trained on three-component chain schedules in which the initial component was either a fixed-interval or variable-interval schedule. The middle and terminal components were varied among fixed-interval fixed-interval, variable-interval variable-interval, and an interdependent variable-interval variable-interval schedule in which the sum of the durations of the two variable-interval components was always equal to the sum of the fixed-interval fixed-interval components. At issue was whether the response rate in the initial component was controlled by its time to primary reinforcement or by the temporal parameters of the stimulus correlated with the middle terminal link. The fixed-interval initial-link schedule maintained much lower response rates than the variable-interval initial-link schedule regardless of the schedules in the middle and terminal links. Nevertheless, the intervening schedules played some role: With fixed-interval schedules in the initial links, response rates were consistently highest with independent variable-interval schedules in the middle and terminal links and intermediate with the interdependent variable-interval schedules; these initial-link differences were predicted by the response rates in the middle link of the chain. With variable-interval schedules in the initial links, response rates were lowest with the fixed-interval fixed-interval schedules following the initial link and were not systematically different for the two types of variable-interval variable-interval schedules. The results suggest that time to reinforcement itself accounts for little if any variance in initial-link responding.     

Conditioned reinforcement and discrimination in second-order schedules

Pigeons were exposed to multiple second-order schedules in which responding on the “main key” was reinforced according to either a variable-interval or fixed-interval schedule by production of a brief stimulus on the “brief-stimulus key”. A response was required to the brief stimulus during its fourth (final) presentation to produce food; responses to the earlier brief stimuli indicated the extent to which the final brief stimulus was discriminated from preceding ones. Main-key response rates were higher in early components of paired brief-stimulus schedules, in which each brief stimulus was the same as that paired with reinforcement, than in comparable unpaired brief-stimulus or tandem schedules. Poor discrimination occurred between paired brief stimuli (Experiment I). When chain stimuli on the main key induced a discrimination between the first two and second two brief stimuli, the response-rate enhancement in the paired brief-stimulus schedule persisted (Experiment II). Rate enhancement diminished when the initial link of the chain included the first three components (Experiment IV). Eliminating the contingency between responding and brief-stimulus production also diminished rate enhancement (Experiment III). The results show that the discriminative and conditioned reinforcing effects of food-paired brief stimuli may be selectively manipulated and suggest that the reinforcing effects are modulated by other reinforcers in the situation.     

Conditioned suppression and variable ratio reinforcement

Conditioned suppression is a decrease in response rate during a relatively short duration stimulus that terminates independently of the animal's behavior and coincidentally with a brief unavoidable shock. The degree of conditioned suppression was measured for each of three birds on three variable ratio schedules; that is, the number of responses required for food reinforcement was varied around a mean of 50, 100, or 200. The results indicated a slight and possibly negligible decrease in the degree of suppression as the mean number of responses required on the schedule was increased from 50, to 100, and 200. In general, it was found that all of the variable ratio schedules tested were quite insensitive to the conditioned suppression procedure, although almost complete suppression was obtained on a few occasions. Since the reinforcement was contingent upon the emission of responses, the birds typically displayed a high rate of response during the pre-shock stimulus on all schedules. In addition, the rate during the pre-shock stimulus often changed abruptly independent of the presentation of a reinforcement. As a result of the high rate of response and the abrupt changes in rate, the degree of suppression from trial to trial was quite variable. A clear analysis of an experimental variable on this baseline is thus difficult.     

Conditioned reinforcement value and choice.

The delay-reduction hypothesis of conditioned reinforcement states that the reinforcing value of a food-associated stimulus is determined by the delay to primary reinforcement signaled by the onset of the stimulus relative to the average delay to primary reinforcement in the conditioning situation. In contrast, most contemporary models of conditioned reinforcement strength posit that the reinforcing strength of a stimulus is some simple function only of the delay to primary reinforcement in the presence of stimulus. The delay-reduction hypothesis diverges from other conditioned reinforcement models in that it predicts that a fixed-duration food-paired stimulus will have different reinforcing values depending on the frequency of its presentation. In Experiment 1, pigeons' key pecks were reinforced according to concurrent-chains schedules with variable-interval 10-second and variable-interval 20-second terminal-link schedules. The initial-link schedule preceding the shorter terminal link was always variable-interval 60 seconds, and the initial-link schedule requirement preceding the longer terminal link was varied between 1 second and 60 seconds across conditions. In Experiment 2, the initial-link schedule preceding the longer of two terminal links was varied for each of three groups of pigeons. The terminal links of the concurrent chains for the three groups were variable-interval 10 seconds and 20 seconds, variable-interval 10 seconds and 30 seconds, and variable-interval 30 seconds and 50 seconds. In both experiments, preference for the shorter terminal link was either a bitonic function or an inverse function of the initial-link schedule preceding the longer terminal-link schedule. Consistent with the predictions of the delay-reduction hypothesis, the relative values of the terminal-link stimuli changed as a function of the overall frequency of primary reinforcement. Vaughan's (1985) melioration model, which was shown to be formally similar to Squires and Fantino's (1971) delay-reduction model, can be modified so as to predict these results without changing its underlying assumptions.     

Conditioned reinforcement dynamics in three-link chained schedules.

In two experiments rats were trained on three-link concurrent-chains schedules of reinforcement. In Experiment 1, additional entries to one terminal link were added during one of the middle links to a baseline schedule that was otherwise equal for the two chains, and, depending on the condition, these additional terminal-link presentations ended either in food or in no food. When food occurred, preference was always in favor of the chain with the additional terminal-link presentations (which also entailed a higher rate of reinforcement). When no food occurred at the end of the additional terminal links, the outcome depended on the nature of the stimuli associated with these additional terminal links. When stimuli different from the reinforced baseline terminal links were used for the no-food terminal links, preference was against the choice alternative that led to the extra periods of extinction. When the same stimulus was used for the two kinds of terminal links, preference was near indifference, that is, significantly greater than when different stimuli were used. In Experiment 2, rats learned repeated reversals of a simultaneous discrimination under a three-link concurrent-chains schedule, in which the food or no-food choice outcomes were delayed until the end of the chain. Different conditions were defined by the point in the chain at which differential stimuli occurred. When the middle and terminal links provided no differential stimuli, discrimination was acquired more slowly than when differential stimuli occurred in both links. When differential stimuli occurred in the middle but not the terminal links, acquisition rates were intermediate. Both experiments together show that the effects of stimuli in a chain schedule are due partly to the time to food correlated with the stimuli and partly to the time to the next conditioned reinforcer in the sequence.     

Conditioned reinforcement value and resistance to change

Three experiments examined the effects of conditioned reinforcement value and primary reinforcement rate on resistance to change using a multiple schedule of observing-response procedures with pigeons. In the absence of observing responses in both components, unsignaled periods of variable-interval (VI) schedule food reinforcement alternated with extinction. Observing responses in both components intermittently produced 15 s of a stimulus associated with the VI schedule (i.e., S+). In the first experiment, a lower-valued conditioned reinforcer and a higher rate of primary reinforcement were arranged in one component by adding response-independent food deliveries uncorrelated with S+. In the second experiment, one component arranged a lower valued conditioned reinforcer but a higher rate of primary reinforcement by increasing the probability of VI schedule periods relative to extinction periods. In the third experiment, the two observing-response components provided similar rates of primary reinforcement but arranged different valued conditioned reinforcers. Across the three experiments, observing-response rates were typically higher in the component associated with the higher valued conditioned reinforcer. Resistance to change was not affected by conditioned reinforcement value, but was an orderly function of the rate of primary reinforcement obtained in the two components. One interpretation of these results is that S+ value does not affect response strength and that S+ deliveries increase response rates through a mechanism other than reinforcement. Alternatively, because resistance to change depends on the discriminative stimulus-reinforcer relation, the failure of S+ value to impact resistance to change could have resulted from a lack of transfer of S+ value to the broader discriminative context.     

Conditioned reinforcement by conditional discriminative stimuli.

A concurrent-chains schedule was used to examine how a delay to conditional discriminative stimuli affects conditioned reinforcement strength. Pigeons' key-peck responses in the initial link produced either of two terminal links according to independent variable-interval 30-s schedules. Each terminal link involved an identical successive conditional discrimination and was segmented into three links: a delay interval (green), a color conditional discriminative stimulus (blue or red), and a line conditional discriminative stimulus (vertical or horizontal lines). Food delivery occurred 45 s after entering the terminal link with a probability of .5, but its conditional probability (1.0 or 0) depended on the combination of the color and the line stimuli. One of the color stimuli occurred independently of further responding, 5 s after entry into the right terminal link, but it occurred 35 s after entry into the left terminal link. One of the line stimuli occurred independently of responding 40 s after entry into either terminal link, synchronized with the offset of the color stimulus. The initial-link relative response rate for the right was consistently higher in comparison with a control condition in which the color stimuli occurred 20 s after entry into either terminal link. The preference for the short delay to the color conditional discriminative stimuli suggests the possibility of conditioned reinforcement by information about the relation between the line conditional discriminative stimuli and the outcomes.     

Conditioned reinforcement: Experimental and theoretical issues

The concept of conditioned reinforcement has received decreased attention in learning textbooks over the past decade, in part because of criticisms of its validity by major behavior theorists and in part because its explanatory function in a variety of different conditioning procedures has become uncertain. Critical data from the major procedures that have been used to investigate the concept (second-order schedules, chain schedules, concurrent chains, observing responses, delay-of-reinforcement procedures) are reviewed, along with the major issues of interpretation. Although the role played by conditioned reinforcement in some procedures remains unresolved, the results taken together leave little doubt that the underlying idea of conditioned value is a critical component of behavior theory that is necessary to explain many different types of data. Other processes (marking, bridging) may also operate to produce effects similar to those of conditioned reinforcement, but these clearly cannot explain the full domain of experimental effects ascribed to conditioned reinforcement and should be regarded as complements to the concept rather than theoretical competitors. Examples of practical and theoretical applications of the concept of conditioned reinforcement are also considered.     

Conditioned inhibition from incomplete reductions in the probability of reinforcement

Five experiments, all using appetitive, discrete trial operant conditioning, studied the properties of a stimulus that signalled a reduction in the probability of reward. Discriminations were trained in which reinforcement was available on 100% of trials signalled by a tone, but only on some occasions when the tone was presented simultaneously with a light. The properties of the light were assessed in summation tests with a clicker. The first two experiments established that if the tone-light compound signalled reinforcement on only 25%, 33% or 50% of trials the light acted as a discriminative inhibitor, suppressing responding maintained by the clicker. In these experiments reinforcement had been available on 86.7% of clicker trials during initial training. Experiments 3, 4 and 5 examined further the properties of the light after animals had been trained on a discrimination where reinforcement was available on 50% of trials signalled by the tone-light compound. The light was evaluated in a summation test with a clicker, which had signalled only a 25% or 15% probability of a reinforcement. In this instance, the light did not suppress the responding maintained by the clicker. Instead, the light acted as an excitatory discriminative stimulus to enhance that responding. This dependence of the properties of the light on the reinforcement history of the clicker poses problems for most contemporary models of inhibitory performance.     

Conditioned reinforcement as a function of duration of stimulus

Pigeons were provided with three keys. Pecking the center key produced grain on a schedule that alternated at unpredictable times between a variable-interval component and extinction. On concurrent variable-interval schedules, pecking either side key produced a stimulus associated with the variable-interval component on the center key provided that said schedule was currently in effect. The independent variable was the length of time this stimulus remained on the keys. Pecking one side key produced the stimulus for 27 seconds, whereas the duration produced by pecking the other key varied for successive blocks of sessions. For the first four birds, the values tested were 3, 9, 27, and 81 seconds. For the second group, numbering three birds, the values tested were 1, 3, 9, and 27 seconds. The dependent variable was the proportion of total side key pecks that occurred on the variable key. For all birds, the function was positive in slope and negative in acceleration. This finding supports a formulation that ascribes the maintenance of observing responses in a normal setting to the fact that the subject exposes itself to the positive discriminative stimulus for a longer mean duration than it does to the negative stimulus.     

Conditioned reinforcement and choice with delayed and uncertain primary reinforcers.

In an adjusting-delay choice procedure, pigeons could peck on either a red key or a green key. A peck on the red key always led to a delay associated with red houselights and then food. The delay was adjusted over trials to estimate an indifference point--a delay at which the two keys were chosen about equally often. In some conditions, a peck on the green key led to food on all trials after delays of either 10 s or 30 s, and green houselights were lit during the delays. In other conditions, food was presented on only half of the green-key trials. If the green houselights continued to occur on both reinforcement and nonreinforcement trials, preference for the green key always decreased. Preference for the green key also decreased if half of the trials had 30-s houselights followed by food and the other half had no green houselights and no food. However, preference for the green key actually increased if half of the trials had 10-s green houselights followed by food and the other half had no green houselights followed by no food. The latter condition therefore demonstrated a case in which preference for an alternative increased when food was removed from half of the trials. The results suggest that the red and green houselights served as conditioned reinforcers. A hyperbolic decay model (Mazur, 1989) provided good predictions for all conditions by assuming that the strength of a conditioned reinforcer is inversely related to the total time spent in its presence before food is delivered.     

Autonomy in chimpanzees

Literature on the mental capacities and cognitive mechanisms of the great apes has been silent about whether they can act autonomously. This paper provides a philosophical theory of autonomy supported by psychological studies of the cognitive mechanisms that underlie chimpanzee behavior to argue that chimpanzees can act autonomously even though their psychological mechanisms differ from those of humans. Chimpanzees satisfy the two basic conditions of autonomy: (1) liberty (the absence of controlling influences) and (2) agency (self-initiated intentional action), each of which is specified here in terms of conditions of understanding, intention, and self-control. In this account, chimpanzees make knowledge-based choices reflecting a richly information-based and socially sophisticated understanding of the world. Finally, two major theories of autonomy (Kantian theory and two-level theory) are rejected as too narrow to adequately address these issues, necessitating the modifications made in the present approach.