Visual Reinforcement Signals Interfere with the Effects of Reinforcer Magnitude Manipulations

Three experiments examined the effect of reinforcement magnitude on free-operant response rates. In Experiment 1, rats that received four food pellets responded faster than rats that received one pellet on a variable ratio 30 schedule. However, when the food hopper was illuminated during reinforcer delivery, there was no difference between the rates of response produced by the two magnitudes of reward. In Experiment 2, there was no difference in response rates emitted by rats receiving either one or four pellets of food as reward on a random interval (RI) 60-s schedule. In Experiment 3, rats responding on an RI 30-s schedule did so at a lower rate with four pellets as reinforcement than with one pellet. This effect was abolished by the illumination of the food hopper during reinforcement delivery. These results indicate that the influence of magnitude is obscured by manipulations which signal the delivery of reinforcement.     

Neural coding of reward magnitude in the orbitofrontal cortex of the rat during a five-odor olfactory discrimination task

The orbitofrontal cortex (OBFc) has been suggested to code the motivational value of environmental stimuli and to use this information for the flexible guidance of goal-directed behavior. To examine whether information regarding reward prediction is quantitatively represented in the rat OBFc, neural activity was recorded during an olfactory discrimination “go”/“no-go” task in which five different odor stimuli were predictive for various amounts of reward or an aversive reinforcer. Neural correlates related to both actual and expected reward magnitude were observed. Responses related to reward expectation occurred during the execution of the behavioral response toward the reward site and within a waiting period prior to reinforcement delivery. About one-half of these neurons demonstrated differential firing toward the different reward sizes. These data provide new and strong evidence that reward expectancy, regardless of reward magnitude, is coded by neurons of the rat OBFc, and are indicative for representation of quantitative information concerning expected reward. Moreover, neural correlates of reward expectancy appear to be distributed across both motor and nonmotor phases of the task.     

Determinants of contrast in the signal-key procedure: Evidence against additivity theory

Two experiments are reported that challenge the interpretation of previous results with the signal-key procedure, in which the discriminative stimuli are located on a response key different from the key associated with the operant response requirement. Experiment 1 replicated the procedure of Keller (1974), and found that contrast effects on the operant key occurred reliably for only one of four subjects. High rates to the signal key initially occurred for only one subject, but modifications of the procedure produced substantial rates to the signal key for all subjects. In all cases, however, signal-key behavior was greatly reduced by the addition of a changeover delay which prevented reinforcement within 2 seconds of the last peck to the signal key, suggesting that signal-key pecking was maintained primarily by adventitious reinforcement. Experiment 2 modified the signal-key procedure by using three response keys, so that the discriminative stimuli on the signal key controlled different responses during all phases of training. With this modification, reliable contrast effects on the operant key occurred for all subjects, suggesting that the failure to find contrast in previous studies has been due to the confounding of changes in the discrimination requirements with changes in relative rate of reinforcement. The results challenge the additivity theory of contrast, and suggest that “elicited” behavior plays a minor role, if any, in the determination of contrast effects in multiple schedules.     

Transfer between downshift in reward magnitude and continuous delay of reward

Two separate experiments were conducted to investigate transfer of persistence between delay and a downshift in reward magnitude. In the first experiment, experimental rats were initially downshifted in reward magnitude and later tested for persistence to continuous delay of large reward. It was found that these rats were more persistent to the effects of delay than control rats which did not receive prior experience with a downshift in reward magnitude. In the second experiment, experimental rats were first trained to receive large reward under delayed conditions and then tested for persistence to a downshift in reward magnitude. Compared to control rats which received no prior experience with delay, the experimental rats showed a significantly smaller negative contrast effect. The results were interpreted as supporting Amsel's theory of persistence as well as Capaldi's recent interpretation of contrast effects.     

DELAYED REINFORCEMENT OF OPERANT BEHAVIOR

The experimental analysis of delay of reinforcement is considered from the perspective of three questions that seem basic not only to understanding delay of reinforcement, but, also, by implication, the contributions of temporal relations between events to operant behavior. The first question is whether effects of the temporal relation between responses and reinforcers can be isolated from other features of the environment that often accompany delays, such as stimuli or changes in the temporal distribution or rate of reinforcement. The second question is that of the effects of delays on operant behavior. Beyond the common denominator of a temporal separation between reinforcers and the responses that produce them, delay of reinforcement procedures differ from one another along several dimensions, making delay effects circumstance dependent. The final question is one of interpreting delay of reinforcement effects. It centers on the role of the response—reinforcer temporal relation in the context of other, concurrently operating behavioral processes.     

Identification of a reinforcement pathway necessary for operant conditioning of head waving in Aplysia californica

The marine mollusc Aplysia californica exhibits a wide range of nonassociative and associative forms of learning. Recently, we found that the learning repertoire of Aplysia includes operant conditioning (Cook & Carew, 1986, 1989b). The behavior we examined is a naturally occurring, side-to-side head-waving response used by Aplysia in seeking food, obtaining a foothold, and egg laying. Aplysia can be operantly conditioned to reduce head-waving to one side of their body if such a response results in exposure to bright uniform-field illumination, which the animals find aversive. An essential step toward achieving a mechanistic understanding of operant conditioning is to identify and characterize the reinforcement pathway used during the learning. Toward this end, we wished to determine which of the peripheral visual pathways in Aplysia are critical for performance of the operant task. Previous experiments indicated that photic input from the optic and rhinophore nerves functionally inhibited motor neurons that participate in the operant response (head-waving), while photic input from the oral veil nerves excited these same motor neurons (Cook & Carew, 1989c). These findings suggested the hypothesis that one or both of these pathways could play an important role in mediating reinforcement during training. To explore this possibility we operantly trained animals that had received chronic bilateral transections of either the optic and rhinophore nerves or the oral veil nerves C1-C3 (in conjunction with transection of the optic and rhinophore nerves). We found that operant conditioning was not disrupted by ablation of input from the eyes and rhinophores. By contrast, ablation of input from the oral veil (together with that from the eyes and rhinophores) abolished operant conditioning. Thus, the oral veil nerves play a critical modulatory role in operant conditioning of head-waving. This observation further suggested that photic input from the oral veil is conveyed to the CNS via the oral veil nerves. In a final experiment we confirmed that stimulation of the oral veil with light evokes increased afferent activity in the oral veil nerves C1-C2. These results support the idea that the oral veil nerves contain processes that are critical components of the reinforcement pathway for operant conditioning of head-waving.     

Contingency tracking during unsignaled delayed reinforcement

Three experiments were conducted with rats in which responses on one lever (labeled the functional lever) produced reinforcers after an unsignaled delay period that reset with each response during the delay. Responses on a second, nonfunctional, lever did not initiate delays, but, in the first and third experiments, such responses during the last 10 s of a delay did postpone food delivery another 10 s. In the first experiment, the location of the two levers was reversed several times. Responding generally was higher on the functional lever, though the magnitude of the difference diminished with successive reversals. In the second experiment, once a delay was initiated by a response on the functional lever, in different conditions responses on the nonfunctional lever either had no effect or postponed food delivery by 30 s. The latter contingency typically lowered response rates on the nonfunctional lever. In the first two experiments, both the functional and nonfunctional levers were identical except for their location; in the third experiment, initially, a vertically mounted, pole-push lever defined the functional response and a horizontally mounted lever defined the nonfunctional response. Higher response rates occurred on the functional lever. These results taken together suggest that responding generally tracked the response-reinforcer contingency. The results further show how nonfunctional-operanda responses are controlled by a prior history of direct reinforcement of such responses, by the temporal delay between such responses and food delivery, and as simple generalization between the two operanda.     

Response-initiated imaging of operant behavior using a digital camera

A miniature digital camera, QuickCam Pro 3000, intended for use with video e-mail, was modified so that snapshots were triggered by operant behavior emitted in a standard experimental chamber. With only minor modification, the manual shutter button on the camera was replaced with a simple switch closure via an I/O interface controlled by a PC computer. When the operant behavior activated the I/O switch, the camera took a snapshot of the subject's behavior at that moment. To illustrate the use of the camera, a simple experiment was designed to examine stereotypy and variability in topography of operant behavior under continuous reinforcement and extinction in 6 rats using food pellets as reinforcement. When a rat operated an omnidirectional pole suspended from the ceiling, it also took a picture of the topography of its own behavior at that moment. In a single session after shaping of pole movement (if necessary), blocks of continuous reinforcement, in which each response was reinforced, alternated with blocks of extinction (no reinforcement), with each block ending when 20 responses had occurred. The software supplied with the camera automatically stored each image and named image files successively within a session. The software that controlled the experiment also stored quantitative data regarding the operant behavior such as consecutive order, temporal location within the session, and response duration. This paper describes how the two data types--image information and numerical performance characteristics-can be combined for visual analysis. The experiment illustrates in images how response topography changes during shaping of pole movement, how response topography quickly becomes highly stereotyped during continuous reinforcement, and how response variability increases during extinction. The method of storing digital response-initiated snapshots should be useful for a variety of experimental situations that are intended to examine behavior change and topography.     

Reinforcement magnitude and responding during treatment with differential reinforcement

Basic findings indicate that the amount or magnitude of reinforcement can influence free-operant responding prior to and during extinction. In this study, the relation between reinforcement magnitude and adaptive behavior was evaluated with 3 children as part of treatment with differential reinforcement. In the first experiment, a communicative response was shaped and maintained by the same reinforcer that was found to maintain problem behavior. Two reinforcement magnitudes (20-s or 60-s access to toys or escape from demands) were compared and found to be associated with similar levels of resistance to extinction. The relation between reinforcement magnitude and response maintenance was further evaluated in the second experiment by exposing the communicative response to 20-s or 300-s access to toys or escape. Results for 2 participants suggested that this factor may alter the duration of postreinforcement pauses.     

Dorsal striatum responses to reward and punishment: Effects of valence and magnitude manipulations

The goal of this research was to further our understanding of how the striatum responds to the delivery of affective feedback. Previously, we had found that the striatum showed a pattern of sustained activation after presentation of a monetary reward, in contrast to a decrease in the hemodynamic response after a punishment. In this study, we tested whether the activity of the striatum could be modulated by parametric variations in the amount of financial reward or punishment. We used an event-related fMRI design in which participants received large or small monetary rewards or punishments after performance in a gambling task. A parametric ordering of conditions was observed in the dorsal striatum according to both magnitude and valence. In addition, an early response to the presentation of feedback was observed and replicated in a second experiment with increased temporal resolution. This study further implicates the dorsal striatum as an integral component of a reward circuitry responsible for the control of motivated behavior, serving to code for such feedback properties as valence and magnitude.     

Positive affect and learning: exploring the “Eureka Effect” in dogs

Animals may experience positive affective states in response to their own achievements. We investigated emotional responses to problem-solving in dogs, separating these from reactions to rewards per se using a yoked control design. We also questioned whether the intensity of reaction would vary with reward type. We examined the response (behavior and heart rate) of dogs as they learned to gain access to different rewards: (1) food (2) human contact, and (3) dog contact. Twelve beagles were assigned to matched pairs, and each dog served as both an experimental and a control animal during different stages of the experiment. We trained all dogs to perform distinct operant tasks and exposed them to additional devices to which they were not trained. Later, dogs were tested in a new context. When acting as an experimental dog, access to the reward was granted immediately upon completion of trained operant tasks. When acting as a control, access to the reward was independent of the dog’s actions and was instead granted after a delay equal to their matched partner’s latency to complete their task. Thus, differences between the two situations could be attributed to experimental dogs having the opportunity to learn to control access to the reward. Experimental dogs showed signs of excitement (e.g., increased tail wagging and activity) in response to their achievements, whereas controls showed signs of frustration (e.g., chewing of the operant device) in response to the unpredictability of the situation. The intensity of emotional response in experimental dogs was influenced by the reward type, i.e., greatest response to food and least to another dog. Our results suggest that dogs react emotionally to problem-solving opportunities and that tail wagging may be a useful indicator of positive affective states in dogs.     

Introduction to the special issue on comparative cognition

Prior research has shown that nonhumans show an extreme preference for variable‐ over fixed‐delays to reinforcement. This well‐established preference for variability occurs because a reinforcer's strength or “value” decreases according to a curvilinear function as its delay increases. The purpose of the present experiments was to investigate whether this preference for variability occurs with human participants making hypothetical choices. In three experiments, participants recruited from Amazon Mechanical Turk made choices between variable and fixed monetary rewards. In a variable‐delay procedure, participants repeatedly chose between a reward delivered either immediately or after a delay (with equal probability) and a reward after a fixed delay (Experiments 1 and 2). In a double‐reward procedure, participants made choices between an alternative consisting of two rewards, one delivered immediately and one after a delay, and a second alternative consisting of a single reward delivered after a delay (Experiments 1 and 3). Finally, all participants completed a standard delay‐discounting task. Although we observed both curvilinear discounting and magnitude effects in the standard discounting task, we found no consistent evidence of a preference for variability—as predicted by two prominent models of curvilinear discounting (i.e., a simple hyperbola and a hyperboloid)—in our variable‐delay and double‐reward procedures. This failure to observe a preference for variability may be attributed to the hypothetical, rule‐governed nature of choices in the present study. In such contexts, participants may adopt relatively simple strategies for making more complex choices.     

Defining reinforcess — A problem in communication for consultation in behavior modification

This paper reviews the respondent (Hull-Spence) and operant (Skinnerian) conditioning definitions of reinforcers and reinforcement and demonstrates the need to keep the systems separate when consulting about behavior modification. The two systems are shown to lead to different modification procedures.One important distinction between the systems is whether a reinforcer is simply associated with a response (respondent) or whether it must follow the response (operant). A second important distinction is the definition of negative reinforcement. In respondent conditioning, negative reinforcement entails presenting an aversive stimulus in association with the response and results in a decrease in response rate. In operant conditioning, negative reinforcement entails the removal of an aversive stimulus following a correct response, which results in an increase in response rate.     

Operant and nonoperant vocal responding in the mynah: Complex schedule control and deprivation-induced responding

Several recent studies have been concerned with operant responses that are also affected by nonoperant factors, (e.g., biological constraints, innate behavior patterns, respondent processes). The major reason for studying mynah vocal responding concerned the special relation of avian vocalizations to nonoperant emotional and reflexive systems. The research strategy was to evaluate operant and nonoperant control by comparing the schedule control obtained with the vocal response to that characteristic of the motor responses of other animals. We selected single, multiple, and chain schedules that ordinarily produce disparate response rates at predictable times. In multiple schedules with one component where vocal responding (“Awk”) was reinforced with food (fixed-ratio or fixed-interval schedule) and one where the absence of vocal responding was reinforced (differential reinforcement of other behavior), response rates never exceeded 15 responses per minute, but clear schedule differences developed in response rate and pause time. Nonoperant vocal responding was evident when responding endured across 50 extinction sessions at 25% to 40% of the rate during reinforcement. The “enduring extinction responding” was largely deprivation induced, because the operant-level of naive mynahs under food deprivation was comparable in magnitude, but without deprivation the operant level was much lower. Food deprivation can induce vocal responding, but the relatively precise schedule control indicated that operant contingencies predominate when they are introduced.     

Feeding behavior of Aplysia: a model system for comparing cellular mechanisms of classical and operant conditioning

Feeding behavior of Aplysia provides an excellent model system for analyzing and comparing mechanisms underlying appetitive classical conditioning and reward operant conditioning. Behavioral protocols have been developed for both forms of associative learning, both of which increase the occurrence of biting following training. Because the neural circuitry that mediates the behavior is well characterized and amenable to detailed cellular analyses, substantial progress has been made toward a comparative analysis of the cellular mechanisms underlying these two forms of associative learning. Both forms of associative learning use the same reinforcement pathway (the esophageal nerve, En) and the same reinforcement transmitter (dopamine, DA). In addition, at least one cellular locus of plasticity (cell B51) is modified by both forms of associative learning. However, the two forms of associative learning have opposite effects on B51. Classical conditioning decreases the excitability of B51, whereas operant conditioning increases the excitability of B51. Thus, the approach of using two forms of associative learning to modify a single behavior, which is mediated by an analytically tractable neural circuit, is revealing similarities and differences in the mechanisms that underlie classical and operant conditioning.     

Self-reinforcement

Self-reinforcement in operant situations generally refers to those arrangements in which the subject delivers to himself a consequence, contingent on his behavior. However, it is noted that the definition of all other types of reinforcement make its delivery contingent on the subject's behavior. What is actually at issue is the agent who defines whether or not the response required for reinforcement has been met. In self-reinforcement, the subject himself defines this. In the laboratory, this requirement is machine-defined; in school examinations, it is teacher-defined; and in many clinical self-control situations, it is also independently defined. A reinforcement contingency presupposes such independence, absent in self-reinforcement. Implications for research and practice are discussed and alternative formulations are offered.     

Transfer across drives of the discriminative effect of a Pavlovian conditioned stimulus

Previous results suggest that a stimulus paired in Pavlovian fashion with reward should exert some discriminative control over an unrelated operant response acquired under a different drive-reward system. In the following experiment, a stimulus was first paired with food reinforcement for a hungry rat. Subsequently, the animal learned to lever-press for water reinforcements when thirsty but not hungry. Finally, the control over lever-pressing of the food-paired stimulus was tested by presenting it at various times during extinction of the lever-pressing response. All animals in the experiment showed the expected effect; each emitted more lever-presses during periods of the food-paired stimulus than during alternate control periods.     

Social discounting and delay discounting

Social discounting was measured as the amount of money a participant was willing to forgo to give a fixed amount (usually $75) to another person. In the first experiment, amount forgone was a hyperbolic function of the social distance between the giver and receiver. In the second experiment, degree of social discounting was an increasing function of reward magnitude whereas degree of delay discounting was a decreasing function of reward magnitude. In the third experiment, the shape of the function relating delayed rewards to equally valued immediate rewards for another person was predicted from individual delay and social discount functions. All in all, the studies show that the social discount function, like delay and probability discount functions, is hyperbolic in form. Copyright © 2007 John Wiley & Sons, Ltd.     

Differential reinforcement of lever-press durations: Effects of deprivation level and reward magnitude

Three experiments investigated the performance of rats on a task involving differential reinforcement of lever-press durations. Experiment 1, which employed a discrete-trials procedure, manipulated deprivation level between subjects and reward magnitude within subjects. The minimum lever-press duration which would result in reward was varied from .4 to 6.4 sec. It was found that low deprivation resulted in longer mean durations and less response variability at the higher criterial values than did high deprivation. The magnitude of reward was not found to affect performance. Experiment 2 manipulated reward magnitude between subjects while holding deprivation level constant, and used the same general procedures as in Experiment 1. Small reward resulted in longer mean lever-press durations and less variability in responding than did large reward at the higher criterial values. The intertrial intervals were omitted in Experiment 3 in which deprivation level was varied between subjects and reinforcement was delivered only for response durations extending between 6.0 and 7.6 sec. Low deprivation resulted in longer mean lever-press durations and less response variability than did high deprivation, but the probability of a rewarded press duration did not differ between groups. The results overall are consistent with the hypothesis that low deprivation and small reward magnitude lead to weaker goal-approach responses and, hence, to less competition with lever holding. The deprivation and reward magnitude manipulations did not appear to influence lever holding performance by affecting the ability of animals to form temporal discriminations.     

The dynamics of operant conditioning

Existing models of operant learning are relatively insensitive to historical properties of behavior and applicable to only limited data sets. This article proposes a minimal set of principles based on short-term and long-term memory mechanisms that can explain the major static and dynamic properties of operant behavior in both single-choice and multiresponse situations. The critical features of the theory are as follows: (a) The key property of conditioning is assessment of the degree of association between responses and reinforcement and between stimuli and reinforcement; (b) the contingent reinforcement is represented by learning expectancy, which is the combined prediction of response-reinforcement and stimulus-reinforcement associations; (c) the operant response is controlled by the interplay between facilitatory and suppressive variables that integrate differences between expected (long-term) and experienced (short-term) events; and (d) very-long-term effects are encoded by a consolidated memory that is sensitive to the entire reinforcement history. The model predicts the major qualitative features of operant phenomena and then suggests an experimental test of theoretical predictions about the joint effects of reinforcement probability and amount of training on operant choice. We hypothesize that the set of elementary principles that we propose may help resolve the long-standing debate about the fundamental variables controlling operant conditioning.