The effects of a local negative feedback function between choice and relative reinforcer rate

Four pigeons were trained on two-key concurrent variable-interval schedules with no changeover delay. In Phase 1, relative reinforcers on the two alternatives were varied over five conditions from .1 to .9. In Phases 2 and 3, we instituted a molar feedback function between relative choice in an interreinforcer interval and the probability of reinforcers on the two keys ending the next interreinforcer interval. The feedback function was linear, and was negatively sloped so that more extreme choice in an interreinforcer interval made it more likely that a reinforcer would be available on the other key at the end of the next interval. The slope of the feedback function was -1 in Phase 2 and -3 in Phase 3. We varied relative reinforcers in each of these phases by changing the intercept of the feedback function. Little effect of the feedback functions was discernible at the local (interreinforcer interval) level, but choice measured at an extended level across sessions was strongly and significantly decreased by increasing the negative slope of the feedback function.     

Choice in a variable environment: effects of unequal reinforcer distributions

Six pigeons were trained in a procedure in which sessions included seven unsignaled components, each offering two pecking keys, and each providing a potentially different reinforcer ratio between the two keys. Across conditions, various combinations of reinforcer ratios and reinforcer-magnitude ratios were used to create unequal reinforcer distributions between the two alternatives when averaged across a session. The results extended previous research using the same basic procedure that had included only reinforcer distributions symmetrical around 1:1. Data analyses suggested that the variables controlling choice operated at a number of levels: First, individual reinforcers had local effects on choice; second, sequences of successive reinforcers obtained at the same alternative (continuations) had cumulative effects; and, third, when these sequences themselves occurred with greater frequency, their effects further cumulated. A reinforcer obtained at the other alternative following a sequence of continuations (a discontinuation) had a large effect and apparently reset choice to levels approximating the sessional reinforcer ratio.     

Choosing among multiple alternatives: Relative and overall reinforcer rates

Choice behavior among two alternatives has been widely researched, but fewer studies have examined the effect of multiple (more than two) alternatives on choice. Two experiments investigated whether changing the overall reinforcer rate affected preference among three and four concurrently scheduled alternatives. Experiment 1 trained six pigeons on concurrent schedules with three alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 9:3:1 with the configuration counterbalanced across pigeons. The overall rate of reinforcement was varied across conditions. Preference between the pair of keys arranging the 9:3 reinforcer ratio was less extreme than the pair arranging the 3:1 reinforcer ratio regardless of overall reinforcer rate. This difference was attributable to the richer alternative receiving fewer responses per reinforcer than the other alternatives. Experiment 2 trained pigeons on concurrent schedules with four alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 8:4:2:1, and the overall reinforcer rate was varied. Next, two of the alternatives were put into extinction and the random interval duration was changed from 60 s to 5 s. The ratio of absolute response rates was independent of interval length across all conditions. In both experiments, an analysis of sequences of visits following each reinforcer showed that the pigeons typically made their first response to the richer alternative irrespective of which alternative was just reinforced. Performance on these three‐ and four‐alternative concurrent schedules is not easily extrapolated from corresponding research using two‐alternative concurrent schedules.     

Melioration revisited: a systematic replication of Vaughan (1981)

Organisms that behave so as to forfeit a relatively higher overall rate of reinforcement in favor of a relatively lower rate are said to engage in suboptimal choice. Suboptimal choice has been linked with maladaptive behavior in humans. Melioration theory offers one explanatory framework for suboptimal choice. Melioration theory suggests behavior is controlled by differences in local reinforcer rates between alternatives. Vaughan (1981) arranged two experimental conditions in which maximizing the overall rate of reinforcement required behavior that was compatible, or incompatible, with melioration. Vaughan found pigeons allocated more time to a locally richer alternative even when doing so resulted in suboptimal choice. However, Vaughan did not show whether these effects could systematically reverse and did not provide within‐session data to show that choice across short time spans remains under the control of differences in local reinforcer rates. The present study used pigeons to replicate and extend Vaughan's findings. We investigated shifts in overall‐ and within‐session choice across repeated conditions, according to arranged local contingencies. Behavior systematically followed changes in local contingencies for most pigeons. Within‐session data suggests that, providing differences in local reinforcer rates are discriminated, pigeons will allocate more time to a locally richer alternative, even if this leads to suboptimal choice. These findings facilitate the more confident use of similar procedures that investigate how melioration contributes to suboptimal choice.     

Fixed-interval performance and self-control in children.

Operant responses of 16 children (mean age 6 years and 1 month) were reinforced according to different fixed-interval schedules (with interreinforcer intervals of 20, 30, or 40 s) in which the reinforcers were either 20-s or 40-s presentations of a cartoon. In another procedure, they received training on a self-control paradigm in which both reinforcer delay (0.5 s or 40 s) and reinforcer duration (20 s or 40 s of cartoons) varied, and subjects were offered a choice between various combinations of delay and duration. Individual differences in behavior under the self-control procedure were precisely mirrored by individual differences under the fixed-interval schedule. Children who chose the smaller immediate reinforcer on the self-control procedure (impulsive) produced short postreinforcement pauses and high response rates in the fixed-interval conditions, and both measures changed little with changes in fixed-interval value. Conversely, children who chose the larger delayed reinforcer in the self-control condition (the self-controlled subjects) exhibited lower response rates and long postreinforcement pauses, which changed systematically with changes in the interval, in their fixed-interval performances.     

A comparison of presession and within-session reinforcement choice.

Single- and concurrent-operants procedures were used to evaluate the effects of two reinforcement conditions on the free-operant responding of 3 individuals with developmental disabilities and 1 with attention deficit disorder. In the presession choice condition, prior to each session the participant chose one item from an array of three different highly preferred stimuli. This item was delivered by the experimenter on each reinforcer delivery during that session. In the within-session choice condition, each reinforcer delivery consisted of placing an array of three different highly preferred stimuli in front of the participant, who was allowed to select one. Only one of the two reinforcement conditions was in effect for any particular session in single-operant phases. Buttons associated with each reinforcement condition were present, and the participant could allocate responses to one or the other in concurrent-operants phases. Data showed substantially more responding to the button associated with within-session choice than presession choice during concurrent-operants phases. This effect was not as apparent during single-operant phases, suggesting that a concurrent-operants procedure provided the more sensitive evaluation of within-session and presession reinforcer choice effects.     

Two‐year‐olds use adults' but not peers' points

In the current study, 24‐ to 27‐month‐old children (N = 37) used pointing gestures in a cooperative object choice task with either peer or adult partners. When indicating the location of a hidden toy, children pointed equally accurately for adult and peer partners but more often for adult partners. When choosing from one of three hiding places, children used adults’ pointing to find a hidden toy significantly more often than they used peers’. In interaction with peers, children's choice behavior was at chance level. These results suggest that toddlers ascribe informative value to adults’ but not peers’ pointing gestures, and highlight the role of children's social expectations in their communicative development.     

Effects of signaled and unsignaled alternative reinforcement on persistence and relapse in children and pigeons

Three experiments explored the impact of different reinforcer rates for alternative behavior (DRA) on the suppression and post‐DRA relapse of target behavior, and the persistence of alternative behavior. All experiments arranged baseline, intervention with extinction of target behavior concurrently with DRA, and post‐treatment tests of resurgence or reinstatement, in two‐ or three‐component multiple schedules. Experiment 1, with pigeons, arranged high or low baseline reinforcer rates; both rich and lean DRA schedules reduced target behavior to low levels. When DRA was discontinued, the magnitude of relapse depended on both baseline reinforcer rate and the rate of DRA. Experiment 2, with children exhibiting problem behaviors, arranged an intermediate baseline reinforcer rate and rich or lean signaled DRA. During treatment, both rich and lean DRA rapidly reduced problem behavior to low levels, but post‐treatment relapse was generally greater in the DRA‐rich than the DRA‐lean component. Experiment 3, with pigeons, repeated the low‐baseline condition of Experiment 1 with signaled DRA as in Experiment 2. Target behavior decreased to intermediate levels in both DRA‐rich and DRA‐lean components. Relapse, when it occurred, was directly related to DRA reinforcer rate as in Experiment 2. The post‐treatment persistence of alternative behavior was greater in the DRA‐rich component in Experiment 1, whereas it was the same or greater in the signaled‐DRA‐lean component in Experiments 2 and 3. Thus, infrequent signaled DRA may be optimal for effective clinical treatment.     

EFFECTS OF A SIGNALED DELAY TO REINFORCEMENT IN THE PREVIOUS AND UPCOMING RATIOS ON BETWEEN-RATIO PAUSING IN FIXED-RATIO SCHEDULES

Domestic hens responded under multiple fixed‐ratio fixed‐ratio schedules with equal fixed ratios. One component provided immediate reinforcement and the other provided reinforcement after a delay, signaled by the offset of the key light. The components were presented quasirandomly so that all four possible transitions occurred in each session. The delay was varied over 0, 4, 8, 16, and 32 s with fixed‐ratio 5 schedules, and over 0, 8 and 32 s with fixed‐ratio 1, 15 and 40 schedules. Main effects of fixed‐ratio value and delay duration were detected on between‐ratio pauses. Pauses were longer when the multiple‐schedule stimulus correlated with a delayed‐reinforcer component was presented, with the longest pauses occurring at the transition from a component with an immediate reinforcer to one with a delayed reinforcer. Pause durations were shortest during immediate components. Overall, both the presence or absence of a delay in the upcoming component, and the presence or absence of a delay in the preceding component affected pause length, but the upcoming delay had the larger effect. Thus changes in delay had similar effects to past reports of the effects of changes in response force, response requirement, and reinforcer magnitude in multiple fixed‐ratio fixed‐ratio schedules.     

Children's choice: Sensitivity to changes in reinforcer density

Two experiments were carried out in which children's sensitivity to changes in reinforcer density (number of reinforcers per session) was measured in a choice paradigm. In Experiment 1, 24 girls (ages 6, 9, and 12 years) performed on concurrent-chain schedules of reinforcement. The initial links were variable-interval 10-s schedules. One terminal link always gave three tokens after 30 s, but the parameters associated with the other were varied. Independent manipulations of reinforcer size (two tokens or four tokens) and prereinforcement delay (25 s or 65 s) led to equal changes in the relative density of tokens that could be earned on the schedules. Subjects at all ages were sensitive to changes in reinforcer density brought about by changes in reinforcer size, whereas only 3 12-year-olds showed sensitivity to the changes brought about by manipulation of prereinforcer delay. In Experiment 2, titration procedures were used to test the extent of this insensitivity to delay in 32 6- and 12-year-old children. In these procedures, a repeated choice of the large reinforcer increased the delay to its delivery, and a repeated choice of the small reinforcer reduced the delay to the delivery of the large reinforcer. Whereas 6-year-old boys and girls tended to maintain a strong preference for the large reinforcer, so increasing the delay to its delivery, 12-year-olds tended to distribute their responses to both alternatives, thus producing a stable level of delay to the large reinforcer. The results from the two experiments support the idea of two stages in the development of adaptive intertemporal choice.     

Sensitivity to reinforcer duration in a self-control procedure

In a concurrent-chains procedure, pigeons' responses on left and right keys were followed by reinforcers of different durations at different delays following the choice responses. Three pairs of reinforcer delays were arranged in each session, and reinforcer durations were varied over conditions. In Experiment 1 reinforcer delays were unequal, and in Experiment 2 reinforcer delays were equal. In Experiment 1 preference reversal was demonstrated in that an immediate short reinforcer was chosen more frequently than a longer reinforcer delayed 6 s from the choice, whereas the longer reinforcer was chosen more frequently when delays to both reinforcers were lengthened. In both experiments, choice responding was more sensitive to variations in reinforcer duration at overall longer reinforcer delays than at overall shorter reinforcer delays, independently of whether fixed-interval or variable-interval schedules were arranged in the choice phase. We concluded that preference reversal results from a change in sensitivity of choice responding to ratios of reinforcer duration as the delays to both reinforcers are lengthened.     

Nonstable concurrent choice in pigeons

Six pigeons were trained on concurrent variable-interval schedules in which the arranged reinforcer ratios changed from session to session according to a 31-step pseudorandom binary sequence. This procedure allows a quantitative analysis of the degree to which performance in an experimental session is affected by conditions in previous sessions. Two experiments were carried out. In each, the size of the reinforcer ratios arranged between the two concurrent schedules was varied between 31-step conditions. In Experiment 1, the concurrent schedules were arranged independently, and in Experiment 2 they were arranged nonindependently. An extended form of the generalized matching law described the relative contribution of past and present events to present-session behavior. Total performance in sessions was mostly determined by the reinforcer ratio in that session and partially by reinforcers that had been obtained in previous sessions. However, the initial exposure to the random sequence produced a lower sensitivity to current-session reinforcers but no difference in overall sensitivity to reinforcement. There was no evidence that the size of the reinforcer ratios available on the concurrent schedules affected either overall sensitivity to reinforcement or the sensitivity to reinforcement in the current session. There was also no evidence of any different performance between independent and nonindependent scheduling. Because of these invariances, this experiment validates the use of the pseudorandom sequence for the fast determination of sensitivity to reinforcement.     

WITHIN‐SESSION TRANSITIONS IN CHOICE: A STRUCTURAL AND QUANTITATIVE ANALYSIS

The present study used within‐session transitions between two concurrent schedules to evaluate choice in transition. Eight female Long‐Evans rats were trained to respond under concurrent schedules of reinforcement during experimental sessions that lasted 22 hr. The generalized matching equation was used to model steady‐state behavior at the end of each session, while transitional behavior that emerged following the change in reinforcement schedules was modeled using a logistic equation. The generalized matching and logistic equations were appropriate models for behavior generated during single‐session transitions. A local analysis of behavior on the two response alternatives during acquisition was used to determine the source of preference as revealed in response ratios. The number of “low‐response” visits, those containing three to five responses, remained stable. Preference ratios largely reflected a sharp increase in the number of visits with long response bouts on the rich alternative and a decrease in the number of such visits to the leaner alternative.     

The effects of fixed-time reinforcement schedules on problem behavior of children with emotional and behavioral disorders in a day-treatment classroom setting

The current study assessed the effects of fixed-time reinforcement schedules on problem behavior of students with emotional-behavioral disorders in a clinical day-treatment classroom setting. Three elementary-aged students with a variety of emotional and behavioral problems participated in the study. Initial functional assessments indicated that social attention was the maintaining reinforcer for their verbally disruptive behavior. Baseline phases were alternated with phases in which attention was provided on fixed-time schedules in the context of an ABAB design. The results indicated that the provision of attention on fixed-time schedules substantially reduced the participants' rate of verbal disruptions. These decreases were maintained during initial thinning of the schedules. The results provide one of the first examples that such an intervention can be successfully implemented in a classroom setting.     

Self-control in children with autism: response allocation during delays to reinforcement.

This study examined the use of a progressive‐delay schedule of reinforcement to increase self‐control and decrease disruptive behavior in children with autism. When initially given the choice between an immediate smaller reinforcer and a larger delayed reinforcer, all participants chose the smaller reinforcer. When access to the larger reinforcer required either no activity or engaging in a concurrent task during the delay, all participants demonstrated both self‐control and preference for a response requirement. Disruptive behavior decreased during delays that required a concurrent task compared to sessions without an activity requirement.     

Dynamic response-by-response models of matching behavior in rhesus monkeys

We studied the choice behavior of 2 monkeys in a discrete-trial task with reinforcement contingencies similar to those Herrnstein (1961) used when he described the matching law. In each session, the monkeys experienced blocks of discrete trials at different relative-reinforcer frequencies or magnitudes with unsignalled transitions between the blocks. Steady-state data following adjustment to each transition were well characterized by the generalized matching law; response ratios undermatched reinforcer frequency ratios but matched reinforcer magnitude ratios. We modelled response-by-response behavior with linear models that used past reinforcers as well as past choices to predict the monkeys' choices on each trial. We found that more recently obtained reinforcers more strongly influenced choice behavior. Perhaps surprisingly, we also found that the monkeys' actions were influenced by the pattern of their own past choices. It was necessary to incorporate both past reinforcers and past choices in order to accurately capture steady-state behavior as well as the fluctuations during block transitions and the response-by-response patterns of behavior. Our results suggest that simple reinforcement learning models must account for the effects of past choices to accurately characterize behavior in this task, and that models with these properties provide a conceptual tool for studying how both past reinforcers and past choices are integrated by the neural systems that generate behavior.     

Local effects of delayed food

Five pigeons were trained on a procedure in which seven concurrent variable-interval schedules arranged seven different food-rate ratios in random sequence in each session. Each of these components lasted for 10 response-produced food deliveries, and components were separated by 10-s blackouts. We varied delays to food (signaled by blackout) between the two response alternatives in an experiment with three phases: In Phase 1, the delay on one alternative was 0 s, and the other was varied between 0 and 8 s; in Phase 2, both delays were equal and were varied from 0 to 4 s; in Phase 3, the two delays summed to 8 s, and each was varied from 1 to 7 s. The results showed that increasing delay affected local choice, measured by a pulse in preference, in the same way as decreasing magnitude, but we found also that increasing the delay at the other alternative increased local preference. This result casts doubt on the traditional view that a reinforcer strengthens a response depending only on the reinforcer's value discounted by any response-reinforcer delay. The results suggest that food guides, rather than strengthens, behavior.     

Response–reinforcer dependency and resistance to change

The effects of the response–reinforcer dependency on resistance to change were studied in three experiments with rats. In Experiment 1, lever pressing produced reinforcers at similar rates after variable interreinforcer intervals in each component of a two‐component multiple schedule. Across conditions, in the fixed component, all reinforcers were response‐dependent; in the alternative component, the percentage of response‐dependent reinforcers was 100, 50 (i.e., 50% response‐dependent and 50% response‐independent) or 10% (i.e., 10% response‐dependent and 90% response‐independent). Resistance to extinction was greater in the alternative than in the fixed component when the dependency in the former was 10%, but was similar between components when this dependency was 100 or 50%. In Experiment 2, a three‐component multiple schedule was used. The dependency was 100% in one component and 10% in the other two. The 10% components differed on how reinforcers were programmed. In one component, as in Experiment 1, a reinforcer had to be collected before the scheduling of other response‐dependent or independent reinforcers. In the other component, response‐dependent and ‐independent reinforcers were programmed by superimposing a variable‐time schedule on an independent variable‐interval schedule. Regardless of the procedure used to program the dependency, resistance to extinction was greater in the 10% components than in the 100% component. These results were replicated in Experiment 3 in which, instead of extinction, VT schedules replaced the baseline schedules in each multiple‐schedule component during the test. We argue that the relative change in dependency from Baseline to Test, which is greater when baseline dependencies are high rather than low, could account for the differential resistance to change in the present experiments. The inconsistencies in results across the present and previous experiments suggest that the effects of dependency on resistance to change are not well understood. Additional systematic analyses are important to further understand the effects of the response–reinforcer relation on resistance to change and to the development of a more comprehensive theory of behavioral persistence.     

Multiple schedules,off‐baseline reinforcement shifts,and resistance to extinction

Resistance to extinction in a target multiple‐schedule component varies inversely with the rate of reinforcement arranged in an alternative component during baseline. The present experiment asked whether changing the reinforcer rate in an alternative component would impact extinction of target component responding if those changes occurred in an off‐baseline phase during which the target component was never experienced. Pigeons' key pecking was studied in three types of conditions, and each condition consisted of three phases. In Phase 1, pecking produced food in the target and alternative components of a multiple schedule according to variable‐interval 60‐s schedules. In Phase 2, the alternative‐component stimulus was presented alone in a single schedule. Pecking during this phase produced the same reinforcer rate as in baseline in the Control condition, a higher rate of food (variable‐interval 15 s) in the High‐Rate condition, or was extinguished in the Extinction condition. Extinction of target‐ and alternative‐component key pecking then was assessed in a multiple schedule during the final phase of each condition. Resistance to extinction of target‐component key pecking was the same between the Control and High‐Rate conditions but lower in the Extinction condition. These findings are discussed in terms of discrimination and generalization processes.     

Concurrent variable‐interval variable‐ratio schedules in a dynamic choice environment

Most studies of operant choice have focused on presenting subjects with a fixed pair of schedules across many experimental sessions. Using these methods, studies of concurrent variable‐ interval variable‐ratio schedules helped to evaluate theories of choice. More recently, a growing literature has focused on dynamic choice behavior. Those dynamic choice studies have analyzed behavior on a number of different time scales using concurrent variable‐interval schedules. Following the dynamic choice approach, the present experiment examined performance on concurrent variable‐interval variable‐ratio schedules in a rapidly changing environment. Our objectives were to compare performance on concurrent variable‐interval variable‐ratio schedules with extant data on concurrent variable‐interval variable‐interval schedules using a dynamic choice procedure and to extend earlier work on concurrent variable‐interval variable‐ratio schedules. We analyzed performances at different time scales, finding strong similarities between concurrent variable‐interval variable‐interval and concurrent variable‐interval variable‐ ratio performance within dynamic choice procedures. Time‐based measures revealed almost identical performance in the two procedures compared with response‐based measures, supporting the view that choice is best understood as time allocation. Performance at the smaller time scale of visits accorded with the tendency seen in earlier research toward developing a pattern of strong preference for and long visits to the richer alternative paired with brief “samples” at the leaner alternative (“fix and sample”).