WITHIN‐SESSION TRANSITIONS IN CHOICE: A STRUCTURAL AND QUANTITATIVE ANALYSIS

The present study used within‐session transitions between two concurrent schedules to evaluate choice in transition. Eight female Long‐Evans rats were trained to respond under concurrent schedules of reinforcement during experimental sessions that lasted 22 hr. The generalized matching equation was used to model steady‐state behavior at the end of each session, while transitional behavior that emerged following the change in reinforcement schedules was modeled using a logistic equation. The generalized matching and logistic equations were appropriate models for behavior generated during single‐session transitions. A local analysis of behavior on the two response alternatives during acquisition was used to determine the source of preference as revealed in response ratios. The number of “low‐response” visits, those containing three to five responses, remained stable. Preference ratios largely reflected a sharp increase in the number of visits with long response bouts on the rich alternative and a decrease in the number of such visits to the leaner alternative.     

Dynamic response-by-response models of matching behavior in rhesus monkeys

We studied the choice behavior of 2 monkeys in a discrete-trial task with reinforcement contingencies similar to those Herrnstein (1961) used when he described the matching law. In each session, the monkeys experienced blocks of discrete trials at different relative-reinforcer frequencies or magnitudes with unsignalled transitions between the blocks. Steady-state data following adjustment to each transition were well characterized by the generalized matching law; response ratios undermatched reinforcer frequency ratios but matched reinforcer magnitude ratios. We modelled response-by-response behavior with linear models that used past reinforcers as well as past choices to predict the monkeys' choices on each trial. We found that more recently obtained reinforcers more strongly influenced choice behavior. Perhaps surprisingly, we also found that the monkeys' actions were influenced by the pattern of their own past choices. It was necessary to incorporate both past reinforcers and past choices in order to accurately capture steady-state behavior as well as the fluctuations during block transitions and the response-by-response patterns of behavior. Our results suggest that simple reinforcement learning models must account for the effects of past choices to accurately characterize behavior in this task, and that models with these properties provide a conceptual tool for studying how both past reinforcers and past choices are integrated by the neural systems that generate behavior.     

Additional free reinforcers increase persistence of problem behavior in a clinical context: A partial replication of laboratory findings

Behavioral momentum theory is a quantitative framework used to characterize the persistence of behavior during response disruptors as a function of baseline stimulus–reinforcer relations. Results of several investigations have shown that alternative reinforcement can increase the resistance to change of a target response during extinction. In the present study, concomitant variable‐interval fixed‐time schedules of reinforcement for problem behavior were employed to simulate naturalistic situations involving the superimposition of response‐independent reinforcers on a baseline schedule of reinforcement for problem behavior, as in the common use of noncontingent reinforcement treatments. Resistance to change of problem behavior was assessed during postsession periods of extinction by comparing response rates in extinction following sessions with and without additional reinforcer deliveries arranged by fixed‐time schedules. For 2 out of 3 participants, problem behavior tended to be more resistant to extinction following periods in which additional fixed‐time reinforcers were delivered. These results are discussed in terms of potential effects of noncontingent reinforcement on problem behavior when the intervention is discontinued or implemented without good treatment integrity.     

Choice in transition: Replication and extension to preschool children in a naturalistic setting

This study replicated previous basic research into the dynamics of choice and extended this analysis to children's behavior in a naturalistic setting. Two preschoolers with disabilities were observed interacting with their teachers at baseline and during an experimental analysis involving four pairs of concurrent variable‐interval schedules of adult attention implemented by an experimenter. Each child was exposed to four experimental phases in which the relative reinforcer rates for on‐ and off‐task behavior were 10:1, 1:1, 1:10, and reversed back to 10:1. The 10:1 phase was designed to mimic the same schedules and types of adult attention observed at baseline. We used the generalized matching equation to model steady‐state behavior at the end of the transition phases and to evaluate changes in sensitivity at various points throughout the phases. Choice in transition was evaluated by plotting log behavior ratios by session, cumulated time on‐ and off‐task and cumulated attention for on‐ and off‐task behavior by session, and interreinforcer behavior ratios following different sequences of the first four reinforcer deliveries. The generalized matching equation accounted for a large proportion of variance in steady‐state responding, sensitivity values increased steadily throughout the phases, patterns of choice in transition were similar to those reported in basic research, and interreinforcer preference generally shifted toward the just‐reinforced alternative. These findings are consistent with previous basic research and support the generality of the dynamics of choice to children's on‐ and off‐task behavior reinforced by adult attention.     

Choosing among multiple alternatives: Relative and overall reinforcer rates

Choice behavior among two alternatives has been widely researched, but fewer studies have examined the effect of multiple (more than two) alternatives on choice. Two experiments investigated whether changing the overall reinforcer rate affected preference among three and four concurrently scheduled alternatives. Experiment 1 trained six pigeons on concurrent schedules with three alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 9:3:1 with the configuration counterbalanced across pigeons. The overall rate of reinforcement was varied across conditions. Preference between the pair of keys arranging the 9:3 reinforcer ratio was less extreme than the pair arranging the 3:1 reinforcer ratio regardless of overall reinforcer rate. This difference was attributable to the richer alternative receiving fewer responses per reinforcer than the other alternatives. Experiment 2 trained pigeons on concurrent schedules with four alternatives available simultaneously. These alternatives arranged reinforcers in a ratio of 8:4:2:1, and the overall reinforcer rate was varied. Next, two of the alternatives were put into extinction and the random interval duration was changed from 60 s to 5 s. The ratio of absolute response rates was independent of interval length across all conditions. In both experiments, an analysis of sequences of visits following each reinforcer showed that the pigeons typically made their first response to the richer alternative irrespective of which alternative was just reinforced. Performance on these three‐ and four‐alternative concurrent schedules is not easily extrapolated from corresponding research using two‐alternative concurrent schedules.     

Using a supervised machine learning algorithm for detecting faking good in a personality self‐report

We developed a supervised machine learning classifier to identify faking good by analyzing item response patterns of a Big Five personality self‐report. We used a between‐subject design, dividing participants (N = 548) into two groups and manipulated their faking behavior via instructions given prior to administering the self‐report. We implemented a simple classifier based on the Lie scale's cutoff score and several machine learning models fitted either to the personality scale scores or to the items response patterns. Results shown that the best machine learning classifier—based on the XGBoost algorithm and fitted to the item responses—was better at detecting faked profiles than the Lie scale classifier.     

Zebrafish choice behavior is sensitive to reinforcer rate,immediacy, and magnitude ratios

Behavioral flexibility has, in part, been defined by choice behavior changing as a function of changes in reinforcer payoffs. We examined whether the generalized matching law quantitatively described changes in choice behavior in zebrafish when relative reinforcer rates, delays/immediacy, and magnitudes changed between two alternatives across conditions. Choice was sensitive to each of the three reinforcer properties. Sensitivity estimates to changes in relative reinforcer rates were greater when 2 variable-interval schedules were arranged independently between alternatives (Experiment 1a) than when a single schedule pseudorandomly arranged reinforcers between alternatives (Experiment 1b). Sensitivity estimates for changes in relative reinforcer immediacy (Experiment 2) and magnitude (Experiment 3) were similar but lower than estimates for reinforcer rates. These differences in sensitivity estimates are consistent with studies examining other species, suggesting flexibility in zebrafish choice behavior in the face of changes in payoff as described by the generalized matching law.     

Choice among two and three alternatives

Although choice between two alternatives has been widely researched, fewer studies have examined choice across multiple (more than two) alternatives. Past models of choice behavior predict that the number of alternatives should not affect relative response allocation, but more recent research has found violations of this principle. Five pigeons were presented with three concurrently scheduled alternatives. Relative reinforcement rates across these alternatives were assigned 9:3:1. In some conditions three keys were available; in others, only two keys were available. The number of available alternatives did not affect relative response rates for pairs of alternatives; there were no significant differences in behavior between the two and three key conditions. For two birds in the three‐alternative conditions and three birds in the two‐alternative conditions, preference was more extreme for the pair of alternatives with the lower overall pairwise reinforcer rate (3:1) than the pair with higher overall reinforcer rate (9:3). However, when responding during the changeover was removed three birds showed the opposite pattern in the three‐alternative conditions; preference was more extreme for the pair of alternatives with the higher overall reinforcer rate. These findings differ from past research and do not support established theories of choice behavior.     

Control by reinforcers across time and space: A review of recent choice research

Reinforcers affect behavior. A fundamental assumption has been that reinforcers strengthen the behavior they follow, and that this strengthening may be context‐specific (stimulus control). Less frequently discussed, but just as evident, is the observation that reinforcers have discriminative properties that also guide behavior. We review findings from recent research that approaches choice using nontraditional procedures, with a particular focus on how choice is affected by reinforcers, by time since reinforcers, and by recent sequences of reinforcers. We also discuss how conclusions about these results are impacted by the choice of measurement level and display. Clearly, reinforcers as traditionally considered are conditionally phylogenetically important to animals. However, their effects on behavior may be solely discriminative, and contingent reinforcers may not strengthen behavior. Rather, phylogenetically important stimuli constitute a part of a correlated compound stimulus context consisting of stimuli arising from the organism, from behavior, and from physiologically detected environmental stimuli. Thus, the three‐term contingency may be seen, along with organismic state, as a correlation of stimuli. We suggest that organisms may be seen as natural stimulus‐correlation detectors so that behavioral change affects the overall correlation and directs the organism toward currently appetitive goals and away from potential aversive goals. As a general conclusion, both historical and recent choice research supports the idea that stimulus control, not reinforcer control, may be fundamental.     

The substitutability of reinforcers

Substitutability is a construct borrowed from microeconomics that describes a continuum of possible interactions among the reinforcers in a given situation. Highly substitutable reinforcers, which occupy one end of the continuum, are readily traded for each other due to their functional similarity. Complementary reinforcers, at the other end of the continuum, tend to be consumed jointly in fairly rigid proportion, and therefore cannot be traded for one another except to achieve that proportion. At the center of the continuum are reinforcers that are independent with respect to each other; consumption of one has no influence on consumption of another. Psychological research and analyses in terms of substitutability employ standard operant conditioning paradigms in which humans and nonhumans choose between alternative reinforcers. The range of reinforcer interactions found in these studies is more readily accommodated and predicted when behavior-analytic models of choice consider issues of substitutability. New insights are gained into such areas as eating and drinking, electrical brain stimulation, temporal separation of choice alternatives, behavior therapy, drug use, and addictions. Moreover, the generalized matching law (Baum, 1974) gains greater explanatory power and comprehensiveness when measures of substitutability are included.     

RISKY CHOICE IN PIGEONS: PREFERENCE FOR AMOUNT VARIABILITY USING A TOKEN-REINFORCEMENT SYSTEM

Pigeons were given repeated choices between variable and fixed numbers of token reinforcers (stimulus lamps arrayed above the response keys), with each earned token exchangeable for food. The number of tokens provided by the fixed‐amount option remained constant within blocks of sessions, but varied parametrically across phases, assuming values of 2, 4, 6, or 8 tokens per choice. The number of tokens provided by the variable‐amount option varied between 0 and 12 tokens per choice, arranged according to an exponential or rectangular distribution. In general, the pigeons strongly preferred the variable option when the fixed option provided equal or greater numbers of tokens than the variable amount. Preference for the variable amount decreased only when the alternatives provided widely disparate amounts favoring the fixed amount. When tokens were removed from the experimental context, preference for the variable option was reduced or eliminated, suggesting that the token presentation played a key role in maintaining risk‐prone choice patterns. Choice latencies varied inversely with preferences, suggesting that local analyses may provide useful ancillary measures of reinforcer value. Overall, the results indicate that systematic risk sensitivity can be attained with respect to reinforcer amount, and that tokens may be critical in the development of such preferences.     

Being there on time: Reinforcer effects on timing and locating

In research on timing, reinforcers often are assumed to influence discrimination of elapsed time. We asked whether changes in choice used to measure timing arise because of joint control by elapsed time and reinforcers, rather than from the direct modification of control by elapsed time by reinforcers. Pigeons worked on a concurrent-choice task in which 1 response was 9 times more likely to produce a reinforcer, reversing between locations when 19 s had elapsed since the marker event. Across conditions, we manipulated the percentage of reinforcers arranged before the probability reversal from 5 to 95%. These changes in reinforcer percentages altered control by location-based elements of the contingency, but not by time-based elements. Choice was well described by a model that assumes that control by the contingency is weakened by generalization across the time and location of reinforcers, and that these generalizations become more likely at later times since a marker. These findings add to a growing body of research that suggests that reinforcers share the same function as other environmental events in determining how the environment controls behavior.     

Choice in a variable environment: visit patterns in the dynamics of choice

Molar and molecular views of behavior imply different approaches to data analysis. The molecular view privileges moment-to-moment analyses, whereas the molar view supports analysis of more and less extended activities. In concurrent performance, the molar view supports study of both extended patterns of choice and more local patterns of visiting the choice alternatives. Analysis of the present data illustrated the usefulness of investigating order at various levels of extendedness. Seven different reinforcer ratios were presented within each session, without cues to identify them, and pigeons pecked at two response keys that delivered food on variable-interval schedules. Choice changed rapidly within components as reinforcers were delivered and, following each reinforcer, shifted toward the alternative that produced it. If several reinforcers were delivered consecutively by one alternative, choice favored that alternative, but shifted more slowly with each new reinforcer. A discontinuation of such a series of reinforcers by the delivery of a reinforcer by the other alternative resulted in a large shift of choice toward that alternative. These effects were illuminated by analysis of visits to the two alternatives. Changes in visit length occurred primarily in the first postreinforcer visit to the repeatedly reinforced alternative. All other visits tended to be brief and equal. Performance showed multiple signs of moving in the direction of a fix-and-sample pattern that characterized steady-state performance in earlier experiments with many sessions of maintaining each schedule pair. The analyses of extended and local patterns illustrate the flexibility of a molar view of behavior.     

Evaluating the effects of discriminability on behavioral persistence during and following time‐based reinforcement

With four children with autism we evaluated a refinement to time‐based reinforcement designed to reduce response persistence when we simultaneously introduced time‐based reinforcement and extinction. We further evaluated whether this refinement mitigated response recurrence when all reinforcer deliveries ceased during an extinction‐only disruptor phase. The refinement involved increasing the saliency of the contingency change from contingent reinforcement (during baseline) to time‐based reinforcement by delivering different colored reinforcers during time‐based reinforcement. Behavioral momentum theory predicts that increasing the discriminability of the change from variable‐interval to variable‐time reinforcement should lead to faster reductions in responding. We present data on four participants, three of whom displayed response patterns consistent with the predictions of behavioral momentum theory during time‐based reinforcement. However, the participants showed more varied patterns of recurrent behavior during extinction. We discuss these results within a translational research framework focusing on strategies used to mitigate treatment relapse for severe destructive behavior, as time‐based reinforcement is one of the most commonly prescribed interventions for destructive behavior displayed by individuals with intellectual and developmental disabilities.     

Response–reinforcer dependency and resistance to change

The effects of the response–reinforcer dependency on resistance to change were studied in three experiments with rats. In Experiment 1, lever pressing produced reinforcers at similar rates after variable interreinforcer intervals in each component of a two‐component multiple schedule. Across conditions, in the fixed component, all reinforcers were response‐dependent; in the alternative component, the percentage of response‐dependent reinforcers was 100, 50 (i.e., 50% response‐dependent and 50% response‐independent) or 10% (i.e., 10% response‐dependent and 90% response‐independent). Resistance to extinction was greater in the alternative than in the fixed component when the dependency in the former was 10%, but was similar between components when this dependency was 100 or 50%. In Experiment 2, a three‐component multiple schedule was used. The dependency was 100% in one component and 10% in the other two. The 10% components differed on how reinforcers were programmed. In one component, as in Experiment 1, a reinforcer had to be collected before the scheduling of other response‐dependent or independent reinforcers. In the other component, response‐dependent and ‐independent reinforcers were programmed by superimposing a variable‐time schedule on an independent variable‐interval schedule. Regardless of the procedure used to program the dependency, resistance to extinction was greater in the 10% components than in the 100% component. These results were replicated in Experiment 3 in which, instead of extinction, VT schedules replaced the baseline schedules in each multiple‐schedule component during the test. We argue that the relative change in dependency from Baseline to Test, which is greater when baseline dependencies are high rather than low, could account for the differential resistance to change in the present experiments. The inconsistencies in results across the present and previous experiments suggest that the effects of dependency on resistance to change are not well understood. Additional systematic analyses are important to further understand the effects of the response–reinforcer relation on resistance to change and to the development of a more comprehensive theory of behavioral persistence.     

A comparison of ratio and interval reinforcement schedules with comparable interreinforcement times

Pigeons were trained to peck keys on fixed-ratio and fixed-interval schedules of food reinforcement. Both schedules produced a pattern of behavior characterized as pause and run, but the relation of pausing to time between reinforcers differed for the two schedules even when mean time between reinforcers was the same. Pausing in the fixed ratio occupied less of the time between reinforcers for shorter interreinforcer times. For two of three birds, the relation was reversed at longer interreinforcer times. As an interreinforcer time elapsed, there was an increasing tendency to return to responding for the fixed interval, but a roughly constant tendency to return to responding for the fixed-ratio schedule. In Experiment 1 these observations were made for both single-reinforcement schedules and multiple schedules of fixed-ratio and fixed-interval reinforcement. In Experiment 2 the observations were extended to a comparison of fixed-ratio versus variable-interval reinforcement schedules, where the distribution of interreinforcement times in the variable interval approximated that for the fixed ratio.     

Concurrent variable‐interval variable‐ratio schedules in a dynamic choice environment

Most studies of operant choice have focused on presenting subjects with a fixed pair of schedules across many experimental sessions. Using these methods, studies of concurrent variable‐ interval variable‐ratio schedules helped to evaluate theories of choice. More recently, a growing literature has focused on dynamic choice behavior. Those dynamic choice studies have analyzed behavior on a number of different time scales using concurrent variable‐interval schedules. Following the dynamic choice approach, the present experiment examined performance on concurrent variable‐interval variable‐ratio schedules in a rapidly changing environment. Our objectives were to compare performance on concurrent variable‐interval variable‐ratio schedules with extant data on concurrent variable‐interval variable‐interval schedules using a dynamic choice procedure and to extend earlier work on concurrent variable‐interval variable‐ratio schedules. We analyzed performances at different time scales, finding strong similarities between concurrent variable‐interval variable‐interval and concurrent variable‐interval variable‐ ratio performance within dynamic choice procedures. Time‐based measures revealed almost identical performance in the two procedures compared with response‐based measures, supporting the view that choice is best understood as time allocation. Performance at the smaller time scale of visits accorded with the tendency seen in earlier research toward developing a pattern of strong preference for and long visits to the richer alternative paired with brief “samples” at the leaner alternative (“fix and sample”).     

On adaptation, maximization, and reinforcement learning among cognitive strategies

Analysis of binary choice behavior in iterated tasks with immediate feedback reveals robust deviations from maximization that can be described as indications of 3 effects: (a) a payoff variability effect, in which high payoff variability seems to move choice behavior toward random choice; (b) underweighting of rare events, in which alternatives that yield the best payoffs most of the time are attractive even when they are associated with a lower expected return; and (c) loss aversion, in which alternatives that minimize the probability of losses can be more attractive than those that maximize expected payoffs. The results are closer to probability matching than to maximization. Best approximation is provided with a model of reinforcement learning among cognitive strategies (RELACS). This model captures the 3 deviations, the learning curves, and the effect of information on uncertainty avoidance. It outperforms other models in fitting the data and in predicting behavior in other experiments.     

Concurrent-schedule performance: Effects of relative and overall reinforcer rate

Six pigeons were trained to respond on two keys, each of which provided reinforcers on an arithmetic variable-interval schedule. These concurrent schedules ran nonindependently with a 2-s changeover delay. Six sets of conditions were conducted. Within each set of conditions the ratio of reinforcers available on the two alternatives was varied, but the arranged overall reinforcer rate remained constant. Each set of conditions used a different overall reinforcer rate, ranging from 0.22 reinforcers per minute to 10 reinforcers per minute. The generalized matching law fit the data from each set of conditions, but sensitivity to reinforcer frequency (a) decreased as the overall reinforcer rate decreased for both time allocation and response allocation based analyses of the data. Overall response rates did not vary with changes in relative reinforcer rate, but decreased with decreases in overall reinforcer rate. Changeover rates varied as a function of both relative and overall reinforcer rates. However, as explanations based on changeover rate seem unable to deal with the changes in generalized matching sensitivity, discrimination accounts of choice may offer a more promising interpretation.