Quasi-dynamic choice models: Melioration and ratio invariance

There is continuing controversy about the behavioral process or processes that underlie the major regularities of free-operant choice such as molar matching and systematic deviations therefrom. A recent interchange between Vaughan and Silberberg and Ziriax concerned the relative merits of melioration, and a computer simulation of molecular maximizing. There are difficulties in evaluating theories expressed as computer programs because many arbitrary decisions must often be made in order to get the programs to operate. I therefore propose an alternative form of model that I term quasi-dynamic as a useful intermediate form of theory appropriate to our current state of knowledge about free-operant choice. Quasi-dynamic models resemble the game-theoretic analyses now commonplace in biology in that they can predict stable and unstable equilibria but not dynamic properties such as learning curves. It is possible to interpret melioration as a quasi-dynamic model. An alternative quasi-dynamic model for probabilistic choice, ratio invariance, has been proposed by Horner and Staddon. The present paper compares the predictions of melioration and ratio invariance for five experimental situations: concurrent variable-interval variable-interval schedules, concurrent variable-interval variable-ratio schedules, the two-armed bandit (concurrent random-ratio schedules), and two types of frequency-dependent schedule. Neither approach easily explains all the data, but ratio invariance seems to provide a better picture of pigeons' response to probabilistic choice procedures. Ratio invariance is also more adaptive (less susceptible to “traps”) and closer to the original expression of the law of effect than pure hill-climbing processes such as momentary maximizing and melioration, although such processes may come in to play on more complex procedures that provide opportunities for temporal discrimination.     

Optimality And Concurrent Variable-interval Variable-ratio Schedules

Despite claims to the contrary, all leading theories about operant choice may be seen as models of optimality. Although melioration is often contrasted with global maximization, both make the same core assumptions as other versions of optimality theory, including momentary maximizing, hill climbing, and the various versions of optimal foraging theory. The present experiment aimed to test melioration against more global optimality and to apply the visit-by-visit analysis suggested by foraging theory. Rats were exposed to concurrent schedules in which one alternative was always variable-ratio 10 and the other alternative was a variable-interval schedule. Although choice relations varied from rat to rat, the overall results roughly confirmed the matching law, a result often taken to support melioration. Pooling the data across sessions and across rats, however, resulted in no increment in unsystematic variance, lending support to the contention by Ziriax and Silberberg (1984) that the choice relation is partly constrained. When the data were analyzed at the level of visits, the results either disconfirmed predictions of melioration or showed regularities about which melioration is silent. Instead, performance tended toward a rough optimization, in which responding favored the variable ratio, but with relatively brief visits to the variable interval. There were no asymmetries in travel or variability that would indicate that different processes were involved in generating visits at the two different schedules. The findings point toward a more global optimality model than melioration and demonstrate the value of per-visit analysis in the study of concurrent performances.     

Interresponse-time sensitivity during discrete-trial and free-operant concurrent variable-interval schedules

Two experiments investigated the sensitivity of pigeons' choice to elapsed time since the last response (i.e., to inter-response time [IRT]) during concurrent variable-interval variable-interval schedules. Experiment 1 used a two-key discrete-trial procedure with variable intertrial intervals. Experiment 2 employed a three-key free-operant procedure. In both experiments choice was found to be a function of the active-schedule IRT, defined as the time since the most recent response. Monte Carlo simulations show how this finding permits the joining of several seemingly incompatible data sets held to both support and contradict a kind of choice strategy, termed momentary maximizing, which attempts to maximize momentary reinforcement probabilities. The studies suggest that only two variables are needed to describe the static molecular structure of concurrent variable-interval choice: active-schedule IRTs and "response states" consisting of the last one or two schedule choices.     

Hill-climbing by pigeons

Pigeons were exposed to two types of concurrent operant-reinforcement schedules in order to determine what choice rules determine behavior on these schedules. In the first set of experiments, concurrent variable-interval, variable-interval schedules, key-peck responses to either of two alternative schedules produced food reinforcement after a random time interval. The frequency of food-reinforcement availability for the two schedules was varied over different ranges for different birds. In the second series of experiments, concurrent variable-ratio, variable-interval schedules, key-peck responses to one schedule produced food reinforcement after a random time interval, whereas food reinforcement occurred for an alternative schedule only after a random number of responses. Results from both experiments showed that pigeons consistently follow a behavioral strategy in which the alternative schedule chosen at any time is the one which offers the highest momentary reinforcement probability (momentary maximizing). The quality of momentary maximizing was somewhat higher and more consistent when both alternative reinforcement schedules were time-based than when one schedule was time-based and the alternative response-count based. Previous attempts to provide evidence for the existence of momentary maximizing were shown to be based upon faulty assumptions about the behavior implied by momentary maximizing and resultant inappropriate measures of behavior.     

Choice behavior on discrete trials: a demonstration of the occurrence of a response strategy

Three pairs of pigeons were trained to peck at two keys presented simultaneoulsy in discrete trials with intertrial intervals of 1, 22, or 120 sec. Left-key responses incremented the probability of reinforcement for the first right-key response and, conversely, right-key responses incremented the probability of reinforcement for the first left-key response. In terms of relative response rates, it was found that all birds' choices were described by a momentary maximizing strategy, but this fact was not reflected in the detailed sequential statistics for birds with the longer (22 or 120 sec) intertrial intervals. It was hypothesized that choice behavior, in general, may be accurately described by a momentary maximizing sequence, but that prior failures to demonstrate this were due to “errors” in executing the momentary maximizing sequence. These misappropriated responses, which are hypothesized to be randomly distributed among the responses defining the momentary maximizing sequence, caused successive choices to appear to be statistically independent when, in fact, they were not.     

Optimal choice

We present a classification and theoretical analysis of discrete-trial and free-operant choice procedures in which reinforcement is assigned to one alternative only, or independently to both, is either always available or conditionally available, and is either "held" or not from trial to trial. Momentary-maximizing and (globally) optimal choice sequences are defined in terms of initializing and marker events. Free-operant choice is analyzed in terms of a clock space whose axes are the times since the last A and B choices. The analysis shows that most molar matching data are derivable from momentary maximizing, and that the momentary-maximizing hypothesis has not been adequately tested in either discrete-trial or free-operant situations.     

Concurrent schedules: Maximization versus reinforcement of changeover behaviour

Pigeons responded on concurrent variable-interval 180-sec variable-interval 36-sec schedules during Conditions 1 and 3 of Experiment 1. Condition 2 arranged variable-interval 60-sec schedules for both response alternatives. The schedule assigned to the alternative that was associated with the variable-interval 36-sec schedule in Conditions 1 and 3 operated only when the subject responded on that alternative. The proportion of time spent responding on the alternative with the conventional variable-interval 60-sec schedule increased during Condition 2, but exclusive choice of that alternative did not develop. This result is inconsistent with maximization of the overall reinforcement rate and with maximization of the momentary probability of reinforcement (momentary maximizing). Increasing time proportions were also found in Experiment 2, which arranged similar conditions, except that reinforcement was provided on a variable-time basis. The time proportions were close to the momentary maximizing prediction in Experiment 2. The results of both experiments can be explained if it is assumed that time allocation is controlled by delayed reinforcement of changeovers between alternatives.     

A test of the effectiveness of the differential-reinforcement-of-low-rate schedule

Pigeons and rats were used in a yoked-control design that equated the reinforcement distributions of differential-reinforcement-of-low-rate and variable-interval schedules. Both a between-subjects design and a within-subjects design found response rate higher for the variable-interval schedule than for the differential-reinforcement-of-low-rate schedule, thus demonstrating the effectiveness of the differential-reinforcement-of-low-rate contingency. The interresponse-time distributions were unimodal for all subjects under the variable-interval schedule and bimodal for pigeons under the differential-reinforcement-of-low-rate schedule. The interresponse-time distributions for rats under the differential-reinforcement-of-low-rate schedule were also bimodal in three of four cases but the height of the modes at the shorter interresponse times were small in both absolute value and in relation to the height of the modes at the shorter interresponse times of the pigeons' distributions.     

Choice, time allocation, and response rate during stimulus generalization

Six pigeons were trained to discriminate between two noise intensities using a procedure that assessed choice, time allocation, and response rate simultaneously and independently. Responses on the left or right key (R1 or R2) were respectively correct in the presence of two different intensities, S1 and S2. After a correct response, reinforcement became available for pecks on the center key. Reinforcement density for R1¦S1 relative to R2¦S2 was varied across experimental conditions. Generalization tests followed extensive training at each condition. As a function of stimulus intensity, proportions of initial choices of R2, of time spent in R2-initiated components, and of center-key responses emitted in R2-initiated components all yielded sigmoidal gradients of similar slope, which shifted slightly in location when relative reinforcement density changed. Changeovers were maximal where initial choice proportions approximated 0.5. Gradients relating the absolute number of center-key responses to stimulus intensity were also roughly sigmoidal, but were more sensitive to changes in reinforcement density. Gradients of momentary response rate also depended on reinforcement density. During training, large but transitory shifts in choice responding occurred when reinforcement density changed, while differences in momentary response rate developed slowly, suggesting separate control of choice and response rate by the contingencies of reinforcement.     

Matching, delay-reduction, and maximizing models for choice in concurrent-chains schedules.

Models of choice in concurrent-chains schedules are derived from melioration, generalized matching, and optimization. The resulting models are compared with those based on Fantino's (1969, 1981) delay-reduction hypothesis. It is found that all models involve the delay reduction factors (T - t2L) and (T - t2R), where T is the expected time to primary reinforcement and t2L, t2R are the durations of the terminal links. In particular, in the case of equal initial links, the model derived from melioration coincides with Fantino's original model for full (reliable) reinforcement and with the model proposed by Spetch and Dunn (1987) for percentage (unreliable) reinforcement. In the general case of unequal initial links, the model derived from melioration differs from the revised model advanced by Squires and Fantino (1971) only in the factors affecting the delay-reduction terms (T - t2L) and (T - t2R). The models of choice obtained by minimizing the expected time to reinforcement depend on the type of feedback functions used. In particular, if power feedback functions are used, the optimization model coincides with that obtained from melioration.     

Momentary maximizing in concurrent schedules with a minimum interchangeover interval

Eight pigeons were trained on concurrent variable-interval variable-interval schedules with a minimum interchangeover time programmed as a consequence of changeovers. In Experiment 1 the reinforcement schedules remained constant while the minimum interchangeover time varied from 0 to 200 s. Relative response rates and relative time deviated from relative reinforcement rates toward indifference with long minimum interchangeover times. In Experiment 2 different reinforcement ratios were scheduled in successive experimental conditions with the minimum interchangeover time constant at 0, 2, 10, or 120 s. The exponent of the generalized matching equation was close to 1.0 when the minimum interchangeover time was 0 s (the typical procedure for concurrent schedules without a changeover delay) and decreased as that duration was increased. The data support the momentary maximizing theory and contradict molar maximizing theories and the melioration theory.     

A local model of concurrent performance

Concurrent procedures may be conceptualized as consisting of two pairs of schedules with only one pair operating at a time. One schedule of each pair arranges reinforcers for staying in the current alternative, and the other schedule arranges reinforcers for switching to the other alternative. These pairs alternate operation as the animal switches between choices. This analysis of the contingencies suggests that variables operating within an alternative produce behavior that conforms to the generalized matching law. Rats were exposed to one pair of stay and switch schedules in each condition, and the probabilities of reinforcement varied across conditions. Both run length and visit duration were power functions of the ratio of the probabilities of reinforcement for staying and switching. The local model, a model of performance on concurrent procedures, was derived from this power function. Performance on concurrent schedules was synthesized from the performances on the separate pairs. Both the generalized matching law and the local model fitted the synthesized concurrent performances. These results are consistent with the view that the contingencies in the alternative, the probability of stay and switch reinforcement, are responsible for performance consistent with the generalized matching law. These results are compatible with momentary maximizing and molar maximizing accounts of concurrent performance. Models of concurrent performance that posit comparisons among the alternatives are not easily applied to these results.     

Molecular maximizing characterizes choice on Vaughan's (1981) procedure

Pigeons keypecked on a two-key procedure in which their choice ratios during one time period determined the reinforcement rates assigned to each key during the next period (Vaughan, 1981). During each of four phases, which differed in the reinforcement rates they provided for different choice ratios, the duration of these periods was four minutes, duplicating one condition from Vaughan's study. During the other four phases, these periods lasted six seconds. When these periods were long, the results were similar to Vaughan's and appeared compatible with melioration theory. But when these periods were short, the data were consistent with molecular maximizing (see Silberberg & Ziriax, 1982) and were incompatible with melioration, molar maximizing, and matching. In a simulation, stat birds following a molecular-maximizing algorithm responded on the short- and long-period conditions of this experiment. When the time periods lasted four minutes, the results were similar to Vaughan's and to the results of the four-minute conditions of this study; when the time periods lasted six seconds, the choice data were similar to the data from real subjects for the six-second conditions. Thus, a molecular-maximizing response rule generated choice data comparable to those from the short- and long-period conditions of this experiment. These data show that, among extant accounts, choice on the Vaughan procedure is most compatible with molecular maximizing.     

Coping with Unfavorable Attribute Values in Choice

This paper examines how decision makers cope when faced with trade-offs between a higher quality alternative and a lower price alternative in situations where both alternatives involve relatively unfavorable versus relatively favorable values for quality. We hypothesize that choices between alternatives defined by unfavorable quality values will generate negative emotion, resulting in emotion-focused coping behavior. Choosing the higher quality alternative (i.e., maximizing the quality attribute in choice) appears to function as a coping mechanism in these situations. These apparently coping-motivated choice effects are found even after methods are implemented to control for more cognitive factors associated with manipulations of quality-attribute value, such as the possibility that unfavorable attribute values are associated with increased attribute ranges and therefore increased relative importance for quality.     

Preference between variable-ratio and fixed-ratio schedules: local and extended relations.

Although it has repeatedly been demonstrated that pigeons, as well as other species, will often choose a variable schedule of reinforcement over an equivalent (or even richer) fixed schedule, the exact nature of that controlling relation has yet to be fully assessed. In this study pigeons were given repeated choices between concurrently available fixed-ratio and variable-ratio schedules. The fixed-ratio requirement (30 responses) was constant throughout the experiment, whereas the distribution of individual ratios making up the variable-ratio schedule changed across phases: The smallest and largest of these components were varied gradually, with the mean variable-ratio requirement constant at 60 responses. The birds' choices of the variable-ratio schedule tracked the size of the smallest variable-ratio component. A minimum variable-ratio component at or near 1 produced strong preference for the variable-ratio schedule, whereas increases in the minimum variable-ratio component resulted in reduced preference for the variable-ratio schedule. The birds' behavior was qualitatively consistent with Mazur's (1984) hyperbolic model of delayed reinforcement and could be described as approximate maximizing with respect to reinforcement value.     

Matching, maximizing, and hill-climbing

In simple situations, animals consistently choose the better of two alternatives. On concurrent variable-interval variable-interval and variable-interval variable-ratio schedules, they approximately match aggregate choice and reinforcement ratios. The matching law attempts to explain the latter result but does not address the former. Hill-climbing rules such as momentary maximizing can account for both. We show that momentary maximizing constrains molar choice to approximate matching; that molar choice covaries with pigeons' momentary-maximizing estimate; and that the “generalized matching law” follows from almost any hill-climbing rule.     

Choice behavior in a discrete-trial concurrent VI-VR: A test of maximizing theories of matching

Rats were trained on a discrete-trial procedure in which one alternative (VR) was correlated with a constant probability of reinforcement while the other was correlated with a VI schedule which ran during the intertrial intervals and held the scheduled reinforcer until they were obtained by the next VI response. Relative reinforcement rate was varied in series of conditions in which the VR schedule was varied and in series in which the VI was varied. Choice behavior was described well by the generalized matching law, although moderate undermatching occurred for all subjects. Contrary to the predictions of molar maximizing (optimality) theories, there was no consistent bias in favor of the ratio alternative, and the sensitivity to reinforcement allocation was not systematically affected by whether the ratio or interval schedule was varied. The results were also contrary to momentary maximizing accounts, as there was no correspondence between the probability of a changeover to the VI behavior and the time since the last response to the VI alternative. Neither variety of maximizing theory appears to provide a general explanation of matching in concurrent schedules.     

Choice between reliable and unreliable outcomes: mixed percentage-reinforcement in concurrent chains.

Pigeons' choices between alternatives that provided different percentages of reinforcement in mixed schedules were studied using the concurrent-chains procedure. In Experiment 1, the alternatives were terminal-link schedules that were equal in delay and magnitude of reinforcement, but that provided different percentages of reinforcement, with one schedule providing, reinforcement twice as reliably as the other. All pigeons preferred the more reliable schedule, and their level of preference was not systematically affected by variation in the absolute percentage values, or in the magnitude of reinforcement. In Experiment 2, preference for a schedule providing 100% reinforcement over one providing 33% reinforcement increased systematically with increases in the duration of the terminal links. In contrast, preference decreased systematically with increases in the duration of the initial links. Experiment 3 examined choice with equal percentages of reinforcement but unequal delays to reinforcement. Preference for the shorter delay to reinforcement was not systematically affected by variation in the absolute percentage of reinforcement. The overall pattern of results supported predictions based on an extension of the delay-reduction hypothesis to choice procedures involving mixed schedules of percentage reinforcement.     

Molar versus local reinforcement probability as determinants of stimulus value.

During one component of a multiple schedule, pigeons were trained on a discrete-trial concurrent variable-interval variable-interval schedule in which one alternative had a high scheduled rate of reinforcement and the other a low scheduled rate of reinforcement. When the choice proportion between the alternatives matched their respective relative reinforcement frequencies, the obtained probabilities of reinforcement (reinforcer per peck) were approximately equal. In alternate components of the multiple schedule, a single response alternative was presented with an intermediate scheduled rate of reinforcement. During probe trials, each alternative of the concurrent schedule was paired with the constant alternative. The stimulus correlated with the high reinforcement rate was preferred over that with the intermediate rate, whereas the stimulus correlated with the intermediate rate of reinforcement was preferred over that correlated with the low rate of reinforcement. Preference on probe tests was thus determined by the scheduled rate of reinforcement. Other subjects were presented all three alternatives individually, but with a distribution of trial frequency and reinforcement probability similar to that produced by the choice patterns of the original subjects. Here, preferences on probe tests were determined by the obtained probabilities of reinforcement. Comparison of the two sets of results indicates that the availability of a choice alternative, even when not responded to, affects the preference for that alternative. The results imply that models of choice that invoke only obtained probability of reinforcement as the controlling variable (e.g., melioration) are inadequate.     

Maximizing Scales Do Not Reliably Predict Maximizing Behavior in Decisions from Experience

In this paper, we explore the relationships between psychometric and behavioral measures of maximization in decisions from experience (DfE). In two experiments, we measured choice behavior in two experimental paradigms of DfE and self‐reported maximizing tendencies using three prominent scales of maximization. In the repeated consequentialist choice paradigm, participants made repeated choices between two unlabeled options and received consequential feedback on each trial. In the sampling paradigm, participants freely sampled from two options and received feedback on their sampling before making a single consequential choice. Individuals exhibited different degrees of maximizing behavior in both paradigms and across different payoff distributions, but none of the maximizing scales predicted this behavior. These results indicate that maximization scales address constructs that are different from the maximization behavior observed in DfE, and that these measures will need to be improved to reflect behavioral aspects of choice and search from experience. Copyright © 2017 John Wiley & Sons, Ltd.