Additional rules for the transformed up-down method in psychophysics

In a classic paper, Levitt (1971) described an adaptive procedure for estimating points on the psychometric function known as thetransformed up-down method. Levitt discussed the assumptions of the method and presented a brief table with simple rules that converge to a few different points on the psychometric function. Levitt’s original table contains only the simplest rules, and sparsely covers the range of the psychometric function. This paper provides a table with previously unpublished rules which cover the range of the psychometric function at 5% intervals. There is a brief review of the major issues in adaptive testing. Technical issues such as the mean length and logical construction of the new rules are discussed.     

Comparing adaptive procedures for estimating the psychometric function for an auditory gap detection task

A subject’s sensitivity to a stimulus variation can be studied by estimating the psychometric function. Generally speaking, three parameters of the psychometric function are of interest: the performance threshold, the slope of the function, and the rate at which attention lapses occur. In the present study, three psychophysical procedures were used to estimate the three-parameter psychometric function for an auditory gap detection task. These were an up–down staircase (up–down) procedure, an entropy-based Bayesian (entropy) procedure, and an updated maximum-likelihood (UML) procedure. Data collected from four young, normal-hearing listeners showed that while all three procedures provided similar estimates of the threshold parameter, the up–down procedure performed slightly better in estimating the slope and lapse rate for 200 trials of data collection. When the lapse rate was increased by mixing in random responses for the three adaptive procedures, the larger lapse rate was especially detrimental to the efficiency of the up–down procedure, and the UML procedure provided better estimates of the threshold and slope than did the other two procedures.     

Estimation of psychometric functions from adaptive tracking procedures

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from-trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to -target-level-for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated-from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Invariance of the psychometric function for character recognition across the visual field

The psychometric function for recognition of singly presented digits as a function of digit contrast was measured at 2° steps across the horizontal meridian of the visual field, under monocular and binocular viewing conditions. A maximum-likelihood staircase procedure was used in a 10-alternative forcedchoice recognition paradigm to gather the data. Both the Weibull and the logistic psychometric functions provide excellent fits to the observed data. The slopes of these functions at their point of inflection ranged from 4.0 to 5.0 proportion-correct/log₁₀-unit contrast, for both monocular and binocular viewing and for all loci in the visual field. These slope values correspond to short-term measurements (around 30 trials, or 1 min) and do not include performance variations of longer duration; the latter are estimated to increase slope by a factor of about 1.5. A single psychometric function shape, centered around a threshold value, therefore describes recognition performance at all retinal loci and binocularity. An empirical comparison of slope results across the literature shows that the function’s slope is about twice that reported for a number of detection tasks. The comparison of recognition contrast thresholds, percentage correct values, and other performance measures across studies requires the knowledge of the psychometric function’s slope, and our results thus provide a firm basis for the study of low-contrast character recognition     

Estimation of psychometric functions from adaptive tracking procedures.

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Bayesian adaptive estimation: The next dimension

We propose a new psychometric model for two-dimensional stimuli, such as color differences, based on parameterizing the threshold of a one-dimensional psychometric function as an ellipse. The Ψ Bayesian adaptive estimation method applied to this model yields trials that vary in multiple stimulus dimensions simultaneously. Simulations indicate that this new procedure can be much more efficient than the more conventional procedure of estimating the psychometric function on one-dimensional lines independently, requiring only one-fourth or less the number of trials for equivalent performance in typical situations. In a real psychophysical experiment with a yes-no task, as few as 22 trials per estimated threshold ellipse were enough to consistently demonstrate certain color appearance phenomena. We discuss the practical implications of the multidimensional adaptation. In order to make the application of the model practical, we present two significantly faster algorithms for running the Ψ method: a discretized algorithm utilizing the Fast Fourier Transform for better scaling with the sampling rates and a Monte Carlo particle filter algorithm that should be able to scale into even more dimensions.     

Invariance of the psychometric function for character recognition across the visual field.

The psychometric function for recognition of singly presented digits as a function of digit contrast was measured at 2 degrees steps across the horizontal meridian of the visual field, under monocular and binocular viewing conditions. A maximum-likelihood staircase procedure was used in a 10-alternative forced-choice recognition paradigm to gather the data Both the Weibull and the logistic psychometric functions provide excellent fits to the observed data. The slopes of these functions at their point of inflection ranged from 4.0 to 5.0 proportion-correct/log10-unit contrast, for both monocular and binocular viewing and for all loci in the visual field. These slope values correspond to short-term measurements (around 30 trials, or 1 min) and do not include performance variations of longer duration; the latter are estimated to increase slope by a factor of about 1.5. A single psychometric function shape, centered around a threshold value, therefore describes recognition performance at all retinal loci and binocularity. An empirical comparison of slope results across the literature shows that the function's slope is about twice that reported for a number of detection tasks. The comparison of recognition contrast thresholds, percentage correct values, and other performance measures across studies requires the knowledge of the psychometric function's slope, and our results thus provide a firm basis for the study of low-contrast character recognition.     

The effect of exposure duration on visual character identification in single, whole, and partial report

The psychometric function of single-letter identification is typically described as a function of stimulus intensity. However, the effect of stimulus exposure duration on letter identification remains poorly described. This is surprising because the effect of exposure duration has played a central role in modeling performance in whole and partial report (Shibuya & Bundesen, 1988). Therefore, we experimentally investigated visual letter identification as a function of exposure duration. We compared the exponential, the gamma, and the Weibull psychometric functions, all with a temporal offset included, as well as the ex-Gaussian, the log-logistic, and finally the squared-logistic, which is a psychometric function that to our knowledge has not been described before. The log-logistic and the squared-logistic psychometric function fit well to experimental data. Also, we conducted an experiment to test the ability of the psychometric functions to fit single-letter identification data, at different stimulus contrast levels; also here the same psychometric functions prevailed. Finally, after insertion into Bundesen's Theory of Visual Attention (Bundesen, 1990), the same psychometric functions enable closer fits to data from a previous whole and partial report experiment.     

Shifts in the psychometric function and their implications for models of timing

This study examined how two models of timing, scalar expectancy theory (SET) and learning to time (LeT), conceptualize the learning process in temporal tasks, and then reports two experiments to test these conceptualizations. Pigeons responded on a two-alternative free-operant psychophysical procedure in which responses on the left key were reinforceable during the first two, but not the last two, quarters of a 60-s trial, and responses on the right key were reinforceable during the last two, but not the first two, quarters of the trial. In Experiment 1 three groups of birds experienced a difference in reinforcement rates between the two keys only at the end segments of the trial (i.e., between the first and fourth quarters), only around the middle segments of the trial (i.e., between the second and third quarters), or in both end and middle segments. In Condition 1 the difference in reinforcement rate favored the left key; in Condition 2 it favored the right key. When the reinforcement rates differed in the end segments of the trial, the psychometric function--the proportion of right responses across the trial--did not shift across conditions; when it occurred around the middle of the trial or in both end and middle segments, the psychometric function shifted across conditions. Experiment 2 showed that the psychometric function shifts even when the overall reinforcement rate for the two keys is equal, provided the rates differ around the middle of the trial. This pattern of shifts of the psychometric function is inconsistent with SET. In contrast, LeT provided a good quantitative fit to the data.     

A test for psychometric function shift

A nonparametric, small-sample-size test for the homogeneity of two psychometric functions against the left- and right-shift alternatives has been developed. The test is designed to determine whether it is safe to amalgamate psychometric functions obtained in different experimental sessions. The sum of the lower and upper p-values of the exact (conditional) Fisher test for several 2 × 2 contingency tables (one for each point of the psychometric function) is employed as the test statistic. The probability distribution of the statistic under the null (homogeneity) hypothesis is evaluated to obtain corresponding p-values. Power functions of the test have been computed by randomly generating samples from Weibull psychometric functions. The test is free of any assumptions about the shape of the psychometric function; it requires only that all observations are statistically independent.     

Statistical properties of staircase estimates

The bias and variability of staircase estimators were studied by means of repeated computer simulations of staircase runs. Both forced-choice and yes-no staircases were simulated. The influence of the shape of the psychometric function, the location and spacing of the stimuli, the number of trials in a run, and the method of deriving the estimate from the data are discussed. The forced-choice staircase is compared to the yes-no staircase, and the limitations of the simulation procedure are outlined.     

Measuring, estimating, and understanding the psychometric function: a commentary.

The psychometric function, relating the subject's response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue of Perception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to the d' function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling.

The psychometric function relates an observer's performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper's principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one's choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling

The psychometric function relates an observer’s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper’s principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one’s choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

Visual detection of signals in the presence of continuous and pulsed backqrounds

The detectability of intensity increments in the presence of continuous and pulsed backgrounds was investigated using a two-alternative, temporal, forced-choice procedure. Differences were found in intensity-duration reciprocity relations, the form of the Weber function, and the shape of the psychometric function between continuous- and pulsed-detection conditions. In a second study, signals were added either to a steady background (simple detection) or to background plus pedestal (pedestal detection). Two unusual phenomena were noted, namely, “negative masking” and the “pedestal effect.” The interpretation of the results in terms of a simple, Poisson-detection model is discussed.     

Operant methods for mouse psychoacoustics

Tone detection thresholds for a 10-kHz tone in NMRI mice were determined in psychoacoustic experiments using both a constant-stimuli procedure and a two-down/one-up adaptive-tracking procedure in the same subjects and applying identical threshold criteria (70.7% response probability). Constant-stimuli thresholds were on average 24 dB lower than adaptive-tracking thresholds, and there was a trend indicating that constant-stimuli thresholds were less variable than adaptive-tracking thresholds. Furthermore, in the constant-stimuli procedure the number of trials constituting the psychometric function could be reduced from 100 to 50 trials without a large loss of accuracy of threshold determination. In the constant-stimuli procedure, the threshold value was affected by the threshold criteria. The lowest and least variable constant-stimuli thresholds were obtained by applying signal detection theory and a criterion ofd′=1. Thus, the constant-stimuli procedure in combination with signal detection theory appears to be better suited than the adaptive-tracking procedure to determine auditory sensory thresholds.     

Measuring, estimating, and understanding the psychometric function: A commentary

The psychometric function, relating the subject’s response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue ofPerception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to thed’ function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Numerosity discrimination in preschool children

We used a numerical bisection procedure to examine preschool children's sensitivity to the numerical attributes of stimuli. In Experiment 1 children performed two tasks. In the Cups Task they earned coins for choosing a green cup after two drumbeats and a blue cup after eight drumbeats. In the Gloves Task they earned coins for raising a red glove on their left hand after two drumbeats and a yellow glove on their right hand after eight drumbeats. Then in each task a psychometric function was obtained by presenting intermediate numerosities and recording the percentage of trials in which children chose the "many" option. In Experiment 2 children's performance in a '2 vs. 8' discrimination was compared with their performance in a "4 vs. 16" discrimination. Results showed that the individual psychometric functions were of two types, one in which the percentage of "many" choices increased gradually with stimulus numerosity and another in which it increased abruptly, in a step-like manner. Although the average point of subjective equality was close to the geometric mean of the anchor numerosities and the average functions for "2 vs. 8" and "4 vs. 16" superimposed when plotted on a common scale (the scalar property), the individual data were highly variable both across tasks (Cups and Gloves) and numerosity ranges ('2 vs. 8' and '4 vs. 16'). It is suggested that between- and within-subjects variability in the psychometric function is related to children's verbalizations about the sample stimulus.     

Bayesian active probabilistic classification for psychometric field estimation

Psychometric functions are typically estimated by fitting a parametric model to categorical subject responses. Procedures to estimate unidimensional psychometric functions (i.e., psychometric curves) have been subjected to the most research, with modern adaptive methods capable of quickly obtaining accurate estimates. These capabilities have been extended to some multidimensional psychometric functions (i.e., psychometric fields) that are easily parameterizable, but flexible procedures for general psychometric field estimation are lacking. This study introduces a nonparametric Bayesian psychometric field estimator operating on subject queries sequentially selected to improve the estimate in some targeted way. This estimator implements probabilistic classification using Gaussian processes trained by active learning. The accuracy and efficiency of two different actively sampled estimators were compared to two non-actively sampled estimators for simulations of one of the simplest psychometric fields in common use: the pure-tone audiogram. The actively sampled methods achieved estimate accuracy equivalent to the non-actively sampled methods with fewer observations. This trend held for a variety of audiogram phenotypes representative of the range of human auditory perception. Gaussian process classification is a general estimation procedure capable of extending to multiple input variables and response classes. Its success with a two-dimensional psychometric field informed by binary subject responses holds great promise for extension to complex perceptual models currently inaccessible to practical estimation.