Estimation of psychometric functions from adaptive tracking procedures.

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Estimation of psychometric functions from adaptive tracking procedures

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from-trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to -target-level-for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated-from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Slope bias of psychometric functions derived from adaptive data.

Several investigators have fit psychometric functions to data from adaptive procedures for threshold estimation. Although the threshold estimates are in general quite correct, one encounters a slope bias that has not been explained up to now. The present paper demonstrates slope bias for parametric and nonparametric maximum-likelihood fits and for Spearman-K?rber analysis of adaptive data. The examples include staircase and stochastic approximation procedures. The paper then presents an explanation of slope bias based on serial data dependency in adaptive procedures. Data dependency is first illustrated with simple two-trial examples and then extended to realistic adaptive procedures. Finally, the paper presents an adaptive staircase procedure designed to measure threshold and slope directly. In contrast to classical adaptive threshold-only procedures, this procedure varies both a threshold and a spread parameter in response to double trials.     

Slope bias of psychometric functions derived from adaptive data

Several investigators have fit psychometric functions to data from adaptive procedures for threshold estimation. Although the threshold estimates are in general quite correct, one encounters a slope bias that has not been explained up to now. The present paper demonstrates slope bias for parametric and nonparametric maximum-likelihood fits and for Spearman-Kärber analysis of adaptive data. The examples include staircase and stochastic approximation procedures. The paper then presents an explanation of slope bias based on serial data dependency in adaptive procedures. Data dependency is first illustrated with simple two-trial examples and then extended to realistic adaptive procedures. Finally, the paper presents an adaptive staircase procedure designed to measure threshold and slope directly. In contrast to classical adaptive threshold-only procedures, this procedure varies both a threshold and a spread parameter in response to double trials.     

Evaluation of maximum-likelihood estimators in nonintensive auditory psychophysics

This is a brief report on the use of maximum-likelihood (ML) estimators in auditory psychophysics. Slope parameters of psychometric functions are characterized for three nonintensive auditory tasks: forced-choice discrimination of interaural time differences (ΔITD), frequency (Δf), and duration (Δt). Using these slope estimates, the ML method is implemented and threshold estimates are obtained for the three tasks and compared with previously published data. ΔITD thresholds were additionally measured for human observers by means of two other psychophysical procedures: the constant-stimuli (CS) and the 2-down 1-up methods (Wetherill & Levitt, 1965). Standard errors were smallest for the ML method. Finally, simulations showed ML estimates to be more efficient than the CS andk-down 1-up procedures fork=2 to 5. For up—down procedures, efficiency was highest fork values of 3 and 4. The entropy (Shannon, 1949) of ML estimates was the smallest of the simulated procedures, but poorer than ideal by 0.5 bits.     

Adaptive psychophysical methods for nonmonotonic psychometric functions

Many psychophysical tasks in current use render nonmonotonic psychometric functions; these include the oddball task, the temporal generalization task, the binary synchrony judgment task, and other forms of the same–different task. Other tasks allow for ternary responses and render three psychometric functions, one of which is also nonmonotonic, like the ternary synchrony judgment task or the unforced choice task. In all of these cases, data are usually collected with the inefficient method of constant stimuli (MOCS), because extant adaptive methods are only applicable when the psychometric function is monotonic. This article develops stimulus placement criteria for adaptive methods designed for use with nonmonotonic psychometric functions or with ternary tasks. The methods are transformations of conventional up–down rules. Simulations under three alternative psychophysical tasks prove the validity of these methods, their superiority to MOCS, and the accuracy with which they recover direct estimates of the parameters determining the psychometric functions, as well as estimates of derived quantities such as the point of subjective equality or the difference limen. Practical recommendations and worked-out examples are provided to illustrate how to use these adaptive methods in empirical research.     

On the theoretical error bound for estimating psychometric functions

The theoretical limits to the amount of error, or the Cramer-Rao bounds, were derived for estimating psychometric functions. These theoretical error bounds were compared with the variability of psychometric functions estimated from human as well as computer-simulated observers. For the simulated observers, due to the limited efficiency of the sampling strategies, including the placement of the signals and the distribution of the trials, the variances of the estimated parameters are seven times the theoretical bound for threshold and 22 times that for slope. For the human observers, the variance is 18 times the theoretical bounds for threshold and 80 times that for slope. Therefore, a major portion of the variances (60% for threshold and 73% for slope) for the human observers is associated with factors other than sampling strategies. Further improvement of the accuracy for estimating psychometric functions will depend on not only optimizing the sampling strategy, but also better understanding the various sources of error related to the behavior of human observers.     

Measuring, estimating, and understanding the psychometric function: A commentary

The psychometric function, relating the subject’s response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue ofPerception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to thed’ function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

The role of parametric assumptions in adaptive Bayesian estimation

Variants of adaptive Bayesian procedures for estimating the 5% point on a psychometric function were studied by simulation. Bias and standard error were the criteria to evaluate performance. The results indicated a superiority of (a) uniform priors, (b) model likelihood functions that are odd symmetric about threshold and that have parameter values larger than their counterparts in the psychometric function, (c) stimulus placement at the prior mean, and (d) estimates defined as the posterior mean. Unbiasedness arises in only 10 trials, and 20 trials ensure constant standard errors. The standard error of the estimates equals 0.617 times the inverse of the square root of the number of trials. Other variants yielded bias and larger standard errors.     

Measuring, estimating, and understanding the psychometric function: a commentary.

The psychometric function, relating the subject's response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue of Perception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to the d' function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Psychophysics with children: Investigating the effects of attentional lapses on threshold estimates

When assessing the perceptual abilities of children, researchers tend to use psychophysical techniques designed for use with adults. However, children’s poorer attentiveness might bias the threshold estimates obtained by these methods. Here, we obtained speed discrimination threshold estimates in 6- to 7-year-old children in UK Key Stage 1 (KS1), 7- to 9-year-old children in Key Stage 2 (KS2), and adults using three psychophysical procedures: QUEST, a 1-up 2-down Levitt staircase, and Method of Constant Stimuli (MCS). We estimated inattentiveness using responses to “easy” catch trials. As expected, children had higher threshold estimates and made more errors on catch trials than adults. Lower threshold estimates were obtained from psychometric functions fit to the data in the QUEST condition than the MCS and Levitt staircases, and the threshold estimates obtained when fitting a psychometric function to the QUEST data were also lower than when using the QUEST mode. This suggests that threshold estimates cannot be compared directly across methods. Differences between the procedures did not vary significantly with age group. Simulations indicated that inattentiveness biased threshold estimates particularly when threshold estimates were computed as the QUEST mode or the average of staircase reversals. In contrast, thresholds estimated by post-hoc psychometric function fitting were less biased by attentional lapses. Our results suggest that some psychophysical methods are more robust to attentiveness, which has important implications for assessing the perception of children and clinical groups.

     

Adaptive psychophysical procedures, loss functions, and entropy

In this article, we present a new strategy for locating a point on a psychometric function (threshold determination) for yes-no procedures, called EntFirst. Our results show that it performs better than many existing strategies for estimating the 50% threshold that are commonly used in nonlaboratory settings. We also provide a review of existing algorithms for finding thresholds, with an emphasis on identifying the types of problems for which each algorithm is useful. Finally, we address a number of issues that are not adequately covered in the literature, including choosing an appropriate loss function to evaluate the performance of an algorithm for a given problem.     

On the limitations of fixed‐step‐size adaptive methods with response confidence

The family of (non‐parametric, fixed‐step‐size) adaptive methods, also known as ‘up–down’ or ‘staircase’ methods, has been used extensively in psychophysical studies for threshold estimation. Extensions of adaptive methods to non‐binary responses have also been proposed. An example is the three‐category weighted up–down (WUD) method (Kaernbach, 2001) and its four‐category extension (Klein, 2001). Such an extension, however, is somewhat restricted, and in this paper we discuss its limitations. To facilitate the discussion, we characterize the extension of WUD by an algorithm that incorporates response confidence into a family of adaptive methods. This algorithm can also be applied to two other adaptive methods, namely Derman's up–down method and the biased‐coin design, which are suitable for estimating any threshold quantiles. We then discuss via simulations of the above three methods the limitations of the algorithm. To illustrate, we conduct a small scale of experiment using the extended WUD under different response confidence formats to evaluate the consistency of threshold estimation.     

Converting between measures of slope of the psychometric function

The psychometric function’s slope provides information about the reliability of psychophysical threshold estimates. Furthermore, knowing the slope allows one to compare, across studies, thresholds that were obtained at different performance criterion levels. Unfortunately, the empirical validation of psychometric function slope estimates is hindered by the bewildering variety of slope measures that are in use. The present article provides conversion formulas for the most popular cases, including the logistic, Weibull, Quick, cumulative normal, and hyperbolic tangent functions as analytic representations, in both linear and log coordinates and to different log bases, the practical decilog unit, the empirically based interquartile range measure of slope, and slope in a? representation of performance.     

Converting between measures of slope of the psychometric function.

The psychometric function's slope provides information about the reliability of psychophysical threshold estimates. Furthermore, knowing the slope allows one to compare, across studies, thresholds that were obtained at different performance criterion levels. Unfortunately, the empirical validation of psychometric function slope estimates is hindered by the bewildering variety of slope measures that are in use. The present article provides conversion formulas for the most popular cases, including the logistic, Weibull, Quick, cumulative normal, and hyperbolic tangent functions as analytic representations, in both linear and log coordinates and to different log bases, the practical decilog unit, the empirically based interquartile range measure of slope, and slope in a d' representation of performance.     

Comparing and unifying slope estimates across psychometric function models

The psychometric function relating stimulus intensity to response probability generally presents itself as a monotonically increasing sigmoid profile. Two summary parameters of the function are particularly important as measures of perceptual performance: the threshold parameter, which defines the location of the function over the stimulus axis (abscissa), and the slope parameter, which defines the (local) rate at which response probability increases with increasing stimulus intensity. In practice, the psychometric function may be modeled by a variety of mathematical structures, and the resulting algebraic expression describing the slope parameter may vary considerably between different functions fitted to the same experimental data. This variation often restricts comparisons between studies that select different functions and compromises the general interpretation of slope values. This article reviews the general characteristics of psychometric function models, discusses three strategies for resolving the issue of slope value differences, and presents mathematical expressions for implementing each strategy.     

Bayesian adaptive estimation: The next dimension

We propose a new psychometric model for two-dimensional stimuli, such as color differences, based on parameterizing the threshold of a one-dimensional psychometric function as an ellipse. The Ψ Bayesian adaptive estimation method applied to this model yields trials that vary in multiple stimulus dimensions simultaneously. Simulations indicate that this new procedure can be much more efficient than the more conventional procedure of estimating the psychometric function on one-dimensional lines independently, requiring only one-fourth or less the number of trials for equivalent performance in typical situations. In a real psychophysical experiment with a yes-no task, as few as 22 trials per estimated threshold ellipse were enough to consistently demonstrate certain color appearance phenomena. We discuss the practical implications of the multidimensional adaptation. In order to make the application of the model practical, we present two significantly faster algorithms for running the Ψ method: a discretized algorithm utilizing the Fast Fourier Transform for better scaling with the sampling rates and a Monte Carlo particle filter algorithm that should be able to scale into even more dimensions.     

Invariance of the psychometric function for character recognition across the visual field

The psychometric function for recognition of singly presented digits as a function of digit contrast was measured at 2° steps across the horizontal meridian of the visual field, under monocular and binocular viewing conditions. A maximum-likelihood staircase procedure was used in a 10-alternative forcedchoice recognition paradigm to gather the data. Both the Weibull and the logistic psychometric functions provide excellent fits to the observed data. The slopes of these functions at their point of inflection ranged from 4.0 to 5.0 proportion-correct/log₁₀-unit contrast, for both monocular and binocular viewing and for all loci in the visual field. These slope values correspond to short-term measurements (around 30 trials, or 1 min) and do not include performance variations of longer duration; the latter are estimated to increase slope by a factor of about 1.5. A single psychometric function shape, centered around a threshold value, therefore describes recognition performance at all retinal loci and binocularity. An empirical comparison of slope results across the literature shows that the function’s slope is about twice that reported for a number of detection tasks. The comparison of recognition contrast thresholds, percentage correct values, and other performance measures across studies requires the knowledge of the psychometric function’s slope, and our results thus provide a firm basis for the study of low-contrast character recognition     

Invariance of the psychometric function for character recognition across the visual field.

The psychometric function for recognition of singly presented digits as a function of digit contrast was measured at 2 degrees steps across the horizontal meridian of the visual field, under monocular and binocular viewing conditions. A maximum-likelihood staircase procedure was used in a 10-alternative forced-choice recognition paradigm to gather the data Both the Weibull and the logistic psychometric functions provide excellent fits to the observed data. The slopes of these functions at their point of inflection ranged from 4.0 to 5.0 proportion-correct/log10-unit contrast, for both monocular and binocular viewing and for all loci in the visual field. These slope values correspond to short-term measurements (around 30 trials, or 1 min) and do not include performance variations of longer duration; the latter are estimated to increase slope by a factor of about 1.5. A single psychometric function shape, centered around a threshold value, therefore describes recognition performance at all retinal loci and binocularity. An empirical comparison of slope results across the literature shows that the function's slope is about twice that reported for a number of detection tasks. The comparison of recognition contrast thresholds, percentage correct values, and other performance measures across studies requires the knowledge of the psychometric function's slope, and our results thus provide a firm basis for the study of low-contrast character recognition.