Slope bias of psychometric functions derived from adaptive data.

Several investigators have fit psychometric functions to data from adaptive procedures for threshold estimation. Although the threshold estimates are in general quite correct, one encounters a slope bias that has not been explained up to now. The present paper demonstrates slope bias for parametric and nonparametric maximum-likelihood fits and for Spearman-K?rber analysis of adaptive data. The examples include staircase and stochastic approximation procedures. The paper then presents an explanation of slope bias based on serial data dependency in adaptive procedures. Data dependency is first illustrated with simple two-trial examples and then extended to realistic adaptive procedures. Finally, the paper presents an adaptive staircase procedure designed to measure threshold and slope directly. In contrast to classical adaptive threshold-only procedures, this procedure varies both a threshold and a spread parameter in response to double trials.     

Tracking of nociceptive thresholds using adaptive psychophysical methods

Psychophysical thresholds reflect the state of the underlying nociceptive mechanisms. For example, noxious events can activate endogenous analgesic mechanisms that increase the nociceptive threshold. Therefore, tracking thresholds over time facilitates the investigation of the dynamics of these underlying mechanisms. Threshold tracking techniques should use efficient methods for stimulus selection and threshold estimation. This study compares, in simulation and in human psychophysical experiments, the performance of different combinations of adaptive stimulus selection procedures and threshold estimation methods. Monte Carlo simulations were first performed to compare the bias and precision of threshold estimates produced by three different stimulus selection procedures (simple staircase, random staircase, and minimum entropy procedure) and two estimation methods (logistic regression and Bayesian estimation). Logistic regression and Bayesian estimations resulted in similar precision only when the prior probability distributions (PDs) were chosen appropriately. The minimum entropy and simple staircase procedures achieved the highest precision, while the random staircase procedure was the least sensitive to different procedure-specific settings. Next, the simple staircase and random staircase procedures, in combination with logistic regression, were compared in a human subject study (n = 30). Electrocutaneous stimulation was used to track the nociceptive perception threshold before, during, and after a cold pressor task, which served as the conditioning stimulus. With both procedures, habituation was detected, as well as changes induced by the conditioning stimulus. However, the random staircase procedure achieved a higher precision. We recommend using the random staircase over the simple staircase procedure, in combination with logistic regression, for nonstationary threshold tracking experiments.     

Measuring, estimating, and understanding the psychometric function: A commentary

The psychometric function, relating the subject’s response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue ofPerception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to thed’ function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Comparing adaptive procedures for estimating the psychometric function for an auditory gap detection task

A subject’s sensitivity to a stimulus variation can be studied by estimating the psychometric function. Generally speaking, three parameters of the psychometric function are of interest: the performance threshold, the slope of the function, and the rate at which attention lapses occur. In the present study, three psychophysical procedures were used to estimate the three-parameter psychometric function for an auditory gap detection task. These were an up–down staircase (up–down) procedure, an entropy-based Bayesian (entropy) procedure, and an updated maximum-likelihood (UML) procedure. Data collected from four young, normal-hearing listeners showed that while all three procedures provided similar estimates of the threshold parameter, the up–down procedure performed slightly better in estimating the slope and lapse rate for 200 trials of data collection. When the lapse rate was increased by mixing in random responses for the three adaptive procedures, the larger lapse rate was especially detrimental to the efficiency of the up–down procedure, and the UML procedure provided better estimates of the threshold and slope than did the other two procedures.     

Measuring, estimating, and understanding the psychometric function: a commentary.

The psychometric function, relating the subject's response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue of Perception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to the d' function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Estimation of psychometric functions from adaptive tracking procedures.

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Estimation of psychometric functions from adaptive tracking procedures

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from-trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to -target-level-for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated-from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Adaptive threshold estimation with unforced-choice tasks.

This paper evaluates an adaptive staircase procedure for threshold estimation that is suitable for unforced-choice tasks-ones with the additional response alternative don't know. Within the framework of a theory of indecision, evidence is developed that fluctuations of the response criterion are much less detrimental to unforced-choice tasks than to yes/no tasks. An adaptive staircase procedure for unforced-choice tasks is presented. Computer simulations show a slight gain in efficiency if don't know responses are allowed, even if response criteria vary. A behavioral comparison with forced-choice and yes/no procedures shows that the new procedure outdoes the other two with respect to reliability. This is especially true for naive participants. For well-trained participants it is also slightly more efficient than the forced-choice procedure, and it produces a smaller systematic error than the yes/no procedure. Moreover, informal observations suggest that participants are more comfortable with unforced tasks than with forced ones.     

Psychophysics with children: Investigating the effects of attentional lapses on threshold estimates

When assessing the perceptual abilities of children, researchers tend to use psychophysical techniques designed for use with adults. However, children’s poorer attentiveness might bias the threshold estimates obtained by these methods. Here, we obtained speed discrimination threshold estimates in 6- to 7-year-old children in UK Key Stage 1 (KS1), 7- to 9-year-old children in Key Stage 2 (KS2), and adults using three psychophysical procedures: QUEST, a 1-up 2-down Levitt staircase, and Method of Constant Stimuli (MCS). We estimated inattentiveness using responses to “easy” catch trials. As expected, children had higher threshold estimates and made more errors on catch trials than adults. Lower threshold estimates were obtained from psychometric functions fit to the data in the QUEST condition than the MCS and Levitt staircases, and the threshold estimates obtained when fitting a psychometric function to the QUEST data were also lower than when using the QUEST mode. This suggests that threshold estimates cannot be compared directly across methods. Differences between the procedures did not vary significantly with age group. Simulations indicated that inattentiveness biased threshold estimates particularly when threshold estimates were computed as the QUEST mode or the average of staircase reversals. In contrast, thresholds estimated by post-hoc psychometric function fitting were less biased by attentional lapses. Our results suggest that some psychophysical methods are more robust to attentiveness, which has important implications for assessing the perception of children and clinical groups.

     

Adaptive threshold estimation with unforced-choice tasks

This paper evaluates an adaptive staircase procedure for threshold estimation that is suitable for unforced-choice tasks—ones with the additional response alternativedon’t know. Within the framework of a theory of indecision, evidence is developed that fluctuations of the response criterion are much less detrimental to unforced-choice tasks than to yes/no tasks. An adaptive staircase procedure for unforced-choice tasks is presented. Computer simulations show a slight gain in efficiency ifdon’t know responses are allowed, even if response criteria vary. A behavioral comparison with forcedchoice and yes/no procedures shows that the new procedure outdoes the other two with respect to reliability. This is especially true for naive participants. For well-trained participants it is also slightly more efficient than the forced-choice procedure, and it produces a smaller systematic error than the yes/no procedure. Moreover, informal observations suggest that participants are more comfortable with unforced tasks than with forced ones.     

Fast and accurate measurement of taste and smell thresholds using a maximum-likelihood adaptive staircase procedure.

This paper evaluates the use of a maximum-likelihood adaptive staircase psychophysical procedure (ML-PEST), originally developed in vision and audition, for measuring detection thresholds in gustation and olfaction. The basis for the psychophysical measurement of thresholds with the ML-PEST procedure is developed. Then, two experiments and four simulations are reported. In the first experiment, ML-PEST was compared with the Wetherill and Levitt up-down staircase method and with the Cain ascending method of limits in the measurement of butyl alcohol thresholds. The four Monte Carlo simulations compared the three psychophysical procedures. In the second experiment, the test-retest reliability of MLPEST for measuring NaCl and butyl alcohol thresholds was assessed. The results indicate that the ML-PEST method gives reliable and precise threshold measurements. Its ability to detect malingerers shows considerable promise. It is recommended for use in clinical testing.     

Fast and accurate measurement of taste and smell thresholds using a maximum-likelihood adaptive staircase procedure

This paper evaluates the use of a maximum-likelihood adaptive staircase psychophysical procedure (ML-PEST), originally developed in vision and audition, for measuring detection thresholds in gustation and olfaction. The basis for the psychophysical measurement of thresholds with the ML-PEST procedure is developed. Then, two experiments and four simulations are reported. In the first experiment, ML-PEST was compared with the Wetherill and Levitt up-down staircase method and with the Cain ascending method of limits in the measurement of butyl alcohol thresholds. The four Monte Carlo simulations compared the three psychophysical procedures. In the second experiment, the test-retest reliability of ML-PEST for measuring NaCl and butyl alcohol thresholds was assessed. The results indicate that the ML-PEST method gives reliable and precise threshold measurements. Its ability to detect malingerers shows considerable promise. It is recommended for use in clinical testing.     

On the limitations of fixed‐step‐size adaptive methods with response confidence

The family of (non‐parametric, fixed‐step‐size) adaptive methods, also known as ‘up–down’ or ‘staircase’ methods, has been used extensively in psychophysical studies for threshold estimation. Extensions of adaptive methods to non‐binary responses have also been proposed. An example is the three‐category weighted up–down (WUD) method (Kaernbach, 2001) and its four‐category extension (Klein, 2001). Such an extension, however, is somewhat restricted, and in this paper we discuss its limitations. To facilitate the discussion, we characterize the extension of WUD by an algorithm that incorporates response confidence into a family of adaptive methods. This algorithm can also be applied to two other adaptive methods, namely Derman's up–down method and the biased‐coin design, which are suitable for estimating any threshold quantiles. We then discuss via simulations of the above three methods the limitations of the algorithm. To illustrate, we conduct a small scale of experiment using the extended WUD under different response confidence formats to evaluate the consistency of threshold estimation.     

Statistical properties of Pollack’s PEST procedure

The bias and variability of threshold estimates produced by Pollack’s PEST procedure were studied by means of repeated computer simulations of PEST runs. The influence of the size of the exit criterion, the bumping-into-zero rule, and three methods for obtaining the estimate from the data are discussed. Pollack’s PEST procedure is found to result in a highly variable number of trials per run, and to be approximately as good as a simpler staircase procedure in terms of the bias and variability of the estimates obtained.     

A comparison of three adaptive psychophysical procedures using inexperienced listeners

Thresholds for a masked 1-kHz tone were obtained in single sessions from 60 inexperienced listeners. A two-alternative forced-choice procedure was used in conjunction with one of three adaptive psychophysical techniques. These techniques comprised two staircase techniques targeting either 70.7% or 79.4% correct detection (Staircase-71 and Staircase-79), and parameter estimation by sequential testing (PEST), targeting 80% correct detection. Listeners were provided with a rationale for maintaining concentration at weak signal levels. Similar threshold values were obtained from the Staircase-79 and PEST groups in equal numbers of trials. The degree of oscillation in the level of the signal around the value finally chosen as the threshold was comparable for both staircase techniques. Subsequent fixed-level testing did not provide a true indication of the subjects' capabilities. The amount by which percent correct in fixed-level testing differed from expectations based on adaptive testing varied among the techniques. Additional thresholds were obtained in a second session from 30 of the original subjects. Thresholds with both staircase techniques improved by about 1 dB on retest, while thresholds with PEST were constant across sessions. The variability of the data compared well with that from studies involving experienced listeners.     

Adaptive procedures in psychophysical research

As research on sensation and perception has grown more sophisticated during the last century, new adaptive methodologies have been developed to increase efficiency and reliability of measurement. An experimental procedure is said to be adaptive if the physical characteristics of the stimuli on each trial are determined by the stimuli and responses that occurred in the previous trial or sequence of trials. In this paper, the general development of adaptive procedures is described, and three commonly used methods are reviewed. Typically, a threshold value is measured using these methods, and, in some cases, other characteristics of the psychometric function underlying perceptual performance, such as slope, may be developed. Results of simulations and experiments with human subjects are reviewed to evaluate the utility of these adaptive procedures and the special circumstances under which one might be superior to another.     

A comparison of sniff bottle staircase and olfactometer-based threshold tests

Olfactometers have been gaining popularity as research tools, but they have yet to replace established testing procedures in a variety of laboratory and clinical settings, including absolute threshold tests. In this research, we designed and operated a simple olfactometer with which to assess threshold. To do this, we used a method-of-adjustment test that was compared to the three-alternative forced choice ascending sniff bottle staircase method, which is currently a standard threshold test procedure. We found that the olfactometer threshold test correlated highly with the staircase method, and that it possessed suitable test–retest reliability. The advantages of the olfactometer threshold test include faster test time and reduced cleaning and reassembly demands. Future use of the olfactometer in olfactory identification and/or detection thresholds amongst odors is also outlined.     

Adaptive procedures in psychophysical research.

As research on sensation and perception has grown more sophisticated during the last century, new adaptive methodologies have been developed to increase efficiency and reliability of measurement. An experimental procedure is said to be adaptive if the physical characteristics of the stimuli on each trial are determined by the stimuli and responses that occurred in the previous trial or sequence of trials. In this paper, the general development of adaptive procedures is described, and three commonly used methods are reviewed. Typically, a threshold value is measured using these methods, and, in some cases, other characteristics of the psychometric function underlying perceptual performance, such as slope, may be developed. Results of simulations and experiments with human subjects are reviewed to evaluate the utility of these adaptive procedures and the special circumstances under which one might be superior to another.     

Statistical properties of staircase estimates

The bias and variability of staircase estimators were studied by means of repeated computer simulations of staircase runs. Both forced-choice and yes-no staircases were simulated. The influence of the shape of the psychometric function, the location and spacing of the stimuli, the number of trials in a run, and the method of deriving the estimate from the data are discussed. The forced-choice staircase is compared to the yes-no staircase, and the limitations of the simulation procedure are outlined.     

Estimating psychometric functions in forced-choice situations: significant biases found in threshold and slope estimations when small samples are used

When a theoretical psychometric function is fitted to experimental data (as in the obtaining of a psychophysical threshold), maximum-likelihood or probit methods are generally used. In the present paper, the behavior of these curve-fitting methods is studied for the special case of forced-choice experiments, in which the probability of a subject's making a correct response by chance is not zero. A mathematical investigation of the variance of the threshold and slope estimators shows that, in this case, the accuracy of the methods is much worse, and their sensitivity to the way data are sampled is greater, than in the case in which chance level is zero. Further, Monte Carlo simulations show that, in practical situations in which only a finite number of observations are made, the mean threshold and slope estimates are significantly biased. The amount of bias depends on the curve-fitting method and on the range of intensity values, but it is always greater in forced-choice situations than when chance level is zero.