Fitting model-based psychometric functions to simultaneity and temporal-order judgment data: MATLAB and R routines

Research on temporal-order perception uses temporal-order judgment (TOJ) tasks or synchrony judgment (SJ) tasks in their binary SJ2 or ternary SJ3 variants. In all cases, two stimuli are presented with some temporal delay, and observers judge the order of presentation. Arbitrary psychometric functions are typically fitted to obtain performance measures such as sensitivity or the point of subjective simultaneity, but the parameters of these functions are uninterpretable. We describe routines in MATLAB and R that fit model-based functions whose parameters are interpretable in terms of the processes underlying temporal-order and simultaneity judgments and responses. These functions arise from an independent-channels model assuming arrival latencies with exponential distributions and a trichotomous decision space. Different routines fit data separately for SJ2, SJ3, and TOJ tasks, jointly for any two tasks, or also jointly for the three tasks (for common cases in which two or even the three tasks were used with the same stimuli and participants). Additional routines provide bootstrap p-values and confidence intervals for estimated parameters. A further routine is included that obtains performance measures from the fitted functions. An R package for Windows and source code of the MATLAB and R routines are available as Supplementary Files.     

Temporal order judgment criteria are affected by synchrony judgment sensitivity

Synchrony perception for audio-visual stimulus pairs is typically studied by using temporal order judgment (TOJ) or synchrony judgment (SJ) tasks. Research has shown that estimates of the point of subjective simultaneity (PSS) obtained using these two methods do not necessarily correspond. Here, we investigate the hypothesis that the PSS estimate obtained in a TOJ task is shifted in the direction of the most sensitive part of the synchrony judgment curve, as obtained in an SJ task. The results confirm that criterion shifts in the TOJ task move the PSS toward regions of the audio-visual temporal interval continuum where discriminations are most sensitive.     

Estimation of psychometric functions from adaptive tracking procedures.

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Estimation of psychometric functions from adaptive tracking procedures

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from-trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to -target-level-for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated-from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

The psychometric function: I. Fitting, sampling, and goodness of fit.

The psychometric function relates an observer's performance to an independent variable, usually some physical quantity of a stimulus in a psychophysical task. This paper, together with its companion paper (Wichmann & Hill, 2001), describes an integrated approach to (1) fitting psychometric functions, (2) assessing the goodness of fit, and (3) providing confidence intervals for the function's parameters and other estimates derived from them, for the purposes of hypothesis testing. The present paper deals with the first two topics, describing a constrained maximum-likelihood method of parameter estimation and developing several goodness-of-fit tests. Using Monte Carlo simulations, we deal with two specific difficulties that arise when fitting functions to psychophysical data. First, we note that human observers are prone to stimulus-independent errors (or lapses). We show that failure to account for this can lead to serious biases in estimates of the psychometric function's parameters and illustrate how the problem may be overcome. Second, we note that psychophysical data sets are usually rather small by the standards required by most of the commonly applied statistical tests. We demonstrate the potential errors of applying traditional chi2 methods to psychophysical data and advocate use of Monte Carlo resampling techniques that do not rely on asymptotic theory. We have made available the software to implement our methods.     

Psychophysics with children: Investigating the effects of attentional lapses on threshold estimates

When assessing the perceptual abilities of children, researchers tend to use psychophysical techniques designed for use with adults. However, children’s poorer attentiveness might bias the threshold estimates obtained by these methods. Here, we obtained speed discrimination threshold estimates in 6- to 7-year-old children in UK Key Stage 1 (KS1), 7- to 9-year-old children in Key Stage 2 (KS2), and adults using three psychophysical procedures: QUEST, a 1-up 2-down Levitt staircase, and Method of Constant Stimuli (MCS). We estimated inattentiveness using responses to “easy” catch trials. As expected, children had higher threshold estimates and made more errors on catch trials than adults. Lower threshold estimates were obtained from psychometric functions fit to the data in the QUEST condition than the MCS and Levitt staircases, and the threshold estimates obtained when fitting a psychometric function to the QUEST data were also lower than when using the QUEST mode. This suggests that threshold estimates cannot be compared directly across methods. Differences between the procedures did not vary significantly with age group. Simulations indicated that inattentiveness biased threshold estimates particularly when threshold estimates were computed as the QUEST mode or the average of staircase reversals. In contrast, thresholds estimated by post-hoc psychometric function fitting were less biased by attentional lapses. Our results suggest that some psychophysical methods are more robust to attentiveness, which has important implications for assessing the perception of children and clinical groups.

     

Measuring, estimating, and understanding the psychometric function: a commentary.

The psychometric function, relating the subject's response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue of Perception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to the d' function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Comparing adaptive procedures for estimating the psychometric function for an auditory gap detection task

A subject’s sensitivity to a stimulus variation can be studied by estimating the psychometric function. Generally speaking, three parameters of the psychometric function are of interest: the performance threshold, the slope of the function, and the rate at which attention lapses occur. In the present study, three psychophysical procedures were used to estimate the three-parameter psychometric function for an auditory gap detection task. These were an up–down staircase (up–down) procedure, an entropy-based Bayesian (entropy) procedure, and an updated maximum-likelihood (UML) procedure. Data collected from four young, normal-hearing listeners showed that while all three procedures provided similar estimates of the threshold parameter, the up–down procedure performed slightly better in estimating the slope and lapse rate for 200 trials of data collection. When the lapse rate was increased by mixing in random responses for the three adaptive procedures, the larger lapse rate was especially detrimental to the efficiency of the up–down procedure, and the UML procedure provided better estimates of the threshold and slope than did the other two procedures.     

Measuring, estimating, and understanding the psychometric function: A commentary

The psychometric function, relating the subject’s response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue ofPerception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to thed’ function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

The psychometric function: I. Fitting, sampling, and goodness of fit

The psychometric function relates an observer’s performance to an independent variable, usually some physical quantity of a stimulus in a psychophysical task. This paper, together with its companion paper (Wichmann & Hill, 2001), describes an integrated approach to (1) fitting psychometric functions, (2) assessing the goodness of fit, and (3) providing confidence intervals for the function’s parameters and other estimates derived from them, for the purposes of hypothesis testing. The present paper deals with the first two topics, describing a constrained maximum-likelihood method of parameter estimation and developing several goodness-of-fit tests. Using Monte Carlo simulations, we deal with two specific difficulties that arise when fitting functions to psychophysical data. First, we note that human observers are prone to stimulus-independent errors (orlapses). We show that failure to account for this can lead to serious biases in estimates of the psychometric function’s parameters and illustrate how the problem may be overcome. Second, we note that psychophysical data sets are usually rather small by the standards required by most of the commonly applied statistical tests. We demonstrate the potential errors of applying traditionalX ² methods to psychophysical data and advocate use of Monte Carlo resampling techniques that do not rely on asymptotic theory. We have made available the software to implement our methods.     

Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type

When an audio-visual event is perceived in the natural environment, a physical delay will always occur between the arrival of the leading visual component and that of the trailing auditory component. This natural timing relationship suggests that the point of subjective simultaneity (PSS) should occur at an auditory delay greater than or equal to 0 msec. A review of the literature suggests that PSS estimates derived from a temporal order judgment (TOJ) task differ from those derived from a synchrony judgment (SJ) task, with (unnatural) auditory-leading PSS values reported mainly for the TOJ task. We report data from two stimulus types that differed in terms of complexity--namely, (1) a flash and a click and (2) a bouncing ball and an impact sound. The same participants judged the temporal order and synchrony of both stimulus types, using three experimental methods: (1) a TOJ task with two response categories ("audio first" or "video first"), (2) an SJ task with two response categories ("synchronous" or "asynchronous"; SJ2), and (3) an SJ task with three response categories ("audio first," "synchronous," or "video first"; SJ3). Both stimulus types produced correlated PSS estimates with the SJ tasks, but the estimates from the TOJ procedure were uncorrelated with those obtained from the SJ tasks. These results suggest that the SJ task should be preferred over the TOJ task when the primary interest is i n perceived audio-visualsynchrony.     

Evaluation of maximum-likelihood estimators in nonintensive auditory psychophysics

This is a brief report on the use of maximum-likelihood (ML) estimators in auditory psychophysics. Slope parameters of psychometric functions are characterized for three nonintensive auditory tasks: forced-choice discrimination of interaural time differences (ΔITD), frequency (Δf), and duration (Δt). Using these slope estimates, the ML method is implemented and threshold estimates are obtained for the three tasks and compared with previously published data. ΔITD thresholds were additionally measured for human observers by means of two other psychophysical procedures: the constant-stimuli (CS) and the 2-down 1-up methods (Wetherill & Levitt, 1965). Standard errors were smallest for the ML method. Finally, simulations showed ML estimates to be more efficient than the CS andk-down 1-up procedures fork=2 to 5. For up—down procedures, efficiency was highest fork values of 3 and 4. The entropy (Shannon, 1949) of ML estimates was the smallest of the simulated procedures, but poorer than ideal by 0.5 bits.     

A comparison of adaptive psychometric procedures based on the theory of optimal experiments and bayesian techniques: implications for cochlear implant testing

Numerous previous studies have focused on the development of quick and efficient adaptive psychometric procedures. In psychophysics, there is often a model of the psychometric function supported by previous studies for the task of interest. The theory of optimal experiments provides a framework for utilizing a model of the process to develop quick and efficient sequential-testing strategies for estimating model parameters, making it appropriate for developing adaptive psychophysical-testing methods. In this study, we investigated the application of sequential parameter search strategies based on the theory of optimal experiments and Bayesian adaptive procedures for measuring psychophysical variables. The results presented in this article suggest that more sophisticated psychometric procedures can expedite the measurement of psychophysical variables. Such techniques for quickly collecting psychophysical data may be particularly useful in cochlear implant research, where a large set of psychophysical variables are useful for characterizing the performance of an implanted device. It is to be hoped that further development of these techniques will make psychophysical measurements available to clinicians for tuning and optimizing the speech processors of individual cochlear implant patients.     

Intensity resolution and subjective magnitude in psychophysical scaling

Several successful theories of psychophysical judgment imply that exponents of power functions in scaling tasks should covary with measures of intensity resolution such asd’ in the same tasks, whereas the prevailing metatheory of ideal psychophysical scaling asserts the independence of the two. In a direct test of this relationship, three prominent psychophysical scaling paradigms were studied: category judgment without an identification function, absolute magnitude estimation, and cross-modality matching with light intensity as the response continuum. Separate groups of subjects for each scaling paradigm made repeated judgments of the loudnesses of the pure tones that constituted each of two stimulus ensembles. The narrow- and wide-range ensembles shared six identical stimulus intensities in the middle of each set. Intensity resolution, as measured byd’-like distances, of these physically identical stimuli was significantly worse for the wide-range set for all three methods. Exponents of power functions fitted to geometric mean responses, and in magnitude estimation and cross-modality matching the geometric mean responses themselves, were also significantly smaller in the wide-range condition. The variation of power function exponents, and of psychophysical scale values, for stimulus intensities that were identical in the two stimulus sets with the intensities of other members of the ensembles is inconsistent with the metatheory on which modern psychophysical scaling practice is based, although it is consistent with other useful approaches to measurement of psychological magnitudes.     

A computer program for Spearman-K?rber and probit analysis of psychometric function data.

PMETRIC is a computer program for the analysis of observed psychometric functions. It can estimate the parameters of these functions, using either probit analysis (a parametric technique) or the Spearman-K?rber method (a nonparametric one). For probit analysis, either a maximum likelihood or a minimum chi 2 criterion may be used for parameter estimation. In addition, standard errors of parameter estimates can be estimated via bootstrapping. The program can be used to analyze data obtained from either yes-no or m-alternative forced-choice tasks. To facilitate the use of PMETRIC in simulation work, an associated program, PMETGEN, is provided for the generation of simulated psychometric function data. Use of PMETRIC is illustrated with data from a duration discrimination task.     

Use of internal consistency coefficients for estimating reliability of experimental task scores

Reliabilities of scores for experimental tasks are likely to differ from one study to another to the extent that the task stimuli change, the number of trials varies, the type of individuals taking the task changes, the administration conditions are altered, or the focal task variable differs. Given that reliabilities vary as a function of the design of these tasks and the characteristics of the individuals taking them, making inferences about the reliability of scores in an ongoing study based on reliability estimates from prior studies is precarious. Thus, it would be advantageous to estimate reliability based on data from the ongoing study. We argue that internal consistency estimates of reliability are underutilized for experimental task data and in many applications could provide this information using a single administration of a task. We discuss different methods for computing internal consistency estimates with a generalized coefficient alpha and the conditions under which these estimates are accurate. We illustrate use of these coefficients using data for three different tasks.     

Time perception with and without a concurrent nontemporal task

Prospective time estimates were obtained from human subjects for stimulus durations ranging from 2 to 23 sec. Presence and absence of a concurrent nontemporal task was manipulated within subjects in three experiments. In addition, location of the task within temporal reproduction trials and psychophysical method were varied between groups in Experiments 2 and 3, respectively. For long-duration stimuli, the results of all three experiments conformed to results in the literature, showing a decrease in perceived duration under concurrent task conditions, in accord with attentional resource allocation models of timing. The effects of task location and psychophysical method on time estimates were also compatible with this analysis. However, psychophysical functions obtained under task conditions were fit well by power functions, an outcome that would not be anticipated on the basis of attention theory. The slopes of the functions under no-task conditions were steeper than those under task conditions. The data support the perceptual hypothesis that different sources of sensory input mediate timing under task and no-task conditions.     

A computer program for Spearman-Kärber and probit analysis of psychometric function data

PMETRIC is a computer program for the analysis of observed psychometric functions. It can estimate the parameters of these functions, using either probit analysis (a parametric technique) or the Spearman-Kärber method (a nonparametric one). For probit analysis, either a maximum likelihood or a minimum χ² criterion may be used for parameter estimation. In addition, standard errors of parameter estimates can be estimated via bootstrapping. The program can be used to analyze data obtained from either yes-no orm-alternative forced-choice tasks. To facilitate the use of PMETRIC in simulation work, an associated program, PMETGEN, is provided for the generation of simulated psychometric function data. Use of PMETRIC is illustrated with data from a duration discrimination task.     

Task bias and the accuracy of judgment: Setting a baseline for expected group performance

When a judgment task evokes unbiased estimates (i.e. the errors in individual judgments are distributed randomly around the true value), mathematical aggregation of individual estimates, even by a simple arithmetic mean, often will outperform all group members. However, when a task evokes biased estimates, mathematical aggregation does not perform so well. In this study, simulated data were accumulated to specify the expected' accuracy of mathematical aggregation relative to the accuracy of observed judgment of individual group members under varying conditions of task bias. Three types of judgment tasks were employed: (1) single-estimate, holistic tasks, (2) multiple-estimate, ranking tasks, and (3) multi-cue, decomposed tasks. Findings indicated across all task types that a large percentage of judgment-making group estimates formed strictly by computing the arithmetic mean of individual estimates performed better than their most capable members when a judgment task evoked little or no bias, a result particularly pronounced for ranking tasks. When the task was more greatly bias-evoking, a large percentage of parallel groups performed more poorly than average (or median) members, again a pattern more starkly evident for ranking tasks. These results suggest that the extent to which a judgment task evokes bias in a population of prospective group members is an important explanatory variable deserving much greater attention in the study of group performance. For example, an assertion about the efficacy of a particular group intervention based on a reliable demonstration of group performance as accurate as the most capable members may be unfounded when a task evokes no bias, since the baseline standard under such conditions should be much higher. By selecting tasks and populations that jointly produced highly biased estimates, researchers can lower the performance floor enough to detect (with reasonably small samples of groups) experimental effects should they occur.     

Auditory phase and frequency discrimination: a comparison of nine procedures

Two auditory discrimination tasks were thoroughly investigated: discrimination of frequency differences from a sinusoidal signal of 200 Hz and discrimination of differences in relative phase of mixed sinusoids of 200 Hz and 400 Hz. For each task psychometric functions were constructed for three observers, using nine different psychophysical measurement procedures. These procedures included yes-no, two-interval forced-choice, and various fixed- and variable-standard designs that investigators have used in recent years. The data showed wide ranges of apparent sensitivity. For frequency discrimination, models derived from signal detection theory for each psychophysical procedure seem to account for the performance differences. For phase discrimination the models do not account for the data. We conclude that for some discriminative continua the assumptions of signal detection theory are appropriate, and underlying sensitivity may be derived from raw data by appropriate transformations. For other continua the models of signal detection theory are probably inappropriate; we speculate that phase might be discriminable only on the basis of comparison or change and suggest some tests of our hypothesis.