Sampling plans for fitting the psychometric function

Research on estimation of a psychometric function psi has usually focused on comparing alternative algorithms to apply to the data, rarely addressing how best to gather the data themselves (i.e., what sampling plan best deploys the affordable number of trials). Simulation methods were used here to assess the performance of several sampling plans in yes-no and forced-choice tasks, including the QUEST method and several variants of up-down staircases and of the method of constant stimuli (MOCS). We also assessed the efficacy of four parameter estimation methods. Performance comparisons were based on analyses of usability (i.e., the percentage of times that a plan yields usable data for the estimation of all the parameters of psi) and of the resultant distributions of parameter estimates. Maximum likelihood turned out to be the best parameter estimation method. As for sampling plans, QUEST never exceeded 80% usability even when 1000 trials were administered and rendered accurate estimates of threshold but misestimated the remaining parameters. MOCS and up-down staircases yielded similar and acceptable usability (above 95% with 400-500 trials) and, although neither type of plan allowed estimating all parameters with optimal precision, each type appeared well suited to estimating a distinct subset of parameters. An analysis of the causes of this differential suitability allowed designing alternative sampling plans (all based on up-down staircases) for yes-no and forced-choice tasks. These alternative plans rendered near optimal distributions of estimates for all parameters. The results just described apply when the fitted psi has the same mathematical form as the actual psi generating the data; in case of form mismatch, all parameters except threshold were generally misestimated but the relative performance of all the sampling plans remained identical. Detailed practical recommendations are given.     

Statistical properties of staircase estimates

The bias and variability of staircase estimators were studied by means of repeated computer simulations of staircase runs. Both forced-choice and yes-no staircases were simulated. The influence of the shape of the psychometric function, the location and spacing of the stimuli, the number of trials in a run, and the method of deriving the estimate from the data are discussed. The forced-choice staircase is compared to the yes-no staircase, and the limitations of the simulation procedure are outlined.     

Estimation of psychometric functions from adaptive tracking procedures

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from-trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to -target-level-for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated-from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Estimation of psychometric functions from adaptive tracking procedures.

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

A computer program for Spearman-K?rber and probit analysis of psychometric function data.

PMETRIC is a computer program for the analysis of observed psychometric functions. It can estimate the parameters of these functions, using either probit analysis (a parametric technique) or the Spearman-K?rber method (a nonparametric one). For probit analysis, either a maximum likelihood or a minimum chi 2 criterion may be used for parameter estimation. In addition, standard errors of parameter estimates can be estimated via bootstrapping. The program can be used to analyze data obtained from either yes-no or m-alternative forced-choice tasks. To facilitate the use of PMETRIC in simulation work, an associated program, PMETGEN, is provided for the generation of simulated psychometric function data. Use of PMETRIC is illustrated with data from a duration discrimination task.     

Simple adaptive testing with the weighted up-down method

This paper proposes a method for adaptive testing that is less complicated than the commonly used transformed up-down methods (1 up 2 down, 1 up 3 down, etc.). In addition, the weighted up-down method can converge to any desired point of the psychometric function. The rule is very simple: Each correct response leads to a decrease in signal level, each incorrect response to an increase. The only difference from the simple up-down method (1 up 1 down) is that the steps upward and the steps downward are of a different size. The straightforward construction of the novel procedure pays off in efficiency and stability: A Monte Carlo simulation reveals a definite advantage, though small, of the weighted up-down method over the 1-up-2-down rule.     

A computer program for Spearman-Kärber and probit analysis of psychometric function data

PMETRIC is a computer program for the analysis of observed psychometric functions. It can estimate the parameters of these functions, using either probit analysis (a parametric technique) or the Spearman-Kärber method (a nonparametric one). For probit analysis, either a maximum likelihood or a minimum χ² criterion may be used for parameter estimation. In addition, standard errors of parameter estimates can be estimated via bootstrapping. The program can be used to analyze data obtained from either yes-no orm-alternative forced-choice tasks. To facilitate the use of PMETRIC in simulation work, an associated program, PMETGEN, is provided for the generation of simulated psychometric function data. Use of PMETRIC is illustrated with data from a duration discrimination task.     

适用于多维迫选测验的IRT计分模型

迫选(forced-choice,FC)测验由于可以控制传统李克特方法带来的反应偏差,被广泛应用于非认知测验中,而迫选测验的传统计分方式会产生自模式数据,这种数据由于不适合于个体间的比较,一直备受批评。近年来,多种迫选IRT模型的发展使研究者能够从迫选测验中获得接近常模性的数据,再次引起了研究者与实践人员对迫选IRT模型的兴趣。首先,依据所采纳的决策模型和题目反应模型对6种较为主流的迫选IRT模型进行分类和介绍。然后,从模型构建思路、参数估计方法两个角度对各模型进行比较与总结。其次,从参数不变性检验、计算机化自适应测验(computerized adaptive testing, CAT)和效度研究3个应用研究方面进行述评。最后提出未来研究可以在模型拓展、参数不变性检验、迫选CAT测验和效度研究4个方向深入。     

Additional rules for the transformed up-down method in psychophysics

In a classic paper, Levitt (1971) described an adaptive procedure for estimating points on the psychometric function known as thetransformed up-down method. Levitt discussed the assumptions of the method and presented a brief table with simple rules that converge to a few different points on the psychometric function. Levitt’s original table contains only the simplest rules, and sparsely covers the range of the psychometric function. This paper provides a table with previously unpublished rules which cover the range of the psychometric function at 5% intervals. There is a brief review of the major issues in adaptive testing. Technical issues such as the mean length and logical construction of the new rules are discussed.     

Spatial-frequency uncertainty and cuing effects on psychometric functions for contrast detection

In the present study, the effects of spatial-frequency uncertainty and cuing on psychometric functions for contrast detection of sinusoidal gratings are examined. For this purpose, psychometric functions were collected from 4 subjects under fixed-frequency, randomized-frequency, and cued-frequency conditions. The experiment was conducted with a temporal two-alternative forced-choice task, and five spatial frequencies in the range of 0.5 and 8.0 c/deg and seven contrast levels for each frequency were used. The results showed that the psychometric functions for the randomized-frequency condition were shallower than those for the fixed-frequency condition, supporting the single-band model for the uncertainty effects (Hübner, 1993a, 1993b). For the cued-frequency condition, the slopes of the functions were not clearly different from those for the randomized condition. These results clearly differ from those of Hübner (1996b), which showed, in the spatial two-alternative forced-choice task, steeper psychometric functions for the randomized-frequency condition than those for the fixed- and cued-frequency conditions, supporting the multiple-band model (Hübner, 1993a, 1993b). The difference suggests that the single-band model applies to the uncertainty effects in the temporal forced-choice task, whereas the multiple-band model does so in the spatial forced-choice task.     

Psychophysics with children: Investigating the effects of attentional lapses on threshold estimates

When assessing the perceptual abilities of children, researchers tend to use psychophysical techniques designed for use with adults. However, children’s poorer attentiveness might bias the threshold estimates obtained by these methods. Here, we obtained speed discrimination threshold estimates in 6- to 7-year-old children in UK Key Stage 1 (KS1), 7- to 9-year-old children in Key Stage 2 (KS2), and adults using three psychophysical procedures: QUEST, a 1-up 2-down Levitt staircase, and Method of Constant Stimuli (MCS). We estimated inattentiveness using responses to “easy” catch trials. As expected, children had higher threshold estimates and made more errors on catch trials than adults. Lower threshold estimates were obtained from psychometric functions fit to the data in the QUEST condition than the MCS and Levitt staircases, and the threshold estimates obtained when fitting a psychometric function to the QUEST data were also lower than when using the QUEST mode. This suggests that threshold estimates cannot be compared directly across methods. Differences between the procedures did not vary significantly with age group. Simulations indicated that inattentiveness biased threshold estimates particularly when threshold estimates were computed as the QUEST mode or the average of staircase reversals. In contrast, thresholds estimated by post-hoc psychometric function fitting were less biased by attentional lapses. Our results suggest that some psychophysical methods are more robust to attentiveness, which has important implications for assessing the perception of children and clinical groups.

     

Measurement of low-signal suppression and other parameters of the forced-choice psychometric function

Empirical forced-choice psychometric functions often depart in shape from those based on the theory of normal sensory-excitation distributions and optimal decision processes. The typical departure is that the detectability of small signals is too low compared to that of high signals. Some possible explanations were reviewed and it was emphasized that the source of the effect is not known with certainty. An extra parameter resembling a threshold was introduced for the measurement of the effect, and a method of moments was proposed for estimating the two parameters of the resulting two-alternative forced-choice psychometric function. The method was tried out with success in the task of detecting a 0.1-sec luminance increment to a steady adapting luminance of a large visual target. There were large amounts of low-signal suppression, indicating the advisability of reviewing some current psychophysical practices.     

Shifts of the psychometric function: Distinguishing bias from perceptual effects

Morgan, Dillenburger, Raphael, and Solomon have shown that observers can use different response strategies when unsure of their answer, and, thus, they can voluntarily shift the location of the psychometric function estimated with the method of single stimuli (MSS; sometimes also referred to as the single-interval, two-alternative method). They wondered whether MSS could distinguish response bias from a true perceptual effect that would also shift the location of the psychometric function. We demonstrate theoretically that the inability to distinguish response bias from perceptual effects is an inherent shortcoming of MSS, although a three-response format including also an “undecided” response option may solve the problem under restrictive assumptions whose validity cannot be tested with MSS data. We also show that a proper two-alternative forced-choice (2AFC) task with the three-response format is free of all these problems so that bias and perceptual effects can easily be separated out. The use of a three-response 2AFC format is essential to eliminate a confound (response bias) in studies of perceptual effects and, hence, to eliminate a threat to the internal validity of research in this area.     

Measuring, estimating, and understanding the psychometric function: a commentary.

The psychometric function, relating the subject's response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue of Perception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to the d' function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Vibrotactile signal detection

The theory of signal detectability was applied to vibrotactile sensitivity in two experiments. The first experiment showed The psychometric function to be satisfactorily linear when a sensitivity index of the d family was plotted against the signal intensity expressed in decibels. The second experiment yielded receiver-operating-characteristic (ROC) curves of a familiar form for the yes-no and rating response methods. Reasonably consistent estimates of sensitivity were obtained in the second experiment from the yes-no. rating, and forced-choice methods. The sensitivity indices examined were d’ and d_e’, based on Gaussian density functions; A, based on Rayleigh density functions; and the distribution-free indices, P(A) and F(C). For each type of index a tendency was observed for the forced-choice value to be lower than the yes-no and rating values.     

Measuring, estimating, and understanding the psychometric function: A commentary

The psychometric function, relating the subject’s response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue ofPerception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to thed’ function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Visual detection of signals in the presence of continuous and pulsed backqrounds

The detectability of intensity increments in the presence of continuous and pulsed backgrounds was investigated using a two-alternative, temporal, forced-choice procedure. Differences were found in intensity-duration reciprocity relations, the form of the Weber function, and the shape of the psychometric function between continuous- and pulsed-detection conditions. In a second study, signals were added either to a steady background (simple detection) or to background plus pedestal (pedestal detection). Two unusual phenomena were noted, namely, “negative masking” and the “pedestal effect.” The interpretation of the results in terms of a simple, Poisson-detection model is discussed.     

Modified test administration using assistive technology: preliminary psychometric findings

This study examined the psychometric properties of test presentation and response formats that were modified to be accessible with the use of assistive technology (AT). First, the stability of psychometric properties was examined in 60 children, ages 6 to 12, with no significant physical or communicative impairments. Population-specific differences were then examined with samples that included 24 children with cerebral palsy and matched control peers. Children were administered standard and modified versions of tests. The type of AT access did not have a statistically significant effect on modified test scores. Measurement stability between the standard and modified versions of quadrant forced-choice format tests was sufficient. The findings support the potential use of AT and accessible procedures for some test instruments in the assessment of children with cerebral palsy.     

Fitting a Thurstonian IRT model to forced-choice data using Mplus

To counter response distortions associated with the use of rating scales (a.k.a. Likert scales), items can be presented in a comparative fashion, so that respondents are asked to rank the items within blocks (forced-choice format). However, classical scoring procedures for these forced-choice designs lead to ipsative data, which presents psychometric challenges that are well described in the literature. Recently, Brown and Maydeu-Olivares (Educational and Psychological Measurement 71: 460–502, 2011a) introduced a model based on Thurstone’s law of comparative judgment, which overcomes the problems of ipsative data. Here, we provide a step-by-step tutorial for coding forced-choice responses, specifying a Thurstonian item response theory model that is appropriate for the design used, assessing the model’s fit, and scoring individuals on psychological attributes. Estimation and scoring is performed using Mplus, and a very straightforward Excel macro is provided that writes full Mplus input files for any forced-choice design. Armed with these tools, using a forced-choice design is now as easy as using ratings.     

Threshold estimation in two-alternative forced-choice (2AFC) tasks: the Spearman-Kärber method

The Spearman-K?rber method can be used to estimate the threshold value or difference limen in two-alternative forced-choice tasks. This method yields a simple estimator for the difference limen and its standard error, so that both can be calculated with a pocket calculator. In contrast to previous estimators, the present approach does not require any assumptions about the shape of the true underlying psychometric function. The performance of this new nonparametric estimator is compared with the standard technique of probit analysis. The Spearman-K?rber method appears to be a valuable addition to the toolbox of psychophysical methods, because it is most accurate for estimating the mean (i.e., absolute and difference thresholds) and dispersion of the psychometric function, although it is not optimal for estimating percentile-based parameters of this function.