Nonparametric goodness-of-fit tests for the rasch model

A Monte Carlo algorithm realizing a family of nonparametric tests for the Rasch model is introduced which are conditional on the item and subject marginals. The algorithm is based on random changes of elements of data matrices without changing the marginals; most powerful tests against all alternative hypotheses are given for which a monotone characteristic may be computed from the data matrix; alternatives may also be composed. Computation times are long, but exactp-values are approximated with the quality of approximation only depending on calculation time, but not on the number of persons. The power and the flexibility of the procedure is demonstrated by means of an empirical example where, among others, indicators for increased item similarities, the existence of subscales, violations of sufficiency of the raw score as well as learning processes were found. Many of the features described are implemented in the program T-Rasch 1.0 by Ponocny and Ponocny-Seliger (1999).The author wishes to thank Alexander Kaba, Birgit Bukasa, and Ulrike Wenninger of Österreichisches Kuratorium für Verkehrssicherheit (Austrian Traffic Safety Board) for allowing a data set to be used for the empirical example, and Elisabeth Ponocny-Seliger and the reviewers for many helpful comments. The menu-driven program T-Rasch 1.0 by Ponocny and Ponocny-Seliger (1999) can be obtained from Assessment Systems Corporation (http: //www.assess.com) or from the authors. (Note that it also performs exact person fit tests.)     

Adaptive psychophysical methods for nonmonotonic psychometric functions

Many psychophysical tasks in current use render nonmonotonic psychometric functions; these include the oddball task, the temporal generalization task, the binary synchrony judgment task, and other forms of the same–different task. Other tasks allow for ternary responses and render three psychometric functions, one of which is also nonmonotonic, like the ternary synchrony judgment task or the unforced choice task. In all of these cases, data are usually collected with the inefficient method of constant stimuli (MOCS), because extant adaptive methods are only applicable when the psychometric function is monotonic. This article develops stimulus placement criteria for adaptive methods designed for use with nonmonotonic psychometric functions or with ternary tasks. The methods are transformations of conventional up–down rules. Simulations under three alternative psychophysical tasks prove the validity of these methods, their superiority to MOCS, and the accuracy with which they recover direct estimates of the parameters determining the psychometric functions, as well as estimates of derived quantities such as the point of subjective equality or the difference limen. Practical recommendations and worked-out examples are provided to illustrate how to use these adaptive methods in empirical research.     

Confidence intervals for the parameters of psychometric functions

A Monte Carlo method for computing the bias and standard deviation of estimates of the parameters of a psychometric function such as the Weibull/Quick is described. The method, based on Efron's parametric bootstrap, can also be used to estimate confidence intervals for these parameters. The method's ability to predict bias, standard deviation, and confidence intervals is evaluated in two ways. First, its predictions are compared to the outcomes of Monte Carlo simulations of psychophysical experiments. Second, its predicted confidence intervals were compared with the actual variability of human observers in a psychophysical task. Computer programs implementing the method are available from the author.     

On the form of psychometric functions for taste

Psychometric functions were determined for the discrimination of weak solutions of quinine hydrochloride or of hydrochloric acid from distilled water. The slopes of these functions were compared with those of some functions previously determined for sodium chloride and for sucrose. In general, the functions for QHCl were the least steep and those for NaCl were the most steep, but the differences were not great. Thus, it appears that the psychometric functions for these different taste qualities have substantially the same form, even though absolute sensitivity varies over several orders of magnitude.     

Modeling psychometric functions in R

We demonstrate some procedures in the statistical computing environment R for obtaining maximum likelihood estimates of the parameters of a psychometric function by fitting a generalized nonlinear regression model to the data. A feature for fitting a linear model to the threshold (or other) parameters of several psychometric functions simultaneously provides a powerful tool for testing hypotheses about the data and, potentially, for reducing the number of parameters necessary to describe them. Finally, we illustrate procedures for treating one parameter as a random effect that would permit a simplified approach to modeling stimulus-independent variability due to factors such as lapses or interobserver differences. These tools will facilitate a more comprehensive and explicit approach to the modeling of psychometric data.     

On the theoretical error bound for estimating psychometric functions

The theoretical limits to the amount of error, or the Cramer-Rao bounds, were derived for estimating psychometric functions. These theoretical error bounds were compared with the variability of psychometric functions estimated from human as well as computer-simulated observers. For the simulated observers, due to the limited efficiency of the sampling strategies, including the placement of the signals and the distribution of the trials, the variances of the estimated parameters are seven times the theoretical bound for threshold and 22 times that for slope. For the human observers, the variance is 18 times the theoretical bounds for threshold and 80 times that for slope. Therefore, a major portion of the variances (60% for threshold and 73% for slope) for the human observers is associated with factors other than sampling strategies. Further improvement of the accuracy for estimating psychometric functions will depend on not only optimizing the sampling strategy, but also better understanding the various sources of error related to the behavior of human observers.     

A note on preliminary tests of equality of variances

Preliminary tests of equality of variances used before a test of location are no longer widely recommended by statisticians, although they persist in some textbooks and software packages. The present study extends the findings of previous studies and provides further reasons for discontinuing the use of preliminary tests. The study found Type I error rates of a two‐stage procedure, consisting of a preliminary Levene test on samples of different sizes with unequal variances, followed by either a Student pooled‐variances t test or a Welch separate‐variances t test. Simulations disclosed that the twostage procedure fails to protect the significance level and usually makes the situation worse. Earlier studies have shown that preliminary tests often adversely affect the size of the test, and also that the Welch test is superior to the t test when variances are unequal. The present simulations reveal that changes in Type I error rates are greater when sample sizes are smaller, when the difference in variances is slight rather than extreme, and when the significance level is more stringent. Furthermore, the validity of the Welch test deteriorates if it is used only on those occasions where a preliminary test indicates it is needed. Optimum protection is assured by using a separate‐variances test unconditionally whenever sample sizes are unequal.     

Estimation of psychometric functions from adaptive tracking procedures

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from-trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to -target-level-for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated-from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Spatial-frequency uncertainty and cuing effects on psychometric functions for contrast detection

In the present study, the effects of spatial-frequency uncertainty and cuing on psychometric functions for contrast detection of sinusoidal gratings are examined. For this purpose, psychometric functions were collected from 4 subjects under fixed-frequency, randomized-frequency, and cued-frequency conditions. The experiment was conducted with a temporal two-alternative forced-choice task, and five spatial frequencies in the range of 0.5 and 8.0 c/deg and seven contrast levels for each frequency were used. The results showed that the psychometric functions for the randomized-frequency condition were shallower than those for the fixed-frequency condition, supporting the single-band model for the uncertainty effects (Hübner, 1993a, 1993b). For the cued-frequency condition, the slopes of the functions were not clearly different from those for the randomized condition. These results clearly differ from those of Hübner (1996b), which showed, in the spatial two-alternative forced-choice task, steeper psychometric functions for the randomized-frequency condition than those for the fixed- and cued-frequency conditions, supporting the multiple-band model (Hübner, 1993a, 1993b). The difference suggests that the single-band model applies to the uncertainty effects in the temporal forced-choice task, whereas the multiple-band model does so in the spatial forced-choice task.     

Temporal order psychometric functions based on confidence-rating data

Psychophysical tasks involving confidence judgments allow the simultaneous generation of a family of psychometric functions. Sternberg, Knoll, and Mallows (1975) have demonstrated the power of the multiple-function approach in evaluating models concerned with specifying the source of errors in judgments of simultaneity and temporal order. In the present paper, data from a temporal order task requiring confidence ratings are examined, and a number of models for successiveness and order judgments evaluated.

     

Estimation of psychometric functions from adaptive tracking procedures.

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Slope bias of psychometric functions derived from adaptive data

Several investigators have fit psychometric functions to data from adaptive procedures for threshold estimation. Although the threshold estimates are in general quite correct, one encounters a slope bias that has not been explained up to now. The present paper demonstrates slope bias for parametric and nonparametric maximum-likelihood fits and for Spearman-Kärber analysis of adaptive data. The examples include staircase and stochastic approximation procedures. The paper then presents an explanation of slope bias based on serial data dependency in adaptive procedures. Data dependency is first illustrated with simple two-trial examples and then extended to realistic adaptive procedures. Finally, the paper presents an adaptive staircase procedure designed to measure threshold and slope directly. In contrast to classical adaptive threshold-only procedures, this procedure varies both a threshold and a spread parameter in response to double trials.     

Different practice effects for males and females by psychometric and chronometric mental-rotation tests

Gender differences in the psychometric mental-rotation test are usually larger than in the chronometric version. In both tests, practice effects appear for males and females. In this study, 104 participants (54 females, 50 males, age: 21.72 years) completed both tests in counterbalanced order. In the chronometric test, only males reacted and rotated significantly faster after the practice with the psychometric test. A strong practice effect independently of gender was found in the psychometric test and a gender difference in accuracy in favour of males. Males reported more confidence and females rated the perceived pressure of the time limit of the psychometric test higher than males. Consequently, differences in confidence after the practice could partly explain the gender differences in the improvements of reaction time and rotational speed. Practice from one mental-rotation test on the performance in another seems to be dependent of participants’ gender and the type of the test.     

Slope bias of psychometric functions derived from adaptive data.

Several investigators have fit psychometric functions to data from adaptive procedures for threshold estimation. Although the threshold estimates are in general quite correct, one encounters a slope bias that has not been explained up to now. The present paper demonstrates slope bias for parametric and nonparametric maximum-likelihood fits and for Spearman-K?rber analysis of adaptive data. The examples include staircase and stochastic approximation procedures. The paper then presents an explanation of slope bias based on serial data dependency in adaptive procedures. Data dependency is first illustrated with simple two-trial examples and then extended to realistic adaptive procedures. Finally, the paper presents an adaptive staircase procedure designed to measure threshold and slope directly. In contrast to classical adaptive threshold-only procedures, this procedure varies both a threshold and a spread parameter in response to double trials.     

Observers can voluntarily shift their psychometric functions without losing sensitivity

Psychometric sensory discrimination functions are usually modeled by cumulative Gaussian functions with just two parameters, their central tendency (μ) and their slope (1/σ). These correspond to Fechner's "constant" and "variable" errors, respectively. Fechner pointed out that even the constant error could vary over space and time and could masquerade as variable error. We wondered whether observers could deliberately introduce a constant error into their performance without loss of precision. In three-dot vernier and bisection tasks with the method of single stimuli, observers were instructed to favour one of the two responses when unsure of their answer. The slope of the resulting psychometric function was not significantly changed, despite a significant change in central tendency. Similar results were obtained when altered feedback was used to induce bias. We inferred that observers can adopt artificial response criteria without any significant increase in criterion fluctuation. These findings have implications for some studies that have measured perceptual "illusions" by shifts in the psychometric functions of sophisticated observers.     

A comparison of six methods to estimate thresholds from psychometric functions

There are many ways in which to estimate thresholds from psychometric functions. However, almost nothing is known about the relationships between these estimates. In the present experiment, Monte Carlo techniques were used to compare psychometric thresholds obtained using six methods. Three psychometric functions were simulated using Naka-Rushton and Weibull functions and a probit/logit function combination. Thresholds were estimated using probit, logit, and normit analyses and least-squares regressions of untransformed orz-score and logit-transformed probabilities versus stimulus strength. Histograms were derived from 100 thresholds using each of the six methods for various sampling strategies of each psychometric function. Thresholds from probit, logit, and normit analyses were remarkably similar. Thresholds fromz-score- and logit-transformed regressions were more variable, and linear regression produced biased threshold estimates under some circumstances. Considering the similarity of thresholds, the speed of computation, and the ease of implementation, logit and normit analyses provide effective alternatives to the current “gold standard”—probit analysis—for the estimation of psychometric thresholds.