The psychometric function: II. Bootstrap-based confidence intervals and sampling

The psychometric function relates an observer’s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper’s principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one’s choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

Confidence intervals for the parameters of psychometric functions

A Monte Carlo method for computing the bias and standard deviation of estimates of the parameters of a psychometric function such as the Weibull/Quick is described. The method, based on Efron's parametric bootstrap, can also be used to estimate confidence intervals for these parameters. The method's ability to predict bias, standard deviation, and confidence intervals is evaluated in two ways. First, its predictions are compared to the outcomes of Monte Carlo simulations of psychophysical experiments. Second, its predicted confidence intervals were compared with the actual variability of human observers in a psychophysical task. Computer programs implementing the method are available from the author.     

The psychometric function: I. Fitting, sampling, and goodness of fit.

The psychometric function relates an observer's performance to an independent variable, usually some physical quantity of a stimulus in a psychophysical task. This paper, together with its companion paper (Wichmann & Hill, 2001), describes an integrated approach to (1) fitting psychometric functions, (2) assessing the goodness of fit, and (3) providing confidence intervals for the function's parameters and other estimates derived from them, for the purposes of hypothesis testing. The present paper deals with the first two topics, describing a constrained maximum-likelihood method of parameter estimation and developing several goodness-of-fit tests. Using Monte Carlo simulations, we deal with two specific difficulties that arise when fitting functions to psychophysical data. First, we note that human observers are prone to stimulus-independent errors (or lapses). We show that failure to account for this can lead to serious biases in estimates of the psychometric function's parameters and illustrate how the problem may be overcome. Second, we note that psychophysical data sets are usually rather small by the standards required by most of the commonly applied statistical tests. We demonstrate the potential errors of applying traditional chi2 methods to psychophysical data and advocate use of Monte Carlo resampling techniques that do not rely on asymptotic theory. We have made available the software to implement our methods.     

The psychometric function: I. Fitting, sampling, and goodness of fit

The psychometric function relates an observer’s performance to an independent variable, usually some physical quantity of a stimulus in a psychophysical task. This paper, together with its companion paper (Wichmann & Hill, 2001), describes an integrated approach to (1) fitting psychometric functions, (2) assessing the goodness of fit, and (3) providing confidence intervals for the function’s parameters and other estimates derived from them, for the purposes of hypothesis testing. The present paper deals with the first two topics, describing a constrained maximum-likelihood method of parameter estimation and developing several goodness-of-fit tests. Using Monte Carlo simulations, we deal with two specific difficulties that arise when fitting functions to psychophysical data. First, we note that human observers are prone to stimulus-independent errors (orlapses). We show that failure to account for this can lead to serious biases in estimates of the psychometric function’s parameters and illustrate how the problem may be overcome. Second, we note that psychophysical data sets are usually rather small by the standards required by most of the commonly applied statistical tests. We demonstrate the potential errors of applying traditionalX ² methods to psychophysical data and advocate use of Monte Carlo resampling techniques that do not rely on asymptotic theory. We have made available the software to implement our methods.     

Graphing within-subjects confidence intervals using SPSS and S-Plus

Within-subjects confidence intervals are often appropriate to report and to display. Loftus and Masson (1994) have reported methods to calculate these, and their use is becoming common. In the present article, procedures for calculating within-subjects confidence intervals in SPSS and S-Plus are presented (an R version is on the accompanying Web site). The procedure in S-Plus allows the user to report the bias corrected and adjusted bootstrap confidence intervals as well as the standard confidence intervals based on traditional methods. The presented code can be easily altered to fit the individual user’s needs.     

Bootstrap standard error and confidence intervals for the correlations corrected for indirect range restriction

The standard Pearson correlation coefficient, r, is a biased estimator of the population correlation coefficient, ρ(XY) , when predictor X and criterion Y are indirectly range-restricted by a third variable Z (or S). Two correction algorithms, Thorndike's (1949) Case III, and Schmidt, Oh, and Le's (2006) Case IV, have been proposed to correct for the bias. However, to our knowledge, the two algorithms did not provide a procedure to estimate the associated standard error and confidence intervals. This paper suggests using the bootstrap procedure as an alternative. Two Monte Carlo simulations were conducted to systematically evaluate the empirical performance of the proposed bootstrap procedure. The results indicated that the bootstrap standard error and confidence intervals were generally accurate across simulation conditions (e.g., selection ratio, sample size). The proposed bootstrap procedure can provide a useful alternative for the estimation of the standard error and confidence intervals for the correlation corrected for indirect range restriction.     

非正态分布下概化理论方差分量变异量估计

方差分量估计是概化理论的必用技术,但受限于抽样,需要对其变异量进行探讨。采用Monte Carlo数据模拟技术,探讨非正态数据分布对四种方法估计概化理论方差分量变异量的影响。结果表明：（1）不同非正态数据分布下,各种估计方法的“性能”表现出差异性;（2）数据分布对方差分量变异量估计有影响,适合于非正态分布数据的方差分量变异量估计方法不一定适合于正态分布数据。     

Bootstrap Confidence Intervals for Ordinary Least Squares Factor Loadings and Correlations in Exploratory Factor Analysis

This article is concerned with using the bootstrap to assign confidence intervals for rotated factor loadings and factor correlations in ordinary least squares exploratory factor analysis. Coverage performances of SE-based intervals, percentile intervals, bias-corrected percentile intervals, bias-corrected accelerated percentile intervals, and hybrid intervals are explored using simulation studies involving different sample sizes, perfect and imperfect models, and normal and elliptical data. The bootstrap confidence intervals are also illustrated using a personality data set of 537 Chinese men. The results suggest that the bootstrap is an effective method for assigning confidence intervals at moderately large sample sizes.     

The effect of exposure duration on visual character identification in single, whole, and partial report

The psychometric function of single-letter identification is typically described as a function of stimulus intensity. However, the effect of stimulus exposure duration on letter identification remains poorly described. This is surprising because the effect of exposure duration has played a central role in modeling performance in whole and partial report (Shibuya & Bundesen, 1988). Therefore, we experimentally investigated visual letter identification as a function of exposure duration. We compared the exponential, the gamma, and the Weibull psychometric functions, all with a temporal offset included, as well as the ex-Gaussian, the log-logistic, and finally the squared-logistic, which is a psychometric function that to our knowledge has not been described before. The log-logistic and the squared-logistic psychometric function fit well to experimental data. Also, we conducted an experiment to test the ability of the psychometric functions to fit single-letter identification data, at different stimulus contrast levels; also here the same psychometric functions prevailed. Finally, after insertion into Bundesen's Theory of Visual Attention (Bundesen, 1990), the same psychometric functions enable closer fits to data from a previous whole and partial report experiment.     

Effect of non-normality on test statistics for one-way independent groups designs

The data obtained from one‐way independent groups designs is typically non‐normal in form and rarely equally variable across treatment populations (i.e. population variances are heterogeneous). Consequently, the classical test statistic that is used to assess statistical significance (i.e. the analysis of variance F test) typically provides invalid results (e.g. too many Type I errors, reduced power). For this reason, there has been considerable interest in finding a test statistic that is appropriate under conditions of non‐normality and variance heterogeneity. Previously recommended procedures for analysing such data include the James test, the Welch test applied either to the usual least squares estimators of central tendency and variability, or the Welch test with robust estimators (i.e. trimmed means and Winsorized variances). A new statistic proposed by Krishnamoorthy, Lu, and Mathew, intended to deal with heterogeneous variances, though not non‐normality, uses a parametric bootstrap procedure. In their investigation of the parametric bootstrap test, the authors examined its operating characteristics under limited conditions and did not compare it to the Welch test based on robust estimators. Thus, we investigated how the parametric bootstrap procedure and a modified parametric bootstrap procedure based on trimmed means perform relative to previously recommended procedures when data are non‐normal and heterogeneous. The results indicated that the tests based on trimmed means offer the best Type I error control and power when variances are unequal and at least some of the distribution shapes are non‐normal.     

Choice As A Function Of Reinforcement Ratios In Delayed Matching-to-sample

Pigeons were studied in two experiments using a delayed matching-to-sample task. In Experiment 1, 4 subjects were exposed to a task in which the proportion of reinforcement associated with matching and nonmatching, and the overall proportion of reinforcement associated with selecting each choice, regardless of the sample stimulus, were varied. Choice was sensitive to both proportions. A least squares regression analysis showed that Wixted's (1989) proportions of reinforcement model closely fit the data from Experiment 1; however, the model failed to make accurate qualitative predictions for some test conditions. In Experiment 2, 4 subjects were exposed to a delayed matching-to-sample task in which the retention intervals and the reduction in delay to reinforcement signaled by the onset of the sample stimulus were independently varied. When the retention interval was short and when the delay-reduction value of the sample stimulus was high, the sample exerted greater control over choice; the control by the overall proportion of reinforcements for selecting each choice stimulus was correspondingly low. Conversely, when the retention interval was long and the delay-reduction value of the sample stimulus was low, the sample exerted relatively less control over choice; control by the overall proportion of reinforcements obtained for selecting each choice stimulus was correspondingly high. A signal detection analysis found that sensitivity to reinforcement varied directly with retention interval. Data were also consistent with misallocation models. No evidence was found to suggest that pigeons ignore the rate at which selecting individual choice stimuli is reinforced, as has been reported in studies with human subjects.     

Constructing second‐order accurate confidence intervals for communalities in factor analysis

In an effort to find accurate alternatives to the usual confidence intervals based on normal approximations, this paper compares four methods of generating second‐order accurate confidence intervals for non‐standardized and standardized communalities in exploratory factor analysis under the normality assumption. The methods to generate the intervals employ, respectively, the Cornish–Fisher expansion and the approximate bootstrap confidence (ABC), and the bootstrap‐t and the bias‐corrected and accelerated bootstrap (BC_a). The former two are analytical and the latter two are numerical. Explicit expressions of the asymptotic bias and skewness of the communality estimators, used in the analytical methods, are derived. A Monte Carlo experiment reveals that the performance of central intervals based on normal approximations is a consequence of imbalance of miscoverage on the left‐ and right‐hand sides. The second‐order accurate intervals do not require symmetry around the point estimates of the usual intervals and achieve better balance, even when the sample size is not large. The behaviours of the second‐order accurate intervals were similar to each other, particularly for large sample sizes, and no method performed consistently better than the others.     

Bootstrap confidence intervals for principal covariates regression

Principal covariate regression (PCOVR) is a method for regressing a set of criterion variables with respect to a set of predictor variables when the latter are many in number and/or collinear. This is done by extracting a limited number of components that simultaneously synthesize the predictor variables and predict the criterion ones. So far, no procedure has been offered for estimating statistical uncertainties of the obtained PCOVR parameter estimates. The present paper shows how this goal can be achieved, conditionally on the model specification, by means of the bootstrap approach. Four strategies for estimating bootstrap confidence intervals are derived and their statistical behaviour in terms of coverage is assessed by means of a simulation experiment. Such strategies are distinguished by the use of the varimax and quartimin procedures and by the use of Procrustes rotations of bootstrap solutions towards the sample solution. In general, the four strategies showed appropriate statistical behaviour, with coverage tending to the desired level for increasing sample sizes. The main exception involved strategies based on the quartimin procedure in cases characterized by complex underlying structures of the components. The appropriateness of the statistical behaviour was higher when the proper number of components were extracted.     

A robust method for the analysis of experiments with ordered treatment levels.

A robust approach for the analysis of experiments with ordered treatment levels is presented as an alternative to existing approaches such as the parametric Abelson-Tukey test for monotone alternatives and the nonparametric Terpstra-Jonckheere test. The method integrates the familiar Spearman rank-order correlation with bootstrap routines to provide magnitude of association measures, p values, and confidence intervals for magnitude of association measures. The advantages of this method relative to five alternative approaches are pointed out.     

基于概化理论的方差分量变异量估计

概化理论广泛应用于心理与教育测量实践中, 方差分量估计是进行概化理论分析的关键。方差分量估计受限于抽样, 需要对其变异量进行探讨。采用蒙特卡洛(Monte Carlo)数据模拟技术, 在正态分布下讨论不同方法对基于概化理论的方差分量变异量估计的影响。结果表明: Jackknife方法在方差分量变异量估计上不足取; 不采取Bootstrap方法的“分而治之”策略, 从总体上看, Traditional方法和有先验信息的MCMC方法在标准误及置信区间这两个变异量估计上优势明显。     

The psychophysics of sugar concentration discrimination and contrast evaluation in bumblebees

The capacity to discriminate between choice options is crucial for a decision-maker to avoid unprofitable options. The physical properties of rewards are presumed to be represented on context-dependent, nonlinear cognitive scales that may systematically influence reward expectation and thus choice behavior. In this study, we investigated the discrimination performance of free-flying bumblebee workers (Bombus impatiens) in a choice between sucrose solutions with different concentrations. We conducted two-alternative free choice experiments on two B. impatiens colonies containing some electronically tagged bumblebees foraging at an array of computer-automated artificial flowers that recorded individual choices. We mimicked natural foraging conditions by allowing uncertainty in the probability of reward delivery while maintaining certainty in reward concentration. We used a Bayesian approach to fit psychometric functions, relating the strength of preference for the higher concentration option to the relative intensity of the presented stimuli. Psychometric analysis was performed on visitation data from individually marked bumblebees and pooled data from unmarked individuals. Bumblebees preferred the more concentrated sugar solutions at high stimulus intensities and showed no preference at low stimulus intensities. The obtained psychometric function is consistent with reward evaluation based on perceived concentration contrast between choices. We found no evidence that bumblebees reduce reward expectations upon experiencing non-rewarded visits. We compare psychometric function parameters between the bumblebee B. impatiens and the flower bat Glossophaga commissarisi and discuss the relevance of psychophysics for pollinator-exerted selection pressures on plants.     

Adaptive psychophysical methods for nonmonotonic psychometric functions

Many psychophysical tasks in current use render nonmonotonic psychometric functions; these include the oddball task, the temporal generalization task, the binary synchrony judgment task, and other forms of the same–different task. Other tasks allow for ternary responses and render three psychometric functions, one of which is also nonmonotonic, like the ternary synchrony judgment task or the unforced choice task. In all of these cases, data are usually collected with the inefficient method of constant stimuli (MOCS), because extant adaptive methods are only applicable when the psychometric function is monotonic. This article develops stimulus placement criteria for adaptive methods designed for use with nonmonotonic psychometric functions or with ternary tasks. The methods are transformations of conventional up–down rules. Simulations under three alternative psychophysical tasks prove the validity of these methods, their superiority to MOCS, and the accuracy with which they recover direct estimates of the parameters determining the psychometric functions, as well as estimates of derived quantities such as the point of subjective equality or the difference limen. Practical recommendations and worked-out examples are provided to illustrate how to use these adaptive methods in empirical research.     

Simplified Estimation and Testing in Unbalanced Repeated Measures Designs

In this paper we propose a simple estimator for unbalanced repeated measures design models where each unit is observed at least once in each cell of the experimental design. The estimator does not require a model of the error covariance structure. Thus, circularity of the error covariance matrix and estimation of correlation parameters and variances are not necessary. Together with a weak assumption about the reason for the varying number of observations, the proposed estimator and its variance estimator are unbiased. As an alternative to confidence intervals based on the normality assumption, a bias-corrected and accelerated bootstrap technique is considered. We also propose the naive percentile bootstrap for Wald-type tests where the standard Wald test may break down when the number of observations is small relative to the number of parameters to be estimated. In a simulation study we illustrate the properties of the estimator and the bootstrap techniques to calculate confidence intervals and conduct hypothesis tests in small and large samples under normality and non-normality of the errors. The results imply that the simple estimator is only slightly less efficient than an estimator that correctly assumes a block structure of the error correlation matrix, a special case of which is an equi-correlation matrix. Application of the estimator and the bootstrap technique is illustrated using data from a task switch experiment based on an experimental within design with 32 cells and 33 participants.

     

Bias-corrected estimation of the Rudas–Clogg–Lindsay mixture index of fit

Rudas, Clogg, and Lindsay (1994, J. R Stat Soc. Ser. B, 56, 623) introduced the so-called mixture index of fit, also known as pi-star (π*), for quantifying the goodness of fit of a model. It is the lowest proportion of ‘contamination’ which, if removed from the population or from the sample, makes the fit of the model perfect. The mixture index of fit has been widely used in psychometric studies. We show that the asymptotic confidence limits proposed by Rudas et al. (1994, J. R Stat Soc. Ser. B, 56, 623) as well as the jackknife confidence interval by Dayton ( 2003 , Br. J. Math. Stat. Psychol., 56, 1) perform poorly, and propose a new bias-corrected point estimate, a bootstrap test and confidence limits for pi-star. The proposed confidence limits have coverage probability much closer to the nominal level than the other methods do. We illustrate the usefulness of the proposed method in practice by presenting some practical applications to log-linear models for contingency tables.     

Standard errors and confidence intervals for correlations corrected for indirect range restriction: A simulation study comparing analytic and bootstrap methods

A frequent topic of psychological research is the estimation of the correlation between two variables from a sample that underwent a selection process based on a third variable. Due to indirect range restriction, the sample correlation is a biased estimator of the population correlation, and a correction formula is used. In the past, bootstrap standard error and confidence intervals for the corrected correlations were examined with normal data. The present study proposes a large-sample estimate (an analytic method) for the standard error, and a corresponding confidence interval for the corrected correlation. Monte Carlo simulation studies involving both normal and non-normal data were conducted to examine the empirical performance of the bootstrap and analytic methods. Results indicated that with both normal and non-normal data, the bootstrap standard error and confidence interval were generally accurate across simulation conditions (restricted sample size, selection ratio, and population correlations) and outperformed estimates of the analytic method. However, with certain combinations of distribution type and model conditions, the analytic method has an advantage, offering reasonable estimates of the standard error and confidence interval without resorting to the bootstrap procedure's computer-intensive approach. We provide SAS code for the simulation studies.