Measuring, estimating, and understanding the psychometric function: A commentary

The psychometric function, relating the subject’s response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue ofPerception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to thed’ function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.     

Measuring Machiavellianism with Mach V: a psychometric investigation

This study re-examines the reliability of the Mach V Scale using two different scoring procedures: Christie's scoring system and an alternative scoring system by Rogers and Semin. Using commonly accepted standards, results for 123 business managers indicate that the Mach V Scale is not a reliable instrument. Regardless of the scoring Scheme or the method of assessment employed, reliability estimates found in this study are well below conventionally accepted criteria and are also well below reliability estimates previously reported in the literature. In addition, the investigation indicates that although the alternative scoring system appears to increase the precision of the Mach V Scale, the effect of this increase is negligible in hypothesis testing.     

Measuring needs with the thematic apperception test: a psychometric study

Three apperception theories that explain how people respond to Thematic Apperception Test cards are proposed: a simple apperception theory, an apperception theory with a dynamic component, and an apperception theory with 2 types of responses. Each theory is translated into an item response theory model and is applied to need for achievement (nAch) data. The analysis indicates that the best fitting model is provided by the apperception theory with 2 types of responses, also referred to as the drop-out apperception theory. The 1st type of response predicted by this theory is determined by the nAch level of the person and the achievement-response-eliciting value of the card; this response is diagnostic for the nAch level of the person. The 2nd type of response is not determined by the 2 aforementioned characteristics and is therefore not diagnostic of the person's nAch level. The results are cross-validated for need for power and need for affiliation.     

The psychometric function: I. Fitting, sampling, and goodness of fit.

The psychometric function relates an observer's performance to an independent variable, usually some physical quantity of a stimulus in a psychophysical task. This paper, together with its companion paper (Wichmann & Hill, 2001), describes an integrated approach to (1) fitting psychometric functions, (2) assessing the goodness of fit, and (3) providing confidence intervals for the function's parameters and other estimates derived from them, for the purposes of hypothesis testing. The present paper deals with the first two topics, describing a constrained maximum-likelihood method of parameter estimation and developing several goodness-of-fit tests. Using Monte Carlo simulations, we deal with two specific difficulties that arise when fitting functions to psychophysical data. First, we note that human observers are prone to stimulus-independent errors (or lapses). We show that failure to account for this can lead to serious biases in estimates of the psychometric function's parameters and illustrate how the problem may be overcome. Second, we note that psychophysical data sets are usually rather small by the standards required by most of the commonly applied statistical tests. We demonstrate the potential errors of applying traditional chi2 methods to psychophysical data and advocate use of Monte Carlo resampling techniques that do not rely on asymptotic theory. We have made available the software to implement our methods.     

Comparing adaptive procedures for estimating the psychometric function for an auditory gap detection task

A subject’s sensitivity to a stimulus variation can be studied by estimating the psychometric function. Generally speaking, three parameters of the psychometric function are of interest: the performance threshold, the slope of the function, and the rate at which attention lapses occur. In the present study, three psychophysical procedures were used to estimate the three-parameter psychometric function for an auditory gap detection task. These were an up–down staircase (up–down) procedure, an entropy-based Bayesian (entropy) procedure, and an updated maximum-likelihood (UML) procedure. Data collected from four young, normal-hearing listeners showed that while all three procedures provided similar estimates of the threshold parameter, the up–down procedure performed slightly better in estimating the slope and lapse rate for 200 trials of data collection. When the lapse rate was increased by mixing in random responses for the three adaptive procedures, the larger lapse rate was especially detrimental to the efficiency of the up–down procedure, and the UML procedure provided better estimates of the threshold and slope than did the other two procedures.     

The psychometric function: I. Fitting, sampling, and goodness of fit

The psychometric function relates an observer’s performance to an independent variable, usually some physical quantity of a stimulus in a psychophysical task. This paper, together with its companion paper (Wichmann & Hill, 2001), describes an integrated approach to (1) fitting psychometric functions, (2) assessing the goodness of fit, and (3) providing confidence intervals for the function’s parameters and other estimates derived from them, for the purposes of hypothesis testing. The present paper deals with the first two topics, describing a constrained maximum-likelihood method of parameter estimation and developing several goodness-of-fit tests. Using Monte Carlo simulations, we deal with two specific difficulties that arise when fitting functions to psychophysical data. First, we note that human observers are prone to stimulus-independent errors (orlapses). We show that failure to account for this can lead to serious biases in estimates of the psychometric function’s parameters and illustrate how the problem may be overcome. Second, we note that psychophysical data sets are usually rather small by the standards required by most of the commonly applied statistical tests. We demonstrate the potential errors of applying traditionalX ² methods to psychophysical data and advocate use of Monte Carlo resampling techniques that do not rely on asymptotic theory. We have made available the software to implement our methods.     

Fitting the psychometric function

A constrained generalized maximum likelihood routine for fitting psychometric functions is proposed, which determines optimum values for the complete parameter set--that is, threshold and slope--as well as for guessing and lapsing probability. The constraints are realized by Bayesian prior distributions for each of these parameters. The fit itself results from maximizing the posterior distribution of the parameter values by a multidimensional simplex method. We present results from extensive Monte Carlo simulations by which we can approximate bias and variability of the estimated parameters of simulated psychometric functions. Furthermore, we have tested the routine with data gathered in real sessions of psychophysical experimenting.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling.

The psychometric function relates an observer's performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper's principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one's choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

On the theoretical error bound for estimating psychometric functions

The theoretical limits to the amount of error, or the Cramer-Rao bounds, were derived for estimating psychometric functions. These theoretical error bounds were compared with the variability of psychometric functions estimated from human as well as computer-simulated observers. For the simulated observers, due to the limited efficiency of the sampling strategies, including the placement of the signals and the distribution of the trials, the variances of the estimated parameters are seven times the theoretical bound for threshold and 22 times that for slope. For the human observers, the variance is 18 times the theoretical bounds for threshold and 80 times that for slope. Therefore, a major portion of the variances (60% for threshold and 73% for slope) for the human observers is associated with factors other than sampling strategies. Further improvement of the accuracy for estimating psychometric functions will depend on not only optimizing the sampling strategy, but also better understanding the various sources of error related to the behavior of human observers.     

The psychometric function: II. Bootstrap-based confidence intervals and sampling

The psychometric function relates an observer’s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper’s principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one’s choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.     

Measuring rail passenger crowding: Scale development and psychometric properties

Research on rail passenger crowding often tacitly subscribes to a measurement of crowding based on density (i.e. physical conditions involving space limitations) and rarely considers the possible role psychological factors may play in measuring this construct. This paper describes the development of an instrument that captures the dimensionality of rail passenger crowding and its relationship to the experience of stress and feelings of exhaustion. The proposed instrument is a 20-item self-rating questionnaire consisting of three sub-scales designed to assess subjective crowding experiences among rail users (n = 525). Findings from the factor analyses generally support the hypothesised three-factor structure of the measurement model (evaluation of the psychosocial aspects of the crowded situation, evaluation of the ambient environment of the crowded situation, and affective reactions to the crowded situation). All sub-scales demonstrate excellent internal consistency and construct validity as well as good convergent and discriminant validity values. The instrument was further tested using structural equation modelling to examine the impact of crowding on commuters’ stress and feelings of exhaustion. With the addition of the “passenger density” variable as an indicator of objective measurement of crowding operating in tandem with the crowding sub-scales, the results reveal that: (1) commuters’ evaluations of the psychosocial aspects of the crowded situation and of its ambient environment, alongside their rating of passenger density, significantly predict affective reactions to the crowded situation; (2) these affective reactions, in turn, significantly predict stress and feelings of exhaustion; and (3) evaluations of the psychosocial aspects of the crowded situation and of its ambient environment as well as passenger density do not directly predict stress and feelings of exhaustion. The link between rail passenger crowding and the negative outcomes therefore does not appear as a simple, direct relationship, but is mediated by affective feelings of crowdedness. Overall, these results provide satisfactory psychometric properties for the proposed instrument and support its use as an assessment tool for measuring crowding experience in the rail setting.     

Converting between measures of slope of the psychometric function.

The psychometric function's slope provides information about the reliability of psychophysical threshold estimates. Furthermore, knowing the slope allows one to compare, across studies, thresholds that were obtained at different performance criterion levels. Unfortunately, the empirical validation of psychometric function slope estimates is hindered by the bewildering variety of slope measures that are in use. The present article provides conversion formulas for the most popular cases, including the logistic, Weibull, Quick, cumulative normal, and hyperbolic tangent functions as analytic representations, in both linear and log coordinates and to different log bases, the practical decilog unit, the empirically based interquartile range measure of slope, and slope in a d' representation of performance.     

Depression and implicit memory: a commentary.

In this invited commentary, we review four studies in which the issue of whether depression affects priming on implicit memory tests was examined. We conclude that a depressive mood does not affect amount of priming on several implicit memory tests under conditions in which marked effects are shown on conscious recollection (explicit memory). The mood congruity effect (depressives remember depression-related words better than controls; controls remember other types of material better than depressives) also largely disappears on perceptual implicit memory tests. We speculate about reasons for discrepancies in the literature, relate the findings to some current theories of individual differences in memory, and suggest some directions for future research.     

Measuring mental toughness in sport: a psychometric examination of the psychological performance inventory-a and its predecessor

Touted as a multidimensional measure of mental toughness in sport, this study explored the psychometric properties of the Psychological Performance Inventory (PPI; Loehr, 1986 ) and its successor the Psychological Performance Inventory-A (PPI-A; Golby, Sheard, & Van Wersch, 2007 ). Confirmatory factor analysis was employed to examine the extent to which data collected with 333 Australian footballers aged between 15 and 18 years (M = 16.88, SD = .71) fitted the a priori measurement models of both inventories. The results did not support the psychometric properties of the PPI both in terms of model fit and internal consistency. Although model fit data for the PPI-A were encouraging, inadequate levels of internal consistency were evidenced. Convergent validity analyses involving measures of achievement goals and global mental toughness generally supported the validity of the PPI and PPI-A subscales. Taken together with previous research (e.g., Middleton et al., 2004 ), caution is urged when considering the use of the PPI as a measure of mental toughness in sport. Although the empirical data were generally supportive of the psychometric properties of the PPI-A, conceptual (e.g., lack of conceptual underpinnings) and methodological (i.e., revalidated a flawed inventory) concerns become important factors when considering the PPI-A as a measure of mental toughness.     

Biological factors and psychometric intelligence: a review.

Results from a synthesis of correlations between psychometric intelligence and two physical traits, head size and body size, are reported. Within-family studies are reviewed for evidence of pleiotropy, the effect of a single genetic factor on two traits. Studies are also reviewed to determine whether prenatal effects bias twin studies, leading to underestimates of genetic influence. An N-weighted mean partial correlation (controlling height) of .10 between intelligence and head size was found. Using a method developed by Van Valen (1974), the correlation of intelligence and brain size was estimated as .29 based on all the intelligence/head-size studies of adults and adolescents, and .44 based on studies measuring intelligence with IQ tests. The N-weighted mean partial correlations (controlling age) of intelligence and height were .18 for children and .22 for adults. The within-family studies indicated that pleiotropy may contribute to the correlation of intelligence with head size and to the correlation of intelligence with body size. Prenatal effects are not an important source of bias in twin studies or for heritability estimates based on them.     

An antisymmetric psychometric function on a logarithmic scale

This very brief report introduces a psychometric function, very suitable for psychophysical data that displays Weber-like behaviour, because it is antisymmetric on a logarithmic scale.     

Measuring the validity and psychometric properties of a short form of the Hypomanic Personality Scale

The Hypomanic Personality Scale (HPS) is used to investigate hypomanic traits and risk for bipolar spectrum disorders; however, the length of the HPS (48 items) may be prohibitive for clinical research and screening purposes. Meads and Bentall (2008) developed a promising 20-item version of the HPS; however, the psychometric properties and validity of the short form have not been thoroughly examined. The present study investigated the construct validity and psychometric properties of the short HPS. A sample of 2713 non-clinically ascertained young adults was used to assess psychometric properties of the short form relative to the original scale. Two non-overlapping subsamples (n = 522; n = 145) were used to investigate the validity of the short HPS using personality and temperament questionnaires and clinical interviews of bipolar psychopathology and diagnoses. The short and original HPS generally had comparable correlations with measures of temperament, personality, impulsivity, borderline personality, grandiosity, psychosocial functioning, and alcohol use, and comparably predicted DSM bipolar and bipolar spectrum diagnoses. Overall, the short HPS was found to be both reliable and valid. However, the short HPS tended to be more strongly correlated with pathological components of hypomanic personality and less strongly correlated with exuberant and potentially adaptive aspects of the construct.     

Masking-level differences and the form of the psychometric function

Functions showing the relation between-the detectability and the energy of a signal were determined for various interaural phase conditions. The empirical relation between d′ and signal energy E is approximately d′=m(E/N₀)^k where m and k are constants for a particular function. The data show that k is fairly constant for a particular O and that m depends upon the interaural condition. That is, the various psychometrie functions for a given O all had the same form, independent of location. Therefore, the magnitude of the MLD=(10/k)log(m_i/m_r), where m_i is the constant for condition i, and m_r corresponds to a reference condition. Consequently, an MLD is relatively independent of the level of performance that is chosen for the determination of that MLD.