Extracting thresholds from noisy psychophysical data.

Psychophysical studies with infants or with patients often are unable to use pilot data, training, or large numbers of trials. To evaluate threshold estimates under these conditions, computer simulations of experiments with small numbers of trials were performed by using psychometric functions based on a model of two types of noise: stimulus-related noise (affecting slope) and extraneous noise (affecting upper asymptote). Threshold estimates were biased and imprecise when extraneous noise was high, as were the estimates of extraneous noise. Strategies were developed for rejecting data sets as too noisy for unbiased and precise threshold estimation; these strategies were most successful when extraneous noise was low for most of the data sets. An analysis of 1,026 data sets from visual function tests of infants and toddlers showed that extraneous noise is often considerable, that experimental paradigms can be developed that minimize extraneous noise, and that data analysis that does not consider the effects of extraneous noise may underestimate test-retest reliability and overestimate interocular differences.     

Response competition effects insame-different judgments

An account ofsame-different discriminations that is based upon a continuous-flow model of visual information processing (C. W. Eriksen & Schultz, 1979) and response competition and inhibition between the responses by which the subject signifies his judgment is presented. We show that a response signifyingsame will on the average be executed faster due to less priming or incipient activation of the competing response,different. In the experiment, the subjects matched letters on the basis of physical identity. The degree of priming ofdifferent responses on same trials and ofsame responses ondifferent trials was manipulated by an extraneous noise letter placed in the display. Latency for judgments onsame trials increased as the feature overlap of noise and target letters decreased. Latencies were shorter ondifferent trials when the noise letter was dissimilar to either target letter than when the noise letter was the same as one of the targets. These results were consistent with the response-competition interpretation.     

Characterizing perceptual learning with external noise

Performance in perceptual tasks often improves with practice. This effect is known as ‘perceptual learning,’ and it has been the source of a great deal of interest and debate over the course of the last century. Here, we consider the effects of perceptual learning within the context of signal detection theory. According to signal detection theory, the improvements that take place with perceptual learning can be due to increases in internal signal strength or decreases in internal noise. We used a combination of psychophysical techniques (external noise masking and double-pass response consistency) that involve corrupting stimuli with externally added noise to discriminate between the effects of changes in signal and noise as observers learned to identify sets of unfamiliar visual patterns. Although practice reduced thresholds by as much as a factor of 14, internal noise remained virtually fixed throughout training, indicating learning served to predominantly increase the strength of the internal signal. We further examined the specific nature of the changes that took place in signal strength by correlating the externally added noise with observer’s decisions across trials (response classification). This technique allowed us to visualize some of the changes that took place in the linear templates used by the observers as learning occurred, as well as test the predictions of a linear template-matching model. Taken together, the results of our experiments offer important new theoretical constraints on models of perceptual learning.     

Psychophysics with children: Investigating the effects of attentional lapses on threshold estimates

When assessing the perceptual abilities of children, researchers tend to use psychophysical techniques designed for use with adults. However, children’s poorer attentiveness might bias the threshold estimates obtained by these methods. Here, we obtained speed discrimination threshold estimates in 6- to 7-year-old children in UK Key Stage 1 (KS1), 7- to 9-year-old children in Key Stage 2 (KS2), and adults using three psychophysical procedures: QUEST, a 1-up 2-down Levitt staircase, and Method of Constant Stimuli (MCS). We estimated inattentiveness using responses to “easy” catch trials. As expected, children had higher threshold estimates and made more errors on catch trials than adults. Lower threshold estimates were obtained from psychometric functions fit to the data in the QUEST condition than the MCS and Levitt staircases, and the threshold estimates obtained when fitting a psychometric function to the QUEST data were also lower than when using the QUEST mode. This suggests that threshold estimates cannot be compared directly across methods. Differences between the procedures did not vary significantly with age group. Simulations indicated that inattentiveness biased threshold estimates particularly when threshold estimates were computed as the QUEST mode or the average of staircase reversals. In contrast, thresholds estimated by post-hoc psychometric function fitting were less biased by attentional lapses. Our results suggest that some psychophysical methods are more robust to attentiveness, which has important implications for assessing the perception of children and clinical groups.

     

Applications of multivariate visualization to behavioral sciences

The complexity of psychological science often requires the collection and analysis of multidimensional data. Such data bring about a corresponding cognitive load that has led scientists to develop techniques of scientific visualization to ease the burden. This paper provides an introduction to scientific visualization techniques, a framework for understanding those techniques, and an assessment of the suitability of this approach for psychology. The framework employed builds on the notion of balancingnoise andsmooth in statistical analysis.     

Noise-robust fixation detection in eye movement data: Identification by two-means clustering (I2MC)

Eye-tracking research in infants and older children has gained a lot of momentum over the last decades. Although eye-tracking research in these participant groups has become easier with the advance of the remote eye-tracker, this often comes at the cost of poorer data quality than in research with well-trained adults (Hessels, Andersson, Hooge, Nyström, & Kemner Infancy, 20, 601–633, 2015; Wass, Forssman, & Leppänen Infancy, 19, 427–460, 2014). Current fixation detection algorithms are not built for data from infants and young children. As a result, some researchers have even turned to hand correction of fixation detections (Saez de Urabain, Johnson, & Smith Behavior Research Methods, 47, 53–72, 2015). Here we introduce a fixation detection algorithm—identification by two-means clustering (I2MC)—built specifically for data across a wide range of noise levels and when periods of data loss may occur. We evaluated the I2MC algorithm against seven state-of-the-art event detection algorithms, and report that the I2MC algorithm’s output is the most robust to high noise and data loss levels. The algorithm is automatic, works offline, and is suitable for eye-tracking data recorded with remote or tower-mounted eye-trackers using static stimuli. In addition to application of the I2MC algorithm in eye-tracking research with infants, school children, and certain patient groups, the I2MC algorithm also may be useful when the noise and data loss levels are markedly different between trials, participants, or time points (e.g., longitudinal research).     

Analysis of methods to determine the latency of online movement adjustments

When studying online movement adjustments, one of the interesting parameters is their latency. We set out to compare three different methods of determining the latency: the threshold, confidence interval, and extrapolation methods. We simulated sets of movements with different movement times and amplitudes of movement adjustments, all with the same known latency. We applied the three different methods in order to determine when the position, velocity, and acceleration of the adjusted movements started to deviate from the values for unperturbed movements. We did so both for averaged data and for the data of individual trials. We evaluated the methods on the basis of their accuracy and precision, and according to whether the latency was influenced by the intensity of the movement adjustment. The extrapolation method applied to average acceleration data gave the most reliable estimates of latency, according to these criteria.     

“Plateau”-related summary statistics are uninformative for comparing working memory models

Performance on visual working memory tasks decreases as more items need to be remembered. Over the past decade, a debate has unfolded between proponents of slot models and slotless models of this phenomenon (Ma, Husain, Bays (Nature Neuroscience 17, 347-356, 2014). Zhang and Luck (Nature 453, (7192), 233-235, 2008) and Anderson, Vogel, and Awh (Attention, Perception, Psychophys 74, (5), 891-910, 2011) noticed that as more items need to be remembered, “memory noise” seems to first increase and then reach a “stable plateau.” They argued that three summary statistics characterizing this plateau are consistent with slot models, but not with slotless models. Here, we assess the validity of their methods. We generated synthetic data both from a leading slot model and from a recent slotless model and quantified model evidence using log Bayes factors. We found that the summary statistics provided at most 0.15 % of the expected model evidence in the raw data. In a model recovery analysis, a total of more than a million trials were required to achieve 99 % correct recovery when models were compared on the basis of summary statistics, whereas fewer than 1,000 trials were sufficient when raw data were used. Therefore, at realistic numbers of trials, plateau-related summary statistics are highly unreliable for model comparison. Applying the same analyses to subject data from Anderson et al. (Attention, Perception, Psychophys 74, (5), 891-910, 2011), we found that the evidence in the summary statistics was at most 0.12 % of the evidence in the raw data and far too weak to warrant any conclusions. The evidence in the raw data, in fact, strongly favored the slotless model. These findings call into question claims about working memory that are based on summary statistics.     

Estimation of psychometric functions from adaptive tracking procedures.

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Estimation of psychometric functions from adaptive tracking procedures

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from-trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to -target-level-for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated-from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.     

Effects of background symmetry onsame-different pattern matching: A compromise-criteria account

Two experiments are reported, which were designed to test predictions of an account ofsame-different matching that assumes that bilaterally symmetric backgrounds provide extraneous evidence towardsame, whereas asymmetric backgrounds provide evidence towarddifferent. When all backgrounds within a block of trials are of the same type, appropriate adjustments of response criteria can be made to accommodate the irrelevant evidence and thus maintain acceptable levels of accuracy. However, when backgrounds of different types are mixed randomly, compromise criteria are adopted. This compromise-criteria account predicts distinctive interaction patterns for reaction times when blocked versus mixed presentations of various background types are compared. The predicted interactions were obtained for asymmetric- and no-noise backgrounds in Experiment 1 and forsymmetric-, asymmetric-, and no-noise backgrounds in Experiment 2. The findings support the general view that extraneous display attributes are weighted into the evidence for same anddifferent, with criteria settings used that minimize errors under the noisiest conditions.     

Auditory temporal summation in infants and adults: Effects of stimulus bandwidth and masking noise

A visually reinforced operant procedure was employed to determine the behavioral thresholds of 6- to 7-month-old infants and adults for stimuli of various bandwidths and durations. Experiment 1 compared absolute thresholds for broadband and 1/3-octavefiltered clicks and 300-msec noise bursts. For adult subjects, the difference in threshold for clicks and noise bursts was -quite comparable in the two bandwidth conditions, but infants’ click-noise threshold differences were significantly larger for broadband than for 1/3-octave stimuli. In Experiment 2, 2-point threshold-duration functions were compared for 4-kHz tones and octave-band noise bursts presented in backgrounds of quiet and continuous noise. Infants’ threshold-duration function for octave-band noise bursts was significantly steeper than the comparable adult function in quiet, but not in masking noise. These results suggest that young infants may have particular difficulty detecting low intensity broadband sounds when durations are very short.     

Factor Analysis of Ordinal Variables: A Comparison of Three Approaches

Theory and methodology for exploratory factor analysis have been well developed for continuous variables. In practice, observed or measured variables are often ordinal. However, ordinality is most often ignored and numbers such as 1, 2, 3, 4, representing ordered categories, are treated as numbers having metric properties, a procedure which is incorrect in several ways. In this article we describe four approaches to factor analysis of ordinal variables which take proper account of ordinality and compare three of them with respect to parameter estimates and fit. The comparison is made both in terms of their relative methodological advantages and in terms of an empirical data example and two generated data examples. In particular, we discuss the issue of how to test the model and to measure model fit.     

The effects of perceptual condition on proofreading for misspellings

In two experiments, subjects proofread text in which misspelled words were created by replacing a single letter with another one. The following perceptual conditions were included: clear-cut photocopies of typewritten text, clear-cut text presented on a CRT screen, ditto copies that varied in legibility, and text that included extraneous noise characters superimposed on some letters. In all conditions, subjects adopted a hierarchical feature test that gave first priority to resolving letter envelope and second priority to discriminating other letter features. When clear-cut text with no extraneous noise was proofread, subjects used a sophisticated-guessing decision rule that tolerated misspellings involving missing letter features (as when c replaced e) but not added ones (as when e replaced c). This asymmetrical rule was modified, however, when subjects were exposed to text that included extraneous noise that was confusable with the letter features. In those circumstances, subjects adopted a decision rule that tolerated misspellings involving missing features or any added features that resembled the noise.     

Response selection,sensitivity, and taste-test performance

Tasters selected the odd stimulus from among sets of three samples of party dip. Two samples came from one batch, and one sample came from another batch. The physicochemical difference between the batches consisted of the presence or absence of added salt. Two different tests of discriminability were undertaken by the same subjects with the same stimuli: the triangle test and the three-alternative forced-choice (3-AFC) method. Although different numbers of correct selections were obtained in the two tasks, an index of discriminability,d’, had the same value when the data were analyzed in accordance with the Thurstone-Ura and signal-detection models, respectively. The average data support Frijters’s (1979b) contention that different models of the discrimination process are appropriate to the results of the triangular and the 3-AFC procedures. Further analysis of the data revealed that discrimination was poorer for trios containing one physicochemically weak stimulus and two stronger stimuli than it was for trios containing one stronger stimulus and two weak stimuli. A two-signal 3-AFC task was undertaken by some subjects, andd’ estimates from this task were lower than expected on the basis of performance in the other tasks.     

Sampling plans for fitting the psychometric function

Research on estimation of a psychometric function psi has usually focused on comparing alternative algorithms to apply to the data, rarely addressing how best to gather the data themselves (i.e., what sampling plan best deploys the affordable number of trials). Simulation methods were used here to assess the performance of several sampling plans in yes-no and forced-choice tasks, including the QUEST method and several variants of up-down staircases and of the method of constant stimuli (MOCS). We also assessed the efficacy of four parameter estimation methods. Performance comparisons were based on analyses of usability (i.e., the percentage of times that a plan yields usable data for the estimation of all the parameters of psi) and of the resultant distributions of parameter estimates. Maximum likelihood turned out to be the best parameter estimation method. As for sampling plans, QUEST never exceeded 80% usability even when 1000 trials were administered and rendered accurate estimates of threshold but misestimated the remaining parameters. MOCS and up-down staircases yielded similar and acceptable usability (above 95% with 400-500 trials) and, although neither type of plan allowed estimating all parameters with optimal precision, each type appeared well suited to estimating a distinct subset of parameters. An analysis of the causes of this differential suitability allowed designing alternative sampling plans (all based on up-down staircases) for yes-no and forced-choice tasks. These alternative plans rendered near optimal distributions of estimates for all parameters. The results just described apply when the fitted psi has the same mathematical form as the actual psi generating the data; in case of form mismatch, all parameters except threshold were generally misestimated but the relative performance of all the sampling plans remained identical. Detailed practical recommendations are given.     

Statistical significance tests for binormal ROC curves

Statistical significance tests are derived and evaluated for measuring apparent differences between an obtained and an expected binormal ROC curve, between two independent binormal ROC curves, and among groups of independent binormal ROC curves. A binormal ROC curve is described by two parameters which represent the spread of the means and the ratio of the standard deviations of the two underlying Gaussian decision variable distributions. To test the significance of apparent differences between or among ROC curves, approximate χ² statistics for each of the three tests were constructed from maximum likelihood estimates of the two parameters defining the binormal ROC curve. The performance of each test statistic was evaluated by simulating five-category rating scale data with equal numbers of noise and signal-plus-noise trials (set at 50, 250, and 500) for each of three typical ROC curves. For the significance test involving only one ROC curve, rating scale data were generated from the chance diagonal of the ROC space also. Although test performance was found to be somewhat dependent on the number of trials and on the location of the ROC curve in the ROC space, comparisons of the obtained and expected fractions of (falsely) significant results at various α levels showed the proposed statistical significance tests to be reliable under practical experimental conditions.     

Recognition memory and attentional selection: serial scanning is not enough

In two experiments, using memory sets of up to 10 letters, the response competition paradigm was employed to investigate the extent to which extraneous visual stimuli interfere with or affect the process of memory search. It was assumed that if selective attention could exclude the effect of noise letters from a Sternberg-type memory comparison process, then there would be an increase in intercept for the reaction time-set size functions but no increase in slope. This result was obtained. However, a large difference in response times to both positive and negative set targets was found when the accompanying noise letters indicated a competing response, as opposed to when they indicated the same response as the target. This implies rapid identification of the nature of both target and noise, independent of a serial comparison process. A modification of a dual process model (Juola, Fischler, Wood, & Atkinson, 1971) in which stimuli activate a familiarity value independent of memory search was suggested to account for these results.     

Methodological issues in short-term motor memory research

Four methodological issues identified and studied by short-term motor memory researchers were examined in this paper: (a) the distributions of three commonly used measures of error (AE, CE, and VE); (b) statistical analysis of these scores as dependent measures in an experiment; (c) within-cell and experiment-wise estimates of the reliability of CE; and (d) the power of the statistical test with CE as the dependent measure. The first two sub-problems were explored theoretically, while the latter two were subjected to empirical examination using four data sets from previously published experiments. The distributions of AE and VE scores were described as non-normal under certain conditions, while the CE score distribution was shown to be normal. Therefore analyses of these three dependent measures may require different statistical procedures. In light of this information, the assumptions affecting estimates of reliability and power in an experimental framework were examined. Based on an analysis of the four data sets, a strategy was proposed in which a relatively small number of subjects could be utilized in an experiment without sacrificing desired experimental controls.     

Estimating Incremental Validity Under Missing Data

A common form of missing data is caused by selection on an observed variable (e.g., Z). If the selection variable was measured and is available, the data are regarded as missing at random (MAR). Selection biases correlation, reliability, and effect size estimates when these estimates are computed on listwise deleted (LD) data sets. On the other hand, maximum likelihood (ML) estimates are generally unbiased and outperform LD in most situations, at least when the data are MAR. The exception is when we estimate the partial correlation. In this situation, LD estimates are unbiased when the cause of missingness is partialled out. In other words, there is no advantage of ML estimates over LD estimates in this situation. We demonstrate that under a MAR condition, even ML estimates may become biased, depending on how partial correlations are computed. Finally, we conclude with recommendations about how future researchers might estimate partial correlations even when the cause of missingness is unknown and, perhaps, unknowable.