Measures of sensitivity based on a single hit rate and false alarm rate: the accuracy, precision, and robustness of d', Az, and A'

Signal detection theory offers several indexes of sensitivity (d', Az, and A') that are appropriate for two-choice discrimination when data consist of one hit rate and one false alarm rate per condition. These measures require simplifying assumptions about how target and lure evidence is distributed. We examine three statistical properties of these indexes: accuracy (good agreement between the parameter and the sampling distribution mean), precision (small variance of the sampling distribution), and robustness (small influence of violated assumptions on accuracy). We draw several conclusions from the results. First, a variety of parameters (sample size, degree of discriminability, and magnitude of hits and false alarms) influence statistical bias in these indexes. Comparing conditions that differ in these parameters entails discrepancies that can be reduced by increasing N. Second, unequal variance of the evidence distributions produces significant bias that cannot be reduced by increasing N-a serious drawback to the use of these sensitivity indexes when variance is unknown. Finally, their relative statistical performances suggest that Az is preferable to A'.     

Memory for temporally dynamic scenes

Recognition memory was investigated for individual frames extracted from temporally continuous, visually rich film segments of 5–15 min. Participants viewed a short clip from a film in either a coherent or a jumbled order, followed by a recognition test of studied frames. Foils came either from an earlier or a later part of the film (Experiment 1) or from deleted segments selected from random cuts of varying duration (0.5 to 30?s) within the film itself (Experiment 2). When the foils came from an earlier or later part of the film (Experiment 1), recognition was excellent, with the hit rate far exceeding the false-alarm rate (.78 vs. 18). In Experiment 2, recognition was far worse, with the hit rate (.76) exceeding the false-alarm rate only for foils drawn from the longest cuts (15 and 30?s) and matching the false-alarm rate for the 5?s segments. When the foils were drawn from the briefest cuts (0.5 and 1.0 s), the false-alarm rate exceeded the hit rate. Unexpectedly, jumbling had no effect on recognition in either experiment. These results are consistent with the view that memory for complex visually temporal events is excellent, with the integrity unperturbed by disruption of the global structure of the visual stream. Disruption of memory was observed only when foils were drawn from embedded segments of duration less than 5?s, an outcome consistent with the view that memory at these shortest durations are consolidated with expectations drawn from the previous stream.     

Decision processes in children's recognition memory

Kindergarteners and third graders were given a continuous recognition memory task involving two-digit numbers. In addition, a rating scale consisting of photographs of various facial expressions was used to obtain confidence judgments from the Ss. Conventional analyses as well as signal detection analyses of the data revealed the following results: (a) the overall performance of the third graders was superior to that of the kindergarteners; (b) memory strength decreased as the number of intervening items increased; (c) there was no difference in the forgetting rates of the two grade levels; (d) the third graders exhibited a more liberal response bias than the kindergarteners; (e) both the hit rate (probability of correctly labeling an old stimulus as old) and the false-alarm rate (probability of incorrectly labeling a new stimulus as old) increased across blocks of items; (f) the increases in the hit rate and the false-alarm rate over blocks were due to a change in criterion from a relatively conservative level to a more lenient one; (g) the lower the S's level of confidence in judging an item as old, the lower was the probability of that item actually being old; (h) the third graders were better than the kindergarteners at gauging the accuracy of their recognition responses. It was concluded that with respect to recognition memory, chidren as young as $\text{5}\text{1}\text{2}$ years old are capable, to some extent, of monitoring their own memory states.     

Skill development in vigilance: effects of event rate and age.

Thirty-six young (19-27 years), middle-aged (40-55 years), and old (70-80 years) adults performed a 30-min vigilance task at low (15 per min) and high (40 per min) event rates for 20 sessions. Skill-acquisition curves modeled on power, hyperbolic, and exponential functions were predicted. With extensive practice, hit rates increased and false-alarm rates decreased to virtually asymptotic levels. Skill development was best described by the hyperbolic function. Practice reduced but did not eliminate the vigilance decrement in all subjects. The event-rate effect--the decrease in hit rate at high event rates--was reduced with practice and eliminated in young subjects. Hit rates decreased and false-alarm rates increased with age, but there was little attenuation of age differences with practice. Implications for theories of vigilance, skill development, and cognitive aging are discussed.     

A bias in favor of the positive response to high-frequency words in recognition memory

A relatively liberal response bias for high-frequency words and a violation of the mirror effect for hit and false-alarm rates were found in a yes-no recognition-memory test. Subjects more frequently responded "old" to high-frequency words than to low-frequency words. Four experiments were conducted to determine the causes of the different response biases and of the violation of the mirror effect. The word-frequency effect on hit rates did not appear, whereas the false-alarm rate for low-frequency words was lower than that for high-frequency words. When low- or high-frequency words were presented separately in distinct halves of a recognition-memory test, the relatively liberal response bias for high-frequency words was diminished. A model for recognition judgment is proposed that assumes the use of a common criterion for low- and high-frequency words.     

Heuristics and biases in diagnostic reasoning: I. Priors,error costs,and test accuracy

In three experiments, undergraduate subjects were asked to evaluate or choose between hypothetical medical tests. Subjects were told the subjective prior probability of a hypothetical disease, the hit rate and false-alarm rate of each test, and the relative subjective cost of the two possible errors that might be made. By varying priors, cost, and test accuracy, we could measure the influence of each parameter on subjects' responses. Subjects overweighed costs relative to both priors and test accuracy. In single-test cases in which the choice was whether to test or do something else (treat or withhold treatment), priors were not systematically misweighed relative to accuracy. When two tests were compared, priors were underweighed relative to accuracy. Justifications agreed with the conclusions reached by analysis of the preferences. When evaluating a test, subjects do not seem to understand that high priors make hit rates more relevant, while low priors make false-alarm rates more relevant. Subjects do, however, understand that a large cost of not treating diseased patients makes hit rates more relevant, while a large cost for treating nondiseased patients makes false-alarm rates more relevant. The overweighing of costs seems to result from the use of a heuristic in which the subject tends to minimize the probability of the worst kind of error, regardless of Other parameters.     

Analysis of RT distributions in the remember—know paradigm

Do remembering and knowing differ qualitatively (reflecting distinct underlying processes) or quantitatively (reflecting different levels of strength)? Broadly speaking, models of remember—know judgments based on these alternatives have been tested by examining the proportion of remember and know responses that are made across conditions or levels of confidence. Here, we consider reaction time (RT) data. We replicate Dewhurst and Conway’s (1994) observation that old judgments followed by remember responses are faster, on average, than those followed by know decisions, but show that this effect is largely due to differing distributions of remember and know responses across confidence levels. In addition, fits of ex-Gaussian distributions of hit RTs followed by either remember or know judgments yield similar parameter values when confidence level is controlled. Thus, these RT data do not provide strong support for the idea that remembering and knowing depend on different processes.     

Lexico-semantic structure and the word-frequency effect in recognition memory

The word-frequency effect (WFE) in recognition memory refers to the finding that more rare words are better recognized than more common words. We demonstrate that a familiarity-discrimination model operating on data from a semantic word-association space yields a robust WFE in data on both hit rates and false-alarm rates. Our modeling results suggest that word frequency is encoded in the semantic structure of language, and that this encoding contributes to the WFE observed in item-recognition experiments.     

Detection of auditory signals occurring at random times: Intensity and duration

Observers attempted to detect a weak auditory signal presented in noise. The onset of the signal was determined by a Poisson process, and only responses occurring 1 sec after signal onset were considered detections. Three latency distributions were measured, the time to a signal onset, the reaction time distribution, and the false-alarm distribution (of responses occurring before a signal onset). A simple two-state model is proposed to account for a discrepancy between the distribution of signal waits and the distribution of false alarms. The hazard functions of the reaction time distributions are considered in detail and a simple accumulation model is proposed to account for the results.     

A strict response criterion yields a mirror effect in the novelty paradigm

According to the novelty/encoding hypothesis (NEH; Tulving & Kroll, 1995), efficacy of encoding information into long-term memory depends on the movelty of the information. Recognition accuracy is higher for novel than for previously familiarized material. This novelty effect is not a mirror effect: the superiority of novel over familiar items is not found in the hit rates but only in the false-alarm rates. The main result in the present replication study was that novel hit rates were higher than familiar ones when the most confident responses were examined separately, and thus a mirror effect could be demonstrated for these data, for both the low- and the high-frequency words. Similarly, the word-frequency effect on hits was stronger when a stricter response criterion was applied. It was concluded that the novelty effect and the word-frequency effect are more similar to one another than has hitherto been thought.     

The ROC Toolbox: A toolbox for analyzing receiver-operating characteristics derived from confidence ratings

Signal-detection theory, and the analysis of receiver-operating characteristics (ROCs), has played a critical role in the development of theories of episodic memory and perception. The purpose of the current paper is to present the ROC Toolbox. This toolbox is a set of functions written in the Matlab programming language that can be used to fit various common signal detection models to ROC data obtained from confidence rating experiments. The goals for developing the ROC Toolbox were to create a tool (1) that is easy to use and easy for researchers to implement with their own data, (2) that can flexibly define models based on varying study parameters, such as the number of response options (e.g., confidence ratings) and experimental conditions, and (3) that provides optimal routines (e.g., Maximum Likelihood estimation) to obtain parameter estimates and numerous goodness-of-fit measures.The ROC toolbox allows for various different confidence scales and currently includes the models commonly used in recognition memory and perception: (1) the unequal variance signal detection (UVSD) model, (2) the dual process signal detection (DPSD) model, and (3) the mixture signal detection (MSD) model. For each model fit to a given data set the ROC toolbox plots summary information about the best fitting model parameters and various goodness-of-fit measures. Here, we present an overview of the ROC Toolbox, illustrate how it can be used to input and analyse real data, and finish with a brief discussion on features that can be added to the toolbox.     

Signal detection theory, the approach of choice: model-based and distribution-free measures and evaluation

New and old methods of analyzing two-choice experiments with confidence ratings are evaluated. These include the theory of signal detectability (TSD), Luce's choice theory,nonparametric techniques based on areas under receiver-operating characteristic (ROC) functions, andmethods based on S' and omega, proposed by Balakrishnan and his colleagues. New methods for assessing the bias of a complete ROC function are proposed, together with an additional area-based measure of response bias. Area measures of both sensitivity and bias proved the most consistent. Response bias for a full ROC function was larger than bias at the cut point and also provided additional information. Participants showed voluntary control of bias for all measures except omega. Unequal variance versions of TSD and choice models gave similar fits to data, with the choice model closer to an equal variance version. Discrimination data from Balakrishnan (1999) formed the empirical test bed.     

Face value: testing the utility of contextual face cues for face recognition

The presence of multiple faces during a crime may provide a naturally-occurring contextual cue to support eyewitness recognition for those faces later. Across two experiments, we sought to investigate mechanisms underlying previously-reported cued recognition effects, and to determine whether such effects extended to encoding conditions involving more than two faces. Participants studied sets of individual faces, pairs of faces, or groups of four faces. At test, participants in the single-face condition were tested only on those individual faces without cues. Participants in the two and four-face conditions were tested using no cues, correct cues (a face previously studied with the target test face), or incorrect cues (a never-before-seen face). In Experiment 2, associative encoding was promoted by a rating task. Neither hit rates nor false-alarm rates were significantly affected by cue type or face encoding condition in Experiment 1, but cuing of any kind (correct or incorrect) in Experiment 2 appeared to provide a protective buffer to reduce false-alarm rates through a less liberal response bias. Results provide some evidence that cued recognition techniques could be useful to reduce false recognition, but only when associative encoding is strong.     

Vertical reversal and lateral reversal as components of inversion in recognition memory for upside-down faces

Subjects viewed upright photographs of faces, then attempted to recognize them unchanged, vertically reversed or inverted. In two of four conditions, hit scores were lower for inverted than vertically reversed faces, suggesting that lateral reversal is a meaningful component of inversion. The effect was not sufficiently strong, however, to overcome a generally high false-alarm rate for upside-down faces.     

Recollection and familiarity through the looking glass: when old does not mirror new

The authors use the qualitative differences logic to demonstrate that 2 separate memory influences underlie performance in recognition memory tasks, familiarity and recollection. The experiments focus on the mirror effect, the finding that more memorable stimulus classes produce higher hit rates but lower false-alarm rates than less memorable stimulus classes. The authors demonstrate across a number of experiments that manipulations assumed to decrease recollection eliminate or even reverse the hit-rate portion of the mirror effect while leaving the false-alarm portion intact. This occurs whether the critical distinction between conditions is created during the test phase or manipulated during the study phase. Thus, when recollection is present, it dominates familiarity so that the hit-rate portion of the mirror effect primarily reflects recollection; when recollection is largely absent, the opposite pattern associated with the familiarity process emerges.     

Statistical inferences about the error variance

This paper is a presentation of an essential part of the sampling theory of the error variance and the standard error of measurement. An experimental assumption is that several equivalent tests with equal variances are available. These may be either final forms of the same test or obtained by dividing one test into several parts. The simple model of independent and normally distributed errors of measurement with zero mean is employed. No assumption is made about the form of the distributions of true and observed scores. This implies unrestricted freedom in defining the population. First, maximum-likelihood estimators of the error variance and the standard error of measurement are obtained, their sampling distributions given, and their properties investigated. Then unbiased estimators are defined and their distributions derived. The accuracy of estimation is given special consideration from various points of view. Next, rigorous statistical tests are developed to test hypotheses about error variances on the basis of one and two samples. Also the construction of confidence intervals is treated. Finally, Bartlett's test of homogeneity of variances is used to provide a multi-sample test of equality of error variances.     

Alternatives to a table of criterion values in signal detection theory

Gardner, Dalsing, Reyes, and Brake (1984) supplied a table of criterion values (β) related to hit and false-alarm rates in signal detection theory. Other methods of calculatingβ are suggested as more accurate alternatives to using that table. A short computer program is provided to calculate β and the sensitivity indexd’.     

On the sampling interpretation of confidence intervals and hypothesis tests in the context of conditional maximum likelihood estimation

In the context ofconditional maximum likelihood (CML) estimation, confidence intervals can be interpreted in three different ways, depending on the sampling distribution under which these confidence intervals contain the true parameter value with a certain probability. These sampling distributions are (a) the distribution of the data given theincidental parameters, (b) the marginal distribution of the data (i.e., with the incidental parameters integrated out), and (c) the conditional distribution of the data given the sufficient statistics for the incidental parameters. Results on the asymptotic distribution of CML estimates under sampling scheme (c) can be used to construct asymptotic confidence intervals using only the CML estimates. This is not possible for the results on the asymptotic distribution under sampling schemes (a) and (b). However, it is shown that theconditional asymptotic confidence intervals are also valid under the other two sampling schemes. I am indebted to Theo Eggen, Norman Verhelst and one of Psychometrika's reviewers for their helpful comments.     

Two-sample models with monotonic likelihood ratios for ordinal regression

Signal detection experiments with human observers frequently generate ordinal data which are evaluated using receiver operating characteristic (ROC) methods. These methods may include regressing a continuous two-distribution model to the data set. Because we assume that human observers will not systematically select observations absent the signal, this model should have a monotonic likelihood ratio between the distributions. This paper gives a general method for constructing pairs of distributions that have monotonic likelihood ratios and a possibly large number flexible parameters. It suggests two specific simple parametric forms of monotonic likelihood ratios, constructs new distributions with those likelihood ratios using other standard distributions, and performs ordinal regression with those new distributions to model some example data from the literature.