ROCs in Eyewitness Identification: Instructions versus Confidence Ratings

From the perspective of signal detection theory, different lineup instructions may induce different levels of response bias. If so, then collecting correct and false identification rates across different instructional conditions will trace out the receiver operating characteristic (ROC)—the same ROC that, theoretically, could also be traced out from a single instruction condition in which each eyewitness decision is accompanied by a confidence rating. We tested whether the two approaches do in fact yield the same ROC. Participants were assigned to a confidence rating condition or to an instructional biasing condition (liberal, neutral, unbiased, or conservative). After watching a video of a mock crime, participants were presented with instructions followed by a six‐person simultaneous photo lineup. The ROCs from both methods were similar, but they were not exactly the same. These findings have potentially important policy implications for how the legal system should go about controlling eyewitness response bias.Copyright © 2017 John Wiley & Sons, Ltd.     

Binary ROCs in perception and recognition memory are curved

In recognition memory, a classic finding is that receiver operating characteristics (ROCs) are curvilinear. This has been taken to support the fundamental assumptions of signal detection theory (SDT) over discrete-state models such as the double high-threshold model (2HTM), which predicts linear ROCs. Recently, however, Br?der and Schütz (2009) challenged this argument by noting that most of the data on which support for SDT is based have involved confidence ratings. The authors argued that certain types of rating scale usage may result in curved ROCs even if the generating process is thresholded in nature. From this point of view, only ROCs constructed via experimental bias manipulations are useful for discriminating between the models. Br?der and Schütz conducted a meta-analysis and new experiments that compared SDT and the 2HTM using binary (yes-no) ROCs and found that many of these functions were linear, supporting 2HTM over SDT. We examine all the data reported by Br?der and Schütz, noting important limitations in their methodology, analyses, and conclusions. We report a new meta-analysis and 2 new experiments to examine the issue more closely while avoiding the limitations of Br?der and Schütz's study. These new data indicate that binary ROCs are curved in recognition, consistent with previous findings in perception and reasoning. Our results support classic arguments in favor of SDT and indicate that curvature in ratings ROCs is not task specific. We recommend the ratings procedure and suggest that analyses based on threshold models be treated with caution.     

Signal-Detection, Threshold, and Dual-Process Models of Recognition Memory: ROCs and Conscious Recollection

Threshold- and signal-detection-based models have dominated theorizing about recognition memory. Building upon these theoretical frameworks, we have argued for a dual-process model in which conscious recollection (a threshold process) and familiarity (a signal-detection process) contribute to memory performance. In the current paper we assessed several memory models by examining the effects of levels of processing and the number of presentations on recognition memory receiver operating characteristics (ROCs). In general, when the ROCs were plotted in probability space they exhibited an inverted U shape; however, when they were plotted inzspace they exhibited a U shape. An examination of the ROCs showed that the dual-process model could account for the observed ROCs, but that models based solely on either threshold or signal-detection processes failed to provide a sufficient account of the data. Furthermore, an examination of subjects' introspective reports using the remember/know procedure showed that subjects were aware of recollection and familiarity and were able to consistently report on their occurrence. The remember/know data were used to accurately predict the shapes of the ROCs, and estimates of recollection and familiarity derived from the ROC data mirrored the subjective reports of these processes.     

Recognition memory ROCs for item and associative information: The contribution of recollection and familiarity

Receiver-operating characteristics (ROCs) were examined in three recognition memory experiments. ROCs for item information (i.e., was this word presented?) were found to be curvilinear. However, ROCs for associative information (i.e., were these two words presented together?) were found to be linear. The results are in agreement with the predictions of a dual-process model that assumes that recognition judgments are -based on familiarity and recollection. Familiarity reflects the assessment of a continuous strength dimension and is well described as a signal detection process, whereas recollection reflects the retrieval of qualitative information about the study episode and behaves like a discrete threshold process. The results showed that memory judgments about items relied on a combination of recollection and familiarity, but that judgments about associations relied primarily on recollection. Further examination of the associative ROCs suggested that subjects were able to recollect that old pairs of items were in the study list, and, under some conditions, that new pairs were not in the study list.     

ROCs in recognition with and without identification

Receiver operating characteristics (ROCs) for the recognition of unidentified word fragments were analysed and compared with ROCs for the recognition of identified word fragments. In all conditions reported here, the z-transformed ROCs (z-ROCs) for recognition without identification were linear, with slopes near 1.0. In contrast, the z-ROCs for recognition with identification each had both a linear and a quadratic component, and a slope of substantially less than 1.0. Because the recognition of unidentified word fragments likely results from a familiarity process, the present study examined how well the effect can be explained by global matching variants of signal detection theory. Global matching models predict that memory should be better for strong items from a mixed list than for strong items from a pure list, and should be worse for weak items from a mixed list than for weak items from a pure list; however, only the latter pattern was shown for the recognition of unidentified items.     

ROCs in recognition with and without identification

Receiver operating characteristics (ROCs) for the recognition of unidentified word fragments were analysed and compared with ROCs for the recognition of identified word fragments. In all conditions reported here, the z-transformed ROCs (z-ROCs) for recognition without identification were linear, with slopes near 1.0. In contrast, the z-ROCs for recognition with identification each had both a linear and a quadratic component, and a slope of substantially less than 1.0. Because the recognition of unidentified word fragments likely results from a familiarity process, the present study examined how well the effect can be explained by global matching variants of signal detection theory. Global matching models predict that memory should be better for strong items from a mixed list than for strong items from a pure list, and should be worse for weak items from a mixed list than for weak items from a pure list; however, only the latter pattern was shown for the recognition of unidentified items.     

Development of differences in response latencies to right and left visual fields

Robust lateralization developed in right-handed adults who were asked to judge letter pairs as "same" or "different" during 4608 trials. By the end of the first two blocks (768 trials) "same" responses were favored when presented in the RVF (transmitted directly to the left hemisphere) and "different" responses were favored when presented in the LVF (transmitted direction to the right hemisphere). This gradually reversed over sessions with "same" responses becoming faster for letters presented in the LVF, and "different" responses becoming faster for stimuli presented in the RVF. The laterality acquired under these conditions was cumulative and reproducible, appeared in all 16 subjects, and was preserved between sessions a week apart. The data suggest that laterality is a flexible and reversible characteristic of the human brain even when stimulus and task remain constant.     

Recollection is a continuous process: Evidence from plurality memory receiver operating characteristics

Is recollection a continuous/graded process or a threshold/all-or-none process? Receiver operating characteristic (ROC) analysis can answer this question as the continuous model and the threshold model predict curved and linear recollection ROCs, respectively. As memory for plurality, an item's previous singular or plural form, is assumed to rely on recollection, the nature of recollection can be investigated by evaluating plurality memory ROCs. The present study consisted of four experiments. During encoding, words (singular or plural) or objects (single/singular or duplicate/plural) were presented. During retrieval, old items with the same plurality or different plurality were presented. For each item, participants made a confidence rating ranging from “very sure old”, which was correct for same plurality items, to “very sure new”, which was correct for different plurality items. Each plurality memory ROC was the proportion of same versus different plurality items classified as “old” (i.e., hits versus false alarms). Chi-squared analysis revealed that all of the plurality memory ROCs were adequately fit by the continuous unequal variance model, whereas none of the ROCs were adequately fit by the two-high threshold model. These plurality memory ROC results indicate recollection is a continuous process, which complements previous source memory and associative memory ROC findings.     

Cerebral Asymmetry and Decision Strategies in Mental Rotation: A Psychophysical Analysis

The accuracy with which observers judged the parity of pairs of rotated images in the two visual fields was determined from receiver operating characteristic (ROC) analysis of confidence ratings. In one experiment, observers judged whether pairs of letters were of the same parity (that is, both normal or both backwards) or of different parities (one normal and one backwards). A small right visual field advantage in the observers' accuracy was found in this mental rotation task. In a second experiment, observers judged whether pairs of pictures were of the same parity or of different parities. Unlike the first experiment, no evidence of a consistent visual field advantage was found in this mental rotation task. The decision strategy adopted by the observers in making same-different judgements about rotated stimuli was examined. The symmetrical shape of the ROCs obtained was consistent with the adoption of the likelihood-ratio decision strategy, a result which supplemented previous evidence that this decision strategy is adopted when same-different judgements are made about multidimensional stimuli.     

Source ROCs are (typically) curvilinear: comment on Yonelinas (1999)

On the basis of his assumption that recollection is a threshold process, A. P. Yonelinas (1999) predicted linear source-identification receiver operating characteristics (ROCs) and recently reported data that were consistent with this prediction. In this article, the authors present data showing curvilinear source-identification ROCs across various encoding and test conditions. On the basis of the source-monitoring framework (e.g., M. K. Johnson, S. Hashtroudi, & D. S. Lindsay, 1993), the authors argue that curvilinearity of source-identification ROCs is a result of differences in the qualitative characteristics of memories rather than simply the influence of undifferentiated familiarity as the dual-process model might suggest.     

Division of attention between simultaneous and successive nonverbal signals varying in discriminability

Impaired performance indexed by accuracy usually accompanies the division of attention between simultaneous difficult signals. Presenting the signals successively reduces the impairment. Presenting easy signals eliminates it. Both variables crucial to these findings, relative time of onset and discriminability, were manipulated in the first experiment using a two-choice recognition paradigm involving bimodal signals. The results showed that, when a difficult signal was followed by two simultaneous easy ones, in a different modality from the first, performance was significantly worse on each than on the same signals presented alone. Simultaneous presentation of all three signals further increased the impairment for the difficult signal, but not for the easy ones. In the second experiment, the discriminability of the following signals was varied. The results showed, that when a difficult signal was followed by either two simultaneous easy signals, in a different modality from the first, or by two difficult ones, performance was significantly worse than on the same difficult signal presented alone. Further, when the preceding difficult signal was auditory, but not when it was visual, performance was better with easy following signals than with difficult ones. It is argued that an adequate account of the results from the two experiments requires the following assumptions concerning the processing of multiple signals: (1) a loss associated both with the acquisition and the storage of information, (2) preferential processing of visual signals over auditory ones. Proposals are made for the incorporation of both assumptions into a single model of divided attention.     

Shifts of attention in the identification and discrimination of intensity

Subjects’ ability to resolve stimuli varying in loudness is studied by manipulating task requirements and sequential dependencies in signal presentations. In the first experiment, subjects are required both to identify and to discriminate the same set of 11 randomly presented signals. On the one hand, resolution performance is virtually unchanged under the differing task instructions. On the other hand, large effects on sensitivity are obtained in Experiments 1, 2, and 3 in both identification and discrimination tasks when sequential dependencies are manipulated. The results are interpreted in terms of a shifting attention band in the intensity dimension, as suggested by Luce, Green, and Weber (1976). Preliminary evidence that subjects can voluntarily focus the attention band on a given region of the intensity continuum is obtained.     

Immediate and delayed effects of victim suffering on the attacker's aggression

On the first day of a two-day experiment, male undergraduates were either angered or not, and they were given either high, low, or no metered pain feedback after each shock they supposedly delivered to their previous evaluator for his errors on a learning task. After the learning task the subjects made a number of ratings, including how much they had enjoyed this first session. On the second day, all subjects were simply required to administer shocks to a different person for his mistakes on the same learning task. The angered subjects were more punitive on both days toward both learners than the nonangered men. On the first day the angered men also increased the intensity of the shocks they delivered over trial blocks. Most interestingly, the angered men showed more enjoyment of the first session of the experiment as their victim's pain increased, and this enjoyment rating was related to the angered subjects' level of aggression on the second day of the experiment when they punished an “innocent” victim. The results were interpreted as consistent with the hypothesized reinforcement process which essentially states that signs and/or knowledge of the victim's suffering can reinforce impulsive or angry aggressive behavior.     

Reduced efficiency and capacity limitations in multidimension signal recognition

The first part of Experiment I showed that subjects recognized difficult frequency and intensity signals significantly less well when these were presented together than when they were presented alone. It was estimated that contingent variability resulting from other dimension signal and response context occurring on the same trial accounted for a third of the reduction, and capacity limitations for the remainder. Efficiency on the two dimensions presented together was found to be related and two data subsets were identified for which no efficiency reduction occurred. In the second part of Experiment I subjects recognized correlated frequency and intensity signals and showed a significant improvement in performance over either dimension performed alone. The gain in efficiency was well predicted by the “integration” model of Green and Swets (1966) from one- but not from two-dimension performance. The result suggested that the reduction in efficiency associated with capacity limitations was a function not of the number of signals but of the number of difficult decisions. Experiment II replicated the first part of Experiment I with easy signals. Efficiency was the same whether the signals were presented together or alone. The result indicated that the reduction in efficiency with difficult signals was not due to response interference or to failures of post-recognition retention. The implications of the results for models of attentional capacity are discussed.     

Identifying stimuli of different perceptual categories in pure and mixed blocks of trials: evidence for stimulus-driven switch costs.

Responding to stimuli of different perceptual categories is usually faster when the categories are presented isolated from each other, in pure blocks, than when they are presented randomly intermixed, in mixed blocks; a difference denoted as perceptual mixing costs. The present study examined the contribution of strategic and stimulus-driven factors to these costs. The first two experiments showed that perceptual mixing costs were not reduced when participants were informed at the start of each trial in mixed blocks about the impending category. Furthermore, Experiments 1 and 3 showed that mixing costs were concentrated on those trials of mixed blocks where the perceptual category was different from that of the preceding trial. These results support the view that perceptual mixing costs derive from stimulus-driven trial-by-trial adjustments in processing. Some general implications for processing models are discussed.     

DOES CONDITIONAL DISCRIMINATION LEARNING BY PIGEONS NECESSARILY INVOLVE HIERARCHICAL RELATIONSHIPS?

Four experiments demonstrate that when putative conditional and discriminative cues are presented simultaneously in the single reversal procedure, it is not possible to ascribe a uniquely conditional or uniquely discriminative function to either of the cues. In Experiment 1, pigeons were trained to respond to a blue key and not to a red key while the houselight was on; then in a different session they learned the reversal of this discrimination with the houselight off (single reversal). Separate groups were tested for color generalization with houselight conditions alternating in blocks of trials or for houselight intensity generalization with blue and red key colors alternating in blocks of trials. Both test procedures revealed a conditional relationship between houselight and key color conditions. Experiment 2 produced the same result following training in which the key colors were held constant across training sessions while the houselight and no houselight conditions varied within sessions. In Experiment 3, separate groups were trained with the two procedures but were tested with randomly ordered combinations of key colors and houselight intensities. The two groups yielded indistinguishable bidimensional generalization gradients with peaks at both previously reinforced stimulus combinations. In Experiment 4 the subjects were switched from one of these training procedures to the other with no decrement in their discriminative performance. We conclude that for successive discriminations between conditional- and discriminative-stimulus combinations, the notion of a hierarchical relation between conditional and discriminative stimuli must be extended to include a symmetrical relationship or the notion should be abandoned altogether.     

Vibrotactile signal detection

The theory of signal detectability was applied to vibrotactile sensitivity in two experiments. The first experiment showed The psychometric function to be satisfactorily linear when a sensitivity index of the d family was plotted against the signal intensity expressed in decibels. The second experiment yielded receiver-operating-characteristic (ROC) curves of a familiar form for the yes-no and rating response methods. Reasonably consistent estimates of sensitivity were obtained in the second experiment from the yes-no. rating, and forced-choice methods. The sensitivity indices examined were d’ and d_e’, based on Gaussian density functions; A, based on Rayleigh density functions; and the distribution-free indices, P(A) and F(C). For each type of index a tendency was observed for the forced-choice value to be lower than the yes-no and rating values.     

On the form of ROCs constructed from confidence ratings

A classical question for memory researchers is whether memories vary in an all-or-nothing, discrete manner (e.g., stored vs. not stored, recalled vs. not recalled), or whether they vary along a continuous dimension (e.g., strength, similarity, or familiarity). For yes-no classification tasks, continuous- and discrete-state models predict nonlinear and linear receiver operating characteristics (ROCs), respectively (D. M. Green & J. A. Swets, 1966; N. A. Macmillan & C. D. Creelman, 1991). Recently, several authors have assumed that these predictions are generalizable to confidence ratings tasks (J. Qin, C. L. Raye, M. K. Johnson, & K. J. Mitchell, 2001; S. D. Slotnick, S. A. Klein, C. S. Dodson, & A. P. Shimamura, 2000, and A. P. Yonelinas, 1999). This assumption is shown to be unwarranted by showing that discrete-state ratings models predict both linear and nonlinear ROCs. The critical factor determining the form of the discrete-state ROC is the response strategy adopted by the classifier.     

The lifespan of time intervals in reference memory

To further explore how memory influences time judgments, we conducted two experiments on the lifespan of temporal representations in memory. Penney et al (2000, Journal of Experimental Psychology Human Perception and Performance 26 1770-1787) reported that the perceived duration of auditorily and visually marked intervals differs only when both marker-type intervals are compared directly. This finding can be explained by a 'memory-mixing' process, whereby the memory trace of previous intervals influences the perception of upcoming ones, which are then added to the memory content. In the experiments discussed here, we manipulated the mixing mode of auditory/visual signal presentations. In experiment 1, signals from the same modality were either grouped by blocks or randomised within blocks. The results showed that the auditory/visual difference decreased but remained present when modalities were grouped by blocks. In experiment 2, we used a line-segmentation task. The results showed that, after a training block was performed in one modality, the perceived duration of signals from the other modality was distorted for at least 30 trials and that the magnitude of the difference decreased as the block went on. The results of both experiments highlight the influence of memory on time judgments, providing empirical support to, and quantitative portrayal of, the memory-mixing process.     

Semantic satiation and lexical ambiguity resolution.

This study examines the effects of semantic satiation on lexical ambiguity resolution. On a given trial, participants were presented with a word triad. The first word (e.g., HEART) was presented on average 2.5, 12.5, or 22.5 times, and then participants received 2 new words for relatedness judgments. The first of the two new words was always a homograph (e.g., "ORGAN") and the other word was a related or unrelated pairmate (e.g., "KIDNEY"). In Experiment 1, when blocks of trials were intermixed with concordant (e.g., "HEART-ORGAN-KIDNEY"), discordant (e.g., "PIANO-ORGAN-KIDNEY"), and neutral (e.g., "CEILING-ORGAN-KIDNEY") trials, participants did not produce evidence of semantic satiation. In a second experiment in which only concordant and neutral trials were presented, however, participants did produce evidence of semantic satiation in the concordant condition. Taken together, Experiments 1 and 2 indicate that semantic satiation of the context-appropriate meaning of a homograph may impede ambiguity resolution.