When 90% confidence intervals are 50% certain: on the credibility of credible intervals

Estimated confidence intervals for general knowledge items are usually too narrow. We report five experiments showing that people have much less confidence in these intervals than dictated by the assigned level of confidence. For instance, 90% intervals can be associated with an estimated confidence of 50% or less (and still lower hit rates). Moreover, interval width appears to remain stable over a wide range of instructions (high and low numeric and verbal confidence levels). This leads to a high degree of overconfidence for 90% intervals, but less for 50% intervals or for free choice intervals (without an assigned degree of confidence). To increase interval width one may have to ask exclusion rather than inclusion questions, for instance by soliciting ‘improbable’ upper and lower values (Experiment 4), or by asking separate ‘more than’ and ‘less than’ questions (Experiment 5). We conclude that interval width and degree of confidence have different determinants, and cannot be regarded as equivalent ways of expressing uncertainty. Copyright © 2005 John Wiley & Sons, Ltd.     

Precision and accuracy of judgmental estimation

Whereas probabilistic calibration has been a central normative concept of accuracy in previous research on interval estimates, we suggest here that normative approaches for the evaluation of judgmental estimates should consider the communicative interaction between the individuals who produce the judgments and those who receive or use them for making decisions. We analyze precision and error in judgment and consider the role of the accuracy–informativeness trade-off (Yaniv and Foster, 1995) in the communication of estimates. The results shed light on puzzling findings reported earlier in the literature concerning the calibration of subjective confidence intervals. © 1997 John Wiley & Sons, Ltd.     

QUAID: A questionnaire evaluation aid for survey methodologists

QUAID (question-understanding aid) is a software tool that assists survey methodologists, social scientists, and designers of questionnaires in improving the wording, syntax, and semantics of questions. The tool identifies potential problems that respondents might have in comprehending the meaning of questions on questionnaires. These problems can be scrutinized by researchers when they revise questions to improve question comprehension and, thereby, enhance the reliability and validity of answers. QUAID was designed to identify nine classes of problems, but only five of these problems are addressed in this article: unfamiliar technical term, vague or imprecise relative term, vague or ambiguous noun phrase, complex syntax, and working memory overload. We compared the output of QUAID with ratings of language experts who evaluated a corpus of questions on the five classes of problems. The corpus consisted of 505 questions on 11 surveys developed by the U.S. Census Bureau. Analyses of hit rates, false alarm rates,d′ scores, recall scores, and precision scores revealed that QUAID was able to identify these five problems with questions, although improvements in QUAID’s performance are anticipated in future research and development.     

Patterns of Preference for Numerical and Verbal Probabilities

We report results of an experiment designed to test a principle formulated by Budescu and Wallsten (1995), that, when communicating uncertainty information, mode choices are sensitive to sources and degrees of vagueness. In addition, we examined subjects’ efficacy in using such uncertainty information as a function of communication mode, source, and vagueness. In phase one of the experiment, subjects in a dyad used precise (numerical) or imprecise (verbal) expressions to communicate to a remote partner precise or vague uncertainty about the likelihoods of events. Spinner outcomes were used to generate precise uncertainty while answers to almanac questions were used to elicit vague uncertainty. In phase two, subjects saw the events paired with their partners’ estimates of similar events, and were asked to gamble on one event from each pair. Communication mode preferences were measured as the relative frequency that subjects chose the numerical mode to either express or receive uncertainty information regarding the events. Efficacy was measured as the relative frequency that subjects choose from the pair the event associated with the objectively more probable uncertainty expression. Underlying uncertainty interacted with direction of communication to affect preferences for modes of expression of the probabilities. Subjects preferred precise (numerical) information, especially for precise events (spinners). For vague events (questions), their preference for precise (numerical) information was stronger when receiving than when communicating information. Similar preferences were reflected in the efficiency of subsequent gamble decisions based on the probability estimates. Specifically, decisions were more efficacious (i.e. consistent with Expected Utility) when degrees of precision in events and estimates matched. © 1997 by John Wiley & Sons, Ltd.     

Learning to perceive differences in solid shape through vision and touch

A single experiment was designed to investigate perceptual learning and the discrimination of 3-D object shape. Ninety-six observers were presented with naturally shaped solid objects either visually, haptically, or across the modalities of vision and touch. The observers' task was to judge whether the two sequentially presented objects on any given trial possessed the same or different 3-D shapes. The results of the experiment revealed that significant perceptual learning occurred in all modality conditions, both unimodal and cross-modal. The amount of the observers' perceptual learning, as indexed by increases in hit rate and d', was similar for all of the modality conditions. The observers' hit rates were highest for the unimodal conditions and lowest in the cross-modal conditions. Lengthening the inter-stimulus interval from 3 to 15 s led to increases in hit rates and decreases in response bias. The results also revealed the existence of an asymmetry between two otherwise equivalent cross-modal conditions: in particular, the observers' perceptual sensitivity was higher for the vision-haptic condition and lower for the haptic-vision condition. In general, the results indicate that effective cross-modal shape comparisons can be made between the modalities of vision and active touch, but that complete information transfer does not occur.     

维族大学生面孔识别的本族效应：群际接触的影响

摘 要 采用“学习-再认”范式,让99名在人口统计学上与汉族不同接触程度的维族大学生再认维汉面孔,并填写社会接触和个性化经验量表,以考察本族效应是否能从种族间拓展到种族内不同民族间,以及群际接触的质和量对本族效应的影响。结果显示,维族大学生中与汉族中低接触者本族效应显著,高接触者未出现本族效应;接触量与个性化经验都与本族效应的幅度显著负相关;且相对优势分析表明个性化经验比接触量更能影响本族效应。     

The appeal of vague financial forecasts

Prior findings suggest managers often choose ranges to communicate uncertainty in future earnings. We analyzed earnings forecasts over 11 years and find higher earnings uncertainty firms are more likely to choose range estimates. We study investors’ attitudes to forecast precision and argue investors’ evaluations of forecasts can be explained by a sequential non-compensatory two-stage process – First, investors determine whether a point or a range estimate is more appropriate for a particular domain based on the congruence principle. Then, they seek the most precise reasonable range to maximize informativeness. Results from three experiments indicate the preference for (im)precision is non-monotonic – it peaks for low levels of imprecision and diminishes when the range gets wider, and is consistent with participants’ desire for congruent and informative estimates, and supports the claim that investors favor forecasts that are as precise as warranted by the information available, but not more precise.     

The robustness of false memory for emotional pictures

Emotional material is commonly reported to be more accurately recognised; however, there is substantial evidence of increased false alarm rates (FAR) for emotional material and several reports of stronger influences on response bias than accuracy. This pattern is more frequently reported for words than pictures. Research on the mechanisms underlying bias differences has mostly focused on word lists under short retention intervals. This article presents four series of experiments examining recognition memory for emotional pictures while varying arousal and the control over the content of the pictures at two retention intervals, and one study measuring the relatedness of the series picture sets. Under the shorter retention interval, emotion increased false alarms and reduced accuracy. Under the longer retention interval emotion increased hit rates and FAR, resulting in reduced accuracy and/or bias. At both retention intervals, the pattern of valence effects differed based on the arousal associated with the picture sets. Emotional pictures were found to be more related than neutral pictures in each set; however, the influence of relatedness alone does not provide an adequate explanation for all emotional differences. The results demonstrate substantial emotional differences in picture recognition that vary based on valence, arousal and retention interval.     

Heuristics and biases in diagnostic reasoning: I. Priors,error costs,and test accuracy

In three experiments, undergraduate subjects were asked to evaluate or choose between hypothetical medical tests. Subjects were told the subjective prior probability of a hypothetical disease, the hit rate and false-alarm rate of each test, and the relative subjective cost of the two possible errors that might be made. By varying priors, cost, and test accuracy, we could measure the influence of each parameter on subjects' responses. Subjects overweighed costs relative to both priors and test accuracy. In single-test cases in which the choice was whether to test or do something else (treat or withhold treatment), priors were not systematically misweighed relative to accuracy. When two tests were compared, priors were underweighed relative to accuracy. Justifications agreed with the conclusions reached by analysis of the preferences. When evaluating a test, subjects do not seem to understand that high priors make hit rates more relevant, while low priors make false-alarm rates more relevant. Subjects do, however, understand that a large cost of not treating diseased patients makes hit rates more relevant, while a large cost for treating nondiseased patients makes false-alarm rates more relevant. The overweighing of costs seems to result from the use of a heuristic in which the subject tends to minimize the probability of the worst kind of error, regardless of Other parameters.     

Bayesian journey-to-crime modelling of juvenile and adult offenders by gender in Manchester

This study tested the Bayesian journey-to-crime (JTC) model by gender and age for serial offenders from Manchester, England. The data were 4056 crimes committed by 171 serial offenders between 2003 and 2006. The data were subdivided by gender and age group to examine whether accuracy and precision varied by these subgroups. In general, the centre of minimum distance was found to be the most accurate measure, but the Bayesian risk and product measures were found to be the most precise measurements. The traditional ‘distance decay’ type of JTC function did not produce estimates that are as accurate nor as precise as the Bayesian approach. Tests were conducted on whether specific gender and age group JTC functions and origin–destination matrices improved predictability. With the exception of juvenile male offenders, the general functions were more accurate and precise. In terms of building an accurate and precise geographical profiling methodology, the results point to the need to include information on the likely predisposition of neighbourhoods to produce offenders as well as information on the crime opportunities available to offenders. Copyright © 2009 John Wiley & Sons, Ltd.     

The fallacy of placing confidence in confidence intervals

Interval estimates – estimates of parameters that include an allowance for sampling uncertainty – have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated samples, on average. The width of confidence intervals is thought to index the precision of an estimate; CIs are thought to be a guide to which parameter values are plausible or reasonable; and the confidence coefficient of the interval (e.g., 95 %) is thought to index the plausibility that the true parameter is included in the interval. We show in a number of examples that CIs do not necessarily have any of these properties, and can lead to unjustified or arbitrary inferences. For this reason, we caution against relying upon confidence interval theory to justify interval estimates, and suggest that other theories of interval estimation should be used instead.     

Recognition memory with little or no remembering: Implications for a detection model

Remembering and knowing are two states of awareness that reflect autonoetic and noetic consciousness. Recent extensions of signal detection theory have attempted to fitremember andknow responses, which measure these states of awareness, to a continuum of trace strength or familiarity. The model assumes there are two response criteria, a remembering criterion, which is more strict, and a recognition criterion, which is more lenient and leads to any positive recognition response. The most important prediction of this model is that bias-free estimates of memory should be the same whether derived from overall hit and false alarm rates or fromremember hit and false alarm rates. We describe evidence that disconfirms this prediction and discuss other findings that the model cannot accommodate.     

‘Earwitness’ voice recognition: Factors affecting accuracy and impact on jurors

Two studies were conducted examining voice recognition testimony and its impact on jurors. In the first experiment, subjects listened to a tape recording of a brief sales pitch. After a retention interval of either 0, 7 or 14 days, subjects were unexpectedly asked to pick the salesperson's voice out of a five-voice taped lineup. Retention interval did not have a significant effect on hit rates or false alarms. Accuracy and pre-lineup confidence were not significantly correlated, although accuracy was related to post-lineup willingness to testify. In the second experiment, undergraduate subjects were asked to read a summary of a trial, describing a situation similar to that studied in experiment 1; the independent variables were the presence of an earwitness, the gender and confidence of the earwitness, and the retention interval. Only the presence of an earwitness had a significant main effect upon mock jurors' verdicts. However, there was a significant interaction between witness confidence and witness gender when an earwitness identification was presented.     

"Don't know" responding to answerable and unanswerable questions during misleading and hypnotic interviews

"Don't know" (DK) responses to interview questions are conceptually heterogeneous, and may represent uncertainty or clear statements about the contents of memory. A study examined the subjective intent of DK responses in relation to the objective status of information queried, in the context of memory distorting procedures. Participants viewed a video and responded to answerable and unanswerable questions phrased in misleading or nonmisleading formats, while hypnotized or not hypnotized. Subjective meanings of DK responses were queried, and a recognition measure assessed the contents of memory. Lower DK and accuracy rates were consistently associated with unanswerable and misleading questions. One-third of DK responses were statements that the information had no not presented. When these were recoded, accuracy estimates for answerable questions decreased and more so for hypnotized participants. These results demonstrate that DK responses convey different types of information, thus accuracy estimates in studies that permit DK responses may be misestimated. Robust risks associated with asking unanswerable questions and asking questions at all were observed. Implications for working with DK responses during interviews are discussed.     

Bias and learning in temporal binding: Intervals between actions and outcomes are compressed by prior bias

It has consistently been shown that agents judge the intervals between their actions and outcomes as compressed in time, an effect named intentional binding. In the present work, we investigated whether this effect is result of prior bias volunteers have about the timing of the consequences of their actions, or if it is due to learning that occurs during the experimental session. Volunteers made temporal estimates of the interval between their action and target onset (Action conditions), or between two events (No-Action conditions). Our results show that temporal estimates become shorter throughout each experimental block in both conditions. Moreover, we found that observers judged intervals between action and outcomes as shorter even in very early trials of each block. To quantify the decrease of temporal judgments in experimental blocks, exponential functions were fitted to participants’ temporal judgments. The fitted parameters suggest that observers had different prior biases as to intervals between events in which action was involved. These findings suggest that prior bias might play a more important role in this effect than calibration-type learning processes.     

Predicting the Sensitivity of the Reaction Time‐based Concealed Information Test

Current lie detection and memory detection tests are imperfect. One possibility is that they work in some individuals but not in others. Here, we investigate whether we can statistically predict the accuracy of the Concealed Information Test (CIT). We examined the CIT's potential by (1) its sensitivity on an unrelated set of autobiographic questions and by (2) using mock crime target‐irrelevant differences. A mock crime CIT that included autobiographical questions was used to detect criminal intent. The mock crime CIT discriminated (receiver operating characteristic a = .87; d = 1.25; hit rate: 86%) between the criminal intent and the control condition. Using target‐irrelevant differences, but not the autobiographical questions, successfully screened out individuals and increased the CIT's accuracy (receiver operating characteristic a = .95; d = 1.67; hit rate: 97%). Our study provides preliminary evidence that we can predict for whom the CIT is likely to work and that this information can be used to increase detection efficiency. Copyright © 2013 John Wiley & Sons, Ltd.     

USING CALIBRATION AND INTEROBSERVER AGREEMENT ALGORITHMS TO ASSESS THE ACCURACY AND PRECISION OF DATA FROM ELECTRONIC AND PEN‐AND‐PAPER CONTINUOUS RECORDING METHODS

Often it is assumed that electronic recording by observers necessarily provides better quality data than pen‐and‐paper methods. Fifteen novice observers recorded rates of responding from 10 role‐played video samples using one of three continuous recording input formats: keyboard (laptop), touchscreen (personal digital assistants), or pen‐and‐paper. We evaluated the quality of the observers' data compared with criterion records using calibration and interobserver agreement algorithms. Results of the calibration analysis revealed that observers in the touchscreen group produced the most consistently accurate and precise data, the keyboard group observers showed wide variation in precision and accuracy, and the pen‐and‐paper group observers were significantly less precise than the touchscreen group. We conclude that although electronic recording has the potential to be as accurate as, and more precise than, pen‐and‐paper methods, this is far from guaranteed. Analyses of observers' errors advise recommendations for improving data accuracy and precision when using each method. Copyright © 2014 John Wiley & Sons, Ltd.     

Overconfidence in interval estimates: What does expertise buy you?

People’s 90% subjective confidence intervals typically contain the true value about 50% of the time, indicating extreme overconfidence. Previous results have been mixed regarding whether experts are as overconfident as novices. Experiment 1 examined interval estimates from information technology (IT) professionals and UC San Diego (UCSD) students about both the IT industry and UCSD. This within-subjects experiment showed that experts and novices were about equally overconfident. Experts reported intervals that had midpoints closer to the true value—which increased hit rate—and that were narrower (i.e., more informative)—which decreased hit rate. The net effect was no change in hit rate and overconfidence. Experiment 2 showed that both experts and novices mistakenly expected experts to be much less overconfident than novices, but they correctly predicted that experts would provide narrower intervals with midpoints closer to the truth. Decisions about whether to consult experts should be based on which aspects of performance are desired.     

Who can spot a potential problem gambler? Testing “it takes one to know one” and acquaintanceship effects in a university student population

How easy is it for individuals to detect low to moderate levels of problem gambling tendencies in others? Are individuals who have problem gambling tendencies themselves, or are close relationship partners, more accurate judges? We examine these questions in two studies involving a total of 336 interacting dyads drawn largely from a university student population. In Study 1 all pairs were strangers, whereas in Study 2 approximately half of the pairs were close. After the “judge” observed the “target” complete a gambling task, the dyad had a face-to-face discussion, with topics including favorite pastimes and personal weaknesses. Judges estimated the target's problem gambling tendencies, and both judges and targets self-reported their own gambling tendencies. There was evidence of modest, albeit somewhat inconsistent, accuracy in individuals’ judgments of the other person's problem gambling tendencies, but no “it takes one to know one” or acquaintanceship effects were apparent. Results also indicated that judges evidenced a projection bias, whereby they saw the target as similar to themselves, especially within close pairs. These results reveal that even after minimal interaction with a stranger individuals can be able to judge the person's gambling tendencies with some accuracy. At the same time, our findings indicating that close others and those with problem gambling tendencies themselves are not more or less tuned in to the early signs of a problem than anyone else suggest that it would be inappropriate to be especially convinced by—or skeptical of—these individuals’ judgments.