Improving judgment accuracy by sequential adjustment

Judging an object’s value based on relevant cues can be challenging. We propose a simple method to improve judgment accuracy: Instead of estimating a value after seeing all available cues simultaneously, individuals view cues sequentially, one after another, making and adjusting their estimate at each step. The sequential procedure may alleviate computational difficulties in cue integration, leading to higher judgment accuracy. We tested this hypothesis in two real-world tasks in which participants judged either the price of diamonds or the fuel economy of cars. Two studies with professional jewelers and car salespeople show that most participants indeed judged more accurately with a sequential than with a simultaneous procedure. Another two studies with college students further support this finding and show additionally that the sequential procedure could raise the judgment accuracy of inexperienced students to the same level as that of professionals judging with the simultaneous procedure.

     

Friends and strangers: acquaintanceship, agreement, and the accuracy of personality judgment

We examined the effect of acquaintanceship on interjudge agreement in personality ratings. Approximately 150 undergraduates described their own personalities using the Q-sort. They were also described by two close acquaintances and by two "strangers" who knew them only via a single, spontaneous interaction viewed on videotape. The effect of acquaintanceship was powerful: Judgments by close acquaintances agreed with each other and with subjects' self-judgments much better than did judgments by strangers, even though strangers' judgments agreed with each other and with subjects' self-judgments beyond a chance level. This result implies that agreement among acquaintances' judgments must derive at least partly from experience with and observation of the person who is judged. The same traits that yielded better agreement among acquaintances also yielded better agreement among strangers and tended to be rated higher in subjective visibility, suggesting that people are intuitively knowledgeable about the traits they can judge with more and less agreement.     

Learning and maintaining rules for decreasing judgment accuracy

A meeting of Psychologists Against Tough-Hearted and Offensive Scientists (PATHOS) was convened to redress the damage done by judgment researchers who have degraded the art of clinical decision making. A special task force drafted a rebuttal and mission statement, which is presented in its entirety.     

Using the conditional grade-of-membership model to assess judgment accuracy

Consider the case whereJ instruments are used to classify each ofI objects relative toK nominal categories. The conditional grade-of-membership (GoM) model provides a method of estimating the classification probabilities of each instrument (or judge) when the objects being classified consist of both pure types that lie exclusively in one ofK nominal categories, and mixtures that lie in more than one category. Classification probabilities are identifiable whenever the sample of GoM vectors includes pure types from each category. When additional, relatively mild, assumptions are made about judgment accuracy, the identifiable correct classification probabilities are the greatest lower bounds among all solutions that might correspond to the observed multinomial process, even when the unobserved GoM vectors do not include pure types from each category. Estimation using the conditional GoM model is illustrated on a simulated data set. Further simulations show that the estimates of the classification probabilities are relatively accurate, even when the sample contains only a small percentage of approximately pure objects.The authors thank Max A. Woodbury, Kenneth G. Manton and H. Dennis Tolley for their help and four anonymous Psychometrika reviewers (including an associate editor) for their beneficial expository and technical suggestions. This work was supported by the Dean's Fund for Summer Research, Owen Graduate School of Management, Vanderbilt University.     

Effects of amount of information on judgment accuracy and confidence

When a person evaluates his or her confidence in a judgment, what is the effect of receiving more judgment-relevant information? We report three studies that show when judges receive more information, their confidence increases more than their accuracy, producing substantial confidence–accuracy discrepancies. Our results suggest that judges do not adjust for the cognitive limitations that reduce their ability to use additional information effectively. We place these findings in a more general framework of understanding the cues to confidence that judges use and how those cues relate to accuracy and calibration.     

Developing a biodata measure and situational judgment inventory as predictors of college student performance

This article describes the development and validation of a biographical data (biodata) measure and situational judgment inventory (SJI) as useful predictors of broadly defined college student performance outcomes. These measures provided incremental validity when considered in combination with standardized college-entrance tests (i.e., SAT/ACT) and a measure of Big Five personality constructs. Racial subgroup mean differences were much smaller on the biodata and SJI measures than on the standardized tests and college grade point average. Female students tended to outperform male students on most predictors and outcomes with the exception of the SAT/ACT. The biodata and SJI measures show promise for student development contexts and for selecting students on a wide range of outcomes with reduced adverse impact.     

The effect of divided attention on global judgment of learning accuracy

This study examined the effect of divided attention (DA) on global judgment of learning (JOL) accuracy in a multitrial list learning paradigm. A word monitoring task was used to divide attention. Participants were assigned to an attention condition (DA at encoding, DA at judgment, DA at retrieval, or focused attention) and completed 4 learning trials, each comprising a study, judgment, and recall phase. Participants showed greater overconfidence in the DA at encoding (Trial 2) and DA at retrieval (Trials 1 and 2) conditions than in the focused attention condition. DA atjudgment did not affect JOL accuracy, and there was no effect of DA in Trials 3 and 4 on JOL accuracy across all attention conditions. Results indicate that participants consider conditions of encoding and retrieval but do not engage in recall when forming global JOLs. These findings suggest that people rely on extrinsic cues when making repeated, global metamemoryjudgments.     

Information quantity and quality affect the realistic accuracy of personality judgment

Triads of unacquainted college students interacted in 1 of 5 experimental conditions that manipulated information quantity (amount of information) and information quality (relevance of information to personality), and they then made judgments of each others' personalities. To determine accuracy, the authors compared the ratings of each judge to a broad-based accuracy criterion composed of personality ratings from 3 types of knowledgeable informants (the self, real-life acquaintances, and clinician-interviewers). Results supported the hypothesis that information quantity and quality would be positively related to objective knowledge about the targets and realistic accuracy. Interjudge consensus and self-other agreement followed a similar pattern. These findings are consistent with expectations based on models of the process of accurate judgment (D. C. Funder, 1995, 1999) and consensus (D. A. Kenny, 1994).     

Task bias and the accuracy of judgment: Setting a baseline for expected group performance

When a judgment task evokes unbiased estimates (i.e. the errors in individual judgments are distributed randomly around the true value), mathematical aggregation of individual estimates, even by a simple arithmetic mean, often will outperform all group members. However, when a task evokes biased estimates, mathematical aggregation does not perform so well. In this study, simulated data were accumulated to specify the expected' accuracy of mathematical aggregation relative to the accuracy of observed judgment of individual group members under varying conditions of task bias. Three types of judgment tasks were employed: (1) single-estimate, holistic tasks, (2) multiple-estimate, ranking tasks, and (3) multi-cue, decomposed tasks. Findings indicated across all task types that a large percentage of judgment-making group estimates formed strictly by computing the arithmetic mean of individual estimates performed better than their most capable members when a judgment task evoked little or no bias, a result particularly pronounced for ranking tasks. When the task was more greatly bias-evoking, a large percentage of parallel groups performed more poorly than average (or median) members, again a pattern more starkly evident for ranking tasks. These results suggest that the extent to which a judgment task evokes bias in a population of prospective group members is an important explanatory variable deserving much greater attention in the study of group performance. For example, an assertion about the efficacy of a particular group intervention based on a reliable demonstration of group performance as accurate as the most capable members may be unfounded when a task evokes no bias, since the baseline standard under such conditions should be much higher. By selecting tasks and populations that jointly produced highly biased estimates, researchers can lower the performance floor enough to detect (with reasonably small samples of groups) experimental effects should they occur.     

Probability judgment accuracy for general knowledge. Cross-national differences and assessment methods

In this study we compare the probability judgment accuracy of subjects from the United States and Turkey. Three different response modes were employed — numerical probabilities, pie diagrams, and odds. The questions employed in the study were restricted to two-alternative, general-knowledge items. The observed pattern of differences in the components of probability judgment accuracy paralleled those of studies that have compared Western and Asian subjects. In particular, Turkish subjects exhibited better discrimination but worse calibration than their US counterparts. This result persisted across all three response modes. These findings lend support to previous assertions that observed cross-national differences arise from socioeconomic rather than Asian versus Western cultural differences. However, the consistency of the observed differences across response modes refutes a previous assertion that observed cultural differences are merely the result of response bias.     

A simple and sensitive method to measure timing accuracy

Timing accuracy in presenting experimental stimuli (visual information on a PC or on a TV) and responding (keyboard presses and mouse signals) is of importance in several experimental paradigms. In this article, a simple system for measuring timing accuracy is described. The system uses two PCs (at least Pentium II, 200 MHz), a photocell, and an amplifier. No additional boards and timing hardware are needed. The first PC, a SlavePC, monitors the keyboard presses or mouse signals from the PC under test and uses a photocell that is placed in front of the screen to detect the appearance of visual stimuli on the display. The software consists of a small program running on the SlavePC. The SlavePC is connected through a serial line with a second PC. This MasterPC controls the SlavePC through an ActiveX control, which is used in a Visual Basic program. The accuracy of our system was investigated by using a similar setup of a SlavePC and a MasterPC to generate pulses and by using a pulse generator card. These tests revealed that our system has a 0.01-msec accuracy. As an illustration, the reaction time accuracy of INQUISIT for a few applications was tested using our system. It was found that in those applications that we investigated, INQUISIT measures reaction times from keyboard presses with millisecond accuracy.     

Frequency judgment accuracy as a function of age and school achievement (learning disabled versus non-learning-disabled) patterns

The accuracy of children's judgments of relative situational frequency was examined in two experiments. In Experiment 1 children with normal achievement in Grades 2 and 3 were compared with such children in Grades K and 5, as well as with three groups of low-achieving children in Grades 2 and 3. These latter groups consisted of children low in reading achievement, those low in math achievement, and those identified as learning disabled. Frequency judgment accuracy increased from kindergarten to Grades 2 and 3. No other comparisons yielded significant differences. Experiment 2 confirmed both the above age difference and the absence of any frequency judgment deficiency on the part of the low-achieving groups.     

The effect of simultaneous and sequential presentation of stimulus dimensions on absolute judgment accuracy

The effects of spatial stimulus repetition, sequential stimulus repetition, spatially separated dimensional redundancy, and sequentially presented dimensional redundancy on absolute judgment accuracy of hue and brightness were compared. Two exposure durations, 0.1 and 2.0 sec, were used. While spatial repetition did not improve accuracy for either dimension, the sequential repetition of brightness produced a small increase in accuracy. The spatial presentation of correlated values of both dimensions increased accuracy only at the 2.0-sec duration. The sequential presentation of both dimensions increased accuracy, but only at the 2.0-sec duration was this gain substantial and greater than that provided by the sequential repetition of brightness alone.     

The influence of visual mental imagery size on metamemory accuracy in judgment of learning

Many studies have found the font size of to-be-remembered words has a significant influence on judgments of learning (JOLs). However, few studies have investigated whether JOLs are affected by the mental imagery size of to-be-remembered words, even when the font sizes themselves are kept identical in study materials. This study investigated whether the visual mental imagery size influences the participants’ JOLs and what the underlying mechanisms are. In Experiments 1 and 2, participants learned words with identical font sizes, mentally generated large or small imageries and then made JOLs. We found that JOLs under the large imagery condition were significantly higher than those under the small imagery condition, but actual recall performance exhibited no significant difference. In Experiment 3, participants pressed a button immediately after mental imagery generation and showed that it took significantly longer to generate large imageries than to generate small imageries, and the difference in JOLs between two conditions was no longer significant. In Experiment 4, we used a questionnaire to investigate the contribution of beliefs and found that participants believed large imageries were easier to remember. These findings indicate that imagery size has a significant impact on JOLs, in which beliefs may play a leading role.     

Some critical observations of the test information function as a measure of local accuracy in ability estimation

The test information function serves important roles in latent trait models and in their applications. Among others, it has been used as the measure of accuracy in ability estimation. A question arises, however, if the test information function is accurate enough for all meaningful levels of ability relative to the test, especially when the number of test items is relatively small (e.g., less than 50). In the present paper, using the constant information model and constant amounts of test information for a finite interval of ability, simulated data were produced for eight different levels of ability and for twenty different numbers of test items ranging between 10 and 200. Analyses of these data suggest that it is desirable to consider some modification of the test information function when it is used as the measure of accuracy in ability estimation.