Confidence in aggregation of expert opinions

We investigate the case of a single decision maker (DM) who obtains probabilistic forecasts regarding the occurrence of a unique target event from J distinct, symmetric, and equally diagnostic expert advisors (judges). The paper begins with a mathematical model of DM's aggregation process of expert opinions, in which confidence in the final aggregate is shown to be inversely related to its perceived variance. As such, confidence is expected to vary as a function of factors such as the number of experts, the total number of cues, the fraction of cues available to each expert, the level of inter-expert overlap in information, and the range of experts' opinions. In the second part of the paper, we present results from two experiments that support the main (ordinal) predictions of the model.     

Subjective randomization in one-and two-person games

Following the study by Rapoport and Budescu (1992), two experiments are reported whose main purpose is to compare the generation of random sequences in one- and two-person games. The first experiment addresses the generation of trinary series in two-person zerosum games with asymmetric players. The second examines the generation of binary series using a between-subjects design, and compares the cognitive processes operating in one- and two-person games. Although both types of games result in similar deviations from randomness, they seem to induce qualitatively different cognitive processes.     

Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change

ABSTRACT— The Intergovernmental Panel on Climate Change (IPCC) assesses information relevant to the understanding of climate change and explores options for adaptation and mitigation. The IPCC reports communicate uncertainty by using a set of probability terms accompanied by global interpretational guidelines. The judgment literature indicates that there are large differences in the way people understand such phrases, and that their use may lead to confusion and errors in communication. We conducted an experiment in which subjects read sentences from the 2007 IPCC report and assigned numerical values to the probability terms. The respondents' judgments deviated significantly from the IPCC guidelines, even when the respondents had access to these guidelines. These results suggest that the method used by the IPCC is likely to convey levels of imprecision that are too high. We propose an alternative form of communicating uncertainty, illustrate its effectiveness, and suggest several additional ways to improve the communication of uncertainty.     

Predicting the directionality of probability words from their membership functions

Teigen and Brun have suggested that distinct from their numerical implications, most probability phrases are either positive or negative, in that they encourage one to think of reasons why the target event will or will not occur. We report two experiments testing our hypotheses that (a) the direction of a phrase can be predicted from properties of its membership function, and (b) this relation is invariant across contexts, and (c) —originally formulated by Teigen and Brun (1999)—that strong modifiers intensify phrase directionality. For each phrase, participants encoded membership functions by judging the degree to which it described the numerical probabilities 0.0, 0.1, …, 1.0, and also completed sentences including the target phrase. The types of reasons given in the sentence completion task were used to determine the phrase's directionality. The results support our hypotheses (a) and (b) regarding the relation between directionality and the membership functions, but we found only limited support for hypothesis (c) regarding the effects of modifiers on directionality. A secondary goal, to validate an efficient method of encoding membership functions, was also achieved. Copyright © 2003 John Wiley & Sons, Ltd.     

Modeling certainty equivalents for imprecise gambles

We present results from two experiments on the relative importance of, and subjects' differential sensitivity to, vagueness on both probabilities and outcomes. Subjects in these studies made certainty equivalent (CE) judgments for precise and vague gambles. In the first study subjects responded to gain gambles only; in the second they judged both gain and loss gambles. Model-free analyses of the results indicate (a) a higher concern for the precision of the outcomes than that of the probabilities, (b) vagueness seeking for positive outcomes and (c) vagueness avoidance for negative outcomes and (d) no strong modal attitude toward vagueness on the probability dimension. The greater salience of the outcomes can be explained by the nature of the response mode (CEs). The reflection of attitudes towards outcome vagueness in the two domains can be explained by the distinct goals of the DMs in the two cases, which cause them to focus on the highest (most desirable) possible gain or the largest (most dreaded) conceivable loss. We propose and fit a new model of decision making with vaguely specified attributes that generalize the Prospect Theory model for the precise case. The new generalized model combines the two submodels (preference among precise lotteries and effects of vagueness) and allows estimation of the vagueness parameters. These estimated parameters are consistent with, and confirm, the patterns uncovered by the qualitative analysis.     

Aggregation of opinions based on correlated cues and advisors

We study the process by which decision makers (DMs) aggregate probabilistic opinions from multiple, correlated sources with a special emphasis on the determinants of the DM's confidence, which is a predictor of the DM's willingness to accept the implications of the aggregation process. Our model assumes that (a) DM combines the advisors' opinions by weighting them according to the amount of information underlying them, and (b) the DM's confidence increases as a function of a variety of factors that reduce the variance of the aggregate. We report results of three studies that manipulate the predictive validity of the cues and their inter‐correlations. Most of the models' predictions are supported but, contrary to the model's prediction, the DMs' confidence is not sensitive to the inter‐cue correlation. The best predictors of the DMs' confidence are the perceived predictability of the event, the level of agreement among the advisors, and the advisors' self‐reports of confidence. This pattern of results is explained by the ‘system neglect’ hypothesis. Copyright © 2006 John Wiley & Sons, Ltd.     

Preference for human or algorithmic forecasting advice does not predict if and how it is used

Past research has found that people treat advice differently depending on its source. In many cases, people seem to prefer human advice to algorithms, but in others, there is a reversal, and people seem to prefer algorithmic advice. Across two studies, we examine the persuasiveness of, and judges' preferences for, advice from different sources when forecasting geopolitical events. We find that judges report domain-specific preferences, preferring human advice in the domain of politics and algorithmic advice in the domain of economics. In Study 2, participants report a preference for hybrid advice, that combines human and algorithmic sources, to either one on it's own regardless of domain. More importantly, we find that these preferences did not affect persuasiveness of advice from these different sources, regardless of domain. Judges were primarily sensitive to quantitative features pertaining to the similarity between their initial beliefs and the advice they were offered, such as the distance between them and the relative advisor confidence, when deciding whether to revise their initial beliefs in light of advice, rather than the source that generated the advice.     

Order in the home: Family routines moderate the impact of financial hardship

The study examined whether frequency of adolescent reported family routines moderated the relation between caregivers' perceptions of financial resources and adolescents' behavioral and academic outcomes among a sample of inner-city African American families. The sample consisted of 115 adolescents (average age = 15.95) and their female caregivers participating in a large scale study on African American family life. The results revealed a significant main effect of financial resources and family routine on adolescent outcomes, such that an increase in both was related to less delinquency and more favorable academic outcomes. Family routines moderated the relation between financial resources and adolescent outcomes. Specifically, the study revealed that the link between lack of financial resources and negative adolescent outcomes was diminished for youth in high-routine homes. Alternatively, family routines did not have a significant impact on youth from high-resource homes. Results highlight the importance of organization and predictability for economically disadvantaged youth and families.     

Markets as a structural solution to knowledge-sharing dilemmas

The authors propose a new structural solution to the knowledge-sharing dilemma. They show that simple auction mechanisms, which impose a rigid set of rules designed to standardize interactions and communication among participants, can prevent some of the detrimental effects associated with conflict of interest in freely interacting groups. The authors report results of two experiments that show that transparent conflicts of interest lead to a breakdown of information sharing, learning, and knowledge transfer in freely interacting groups, but not in simple markets and auctions. In these settings, participants were able to identify the best candidate in a voting game and to learn the solution to an intellective reasoning task, allowing participants to successfully transfer their insights to a series of new intellective tasks (tackled up to four weeks later), despite the conflicts of interests among traders. The authors explain their findings within the theoretical framework of collective induction.