Task bias and the accuracy of judgment: Setting a baseline for expected group performance

When a judgment task evokes unbiased estimates (i.e. the errors in individual judgments are distributed randomly around the true value), mathematical aggregation of individual estimates, even by a simple arithmetic mean, often will outperform all group members. However, when a task evokes biased estimates, mathematical aggregation does not perform so well. In this study, simulated data were accumulated to specify the expected' accuracy of mathematical aggregation relative to the accuracy of observed judgment of individual group members under varying conditions of task bias. Three types of judgment tasks were employed: (1) single-estimate, holistic tasks, (2) multiple-estimate, ranking tasks, and (3) multi-cue, decomposed tasks. Findings indicated across all task types that a large percentage of judgment-making group estimates formed strictly by computing the arithmetic mean of individual estimates performed better than their most capable members when a judgment task evoked little or no bias, a result particularly pronounced for ranking tasks. When the task was more greatly bias-evoking, a large percentage of parallel groups performed more poorly than average (or median) members, again a pattern more starkly evident for ranking tasks. These results suggest that the extent to which a judgment task evokes bias in a population of prospective group members is an important explanatory variable deserving much greater attention in the study of group performance. For example, an assertion about the efficacy of a particular group intervention based on a reliable demonstration of group performance as accurate as the most capable members may be unfounded when a task evokes no bias, since the baseline standard under such conditions should be much higher. By selecting tasks and populations that jointly produced highly biased estimates, researchers can lower the performance floor enough to detect (with reasonably small samples of groups) experimental effects should they occur.     

Confidence depends on level of aggregation

The credible intervals that people set around their point estimates are typically too narrow (cf. Lichtenstein, Fischhoff, & Phillips, 1982). That is, a set of many such intervals does not contain the actual values of the criterion variables as often as it should given the probability assigned to this event for each estimate. The typical interpretation of such data is that people are overconfident about the accuracy of their judgments. This paper presents data from two studies showing the typical levels of overconfidence for individual estimates of unknown quantities. However, data from the same subjects on a different measure of confidence for the same items, their own global assessment for the set of multiple estimates as a whole, showed significantly lower levels of confidence and overconfidence than their average individual assessment for items in the set. It is argued that the event and global assessments of judgment quality are fundamentally different and are affected by unique psychological processes. Finally, we discuss the implications of a difference between confidence in single and multiple estimates for confidence research and theory.     

A Theoretical Framework for Analysing the Notion of Relative Importance of Criteria

Multiple-criteria decision aid almost always requires the use of weights, importance coefficients or even a hierarchy of criteria, veto thresholds, etc. These are importance parameters that are used to differentiate the role devoted to each criterion in the construction of comprehensive preferences. Many researchers have studied the problem of how to assign values to such parameters, but few of them have tried to analyse in detail what underlies the notion of importance of criteria and to give a clear formal definition of it. In this paper our purpose is to define a theoretical framework so as to analyse the notion of the importance of criteria under very general conditions. Within this framework it clearly appears that the importance of criteria is taken into account in very different ways in various aggregation procedures. This framework also allows us to shed new light on fundamental questions such as: Under what conditions is it possible to state that one criterion is more important than another? Are importance parameters of the various aggregation procedures dependent on or independent of the encoding of criteria? What are the links between the two concepts of the importance of criteria and the compensatoriness of preferences? This theoretical framework seems to us sufficiently general to ground further research in order to define theoretically valid elicitation methods for importance parameters.     

Moderating Loss Aversion: Loss Aversion Has Moderators,But Reports of its Death are Greatly Exaggerated

Loss aversion, the principle that losses impact decision making more than equivalent gains, is a fundamental idea in consumer behavior and decision making, though its existence has recently been called into question. Across five unique samples (N_total = 17,720), we tested several moderators of loss aversion, which supported a preference construction account. Across studies, more domain knowledge and experience were associated with lower loss aversion, though people of all knowledge and experience levels were loss averse. Among car buyers, those who knew more about a particular car attribute (e.g., fuel economy) were less loss averse for that attribute but not other attributes (e.g., comfort), consistent with the idea that people with less attribute knowledge are more likely to construct preferences, thereby increasing loss aversion. Additionally, older consumers were more loss averse across different loss aversion measures and studies. We discuss implications for several accounts of loss aversion, including accounts rooted in status quo bias, emotion, or ownership. In addition to discovering loss aversion moderators, we cast doubt on recent claims that loss aversion is a fallacy or is fully explained by status quo bias, risk aversion, or the educated laboratory samples often used to study loss aversion.     

Persistence during extinction: Are judgments of persistence affected by contingency information?

The partial reinforcement extinction effect (PREE), defined as increased behavioral persistence following intermittent reward, is considered an important outcome of instrumental learning contingencies, both inside and outside the laboratory. Since adults have a rich experience with situations in which desired outcomes depend on instrumental responding, we asked whether that experience affects judgments of persistence when relevant contingency information is manipulated. Subjects read simple scenarios with information about behaviors generated by high vs. low reward rate, and then judged the resultant persistence of these behaviors under no-reward conditions. Studies 1 and 2 found no evidence that persistence judgments were affected by contingency information in naive subjects. Studies 3 and 4 compared groups with and without explicit knowledge about behavioral psychology and thus tested possible effects of that knowledge for persistence more directly. Judgments in naive subjects were not reliably influenced by reward rate information, but subjects possessing expert knowledge demonstrated judgments that were reliably affected by contingency information. The results indicate that people do not generate generalized knowledge from normal experience with occasional vs. regular reward. Possible explanations and implications of these findings are discussed.     

Logic and Aggregation

Paraconsistent logic is an area of philosophical logic that has yet to find acceptance from a wider audience. The area remains, in a word, disreputable. In this essay, we try to reassure potential consumers that it is not necessary to become a radical in order to use paraconsistent logic. According to the radicals, the problem is the absurd classical account of contradiction: Classically inconsistent sets explode only because bourgeois classical semantics holds, in the face of overwhelming evidence to the contrary, that both A and A cannot simultaneously be true! We suggest (more modestly) that there is, at least sometimes, something else worth preserving, even in an inconsistent, unsatisfiable premise set. In this paper we present, in a new guise, a very general version of this preservationist approach to paraconsistency.     

Preliminary investigation of the Kitchen Picture Test (KPT): A new screening test of practical judgment for older adults

Judgment is widely recognized as an important executive function, and deficits in judgment can lead to health risks, safety concerns, and hospitalizations. Surprisingly, relatively few tests of judgment have been developed specifically for older adults—a population particularly vulnerable to executive and functional declines. The Kitchen Picture Test (KPT) is a new screening measure of practical judgment. In two independent studies (Study 1, N = 99 nursing home patients; Study 2, N = 163 nursing home and assisted living patients), psychometric analyses confirmed strong evidence for reliability, construct validity, and predictive validity. A receiver operating characteristic (ROC) curve was calculated from sensitivity and 1-specificity values for diagnoses of dementia versus no dementia. A KPT cut score can be used for identifying persons to be referred to appropriate health-care professionals who have specific expertise in the evaluation and treatment of cognitive impairment.     

Análisis factorial de ítems de respuesta forzada: una revisión y un ejemplo

Forced-choice tests are widely used in order to reduce the impact of different response set biases typically associated to psychological tests (e.g. acquiescence or social desirability). However, these tests produce ipsative data which have undesirable properties, thereby making an inappropriate application of classical factor analysis techniques for psychometric evaluation commonly used by researchers. This paper explains the analytical properties of forced-choice tests, along with an example that illustrates how these properties have an impact on the application of conventional statistical techniques and produce improper results. Additionally, one of the current proposals is presented in order to analyze these data based on the comparative judgment model by Thurstone, along with the results of a simulation study which illustrates its implementation and effectiveness in recovering the original factor structure.     

The vision heuristic: Judging music ensembles by sight alone

Team effectiveness and group performance are often defined by standards set by domain experts. Professional musicians consistently report that sound output is the most important standard for evaluating the quality of group performance in the domain of music. However, across six studies, visual information dominated rapid judgments of group performance. Participants (1062 experts and novices) were able to select the actual winners of live ensemble competitions and distinguish top-ranked orchestras from non-ranked orchestras based on 6-s silent video recordings yet were unable to do so from sound recordings or recordings with both video and sound. These findings suggest that judgments of group performance in the domain of music are driven at least in part by visual cues about group dynamics and leadership.