Using Ordinal Data to Estimate Cardinal Values

We investigate methods developed in multiple criteria decision‐making that use ordinal information to estimate numerical values. Such methods can be used to estimate attribute weights, attribute values, or event probabilities given ranks or partial ranks. We first review related studies and then develop a generalized rank‐sum (GRS) approach in which we provide a derivation of the rank‐sum approach that had been previously proposed. The GRS approach allows for incorporating the concept of degree of importance (or, difference in likelihood with respect to probabilities and difference in value for attribute values), information that most other rank‐based formulas do not utilize. We then present simulation results comparing the GRS method with other rank‐based formulas such as the rank order centroid method and comparing the GRS methods using as many as three levels of importance (i.e., GRS‐3) with Simos' procedure (which can also incorporate degree of importance). To our surprise, our results show that the incorporation of additional information (i.e., the degree of the importance), both GRS‐3 and Simos' procedure, did not result in better performance than rank order centroid or GRS. Further research is needed to investigate the modelling of such extra information. We also explore the scenario when a decision‐maker has indifference judgments and cannot provide a complete rank order. Copyright © 2014 John Wiley & Sons, Ltd.     

SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement

This paper presents two approximate methods for multiattribute utility measurement, SMARTS and SMARTER, each based on an elicitation procedure for weights. Both correct an error in SMART, originally proposed by Edwards in 1977, and in addition SMARTER is simpler to use. SMARTS uses linear approximations to single-dimension utility functions, an additive aggregation model, and swing weights. The paper proposes tests for the usability of these approximations. SMARTER, based on a formally justifiable weighting procedure developed by Barron and Barrett, uses the same procedures as SMARTS except that it omits the second of two elicitation steps in swing weights, substituting calculations based on ranks. It can be shown to perform about 98% as well as SMARTS does, without requiring any difficult judgments from elicitees.     

The Effectiveness of Partial Information about Attribute Weights for Ranking Alternatives in Multiattribute Decision Making

Weighted additive evaluation functions are widely used to rank alternatives in decision making under certainty with multiple evaluation attributes. Some researchers have suggested that approximate attribute weights may be adequate to accurately rank alternatives. Use of approximate weights would simplify decision analysis since detailed elicitation of weights can be time consuming and controversial. This article investigates the degree to which partial information about the relative magnitudes of attribute weights is sufficient to rank alternatives as a function of the number of decision alternatives, the number of attributes, and the number of allowed levels for each attribute. A simulation analysis, as well as a reanalysis of an actual application, shows that partial information about weights is often not sufficient to determine the most preferred alternative for realistic decision problems. Hence, approximation procedures for specifying weights may lead to errors. However, our work also shows that a simple analysis procedure can be used to accurately determine whether partial information about weights is adequate to correctly specify the most preferred alternative. This procedure can be useful for identifying situations in which detailed elicitation of weights is not needed.     

Selecting a best multiattribute alternative with partial information about attribute weights

Use of approximate weights would greatly simplify decision analysis under certainty since detailed weight elicitation could be avoided. This paper examines the degree to which rank order information about weights can be used to identify a best alternative, or falling uniqueness prescribes an easily implemented rule for selecting a ‘best’ alternative. The prescribed rule uses as weights the centroid of the feasible region defined by the rank order information. In conjunction with the rule, the value of the rank order information can be determined using an ‘expected gain from weight precision’ (EGWP) measure, analogous to ‘expected value of perfect information’ in decision analysis under uncertainty.     

Rank disagreement: A comparison of multi-criteria methodologies

A number of multi-criteria decision support techniques have emerged in recent years that use varying computational approaches to arrive at the most desirable solution and thereby ‘recommend’ a course of action. Decision makers who use the results of this analytic work should be assured that the computational schemes used by their supporting analysts or decision support software produce the appropriate solutions. We conducted a series of simulation experiments that compared the top-ranked options resulting from the computational algorithms that support Multi-Attribute Value Theory (MAVT) and three methods that are reported in the literature that allow rank reversals, the change in rank order of two options when an unrelated option is added or deleted from the analysis: the Analytical Hierarchy Process (AHP), Percentaging and the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). We also included a Fuzzy algorithm proposed by Yager to gauge its consistency with the other algorithms, even though it is not subject to rank reversals. These experiments demonstrated that the MAVT and AHP techniques, when provided with the same decision outcome data, very often identify the same alternative as ‘best’. The other techniques are noticeably less consistent with MAVT, the Fuzzy algorithm being the least consistent. The situations under which the most frequent and significant differences occurred were dependent upon the method. The results of our experiments indicate that other issues (e.g. the processes used for problem structuring and the elicitation of value weights) are likely to be of greater significance to problem outcome (based on our experience) than the choice between the computational algorithms of MAVT and AHP. The results cause us to be concerned about the use of the other methods.     

COLOR CONSTANCY AND THE METHOD OF CATEGORICAL JUDGMENT

Some completion is given on a method for measuring color constancy criticized by Pietarinen (1966). Pietarinen's assumption, that our rating categories did not represent a single rank order, is untenable. A closer examination of the stimuli of the original study shows that the color sets included in, e.g., the Ostwald system allow at least for ordinal ranking of the responses, which was the main purpose of our investigation.     

Using a Difficulty‐Anchored Rating Scale in Performing Angoff Ratings

While the Angoff (1971) is a commonly used cut score method, critics ( Berk, 1996; Impara & Plake, 1997 ) argue the Angoff places too‐high cognitive demands on raters. In response to criticisms of the Angoff, a number of modifications to the method have been proposed. Some suggested Angoff modifications include using an iterative rating process, presenting judges with normative data about item performance, revising the rating judgment into a Yes/No decision, assigning relative weights to dimensions within a test, and using item response theory in setting cut scores. In this study, subject matter expert raters were provided with a ‘difficulty anchored’ rating scale to use while making Angoff ratings; this scale can be viewed as a variation of the Angoff normative data modification. The rating scale presented test items having known p‐values as anchors, and served as a simple means of providing normative information to guide the Angoff rating process. Results are discussed regarding reliability of the mean Angoff rating (.73) and the correlation of mean Angoff ratings with item difficulty (observed r ranges from .65 to .73).     

A Psychometric Analysis of the “Divide and Conquer” Principle in Multicriteria Decision Making

The principle of “Divide and Conquer” (DAC) suggests that: (1) complex decision problems should be decomposed into smaller, more manageable parts and (2) these smaller parts should be logically aggregated to derive an overall value for each alternative. Typically, decompositional procedures have been compared to holistic evaluations that require decision makers to simultaneously consider all of the relevant attributes in the evaluation of the objects under consideration. These comparisons between decompositional and holistic judgments have primarily used a variant of Multiattribute Utility Theory (MAUT).We presented a general experimental framework that allows for a more extensive assessment of the DAC principle, as well as the effects of decision complexity on both holistic and decompositional procedures. We illustrate this approach with a study that uses the Simple Multiattribute Rating Technique with Swing Weights (SMARTS; Edwards & Barron, 1994) and the Analytic Hierarchy Process (AHP; Saaty, 1980). We report data comparing the convergent validity (e.g the agreement between decompositional and holistic strategies) and the temporal stability for decompositional and holistic judgments on a variety of dependent measures. Decision complexity did not significantly affect the correspondence between decompositional and holistic judgments for both SMARTS and AHP judgments. Results from an ordinal measure of temporal stability indicated the DAC principle was violated for the AHP judgments. For a linear measure of temporal stability, trends in the data indicated that the predicted effects of decision complexity on the DAC principle was violated for the SMARTS judgments.     

On the Death of SMART and the Birth of GRAPA

Should SMART be dead—as Edwards and Barron affirmed in 1994? This article is a comparative study of the methods SMART and SMARTS, in relation to the eliciting of weights. It compares the weights and final rankings of alternatives. To these two techniques has been added a graphical version of the PA (GRAPA), which is simpler and more intuitive. My study is designed to orient nonprofessional decision-makers with regard to which technique to employ, when what is sought is simplicity and ease of use without loss of rigor.     

Do Ranks Suffice? A Comparison of Alternative Weighting Approaches in Value Elicitation

We compared both attribute weights and overall evaluations for students′ preferences among apartments described to them in terms of nine independent attributes. Methods used for eliciting attribute weights were (a) 7-point scales; (b) value hierarchy; (c) swing weights; and two methods using importance rankings only: (d) rank order centroid and (e) rank sum weights. Multiple linear regression was also used to infer attribute weights. Test–retest reliability of overall evaluations was found to be modest. Evaluation models based on all five weight elicitation methods were superior to an equal weights model, with rank order centroid weights modestly superior to other methods.     

Stochastic pairwise comparative judgements and direct ratings of alternatives in the REMBRANDT system

The REMBRANDT system for multicriteria decision analysis consists of both the multiplicative variant of the AHP (which employs a method of pairwise comparative judgements by a decision maker to arrive at final impact scores for the alternatives under consideration) and SMART, the simple multiattribute rating technique (which utilizes direct rating of alternatives to achieve final impact scores). This paper examines the effect of imprecision or uncertainty in the decision maker's pairwise judgements or ratings of alternatives by expressing each pairwise judgement or rating as a probability distribution, and the structure of REMBRANDT's component models is exploited to derive interval judgements or interval ratings of the alternatives’ final impact scores. These interval judgements or interval ratings can be used to determine the probability of rank reversal amongst alternatives, i.e. to assess the stability of the final impact score vector. © 1998 John Wiley & Sons, Ltd.     

Aggregating preference rankings: An AHP-based approach

The problem considered here involves the selection of an alternative by a group of decision makers in a context that involves the aggregating of votes from a preferential ballot. A popular approach for solving such problems is the well-known weighting problem formulation. However, with such a formulation two issues become of critical importance: (a) the weights should reflect the decision-making group's belief on the importance of each rank position and (b) the weights should accord each alternative a fair (if not its best) assessment. Many researchers have studied this problem and proposed various solution strategies. None of these strategies, however, appears to have adequately addressed both issues. The objective of this research is to develop a group decision-making model that solves the problem in a manner that adequately addresses the two critical issues identified, as well as allows for sensitivity analysis and is fairly robust with respect to the various paradoxes identified in voting systems.     

Time complexity and consistency issues in using the AHP for making group decisions

Two different ways of using the AHP in making group decisions are compared and evaluated. The first method combines different experts’ opinions before applying an eigenvalue method to obtain final weights for decision alternatives. The second, in contrast, derives each expert's rating for the decision alternatives before combining them. Both methods take into account the relative importance of different experts in making decisions. Comparison and evaluation of these two methods are made by using two criteria: time complexity and consistency indices. A numerical example is provided to illustrate the use of these two methods, and results of a mathematical simulation are presented for comparing the time complexity in different-sized problems. © 1998 John Wiley & Sons, Ltd.     

Power Relations and Group Aggregation in the Multiplicative AHP and SMART

The relative power of the members in a group of decision makers can be incorporated in the multiplicative AHP via power coefficients in the logarithmic least squares whereby we analyse the pairwise comparison matrices. When each decision maker judges every pair of alternatives under each of the criteria, aggregation over the criteria and over the decision makers proceeds via a sequence of geometric-mean calculations which can be carried out in any order, at least with predetermined criterion weights and power coefficients. Hence, since we preserve the rank order of the alternatives, we avoid a deficiency of the original AHP. We also consider SMART, an additive method which is logarithmically related to the multiplicative AHP so that power relations can easily be incorporated in it. Finally, in order to illustrate the proposed model, we analyse a generalized version of the well-known example of Belton and Gear as well as the power relations between member countries of the European Community. © 1997 by John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 6 : 155–165 (1997) No. of Figures: 0. No. of Tables: 6. No. of References: 23.     

Attribute weighting methods and decision quality in the presence of response error: a simulation study

This paper uses a simulation approach to investigate how different attribute weighting techniques affect the quality of decisions based on multiattribute value models. The weighting methods considered include equal weighting of all attributes, two methods for using judgments about the rank ordering of weights, and a method for using judgments about the ratios of weights. The question addressed is: How well does each method perform when based on judgments of attribute weights that are unbiased but subject to random error? To address this question, we employ simulation methods. The simulation results indicate that ratio weights were either better than rank order weights (when error in the ratio weights was small or moderate) or tied with them (when error was large). Both ratio weights and rank order weights were substantially superior to the equal weights method in all cases studied. Our findings suggest that it will usually be worth the extra time and effort required to assess ratio weights. In cases where the extra time or effort required is too great, rank order weights will usually give a good approximation to the true weights. Comparisons of the two rank-order weighting methods favored the rank-order-centroid method over the rank-sum method. © 1998 John Wiley & Sons, Ltd.     

Statistical significance tests for binormal ROC curves

Statistical significance tests are derived and evaluated for measuring apparent differences between an obtained and an expected binormal ROC curve, between two independent binormal ROC curves, and among groups of independent binormal ROC curves. A binormal ROC curve is described by two parameters which represent the spread of the means and the ratio of the standard deviations of the two underlying Gaussian decision variable distributions. To test the significance of apparent differences between or among ROC curves, approximate χ² statistics for each of the three tests were constructed from maximum likelihood estimates of the two parameters defining the binormal ROC curve. The performance of each test statistic was evaluated by simulating five-category rating scale data with equal numbers of noise and signal-plus-noise trials (set at 50, 250, and 500) for each of three typical ROC curves. For the significance test involving only one ROC curve, rating scale data were generated from the chance diagonal of the ROC space also. Although test performance was found to be somewhat dependent on the number of trials and on the location of the ROC curve in the ROC space, comparisons of the obtained and expected fractions of (falsely) significant results at various α levels showed the proposed statistical significance tests to be reliable under practical experimental conditions.     

Multiattribute Choice for the Lazy Decision Maker: Let the Alternatives Decide!

In this article we present a method of multi-attribute choice, based on an application of linear programming called Data Envelopment Analysis (DEA). The first part of the method, which is straightforward DEA, can be thought of as an idealized process of self-evaluation in which each alternative weights the attributes in order to maximize its own desirability relative to the other alternatives. These weights are taken as defining the preferences of a fragment of the market. The second step is to use each alternative′s optimal weights to reconstruct the entire market and thus to infer the preferences of an average decision maker (DM) who needs to choose from among these alternatives. We show how this process is equivalent to an idealized peer evaluation; each alternative applies its own DEA-derived best weights to each of the other alternatives (alternative cross-evaluation, or AXE), then the average of the cross-evaluations that get placed on an alternative is taken as an index of its overall desirability (i.e., we are "letting the alternatives decide"). We use a large data set to examine the workings of the method and to compare our results with the published results of the data set′s compiler. AXE is also able to make use of partial information about the ordering of importance of the attributes specific to a particular DM. Taking an ecological perspective we argue that the method is sensitive to cues already in the data and uses them to amplify helpful bias and attenuate unhelpful bias. We also show how different kinds sensitivity analysis may be performed to check the robustness of the results.     

ROCs in Eyewitness Identification: Instructions versus Confidence Ratings

From the perspective of signal detection theory, different lineup instructions may induce different levels of response bias. If so, then collecting correct and false identification rates across different instructional conditions will trace out the receiver operating characteristic (ROC)—the same ROC that, theoretically, could also be traced out from a single instruction condition in which each eyewitness decision is accompanied by a confidence rating. We tested whether the two approaches do in fact yield the same ROC. Participants were assigned to a confidence rating condition or to an instructional biasing condition (liberal, neutral, unbiased, or conservative). After watching a video of a mock crime, participants were presented with instructions followed by a six‐person simultaneous photo lineup. The ROCs from both methods were similar, but they were not exactly the same. These findings have potentially important policy implications for how the legal system should go about controlling eyewitness response bias.Copyright © 2017 John Wiley & Sons, Ltd.     

Marginally Represented Patients and the Moral Authority of Surrogates

Abstract

Incapacitated adult patients are commonly divided into two groups for purposes of decision making; those with a surrogate and those without. Respectively, these groups are often referred to as represented and unrepresented, and the relative ethics of decision making between them raises two particular issues. The first issue involves the differential application of the best interests standard between groups. Second is the prevailing notion that representedness and unrepresentedness are categorical phenomena, though it is more aptly understood as a multidimensional and continuous variable based on relational moral authority. This paper examines the nature of representedness as it relates to ethical norms of surrogate decision making.     

Attribute‐based differentiation of alternatives

An intermediate step is introduced to the dialogue decision process for decision analysis. Alternatives are refined after they have been generated within a strategy table but before they are subject to more detailed evaluation. Two or more judges create a subjective mapping from alternatives to attributes that will later be mapped to criteria. In strategy tables, each of the alternative strategies consists of a coherent set of choices made across several decisions that are to be coordinated. These strategic alternatives are modified so as to increase their differentiation in the attribute space, rather than in the decision space alone. When criteria weights are unknown, the best alternative from the modified set may be superior to the best alternative from the original set. Furthermore, analysis of the resulting alternatives may yield a better mapping of the value response surface for the action space, in the sense that this mapping leads to eventual construction of a higher value alternative. Results are reported for a consulting engagement incorporating the proposed step. Copyright © 2003 John Wiley & Sons, Ltd.