Coherence reasoning and reliability: a defense of the Shogenji measure

A measure of coherence is said to be reliability conducive if and only if a higher degree of coherence (as measured) results in a higher likelihood that the witnesses are reliable. Recently, it has been proved that several coherence measures proposed in the literature are reliability conducive in a restricted scenario (Olsson and Schubert 2007, Synthese 157:297?C308). My aim is to investigate which coherence measures turn out to be reliability conducive in the more general scenario where it is any finite number of witnesses that give equivalent reports. It is shown that only the so-called Shogenji measure is reliability conducive in this scenario. I take that to be an argument for the Shogenji measure being a fruitful explication of coherence.     

Is coherence conducive to reliability?

A measure of coherence is said to be reliability conducive if and only if a higher degree of coherence (as measured) of a set of testimonies implies a higher probability that the witnesses are reliable. Recently, it has been proved that the Shogenji measure of coherence is reliability conducive in restricted scenarios (e.g., Olsson and Schubert, Synthese, 157:297?C308, 2007). In this article, I investigate whether the Shogenji measure, or any other coherence measure, is reliability conducive in general. An impossibility theorem is proved to the effect that this is not the case. I conclude that coherence is not reliability conducive.     

Reliability conducive measures of coherence

A measure of coherence is said to be truth conducive if and only if a higher degree of coherence (as measured) results in a higher likelihood of truth. Recent impossibility results strongly indicate that there are no (non-trivial) probabilistic coherence measures that are truth conducive. Indeed, this holds even if truth conduciveness is understood in a weak ceteris paribus sense (Bovens & Hartmann, 2003, Bayesian epistemology. New York, Oxford: Oxford University Press; Olsson, 2005, Against coherence: Truth probability and justification. Oxford: Oxford University Press). This raises the problem of how coherence could nonetheless be an epistemically important property. Our proposal is that coherence may be linked in a certain way to reliability. We define a measure of coherence to be reliability conducive if and only if a higher degree of coherence (as measured) results in a higher probability that the information sources are reliable. Restricting ourselves to the most basic case, we investigate which coherence measures in the literature are reliability conducive. It turns out that, while a number of measures fail to be reliability conducive, except possibly in a trivial and uninteresting sense, Shogenji’s measure and several measures generated by Douven and Meijs’s recipe are notable exceptions to this rule.     

Coherence,striking agreement,and reliability

Striving for a probabilistic explication of coherence, scholars proposed a distinction between agreement and striking agreement. In this paper I argue that only the former should be considered a genuine concept of coherence. In a second step the relation between coherence and reliability is assessed. I show that it is possible to concur with common intuitions regarding the impact of coherence on reliability in various types of witness scenarios by means of an agreement measure of coherence. Highlighting the need to separate the impact of coherence and specificity on reliability it is finally shown that a recently proposed vindication of the Shogenji measure qua measure of coherence vanishes.     

Confirmation and justification. A commentary on Shogenji’s measure

So far no known measure of confirmation of a hypothesis by evidence has satisfied a minimal requirement concerning thresholds of acceptance. In contrast, Shogenji’s new measure of justification (Shogenji, Synthese, this number 2009) does the trick. As we show, it is ordinally equivalent to the most general measure which satisfies this requirement. We further demonstrate that this general measure resolves the problem of the irrelevant conjunction. Finally, we spell out some implications of the general measure for the Conjunction Effect; in particular we give an example in which the effect occurs in a larger domain, according to Shogenji justification, than Carnap’s measure of confirmation would have led one to expect.     

Equivalent testimonies as a touchstone of coherence measures

Over the past years, a number of probabilistic measures of coherence have been proposed. As shown in the paper, however, many of them do not conform to the intuitition that equivalent testimonies are highly coherent, regardless of their prior probability.     

EXPLAINING THE LIMITS OF OLSSON'S IMPOSSIBILITY RESULT

In his groundbreaking book, Against Coherence (2005), Erik Olsson presents an ingenious impossibility theorem that appears to show that there is no informative relationship between probabilistic measures of coherence and higher likelihood of truth. Although Olsson's result provides an important insight into probabilistic models of epistemological coherence, the scope of his negative result is more limited than generally appreciated. The key issue is the role conditional independence conditions play within the witness testimony model Olsson uses to establish his result. Olsson maintains that his witness model yields charitable ceteris paribus conditions for any theory of probabilistic coherence. Not so. In fact, Olsson's model, like Bayesian witness models in general, selects a peculiar class of models that are in no way representative of the range of options available to coherence theorists. Recent positive results suggest that there is a way to develop a formal theory of coherence after all. Further, although Bayesian witness models are not conducive to the truth, they are conducive to reliability.     

The Impossibility of Coherence

There is an emerging consensus in the literature on probabilistic coherence that such coherence cannot be truth conducive unless the information sources providing the cohering information are individually credible and collectively independent. Furthermore, coherence can at best be truth conducive in a ceteris paribus sense. Bovens and Hartmann have argued that there cannot be any measure of coherence that is truth conducive even in this very weak sense. In this paper, I give an alternative impossibility proof. I provide a relatively detailed comparison of the two results, which turn out to be logically unrelated, and argue that my result answers a question raised by Bovens and Hartmann’s study. Finally, I discuss the epistemological ramifications of these findings and try to make plausible that a shift to an explanatory framework such as Thagard’s is unlikely to turn the impossibility into a possibility.     

Probabilistic Measures of Coherence and the Problem of Belief Individuation

Coherentism in epistemology has long suffered from lack of formal and quantitative explication of the notion of coherence. One might hope that probabilistic accounts of coherence such as those proposed by Lewis, Shogenji, Olsson, Fitelson, and Bovens and Hartmann will finally help solve this problem. This paper shows, however, that those accounts have a serious common problem: the problem of belief individuation. The coherence degree that each of the accounts assigns to an information set (or the verdict it gives as to whether the set is coherent tout court) depends on how beliefs (or propositions) that represent the set are individuated. Indeed, logically equivalent belief sets that represent the same information set can be given drastically different degrees of coherence. This feature clashes with our natural and reasonable expectation that the coherence degree of a belief set does not change unless the believer adds essentially new information to the set or drops old information from it; or, to put it simply, that the believer cannot raise or lower the degree of coherence by purely logical reasoning. None of the accounts in question can adequately deal with coherence once logical inferences get into the picture. Toward the end of the paper, another notion of coherence that takes into account not only the contents but also the origins (or sources) of the relevant beliefs is considered. It is argued that this notion of coherence is of dubious significance, and that it does not help solve the problem of belief individuation.     

Reliability of a Longitudinal Sequence of Scale Ratings

Reliability captures the influence of error on a measurement and, in the classical setting, is defined as one minus the ratio of the error variance to the total variance. Laenen, Alonso, and Molenberghs (Psychometrika 73:443–448, 2007) proposed an axiomatic definition of reliability and introduced the R _T coefficient, a measure of reliability extending the classical approach to a more general longitudinal scenario. The R _T coefficient can be interpreted as the average reliability over different time points and can also be calculated for each time point separately. In this paper, we introduce a new and complementary measure, the so-called R _Λ , which implies a new way of thinking about reliability. In a longitudinal context, each measurement brings additional knowledge and leads to more reliable information. The R _Λ captures this intuitive idea and expresses the reliability of the entire longitudinal sequence, in contrast to an average or occasion-specific measure. We study the measure’s properties using both theoretical arguments and simulations, establish its connections with previous proposals, and elucidate its performance in a real case study. The authors are grateful to J&J PRD for kind permission to use their data. We gratefully acknowledge support from Belgian IUAP/PAI network “Statistical Techniques and Modeling for Complex Substantive Questions with Complex Data.”     

Advances in the assessment of cognitive skills using computer-based measurement

The impact of technology in the field of intellectual assessment has, for the most part, been limited to computerized administration and scoring. Anderson’s (2001) theory of intelligence suggests that performance on traditional IQ measures is acquired via two main routes, thinking and dedicated processing systems known as modules. Empirical data used to support this relationship between the speed of basic processing and intellectual functioning have been evidenced primarily by correlations between measures of general intelligence and measures of inspection time (IT). These IT measures allow individuals to make a forced choice discrimination task without a motor component. Because only the time used to cognitively solve the problem is recorded, these responses typically occur in milliseconds. Many theorists (e.g., Burns & Nettelbeck, 2002; Deary, 2000; Jensen, 2006) consider IT to be a more “pure” measure of intelligence, because the influences of verbal skills, memory, and socialization are minimized and results are therefore considered to be more culture-fair. Until relatively recently, IT measurement was restricted to complex and expensive specialized laboratory equipment. This article describes the theoretical background and developmental process of a computer-based IT measure that is easily adaptable to accommodate the needs of the researcher.     

Ways in which coherence is confirmation conducive

Recent works in epistemology show that the claim that coherence is truth conducive – in the sense that, given suitable ceteris paribus conditions, more coherent sets of statements are always more probable – is dubious and possibly false. From this, it does not follows that coherence is a useless notion in epistemology and philosophy of science. Dietrich and Moretti (Philosophy of science 72(3): 403–424, 2005) have proposed a formal of account of how coherence is confirmation conducive—that is, of how the coherence of a set of statements facilitates the confirmation of such statements. This account is grounded in two confirmation transmission properties that are satisfied by some of the measures of coherence recently proposed in the literature. These properties explicate everyday and scientific uses of coherence. In his paper, I review the main findings of Dietrich and Moretti (2005) and define two evidence-gathering properties that are satisfied by the same measures of coherence and constitute further ways in which coherence is confirmation conducive. At least one of these properties vindicates important applications of the notion of coherence in everyday life and in science.     

Problems with Priors in Probabilistic Measures of Coherence

Two of the probabilistic measures of coherence discussed in this paper take probabilistic dependence into account and so depend on prior probabilities in a fundamental way. An example is given which suggests that this prior-dependence can lead to potential problems. Another coherence measure is shown to be independent of prior probabilities in a clearly defined sense and consequently is able to avoid such problems. The issue of prior-dependence is linked to the fact that the first two measures can be understood as measures of coherence as striking agreement, while the third measure represents coherence as agreement. Thus, prior (in)dependence can be used to distinguish different conceptions of coherence.     

Why There Cannot be a Single Probabilistic Measure of Coherence

Bayesian Coherence Theory of Justification or, for short, Bayesian Coherentism, is characterized by two theses, viz. (i) that our degree of confidence in the content of a set of propositions is positively affected by the coherence of the set, and (ii) that coherence can be characterized in probabilistic terms. There has been a longstanding question of how to construct a measure of coherence. We will show that Bayesian Coherentism cannot rest on a single measure of coherence, but requires a vector whose components exhaustively characterize the coherence properties of the set. Our degree of confidence in the content of the information set is a function of the reliability of the sources and the components of the coherence vector. The components of this coherence vector are weakly but not strongly separable, which blocks the construction of a single coherence measure.     

Design,data, and theory regarding a digital hand inclinometer: a portable device for studying slant perception

Palm boards are often used as a nonverbal measure in human slant perception studies. It was recently found that palm boards are biased and relatively insensitive measures, and that an unrestricted hand gesture provides a more sensitive response (Durgin, Hajnal, Li, Tonge, & Stigliani, Acta Psychologica, 134, 182-197, 2010a). In this article, we describe an original design for a portable lightweight digital device for measuring hand orientation. This device is microcontroller-based and uses a micro inclinometer chip as its inclination sensor. The parts are fairly inexpensive. This device, used to measure hand orientation, provides a sensitive nonverbal method for studying slant perception, which can be used in both indoor and outdoor environments. We present data comparing the use of a free hand to palm-board and verbal measures for surfaces within reach and explain how to interpret free-hand measures for outdoor hills.     

A Psychometric Review of Measures of Irrational Beliefs: Implications for Psychotherapy

In this study, measures of irrational beliefs based on the principles of Rational Emotive Behavior Therapy (REBT) were reviewed and contrasted based on psychometric characteristics and quality of guidelines for interpretation. The use of measures of irrational beliefs is an important component of the assessment of presenting problems in psychotherapy and may inform clinical decision making. Additionally, these measures are a helpful way to identify change in irrational beliefs and to measure the effectiveness of specific interventions to facilitate such change. While there are a number of measures of beliefs, there is considerable variability in their psychometric properties and utility for assessment. The majority of the tests are not produced by a publishing company, nor do they provide test manuals. The reliability and validity evidence presented in the publications varies considerably as do the size and quality of standardization samples, which impacts upon the ability to draw normative conclusions. Analyses demonstrated that most measures of irrational beliefs do not provide the evidence needed to adequately address the Standards for Educational and Psychological Testing (Standards; American Educational Research Association [AERA], American Psychological Association [APA], National Council on Measurement in Education [NCME] 1999). Discussion of the implications of this review for the development of future measures of irrational thinking by researchers as well as recommendations in the test selection process for an REBT practitioner is offered.     

Plausibilistic coherence

Why should coherence be an epistemic desideratum? One response is that coherence is truth-conducive: mutually coherent propositions are more likely to be true, ceteris paribus, than mutually incoherent ones. But some sets of propositions are more coherent, while others are less so. How could coherence be measured? Probabilistic measures of coherence exist; some are identical to probabilistic measures of confirmation, while others are extensions of such measures. Probabilistic measures of coherence are fine when applicable, but many situations are so information-poor that the requisite probabilities cannot be determined. To measure coherence in these cognitively impoverished situations, this article proposes that the discussion be broadened to include plausibilistic measures of coherence. It shows how plausibilistic measures of coherence can be defined using plausibilistic measures of confirmation. It then illustrates how plausibilisic coherence can be measured in situations where probabilistic coherence cannot be determined. The coherence values obtained through the use of plausibilistic measures are often, though not always, comparable. The article also shows that coherence can be instantiated on different levels, one of which permits connections to inductive strength and deductive validity.     

Belief Change in Branching Time: AGM-consistency and Iterated Revision

We study belief change in the branching-time structures introduced in Bonanno (Artif Intell 171:144–160, 2007). First, we identify a property of branching-time frames that is equivalent (when the set of states is finite) to AGM-consistency, which is defined as follows. A frame is AGM-consistent if the partial belief revision function associated with an arbitrary state-instant pair and an arbitrary model based on that frame can be extended to a full belief revision function that satisfies the AGM postulates. Second, we provide a set of modal axioms that characterize the class of AGM-consistent frames within the modal logic introduced in Bonanno (Artif Intell 171:144–160, 2007). Third, we introduce a generalization of AGM belief revision functions that allows a clear statement of principles of iterated belief revision and discuss iterated revision both semantically and syntactically.     

Reliability and expected loss: A unifying principle

We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the proportional reduction in loss (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.Bruce Cooil is an Associate Professor of Statistics, and Roland T. Rust is a Professor and area head for Marketing. The authors thank three anonymous reviewers and an Associate Editor for their helpful comments and suggestions. This work was supported in part by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.