Decisions as statistical evidence and Birnbaum's ‘confidence concept’

Conclusion To whatever extent the use of a behavioral, not an evidential, interpretation of decisions in the Lindley-Savage argument for Bayesian theory undermines its cogency as a criticism of typical standard practice, it also undermines the Neyman-Pearson theory as a support for typical standard practice. This leaves standard practice with far less theoretical support than Bayesian methods. It does nothing to resolve the anomalies and paradoxes of standard methods. (Similar statements apply to the common protestation that the models are not real anyway.) The appropriate interpretation of tests as evidence, if possible at all, is difficult and counterintuitive. Any attempt to support tests as more than rules of thumb is doomed to failure.     

An Examination of the Biblical Evidence for Open Theism

Open Theist theologians argue that their view of divine foreknowledge and providence is the correct interpretation of the Bible, and suggest some biblical evidence to support this claim. Among these theologians, Gregory A. Boyd’s case is the most systematic, and also the most comprehensive and rigorous. Taking into consideration (1) the main philosophical claims of Open Theism and its main rivals, namely Calvinism and Molinism, and (2) Open Theist theologians’ interpretative principles for the Bible, the biblical evidence for Open Theism suggested by Boyd can be classified into three groups according to their strength. I will argue that the first group of themes has no evidential value for Open Theism, since these themes can be interpreted just as plausibly from a Calvinist or Molinist perspective. By contrast, the second group of themes has some evidential value and thus constitutes prima facie evidence for Open Theism, since these themes make most sense under an Open Theist interpretation. However, these themes also make some sense on a Molinist reading. The third group of themes, I argue, has most evidential value for Open Theism, since it seems hard to reconcile these themes either with Calvinism or Molinism.     

Interpreting statistical testing: Process and propensity, not population and random sampling

The standard textbook treatment of conventional statistical tests assumes random sampling from a population and interprets the outcome of the statistical testing as being about a population. Problems with this interpretation include that (1) experimenters rarely make any attempt to randomly sample, (2) if random sampling occurred, conventional statistical tests would not precisely describe the population, and (3) experimenters do not use statistical testing to generalize to a population. The assumption of random sampling can be replaced with the assumption that scores were produced by a process. Rejecting the null hypothesis then leads to a conclusion about process, applying to only the subjects in the experiment (e.g., that some difference in the treatment of two groups caused the difference in average scores). This interpretation avoids the problems noted and fits how statistical testing is used in psychology.     

A practical solution to the pervasive problems of<Emphasis Type="Italic">p</Emphasis> values

In the field of psychology, the practice ofp value null-hypothesis testing is as widespread as ever. Despite this popularity, or perhaps because of it, most psychologists are not aware of the statistical peculiarities of thep value procedure. In particular,p values are based on data that were never observed, and these hypothetical data are themselves influenced by subjective intentions. Moreover,p values do not quantify statistical evidence. This article reviews thesep value problems and illustrates each problem with concrete examples. The three problems are familiar to statisticians but may be new to psychologists. A practical solution to thesep value problems is to adopt a model selection perspective and use the Bayesian information criterion (BIC) for statistical inference (Raftery, 1995). The BIC provides an approximation to a Bayesian hypothesis test, does not require the specification of priors, and can be easily calculated from SPSS output.     

THE COUNTERNULL VALUE OF AN EFFECT SIZE:

We introduce a new, readily computed statistic, the counternull value of an obtained effect size, which is the nonnull magnitude of effect size that is supported by exactly the same amount of evidence as supports the null value of the effect size In other words, if the counternull value were taken as the null hypothesis, the resulting p value would be the same as the obtained p value for the actual null hypothesis Reporting the counternull, in addition to the p value, virtually eliminates two common errors (a) equating failure to reject the null with the estimation of the effect size as equal to zero and (b) takmg the rejection of a null hypothesis on the basis of a significant p value to imply a scientifically important finding In many common situations with a one-degree-of-freedom effect size, the value of the counternull is simply twice the magnitude of the obtained effect size, but the counternull is defined in general, even with multidegree-of-freedom effect sizes, and therefore can be applied when a confidence interval cannot be The use of the counternull can be especially useful in meta-analyses when evaluating the scientific importance of summary effect sizes     

Studying, Practicing, and Mastering: A Test of the Model of Goal-Directed Behavior (MGB) in the Software Learning Domain

The present study tested the model of goal-directed behavior (MGB). The model proposes that behavioral intentions to perform instrumental behaviors are primarily motivated by desires to perform the acts. In turn, desires mediate the effects of attitudes, subjective norms, perceived control, and anticipated emotions on intentions. Construct validity for MGB variables is assessed, and the predictive utility of the MGB is compared with that of the theory of planned behavior (TPB). College students ( N = 102) provided measures for MGB and TPB variables while participating in a training program to use statistical software. We focused on two goal-related instrumental behaviors: studying handbooks and practicing with the package. The results show that the MGB accounts for a greater proportion of variance in intentions and instrumental behaviors than does TPB. Although desires mediate most of the effects of other predictors on intentions to perform the instrumental behaviors, it is proposed that when the behavior is normatively relevant, or when self-efficacy appraisals play a major role, subjective norms and perceived behavioral control may directly affect intentions.     

Misunderstandings and omissions in textbook accounts of effect sizes

There have been frequent attempts in psychology to reduce the reliance on null hypothesis significance testing (NHST) as the criterion for establishing the importance of results. Many authorities now recommend the reporting of effect sizes (ESs) as a supplement or alternative to NHST. However, there is extensive specialist literature highlighting problems associated with the use and interpretation of ESs. A review of the coverage of ESs in over 100 textbooks on statistical analysis in behavioural science revealed widespread neglect of ESs and the relevant critical issues that have widespread coverage in the more specialist literature. For example, many textbooks claim that ESs should be interpreted as a simple measure of the practical real-world importance of a result despite the fact that ESs are profoundly influenced by features of design and analysis strategy. We seek to highlight areas of misunderstanding about ESs found in the pedagogical literature in the light of the more specialist literature and make recommendations to researchers for the appropriate use and interpretation of ESs. This is critical as statistics textbooks have a crucial role in the education of researchers.     

Analysis of group differences in processing speed: Brinley plots,Q-Q plots,and other conspiracies

Researchers in a growing number of areas (including cognitive development, aging, and neuropsychology) use Brinley plots to compare the processing speed of different groups. Ratcliff, Spieler, and McKoon (2000) argued that a Brinley plot is a quantile-quantile (Q-Q) plot and that therefore Brinley plot regression slopes measure standard deviation ratios rather than relative speed of processing. We show that this argument is incorrect. Brinley plots, by definition, are not Q-Q plots; the former are based on unranked data and the latter are based on ranked data. Furthermore, the relationship between standard deviation ratios and slopes is a general property of regression lines and has no implications for the use of Brinley plot regression slopes as processing speed measures. We also show that the relative speed interpretation of Brinley plot slopes is strongly supported by converging evidence from a metaanalysis of visual search, mental rotation, and memory scanning in young and older adults. As to Ratcliff et al.'s hypothesis that age differences in response time are attributable to greater cautiousness on the part of the elderly, rather than true processing speed differences, this hypothesis has been extensively tested in previous studies and found wanting.     

Précis of statistical significance: rationale, validity, and utility

The null-hypothesis significance-test procedure (NHSTP) is defended in the context of the theory-corroboration experiment, as well as the following contrasts: (a) substantive hypotheses versus statistical hypotheses, (b) theory corroboration versus statistical hypothesis testing, (c) theoretical inference versus statistical decision, (d) experiments versus nonexperimental studies, and (e) theory corroboration versus treatment assessment. The null hypothesis can be true because it is the hypothesis that errors are randomly distributed in data. Moreover, the null hypothesis is never used as a categorical proposition. Statistical significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor responsible. The experimental conclusion is drawn with the inductive principle underlying the experimental design. A chain of deductive arguments gives rise to the theoretical conclusion via the experimental conclusion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than real. The absolute size of the effect is not an index of evidential support for the substantive hypothesis. Nor is the effect size, by itself, informative as to the practical importance of the research result. Being a conditional probability, statistical power cannot be the a priori probability of statistical significance. The validity of statistical power is debatable because statistical significance is determined with a single sampling distribution of the test statistic based on H0, whereas it takes two distributions to represent statistical power or effect size. Sample size should not be determined in the mechanical manner envisaged in power analysis. It is inappropriate to criticize NHSTP for nonstatistical reasons. At the same time, neither effect size, nor confidence interval estimate, nor posterior probability can be used to exclude chance as an explanation of data. Neither can any of them fulfill the nonstatistical functions expected of them by critics.     

Interpretation of Poverty in St. Louis County,Minnesota

The paper analyzes subjective poverty in St. Louis County, Minnesota with the methods of systematic data collection. The initial hypothesis states that American people blame the individuals for their poor living conditions. Based on the Easterlin paradox, the next hypothesis states that subjective and absolute poverty lines are independent. Taking into account that the USA is a developed country, subjective wellbeing is supposed to be associated with relative deprivation. The paper concludes that the items most related to poverty are “no job”, “no access to basic needs” and “addiction”. The main causes of poverty are “cycle of poverty”, “no job” and “addiction”. The main consequences of poverty are “no access to basic needs”, “poor health” and “minimum education”. The results revealed that the most important items related to poverty and the main causes and consequences of poverty cannot be classified unambiguously as being individualist, structural or fatal. The analysis showed that subjective poverty line is different from the objective lines. This result supports the Easterlin paradox and implies that people can be dissatisfied with their lives even when they are above the poverty line according to the official statistics. Elimination of absolute poverty may therefore not be enough to improve subjective well-being. The evidence that relative income assessment is not correlated with subjective well-being implies that people are not concerned with their relative income position.     

How Pseudo-hypotheses Defeat a Non-Bayesian Theory of Evidence: Reply to Bandyopadhyay,Taper, and Brittan

Bandyopadhyay, Taper, and Brittan (BTB) advance a measure of evidential support that first appeared in the statistical and philosophical literature four decades ago and have been extensively discussed since. I have argued elsewhere, however, that it is vulnerable to a simple counterexample. BTB claim that the counterexample is flawed because it conflates evidence with confirmation. In this reply, I argue that the counterexample stands, and is fatal to their theory.     

Individual Differences in the Predictions of Behavioral Intentions

Abstract

We examined individual differences in the formation of behavioral intentions by American female students. Within the frameworks of the Fishbein-Ajzen and Triandis models of behavioral intentions, we measured the relative emphasis given to intention predictors for 40 behavioral intentions and examined how this emphasis was related to authoritarianism. In addition, empirical evidence was sought for the conceptual distinction between a personal norm (internalized norm) and a subjective norm (perceptions of significant other' opinions concerning a given action). Relative to low authoritarians, high authoritarians took subjective norms less into account, placed greater emphasis on personal norms, and emphasized less the consequences of action in forming behavioral intentions. The results suggest that subjective and personal norms can be distinguished empirically.     

Underpowered samples,false negatives,and unconscious learning

The scientific community has witnessed growing concern about the high rate of false positives and unreliable results within the psychological literature, but the harmful impact of false negatives has been largely ignored. False negatives are particularly concerning in research areas where demonstrating the absence of an effect is crucial, such as studies of unconscious or implicit processing. Research on implicit processes seeks evidence of above-chance performance on some implicit behavioral measure at the same time as chance-level performance (that is, a null result) on an explicit measure of awareness. A systematic review of 73 studies of contextual cuing, a popular implicit learning paradigm, involving 181 statistical analyses of awareness tests, reveals how underpowered studies can lead to failure to reject a false null hypothesis. Among the studies that reported sufficient information, the meta-analytic effect size across awareness tests was d _z = 0.31 (95 % CI 0.24–0.37), showing that participants’ learning in these experiments was conscious. The unusually large number of positive results in this literature cannot be explained by selective publication. Instead, our analyses demonstrate that these tests are typically insensitive and underpowered to detect medium to small, but true, effects in awareness tests. These findings challenge a widespread and theoretically important claim about the extent of unconscious human cognition.     

The significance test controversy

Summary The pre-designationist, anti-inductivist and operationalist tenor of Neyman-Pearson theory give that theory an obvious affinity to several currently influential philosophies of science, most particularly, the Popperian. In fact, one might fairly regard Neyman-Pearson theory as the statistical embodiment of Popperian methodology. The difficulties raised in this paper have, then, wider purport, and should serve as something of a touchstone for those who would construct a theory of evidence adequate to statistics without recourse to the notion of inductive probability.     

The error of accepting the "theoretical" null hypothesis: the rise, fall, and resurrection of commonsense hypotheses in psychology

When psychologists test a commonsense (CS) hypothesis and obtain no support, they tend to erroneously conclude that the CS belief is wrong. In many such cases it appears, after many years, that the CS hypothesis was valid after all. It is argued that this error of accepting the "theoretical" null hypothesis reflects confusion between the operationalized hypothesis and the theory or generalization that it is designed to test. That is, on the basis of reliable null data one can accept the operationalized null hypothesis (e.g., "A measure of attitude x is not correlated with a measure of behavior y"). In contrast, one cannot generalize from the findings and accept the abstract or theoretical null (e.g., "We know that attitudes do not predict behavior"). The practice of accepting the theoretical null hypothesis hampers research and reduces the trust of the public in psychological research.     

An inferential confidence interval method of establishing statistical equivalence that corrects Tryon's (2001) reduction factor

Evidence of group matching frequently takes the form of a nonsignificant test of statistical difference. Theoretical hypotheses of no difference are also tested in this way. These practices are flawed in that null hypothesis statistical testing provides evidence against the null hypothesis and failing to reject H-sub-0 is not evidence supportive of it. Tests of statistical equivalence are needed. This article corrects the inferential confidence interval (ICI) reduction factor introduced by W. W. Tryon (2001) and uses it to extend his discussion of statistical equivalence. This method is shown to be algebraically equivalent with D. J. Schuirmann's (1987) use of 2 one-sided t tests, a highly regarded and accepted method of testing for statistical equivalence. The ICI method provides an intuitive graphic method for inferring statistical difference as well as equivalence. Trivial difference occurs when a test of difference and a test of equivalence are both passed. Statistical indeterminacy results when both tests are failed. Hybrid confidence intervals are introduced that impose ICI limits on standard confidence intervals. These intervals are recommended as replacements for error bars because they facilitate inferences.     

Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011)

Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.     

We are All Bayesian,Everyone is Not a Bayesian

Medical research makes intensive use of statistics in order to support its claims. In this paper we make explicit an epistemological tension between the conduct of clinical trials and their interpretation: statistical evidence is sometimes discarded on the basis of an (often) underlined Bayesian reasoning. We suggest that acknowledging the potentiality of Bayesian statistics might contribute to clarify and improve comprehension of medical research. Nevertheless, despite Bayesianism may provide a better account for scientific inference with respect to the standard frequentist approach, Bayesian statistics is rarely adopted in clinical research. The main reason lies in the supposed subjective elements characterizing this perspective. Hence, we discuss this objection presenting the so-called Reference analysis, a formal method which has been developed in the context of objective Bayesian statistics in order to define priors which have a minimal or null impact on posterior probabilities. Furthermore, according to this method only available data are relevant sources of information, so that it resists the most common criticisms against Bayesianism.

     

The deep bodily origins of the subjective perspective: models and their problems

The naturalization of consciousness and the way a subjective perspective arises are hotly debated both in the cognitive sciences and in more strictly philosophical contexts. A number of these debates, mainly inspired by neuroscientific findings, focus on the 'visceral' dimension of the body in order to formulate a hypothesis for the coming about of consciousness. This focus on what might be called the 'in-depth body' (which is usually not governed by the intentions of the subject) shows that consciousness or the subjective perspective is intimately linked with vital and visceral regulatory processes. I join the debate by arguing that representationalist accounts of the origin of consciousness in the in-depth body exhibit a number of flaws hitherto mainly unnoticed. Furthermore, some aspects of neuroscientific theories are explored as possible validations of a nonrepresentationalist model of consciousness and the subjective perspective. Inspired by phenomenological (more specifically Husserlian) philosophy, I present a hypothesis in which the subjective perspective constitutes itself (in a process of auto-constitution) and in which the 'in-depth' body is not the object of representations in the brain. Rather, the 'in-depth body' is in a non-objectified way built-in in the subjective perspective itself. In this account, therefore, the subjective perspective looses its transparency and gains 'interoceptive thickness'.     

Integration of base rates and new information in an abstract hypothesis‐testing task

In two studies, we investigated how people use base rates and the presence versus the absence of new information to judge which of two hypotheses is more likely. Participants were given problems based on two decks of cards printed with 0–4 letters. A table showed the relative frequencies of the letters on the cards within each deck. Participants were told the letters that were printed on or absent from a card the experimenter had drawn. Base rates were conveyed by telling participants that the experimenter had chosen the deck by drawing from an urn containing, in different proportions, tickets marked either ‘deck 1’ or ‘deck 2’. The task was to judge from which of the two decks the card was most likely drawn. Prior probabilities and the evidential strength of the subset of present clues (computed as ‘weight of evidence’) were the only significant predictors of participants’ dichotomous (both studies) and continuous (Study 2) judgments. The evidential strength of all clues was not a significant predictor of participants’ judgments in either study, and no significant interactions emerged. We discuss the results as evidence for additive integration of base rates and the new present information in hypothesis testing.