Four reasons to prefer Bayesian analyses over significance testing

Inference using significance testing and Bayes factors is compared and contrasted in five case studies based on real research. The first study illustrates that the methods will often agree, both in motivating researchers to conclude that H1 is supported better than H0, and the other way round, that H0 is better supported than H1. The next four, however, show that the methods will also often disagree. In these cases, the aim of the paper will be to motivate the sensible evidential conclusion, and then see which approach matches those intuitions. Specifically, it is shown that a high-powered non-significant result is consistent with no evidence for H0 over H1 worth mentioning, which a Bayes factor can show, and, conversely, that a low-powered non-significant result is consistent with substantial evidence for H0 over H1, again indicated by Bayesian analyses. The fourth study illustrates that a high-powered significant result may not amount to any evidence for H1 over H0, matching the Bayesian conclusion. Finally, the fifth study illustrates that different theories can be evidentially supported to different degrees by the same data; a fact that P-values cannot reflect but Bayes factors can. It is argued that appropriate conclusions match the Bayesian inferences, but not those based on significance testing, where they disagree.     

Developing and testing a model of loneliness

This article presents a model of loneliness that incorporates characteristics of the social network, background variables, personality characteristics, and evaluative aspects. The most salient aspect of this approach is its emphasis on cognitive processes that mediate between characteristics of the social network and the experience of loneliness. A total of 554 adult men and women served as respondents. The program LISREL, a causal modelling approach, was used to analyze the data. The LISREL program includes a goodness-of-fit test that indicates the degree of fit between a particular model and the data. The hypothesized model made a valuable contribution to the understanding of loneliness: It accounted for 52.3% of the variance in the data set. One of the model's major advantages is its ability to disentangle both the direct and the indirect causal influences of the various factors on loneliness.     

The Fisher-Pitman permutation test when testing for differences in mean and variance

The Fisher-Pitman permutation test can detect any type of difference between two samples: hence, a significant Fisher-Pitman permutation test does not necessarily provide evidence for a difference in means. It is possible, however, to test separately for differences in means and variances. Here, we present a recently proposed two-stage procedure to decide whether there are differences in means or variances that can be applied when samples may come from nonnormal distributions with possibly unequal variances.     

Analysis of trend: a permutation alternative to the F test

When the categories of the independent variable in an analysis of variance are quantitative, it is more informative to evaluate the trends in the treatment means than to simply compare differences among the treatment means. A permutation alternative to the conventional F test is shown to possess significant advantages when analyzing trend among quantitative treatments in a one-way analysis of variance. An example with and without an extreme data point illustrates the effectiveness of the permutation alternative for the analysis of trend when homogeneity of variance is compromised.     

Applying the permutation test to factorial designs

The permutation test follows directly from the procedure in a comparative experiment, does not depend on a known distribution for error, and is sometimes more sensitive to real effects than are the corresponding parametric tests. Despite its advantages, the permutation test is seldom (if ever) applied to factorial designs because of the computational load that they impose. We propose two methods to limit the computation load. We show, first, that orthogonal contrasts limit the computational load and, second, that when combined with Gill’s (2007) algorithm, the factorial permutation test is both practical and efficient. For within-subjects designs, the factorial permutation test is equivalent to an ANOVA when the latter’s assumptions have been met. For between-subjects designs, the factorial test is conservative. Code to execute the routines described in this article may be downloaded from http://brm.psychonomic-journals.org/content/supplemental.     

A general diagnostic model applied to language testing data

Probabilistic models with one or more latent variables are designed to report on a corresponding number of skills or cognitive attributes. Multidimensional skill profiles offer additional information beyond what a single test score can provide, if the reported skills can be identified and distinguished reliably. Many recent approaches to skill profile models are limited to dichotomous data and have made use of computationally intensive estimation methods such as Markov chain Monte Carlo, since standard maximum likelihood (ML) estimation techniques were deemed infeasible. This paper presents a general diagnostic model (GDM) that can be estimated with standard ML techniques and applies to polytomous response variables as well as to skills with two or more proficiency levels. The paper uses one member of a larger class of diagnostic models, a compensatory diagnostic model for dichotomous and partial credit data. Many well‐known models, such as univariate and multivariate versions of the Rasch model and the two‐parameter logistic item response theory model, the generalized partial credit model, as well as a variety of skill profile models, are special cases of this GDM. In addition to an introduction to this model, the paper presents a parameter recovery study using simulated data and an application to real data from the field test for TOEFL^® Internet‐based testing.     

Introduction to the model guidelines for preemployment integrity testing

TheModel Guidelines for Preemployment Integrity Testing Programs are described. These guidelines have implications for both test publishers and test users.Companies may request a copy of the Association of Personnel Test Publisher'sModel Guidelines for Preemployment Integrity Testing Programs by writingModel Guidelines, APTP, 655 Fifteenth Street, N.W., Suite 320, Washington, D.C. 20005.     

Partially testing a process model for understanding victim responses to an anticipated worksite closure

This study partially tested a recent process model for understanding victim responses to worksite/function closure (W/FC) proposed by Blau [Blau, G. (2006). A process model for understanding victim responses to worksite/function closure. Human Resource Management Review, 16, 12–28], in a pharmaceutical manufacturing site. Central to the model are the Kubler-Ross [Kubler-Ross, E. (1969). On death and dying. New York: Macmillan] grieving stages, which have not been formally measured and applied to downsizing research. Following Blau (2006), individual grieving stages were successfully measured and clustered into more general grieving categories, i.e., negative (denial, anger, bargaining depression) and positive (exploration, acceptance). Across four waves of data 53 respondents constituted the complete data sample. The Time 1 personal factors had minimal impact on any type of response. However, Time 1 situational factors did have an impact, paced by higher perceived contract violation leading to greater strain, work incivility, organizational deviance, and intent to sue employer, and lower transactional obligations and employer endorsement. Earlier Time 2 grieving stages were used as individual antecedents in regression analyses to explain Time 3 (N = 77) victim responses (general strain, work incivility, interpersonal deviance, organizational deviance, transactional obligations, relational obligations) and also Time 4 (N = 53) prior to closure responses (intent to sue employer, employer endorsement). Within negative grieving, results indicated that greater anger was the most influential grieving stage, since it led to greater strain, work incivility, organizational deviance, and intent to sue, as well as lower transactional obligations and lower endorsement. Within positive grieving acceptance was the most influential, since it led to lower strain, lower work incivility, lower organizational deviance, and lower intent to sue. Study limitations and future research issues are discussed.     

Partially testing a process model for understanding victim responses to an anticipated worksite closure

This study partially tested a recent process model for understanding victim responses to worksite/function closure (W/FC) proposed by Blau [Blau, G. (2006). A process model for understanding victim responses to worksite/function closure. Human Resource Management Review, 16, 12-28], in a pharmaceutical manufacturing site. Central to the model are the Kubler-Ross [Kubler-Ross, E. (1969). On death and dying. New York: Macmillan] grieving stages, which have not been formally measured and applied to downsizing research. Following Blau (2006), individual grieving stages were successfully measured and clustered into more general grieving categories, i.e., negative (denial, anger, bargaining depression) and positive (exploration, acceptance). Across four waves of data 53 respondents constituted the complete data sample. The Time 1 personal factors had minimal impact on any type of response. However, Time 1 situational factors did have an impact, paced by higher perceived contract violation leading to greater strain, work incivility, organizational deviance, and intent to sue employer, and lower transactional obligations and employer endorsement. Earlier Time 2 grieving stages were used as individual antecedents in regression analyses to explain Time 3 (N = 77) victim responses (general strain, work incivility, interpersonal deviance, organizational deviance, transactional obligations, relational obligations) and also Time 4 (N = 53) prior to closure responses (intent to sue employer, employer endorsement). Within negative grieving, results indicated that greater anger was the most influential grieving stage, since it led to greater strain, work incivility, organizational deviance, and intent to sue, as well as lower transactional obligations and lower endorsement. Within positive grieving acceptance was the most influential, since it led to lower strain, lower work incivility, lower organizational deviance, and lower intent to sue. Study limitations and future research issues are discussed.     

Cognition in action: testing a model of limb apraxia

Assessment of limb apraxia is still suffering from Liepmann's legacy and performance in gesture-processing tests is generally rendered by classifying patients' profile according to the classic clinical labels of ideomotor and ideational apraxia. At odds with other cognitive functions, interpretation of apraxia has suffered from a lack of a reliable model which does justice to its complexity. Recently such a model has been proposed (Rothi et al., 1991, 1997). In this article a modified version of this model is presented and predictions are made according to its functional architecture. Five different patterns of impairment of gesture processing are postulated. To validate the predicted performance profiles, 19 left-hemisphere-damaged patients were assessed by means of an ad hoc battery of four praxis tests. Four of the five predicted apraxia patterns were observed, the fifth being more equivocal. These results support the need to overcome the simplistic dichotomous view of apraxia and confirm the fruitfulness of a model of normal gesture processing in order to understand dissociations in apraxia.     

Bayesian nonparametric model selection and model testing

This article examines a Bayesian nonparametric approach to model selection and model testing, which is based on concepts from Bayesian decision theory and information theory. The approach can be used to evaluate the predictive-utility of any model that is either probabilistic or deterministic, with that model analyzed under either the Bayesian or classical-frequentist approach to statistical inference. Conditional on an observed set of data, generated from some unknown true sampling density, the approach identifies the “best” model as the one that predicts a sampling density that explains the most information about the true density. Furthermore, in the approach, the decision is to reject a model when it does not explain enough information about the true density (according to a straightforward calibration of the Kullback-Leibler divergence measure). The posterior estimate of the true density is based on a Bayesian nonparametric prior that can give positive support to the entire space of sampling densities (defined on some sample space). This article also discusses the theoretical and practical advantages of the Bayesian nonparametric approach over all other types of model selection procedures, and over any model testing procedure that depends on interpreting a p-value. Finally, the Bayesian nonparametric approach is illustrated on four real data sets, in the comparison and testing of order-constrained models, cognitive models, models of choice-behavior, and a test of a general psychometric model.     

From rotation to disfiguration: testing a dual-strategy model for recognition of faces across view angles

A study is reported of the effect of distinctive marks on the recognition of unfamiliar faces across view angles. Subjects were asked to memorize a set of target faces, half of which had distinctive marks. Recognition was assessed by presenting the target faces, either in the same orientation, or after 90 degrees rotation, mixed with an equal number of distractors. Results show that the effect of distinctive marks depends on the view presented during learning. When a frontal view was learned, as predicted by the dual-strategy model [Valentin et al, in press, in Computational, Geometric, and Process Perspectives on Facial Cognition: Context and Challenges Eds T Wenger, J Townsend (Hillsdale, NJ: Lawrence Erlbaum Associates)], distinctive marks improve recognition performance in the 90 degrees condition but not in the 0 degree condition. However, when a profile view was learned, distinctive marks have no effect on recognition performance, even in the 90 degrees condition where a frontal view is tested.     

Cross-cultural model testing: toward a solution of the etic-emic dilemma

A model for the prediction of behavior from attitudinal components, developed by Triandis, was tested with samples of U.S. and Mexican women, and with fertility relevant behaviors. The elements of the model are etic, but the operationalizations of the various variables were done emically. Results support the model in both cultures. While the predictive utility of the model is equivalent in two cultures, there are social class differences on which component of the model is most emphasized. The U.S. upper-middle-class sample and the Mexican upper-middle-class sample emphasized the person's attitude toward the act, while the Mexican lower SES (socio-economic status) sample emphasized the person's normative beliefs (moral obligations).     

Offering predictive testing for Huntington disease in a medical genetics clinic: Practical applications

Predictive testing for Huntington disease is presently offered in a select few medical genetics centers in the United States. This is in part due to the labor intensive counseling and psychological testing suggested by the research protocols. We discuss some specific suggestions for establishing programs for Huntington disease predictive testing within pre-existing medical genetics clinics to encourage more centers to offer presymptomatic testing. This will allow more at risk individuals the opportunity to consider predictive testing and cut down the expenses of traveling to the few predictive testing centers that currently exist. The counseling principals will remain similar to those discussed here, even following the identification of the Huntington disease mutation.     

Evaluation of global testing procedures for item fit to the Rasch model

Two types of global testing procedures for item fit to the Rasch model were evaluated using simulation studies. The first type incorporates three tests based on first‐order statistics: van den Wollenberg's Q₁ test, Glas's R₁ test, and Andersen's LR test. The second type incorporates three tests based on second‐order statistics: van den Wollenberg's Q₂ test, Glas's R₂ test, and a non‐parametric test proposed by Ponocny. The Type I error rates and the power against the violation of parallel item response curves, unidimensionality and local independence were analysed in relation to sample size and test length. In general, the outcomes indicate a satisfactory performance of all tests, except the Q₂ test which exhibits an inflated Type I error rate. Further, it was found that both types of tests have power against all three types of model violation. A possible explanation is the interdependencies among the assumptions underlying the model.     

Indirect scaling methods for testing quantitative emotion theories

Two studies investigated the utility of indirect scaling methods, based on graded pair comparisons, for the testing of quantitative emotion theories. In Study 1, we measured the intensity of relief and disappointment caused by lottery outcomes, and in Study 2, the intensity of disgust evoked by pictures, using both direct intensity ratings and graded pair comparisons. The stimuli were systematically constructed to reflect variables expected to influence the intensity of the emotions according to theoretical models of relief/disappointment and disgust, respectively. Two probabilistic scaling methods were used to estimate scale values from the pair comparison judgements: Additive functional measurement (AFM) and maximum likelihood difference scaling (MLDS). The emotion models were fitted to the direct and indirect intensity measurements using nonlinear regression (Study 1) and analysis of variance (Study 2). Both studies found substantially improved fits of the emotion models for the indirectly determined emotion intensities, with their advantage being evident particularly at the level of individual participants. The results suggest that indirect scaling methods yield more precise measurements of emotion intensity than rating scales and thereby provide stronger tests of emotion theories in general and quantitative emotion theories in particular.