Data, model, conclusion, doing it again

This paper explores the robustness of conclusions from a statistical model against variations in model choice (rather than variations in random sampling and random assignment to treatments, which are the usual variations covered by inferential statistics). After the problem formulation in section 1, section 2 presents an example from Box and Tiao in which variation in parent distribution is modeled for a one sample location problem. An adaptive Bayesian procedure permits to use a weighted mixture of parent distributions rather than choosing just one, such as a normal or a uniform distribution.In section 3 similar considerations are applied to an event history model for the influence of education and gender on age at first marriage, but here the conclusions are hardly influenced by the choice of the duration distribution. In section 4 a brief discussion of model choice in factor analysis and structural equation models is followed by a more elaborate treatment of the choice of integer valued slopes for item response functions in the OPLM model which is an extension of the Rasch model. A modest simulation study suggests that Adaptive Bayesian Modeling with a mixture of sets of slopes works better than fixing one set of postulated slopes.The paper concludes with some remarks on the roles and sources of prior distributions followed by a short epilogue which argues that simultaneous consideration of a class of models for the same data is sometimes superior to exclusively analyzing the data under one specific model chosen from such a class.This article is based on the Presidential Address Ivo W. Molenaar gave on June 20, 1998 at the 1998 Annual Meeting of the Psychometric Society held at the University of Illinois in Champaign, Illinois. Thanks is given to John Wiley & Sons and George C. Tiao for granting permission to reprint three figures from the book George C. Tiao wrote with George E. P. Box titled Bayesian Inference in Statistical Analysis.—EditorThanks are due to Anne Boomsma, Jeffrey Hoogland, Mark Huisman, Tom Snijders, Marijtje Van Duijn, and Norman Verhelst for suggesting improvements and/or assisting with data analyses.