MorePower 6.0 for ANOVA with relational confidence intervals and Bayesian analysis

MorePower 6.0 is a flexible freeware statistical calculator that computes sample size, effect size, and power statistics for factorial ANOVA designs. It also calculates relational confidence intervals for ANOVA effects based on formulas from Jarmasz and Hollands (Canadian Journal of Experimental Psychology 63:124–138, 2009), as well as Bayesian posterior probabilities for the null and alternative hypotheses based on formulas in Masson (Behavior Research Methods 43:679–690, 2011). The program is unique in affording direct comparison of these three approaches to the interpretation of ANOVA tests. Its high numerical precision and ability to work with complex ANOVA designs could facilitate researchers’ attention to issues of statistical power, Bayesian analysis, and the use of confidence intervals for data interpretation. MorePower 6.0 is available at https://wiki.usask.ca/pages/viewpageattachments.action?pageId=420413544.     

Reasons for the trend toward null findings in research on Type A behavior.

The findings of many studies conducted before 1978 suggest that Type A behavior (TAB) contributes to the development of coronary heart disease (CHD). In contrast, many recent studies have found no association between these variables. Through meta-analysis, several reasons for null findings are identified. First, a type of range restriction bias, disease-based spectrum (DBS) bias, produced many null findings. A study is vulnerable to DBS bias when researchers select only high-risk or diseased Ss for study. Second, self-report measures of TAB were often associated with null findings. Finally, null results were found for all studies that used fatal myocardial infarction as a disease criterion. In addition to identifying the reasons for null findings, this research suggests that TAB, as assessed by the structured interview, is associated with CHD. More Type As (70%) were found in diseased populations of middle-aged men than in healthy populations of middle-aged men (46%).     

Norms of age of acquisition and concreteness for 30,000 Dutch words

Word processing studies increasingly make use of regression analyses based on large numbers of stimuli (the so-called megastudy approach) rather than experimental designs based on small factorial designs. This requires the availability of word features for many words. Following similar studies in English, we present and validate ratings of age of acquisition and concreteness for 30,000 Dutch words. These include nearly all lemmas language researchers are likely to be interested in. The ratings are freely available for research purposes.     

Fractional factorial designs for legal psychology

Researchers considering novel or exploratory psycholegal research are often able to easily generate a sizable list of independent variables (IVs) that might influence a measure of interest. Where the research question is novel and the literature is not developed, however, choosing from among a long list of potential variables those worthy of empirical investigation often presents a formidable task. Many researchers may feel compelled by legal psychology's heavy reliance on full-factorial designs to narrow the IVs under investigation to two or three in order to avoid an expensive and unwieldy design involving numerous high-order interactions. This article suggests that fractional factorial designs provide a reasonable alternative to full-factorial designs in such circumstances because they allow the psycholegal researcher to examine the main effects of a large number of factors while disregarding high-order interactions. An introduction to the logic of fractional factorial designs is provided and several examples from the social sciences are presented.     

A Note on Measurement Issues in Type A Research: Let's Not Throw Out the Baby With the Bath Water

It has been suggested that researchers rely exclusively on the structured interview (SI) to assess Type A behavior instead of using objective self-report measures, because the SI is the only prospectively validated instrument currently available. This article considers the costs and benefits of relying solely on the SI to measure Type A behavior. Although using only the SI would assure that researchers measure, in a relatively unobtrusive fashion, actual Type A behaviors known to be predictive of heart disease, it would dramatically increase research costs, impede longitudinal studies of changes in Type A behavior, reduce the validity of statistical conclusions, restrict the convergent and discriminant validity of the Type A construct, and ultimately inhibit our ability to improve accuracy in predicting heart disease. A set of recommendations is proposed for improving the quality of measurement in Type A research.     

Statistical Power Analysis in Psychological Research

Reviews of the psychological literature suggest that many studies lack sufficient statistical power to detect effects of interest. Increased attention to statistical power by journal editors, reviewers, and funding agencies has led to a need for researchers to consider power carefully when designing studies. Our goal is to present an overview of issues that influence statistical power in the context of traditional research designs and analytic methods. We then extend the discussion of statistical power to complex designs and analyses providing readers with sources useful for evaluating power in the design stage of conducting research. Finally, we advocate the use of simulation and Monte Carlo methods as a flexible general strategy for designing research studies with adequate statistical power.     

Assumptions in research in sport and exercise psychology

ObjectivesThe aim of this article is to outline how certain key assumptions affect the quality and interpretation of research in quantitative sport and exercise psychology.MethodsA review of three common assumptions made in the sport and exercise psychology literature was conducted. The review focused on three assumptions relating to research validity and the treatment and interpretation of observations. A central theme to this discussion is the assumption that research observations reflect true effects in a population.ResultsAssumptions often made in sport and exercise psychology research were identified in three key areas: (1) validity, (2) inferences of causality, and (3) effect size and the “practical significance” of research findings. Findings indicated that many studies made assumptions about the validity of the self-report psychological measures adopted and few provided a comprehensive evaluation of the validity of these measures. Researchers adopting correlational designs in sport and exercise psychology often infer causality despite such conclusions being based on theory or speculation rather than empirical evidence. Research reports still do not include effect size statistics as standard and confine the discussion of findings to statistical significance alone rather than commenting on “practical significance”.ConclusionResearch quality can only be evaluated with due consideration of the common assumptions that limits empirical investigation in sport and exercise psychology. We offer some practical advice for researchers, reviewers, and journal editors to minimise the impact of these assumptions and enhance the quality of research findings in sport and exercise psychology.     

Marginal mean weighting through stratification: a generalized method for evaluating multivalued and multiple treatments with nonexperimental data

Propensity score matching and stratification enable researchers to make statistical adjustment for a large number of observed covariates in nonexperimental data. These methods have recently become popular in psychological research. Yet their applications to evaluations of multi-valued and multiple treatments are limited. The inverse-probability-of-treatment weighting method, though suitable for evaluating multi-valued and multiple treatments, often generates results that are not robust when only a portion of the population provides support for causal inference or when the functional form of the propensity score model is misspecified. The marginal mean weighting through stratification (MMW-S) method promises a viable nonparametric solution to these problems. By computing weights on the basis of stratified propensity scores, MMW-S adjustment equates the pretreatment composition of multiple treatment groups under the assumption that unmeasured covariates do not confound the treatment effects given the observed covariates. Analyzing data from a weighted sample, researchers can estimate a causal effect by computing the difference between the estimated average potential outcomes associated with alternative treatments within the analysis of variance framework. After providing an intuitive illustration of the theoretical rationale underlying the weighting method for causal inferences, the article demonstrates how to apply the MMW-S method to evaluations of treatments measured on a binary, ordinal, or nominal scale approximating a completely randomized experiment; to studies of multiple concurrent treatments approximating factorial randomized designs; and to moderated treatment effects approximating randomized block designs. The analytic procedure is illustrated with an evaluation of educational services for English language learners attending kindergarten in the United States.     

It has been said that, “There are three degrees of falsehoods: Lies,damn lies,and statistics”

Applied behavior analysis (ABA) researchers have historically eschewed population-based, inferential statistics, preferring to conduct and analyze repeated observations of each participant's responding under carefully controlled and manipulated experimental conditions using single-case designs (SCDs). In addition, early attempts to adapt traditional statistical procedures for use with SCDs often involved trade-offs between experimental and statistical control that most ABA researchers have found undesirable. The statistical methods recommended for use with SCDs in the current special issue represent a welcome departure from such prior suggestions in that the current authors are proposing that SCD researchers add statistical methods to their current practices in ways that do not alter traditional SCD practices. Further refinement and use of such methods would (a) facilitate the inclusion of research using SCDs in meta-analyses and (b) aid in the development and planning of grant-funded research using SCD methods. Collaboration between SCD researchers and statisticians, particularly on research that demonstrates the benefit of these methods, may help promote their acceptance and use in ABA.     

Probing the effects of individual components in multiple component prevention programs

Assessing the contributions of individual components in multi-component interventions poses complex challenges for prevention researchers. We review the strengths and weaknesses of designs and analyses that may be useful in answering three questions: (1) Is each of the individual components contributing to the outcome? (2) Is the program optimal? and (3), Through what processes are the components of the program achieving their effects? Factorial and fractional factorial designs in which a systematically selected portion of all possible treatment combinations is implemented are used to address question 1. Response surface designs in which each component is quantitatively scaled are explored in relation to question 2. Mediational analysis, a hybrid of experimental and correlational approaches, is considered in relation to question 3. Design enhancements are offered that may further strengthen some of these techniques. These techniques offer promise of enhancing both the basic science and applied science contributions of prevention research.     

Rank‐based permutation approaches for non‐parametric factorial designs

Inference methods for null hypotheses formulated in terms of distribution functions in general non‐parametric factorial designs are studied. The methods can be applied to continuous, ordinal or even ordered categorical data in a unified way, and are based only on ranks. In this set‐up Wald‐type statistics and ANOVA‐type statistics are the current state of the art. The first method is asymptotically exact but a rather liberal statistical testing procedure for small to moderate sample size, while the latter is only an approximation which does not possess the correct asymptotic α level under the null. To bridge these gaps, a novel permutation approach is proposed which can be seen as a flexible generalization of the Kruskal–Wallis test to all kinds of factorial designs with independent observations. It is proven that the permutation principle is asymptotically correct while keeping its finite exactness property when data are exchangeable. The results of extensive simulation studies foster these theoretical findings. A real data set exemplifies its applicability.     

结构方程模型统计检验力分析：原理与方法

结构方程模型是心理学、管理学、社会学等学科中重要的统计工具之一。然而, 大量使用结构方程模型的研究忽视了对该方法的统计检验力进行必要的分析和报告, 在一定程度上降低了这些研究的结果的证明效力。结构方程模型的统计检验力分析方法主要有Satorra-Saris法、MacCallum法与Monte Carlo法三类。其中Satorra-Saris法适用于备择模型清晰、检验对象相对简单、检验方法基于χ²分布的情形; MacCallum法适用于基于χ²分布的模型拟合检验且备择模型不明的情形; Monte Carlo法适用于检验对象相对复杂、采用模拟或重抽样方法进行检验的情形。在实际应用中, 研究者应当首先判断检验的目的、方法以及是否有明确的备择模型, 并根据这些信息选择具体的分析方法。     

A note on measurement issues in type A research: let's not throw out the baby with the bath water

It has been suggested that researchers rely exclusively on the structured interview (SI) to assess Type A behavior instead of using objective self-report measures, because the SI is the only prospectively validated instrument currently available. This article considers the costs and benefits of relying solely on the SI to measure Type A behavior. Although using only the SI would assure that researchers measure, in a relatively unobtrusive fashion, actual Type A behaviors known to be predictive of heart disease, it would dramatically increase research costs, impede longitudinal studies of changes in Type A behavior, reduce the validity of statistical conclusions, restrict the convergent and discriminant validity of the Type A construct, and ultimately inhibit our ability to improve accuracy in predicting heart disease. A set of recommendations is proposed for improving the quality of measurement in Type A research.     

Solutions to the problems of incremental validity testing in relationship science

Incremental validity testing (i.e., testing whether a focal predictor is associated with an outcome above and beyond a covariate) is common (e.g., 57% of Personal Relationships articles in 2017), yet it is fraught with conceptual and statistical problems. First, researchers often use it to overemphasize the novelty or counterintuitiveness of findings, which hinders cumulative understanding. Second, incremental validity testing requires that the focal predictor and the covariate represent separate constructs; researchers risk committing the “jangle fallacy” without such evidence. Third, the most common approach to incremental validity testing (i.e., standard multiple regression, 88% of articles) inflates Type I error and can produce invalid conclusions. This article also discusses the relevance of these issues to dyadic/longitudinal designs and offers concrete solutions.     

Testing for negligible interaction: A coherent and robust approach

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap‐based intersection–union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a non‐significant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per‐contrast power of the proposed approach to the non‐centrality‐based negligible interaction test of Cheng and Shao (2007, Statistica Sinica, 17, 1441). For 2 × 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied; however, only our proposed bootstrapping approach was satisfactory under non‐normality and/or variance heterogeneity. In general a × b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection–union approach.     

Using analysis of covariance (ANCOVA) with fallible covariates

Analysis of covariance (ANCOVA) is used widely in psychological research implementing nonexperimental designs. However, when covariates are fallible (i.e., measured with error), which is the norm, researchers must choose from among 3 inadequate courses of action: (a) know that the assumption that covariates are perfectly reliable is violated but use ANCOVA anyway (and, most likely, report misleading results); (b) attempt to employ 1 of several measurement error models with the understanding that no research has examined their relative performance and with the added practical difficulty that several of these models are not available in commonly used statistical software; or (c) not use ANCOVA at all. First, we discuss analytic evidence to explain why using ANCOVA with fallible covariates produces bias and a systematic inflation of Type I error rates that may lead to the incorrect conclusion that treatment effects exist. Second, to provide a solution for this problem, we conduct 2 Monte Carlo studies to compare 4 existing approaches for adjusting treatment effects in the presence of covariate measurement error: errors-in-variables (EIV; Warren, White, & Fuller, 1974), Lord's (1960) method, Raaijmakers and Pieters's (1987) method (R&P), and structural equation modeling methods proposed by S?rbom (1978) and Hayduk (1996). Results show that EIV models are superior in terms of parameter accuracy, statistical power, and keeping Type I error close to the nominal value. Finally, we offer a program written in R that performs all needed computations for implementing EIV models so that ANCOVA can be used to obtain accurate results even when covariates are measured with error.     

Examining Spirituality Over Time: Latent Growth Curve and Individual Growth Curve Analyses

SUMMARY

Research on spirituality and religiousness has gained growing attention in recent years; however, most studies have used cross-sectional designs. As research on this topic evolves, there has been increasing recognition of the need to examine these constructs and their effects through the use of longitudinal designs. Beyond repeated-measures ANOVA and OLS regression models, what tools are available to examine these constructs over time? The purpose of this paper is to provide an overview of two cutting-edge statistical techniques that will facilitate longitudinal investigations of spirituality and religiousness: latent growth curve analysis using structural equation modeling (SEM) and individual growth curve models. The SEM growth curve approach examines change at the group level, with change over time expressed as a single latent growth factor. In contrast, individual growth curve models consider longitudinal change at the level of the person. While similar results may be obtained using either method, researchers may opt for one over the other due to the strengths and weaknesses associated with these methods. Examples of applications of both approaches to longitudinal studies of spirituality and religiousness are presented and discussed, along with design and data considerations when employing these modeling techniques.     

Effectively Analyzing Change over Time in Laboratory Research on Stress and Health: A Multilevel Modeling Approach

Stress and health researchers often utilize standardized laboratory stress tasks to evaluate the physical and psychological consequences of challenging experiences. These laboratory sessions usually include multiple measurements of physical and psychological responses collected over time. Multilevel modeling allows researchers to make use of all available data points to model the trajectory of change over time, and within distinct task periods such as baseline, stressor, and recovery. To effectively predict future health outcomes it is important to examine both stress‐related reactivity and recovery. In this paper, we review the analytic approaches used in recent laboratory stress research and note that many recent articles have aggregated multiple responses, used difference scores, or conducted repeated measures analysis of variance (ANOVA). Relatively few studies used a multilevel modeling approach. We highlight the advantages of a multilevel modeling approach and provide an example for using this approach as an alternative to repeated measures ANOVA and difference scores.     

Analysis and meta-analysis of single-case designs: An introduction

The last 10 years have seen great progress in the analysis and meta-analysis of single-case designs (SCDs). This special issue includes five articles that provide an overview of current work on that topic, including standardized mean difference statistics, multilevel models, Bayesian statistics, and generalized additive models. Each article analyzes a common example across articles and presents syntax or macros for how to do them. These articles are followed by commentaries from single-case design researchers and journal editors. This introduction briefly describes each article and then discusses several issues that must be addressed before we can know what analyses will eventually be best to use in SCD research. These issues include modeling trend, modeling error covariances, computing standardized effect size estimates, assessing statistical power, incorporating more accurate models of outcome distributions, exploring whether Bayesian statistics can improve estimation given the small samples common in SCDs, and the need for annotated syntax and graphical user interfaces that make complex statistics accessible to SCD researchers. The article then discusses reasons why SCD researchers are likely to incorporate statistical analyses into their research more often in the future, including changing expectations and contingencies regarding SCD research from outside SCD communities, changes and diversity within SCD communities, corrections of erroneous beliefs about the relationship between SCD research and statistics, and demonstrations of how statistics can help SCD researchers better meet their goals.     

Power analysis for multivariate and repeated measures designs: A flexible approach using the SPSS MANOVA procedure

Although power analysis is an important component in the planning and implementation of research designs, it is often ignored. Computer programs for performing power analysis are available, but most have limitations, particularly for complex multivariate designs. An SPSS procedure is presented that can be used for calculating power for univariate, multivariate, and repeated measures models with and without time-varying and time-constant covariates. Three examples provide a framework for calculating power via this method: an ANCOVA, a MANOVA, and a repeated measures ANOVA with two or more groups. The benefits and limitations of this procedure are discussed.