Evaluating and Reporting Statistical Power in Counseling Research

Despite recommendations from the Publication Manual of the American Psychological Association (6th ed.) to include information on statistical power when publishing quantitative results, authors seldom include analysis or discussion of statistical power. The rationale for discussing statistical power is addressed, approaches to using G*Power to report statistical power are presented, and examples for reporting statistical power are provided.     

An inferential confidence interval method of establishing statistical equivalence that corrects Tryon's (2001) reduction factor

Evidence of group matching frequently takes the form of a nonsignificant test of statistical difference. Theoretical hypotheses of no difference are also tested in this way. These practices are flawed in that null hypothesis statistical testing provides evidence against the null hypothesis and failing to reject H-sub-0 is not evidence supportive of it. Tests of statistical equivalence are needed. This article corrects the inferential confidence interval (ICI) reduction factor introduced by W. W. Tryon (2001) and uses it to extend his discussion of statistical equivalence. This method is shown to be algebraically equivalent with D. J. Schuirmann's (1987) use of 2 one-sided t tests, a highly regarded and accepted method of testing for statistical equivalence. The ICI method provides an intuitive graphic method for inferring statistical difference as well as equivalence. Trivial difference occurs when a test of difference and a test of equivalence are both passed. Statistical indeterminacy results when both tests are failed. Hybrid confidence intervals are introduced that impose ICI limits on standard confidence intervals. These intervals are recommended as replacements for error bars because they facilitate inferences.     

Statistical approaches to achieving sufficiently high test score reliabilities for research purposes

The author provides statistical approaches to aid investigators in assuring that sufficiently high test score reliabilities are achieved for specific research purposes. The statistical approaches use tests of statistical significance between the obtained reliability and lowest population reliability that an investigator will tolerate. The statistical approaches work for coefficient alpha and related coefficients and for alternate-forms, split-half (2-part alpha), and retest reliabilities. The author shows that, in some instances, a formula can help to estimate the sample size necessary for the statistical test.     

Statistical analysis packages for the microcomputer: New technologies for industry

Major companies in the mainframe statistical software business have started producing powerful versions of their programs for microcomputers. Five of the leading microcomputer statistical packages are reviewed. A statistical software vendor directory is provided. These statistical packages can be used by properly trained personnel to make more reliable business decisions.     

Integrating statistical software into laboratories and laboratory courses

Reasonable quality statistical software that is easy to use and easy to learn to use is available, and more will become available. This paper describes nine features of statistical software programs that make such programs easy to use. Some methods for evaluating the capability, accuracy, and speed of statistical software are discussed. In addition, some issues about university and software company support are considered. Finally, a technique for integrating software into laboratories is presented.     

基于一般线性模型框架的心理统计教学

心理统计学教学中,不同的统计方法常常是独立教学,致使学生不易理解各种方法之间的关系。事实上,t检验、方差分析和多元线性回归等方法都可以统一到一般线性模型的框架下,而结构方程是对这个框架的最一般化的描述,且结构方程路径图是呈现这个框架的形象工具。因此,本文尝试用路径图的方式来呈现心理学研究中最常用的统计方法,并将结构方程分析结果与传统分析结果进行对照,帮助学生建立一般线性模型上位概念,将以往孤立的统计方法联系起来。     

The power of statistical tests for moderators in meta-analysis

Calculation of the statistical power of statistical tests is important in planning and interpreting the results of research studies, including meta-analyses. It is particularly important in moderator analyses in meta-analysis, which are often used as sensitivity analyses to rule out moderator effects but also may have low statistical power. This article describes how to compute statistical power of both fixed- and mixed-effects moderator tests in meta-analysis that are analogous to the analysis of variance and multiple regression analysis for effect sizes. It also shows how to compute power of tests for goodness of fit associated with these models. Examples from a published meta-analysis demonstrate that power of moderator tests and goodness-of-fit tests is not always high.     

Using the microcomputer as a visual aid in the statistics classroom

A microcomputer running a spreadsheet program makes an excellent visual aid for demonstrating statistical concepts and procedures. Some advantages of this setup over traditional visual aids are described. Several statistical demonstrations that I have found to work well are presented.     

Judgmental forecasts of time series affected by special events: does providing a statistical forecast improve accuracy?

Time series found in areas such as marketing and sales often have regular established patterns which are occasionally affected by exogenous influences, such as sales promotions. While statistical forecasting methods are adept at extrapolating regular patterns in series, judgmental forecasters have a potential advantage in that they can take into account the effect of these external influences, which may occur too infrequently for reliable statistical estimation. This suggests that a combination of statistical method and judgment is appropriate. An experiment was conducted to examine how judgmental forecasters make use of statistical time series forecasts when series are subject to sporadic special events. This was investigated under different conditions which were created by varying the complexity of the time series signal, the level of noise in the series, the salience of the cue, the predictive power of the cue information and the availability and presentation of the statistical forecast. Although the availability of a statistical forecast improved judgment under some conditions, the use the judgmental forecasters made of these forecasts was far from optimal. They changed the statistical forecasts when they were highly reliable and ignored them when they would have formed an ideal base‐line for adjustment. Copyright © 1999 John Wiley & Sons, Ltd.     

Inductive inference or inductive behavior: Fisher and Neyman-Pearson approaches to statistical testing in psychological research (1940-1960)

The application of statistical testing in psychological research over the period of 1940-1960 is examined in order to address psychologists' reconciliation of the extant controversy between the Fisher and Neyman-Pearson approaches. Textbooks of psychological statistics and the psychological journal literature are reviewed to examine the presence of what Gigerenzer (1993) called a hybrid model of statistical testing. Such a model is present in the textbooks, although the mathematically incomplete character of this model precludes the appearance of a similarly hybridized approach to statistical testing in the research literature. The implications of this hybrid model for psychological research and the statistical testing controversy are discussed.     

The power of statistical tests in meta-analysis

Calculations of the power of statistical tests are important in planning research studies (including meta-analyses) and in interpreting situations in which a result has not proven to be statistically significant. The authors describe procedures to compute statistical power of fixed- and random-effects tests of the mean effect size, tests for heterogeneity (or variation) of effect size parameters across studies, and tests for contrasts among effect sizes of different studies. Examples are given using 2 published meta-analyses. The examples illustrate that statistical power is not always high in meta-analysis.     

Recommendations for Describing Statistical Studies and Results in General Readership Science and Engineering Journals

This paper recommends how authors of statistical studies can communicate to general audiences fully, clearly, and comfortably. The studies may use statistical methods to explore issues in science, engineering, and society or they may address issues in statistics specifically. In either case, readers without explicit statistical training should have no problem understanding the issues, the methods, or the results at a non-technical level. The arguments for those results should be clear, logical, and persuasive. This paper also provides advice for editors of general journals on selecting high quality statistical articles without the need for exceptional work or expense. Finally, readers are also advised to watch out for some common errors or misuses of statistics that can be detected without a technical statistical background.     

Influences of credibility of testimony and strength of statistical evidence on children’s and adolescents’ reasoning

According to dual-process models that include analytic and heuristic modes of processing, analytic processing is often expected to become more common with development. Consistent with this view, on reasoning problems, adolescents are more likely than children to select alternatives that are backed by statistical evidence. It is shown here that this pattern depends on the quality of the statistical evidence and the quality of the testimonial that is the typical alternative to statistical evidence. In Experiment 1, 9- and 13-year-olds (N = 64) were presented with scenarios in which solid statistical evidence was contrasted with casual or expert testimonial evidence. When testimony was casual, children relied on it but adolescents did not; when testimony was expert, both children and adolescents relied on it. In Experiment 2, 9- and 13-year-olds (N = 83) were presented with scenarios in which casual testimonial evidence was contrasted with weak or strong statistical evidence. When statistical evidence was weak, children and adolescents relied on both testimonial and statistical evidence; when statistical evidence was strong, most children and adolescents relied on it. Results are discussed in terms of their implications for dual-process accounts of cognitive development.     

Statistical consistency and hypothesis testing for nonmetric multidimensional scaling

The properties of nonmetric multidimensional scaling (NMDS) are explored by specifying statistical models, proving statistical consistency, and developing hypothesis testing procedures. Statistical models with errors in the dependent and independent variables are described for quantitative and qualitative data. For these models, statistical consistency often depends crucially upon how error enters the model and how data are collected and summarized (e.g., by means, medians, or rank statistics). A maximum likelihood estimator for NMDS is developed, and its relationship to the standard Shepard-Kruskal estimation method is described. This maximum likelihood framework is used to develop a method for testing the overall fit of the model.     

Co‐occurrence statistics as a language‐dependent cue for speech segmentation

To what extent can language acquisition be explained in terms of different associative learning mechanisms? It has been hypothesized that distributional regularities in spoken languages are strong enough to elicit statistical learning about dependencies among speech units. Distributional regularities could be a useful cue for word learning even without rich language‐specific knowledge. However, it is not clear how strong and reliable the distributional cues are that humans might use to segment speech. We investigate cross‐linguistic viability of different statistical learning strategies by analyzing child‐directed speech corpora from nine languages and by modeling possible statistics‐based speech segmentations. We show that languages vary as to which statistical segmentation strategies are most successful. The variability of the results can be partially explained by systematic differences between languages, such as rhythmical differences. The results confirm previous findings that different statistical learning strategies are successful in different languages and suggest that infants may have to primarily rely on non‐statistical cues when they begin their process of speech segmentation.     

An Alternate Method of Deriving Priorities and Related Inferences for Group Decision Making in Analytic Hierarchy Process

Some work has been carried out in the past on statistically deriving priorities in Analytic Hierarchy Process (AHP). In AHP, the aggregated worths of the alternatives, when compared with respect to several criteria, are estimated in a hierarchical comparisons model introduced by Saaty. In this setup, statistical models are used for Saaty's method of scaling in paired comparisons experiments in any level of the hierarchy. At the end, the final priority weights of the alternatives and related inferences are developed with appropriate statistical methods. Existing statistical methods in the literature assume independence of the entries of the paired comparison matrix. However, these entries are highly dependent among themselves. In this article, we propose a statistical method that allows for the dependence among the entries of the pairwise comparisons matrix. The proposed method is then illustrated with a numerical example. Copyright © 2011 John Wiley & Sons, Ltd.     

How meta-analysis increases statistical power

One of the most frequently cited reasons for conducting a meta-analysis is the increase in statistical power that it affords a reviewer. This article demonstrates that fixed-effects meta-analysis increases statistical power by reducing the standard error of the weighted average effect size (T.) and, in so doing, shrinks the confidence interval around T.. Small confidence intervals make it more likely for reviewers to detect nonzero population effects, thereby increasing statistical power. Smaller confidence intervals also represent increased precision of the estimated population effect size. Computational examples are provided for 3 effect-size indices: d (standardized mean difference), Pearson's r, and odds ratios. Random-effects meta-analyses also may show increased statistical power and a smaller standard error of the weighted average effect size. However, the authors demonstrate that increasing the number of studies in a random-effects meta-analysis does not always increase statistical power.     

THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH

We first assess the current level of statistical power across articles in seven leading journals that represent a broad sample of applied psychology and management research. We next survey the authors of these articles to examine their perceptions and usage of statistical power analysis. Finally, we examine the perceptions and usage of power analysis in a survey of authors of regression-based research appearing in leading journals. Findings from the assessment of power and surveys of researchers indicate that power analyses are not typically conducted, researchers perceive little need for statistical power, and power in published research is low. We conclude by discussing implications of low power for the field and recommending avenues for improving researchers' awareness and usage of statistical power.     

Applications of a computer algebra system in introductory statistics

Maple, a computer algebra system, is employed in introductory statistics courses to promote conceptual learning by students of statistical principles without direct use of mathematics. Maple’s symbolic computation, graphic display, and animation capabilities support an integrated set of procedures for active study of sampling distributions and concepts related to samples, populations, and statistical decision making, error, and power. Students select parameter values for one-line commands and examine the effects of alternatives on computer-generated graphical representations of distributions. The 20 procedures written to support active student exploration of basic statistical concepts are described, and examples of exercises to support their use are provided.     

An overall statistic for testing symmetry in social interactions

The present work focuses on the skew-symmetry index as a measure of social reciprocity. This index is based on the correspondence between the amount of behaviour that individuals address toward their partners and what they receive in return. Although the skew-symmetry index enables researchers to describe social groups, statistical inferential tests are required. This study proposes an overall statistical technique for testing symmetry in experimental conditions, calculating the skew-symmetry statistic (Phi) at group level. Sampling distributions for the skew-symmetry statistic were estimated by means of a Monte Carlo simulation to allow researchers to make statistical decisions. Furthermore, this study will allow researchers to choose the optimal experimental conditions for carrying out their research, as the power of the statistical test was estimated. This statistical test could be used in experimental social psychology studies in which researchers may control the group size and the number of interactions within dyads.