Sample size determination for at test given at value from a previous study: A FORTRAN 77 program

When uncertain about the magnitude of an effect, researchers commonly substitute in the standard sample-size-determination formula an estimate of effect size derived from a previous experiment. A problem with this approach is that the traditional sample-size-determination formula was not designed to deal with the uncertainty inherent in an effect-size estimate. Consequently, estimate-substitution in the traditional sample-size-determination formula can lead to a substantial loss of power. A method of sample-size determination designed to handle uncertainty in effect-size estimates is described. The procedure uses thet value and sample size from a previous study, which might be a pilot study or a related study in the same area, to establish a distribution of probable effect sizes. The sample size to be employed in the new study is that which supplies an expected power of the desired amount over the distribution of probable effect sizes. A FORTRAN 77 program is presented that permits swift calculation of sample size for a variety oft tests, including independentt tests, relatedt tests,t tests of correlation coefficients, andt tests of multiple regressionb coefficients.     

Positive mood and helping behavior: a test of six hypotheses

Past research has shown rather consistently that positive mood states lead to increased helpfulness. In an expanded analysis of the published literature, we examined six distinct views about this relation: the focus of attention, objective self-awareness, separate process, social outlook, mood maintenance, and concomitance hypotheses. For each of 61 positive affect conditions in which it was possible to generate an effect-size estimate corresponding to the relative degree of helpfulness exhibited by positive mood subjects (compared with neutral affect subjects), judges assessed the contextual levels of variables relevant to each of the six hypotheses by reading the Method section of each article. Higher-order partial correlation coefficients were then calculated to isolate the independent contribution of each of the theoretically relevant variables to the variation among the 61 effect sizes. The results support the focus of attention, separate process, social outlook, and mood maintenance hypotheses, and partially support the objective self-awareness and concomitance hypotheses.     

When effect sizes disagree: the case of r and d

The increased use of effect sizes in single studies and meta-analyses raises new questions about statistical inference. Choice of an effect-size index can have a substantial impact on the interpretation of findings. The authors demonstrate the issue by focusing on two popular effect-size measures, the correlation coefficient and the standardized mean difference (e.g., Cohen's d or Hedges's g), both of which can be used when one variable is dichotomous and the other is quantitative. Although the indices are often practically interchangeable, differences in sensitivity to the base rate or variance of the dichotomous variable can alter conclusions about the magnitude of an effect depending on which statistic is used. Because neither statistic is universally superior, researchers should explicitly consider the importance of base rates to formulate correct inferences and justify the selection of a primary effect-size statistic.     

Contrasts and correlations in effect-size estimation

This article describes procedures for presenting standardized measures of effect size when contrasts are used to ask focused questions of data. The simplest contrasts consist of comparisons of two samples (e.g., based on the independent t statistic). Useful effect-size indices in this situation are members of the g family (e.g., Hedges's g and Cohen's d ) and the Pearson r . We review expressions for calculating these measures and for transforming them back and forth, and describe how to adjust formulas for obtaining g or d from t , or r from g , when the sample sizes are unequal. The real-life implications of d or g calculated from t become problematic when there are more than two groups, but the correlational approach is adaptable and interpretable, although more complex than in the case of two groups. We describe a family of four conceptually related correlation indices: the alerting correlation, the contrast correlation, the effect-size correlation, and the BESD (binomial effect-size display) correlation. These last three correlations are identical in the simple setting of only two groups, but differ when there are more than two groups.     

A FORTRAN 77 program for sample-size determination in replication attempts when effect size is uncertain

A method of sample-size determination for use in attempts to replicate experiments is described. It is appropriate in situations where there is uncertainty about the magnitude of the effect under investigation. The procedure uses information supplied by the original experiment to establish a distribution of probable effect sizes. The sample size to be used in a replication study is that which provides an expected power of the desired amount over the distribution of probable effect sizes A FORTRAN 77 program is presented that permits rapid calculation of sample size in replication attempts employing comparisons of means, correlation coefficients, or proportions.     

Statistical properties of four effect-size measures for mediation models

This project examined the performance of classical and Bayesian estimators of four effect size measures for the indirect effect in a single-mediator model and a two-mediator model. Compared to the proportion and ratio mediation effect sizes, standardized mediation effect-size measures were relatively unbiased and efficient in the single-mediator model and the two-mediator model. Percentile and bias-corrected bootstrap interval estimates of ab/s _Y , and ab(s _X )/s _Y in the single-mediator model outperformed interval estimates of the proportion and ratio effect sizes in terms of power, Type I error rate, coverage, imbalance, and interval width. For the two-mediator model, standardized effect-size measures were superior to the proportion and ratio effect-size measures. Furthermore, it was found that Bayesian point and interval summaries of posterior distributions of standardized effect-size measures reduced excessive relative bias for certain parameter combinations. The standardized effect-size measures are the best effect-size measures for quantifying mediated effects.     

A note on testing for homogeneity among effect sizes sharing a common control group

L. V. Hedges and I. Olkin (1985) presented a statistic to test for homogeneity among correlated effect sizes and L. J. Gleser and I. Olkin (1994) presented a large-sample approximation to the covariance matrix of the correlated effect sizes. This article presents a more exact expression for this covariance matrix, assuming normally distributed data but not large samples, for the situation where effect sizes are correlated because a single control group was compared with more than one treatment group. After the correlation between effect sizes has been estimated, the standard Q statistic for correlated effect sizes can be used to test for homogeneity. This method is illustrated using results from schizophrenia research.     

A Computationally More Efficient and More Accurate Stepwise Approach for Correcting for Sampling Error and Measurement Error

Over the last decade or two, multilevel structural equation modeling (ML-SEM) has become a prominent modeling approach in the social sciences because it allows researchers to correct for sampling and measurement errors and thus to estimate the effects of Level 2 (L2) constructs without bias. Because the latent variable modeling software Mplus uses maximum likelihood (ML) by default, many researchers in the social sciences have applied ML to obtain estimates of L2 regression coefficients. However, one drawback of ML is that covariance matrices of the predictor variables at L2 tend to be degenerate, and thus, estimates of L2 regression coefficients tend to be rather inaccurate when sample sizes are small. In this article, I show how an approach for stabilizing covariance matrices at L2 can be used to obtain more accurate estimates of L2 regression coefficients. A simulation study is conducted to compare the proposed approach with ML, and I illustrate its application with an example from organizational research.     

The meaning and suitability of various effect sizes for structured Rater x Ratee designs

Four types of analysis are commonly applied to data from structured Rater x Ratee designs. These types are characterized by the unit of analysis, which is either raters or ratees, and by the design used, which is either between-units or within-unit design. The 4 types of analysis are quite different, and therefore they give rise to effect sizes that differ in their substantive interpretations. In most cases, effect sizes based on between-ratee analysis have the least ambiguous meaning and will best serve the aims of meta-analysts and primary researchers. Effect sizes that arise from within-unit designs confound the strength of an effect with its homogeneity. Nonetheless, the authors identify how a range of effect-size types such as these serve the aims of meta-analysis appropriately.     

Biases and Standard Errors of Standardized Regression Coefficients

The paper obtains consistent standard errors (SE) and biases of order O(1/n) for the sample standardized regression coefficients with both random and given predictors. Analytical results indicate that the formulas for SEs given in popular text books are consistent only when the population value of the regression coefficient is zero. The sample standardized regression coefficients are also biased in general, although it should not be a concern in practice when the sample size is not too small. Monte Carlo results imply that, for both standardized and unstandardized sample regression coefficients, SE estimates based on asymptotics tend to under-predict the empirical ones at smaller sample sizes.     

Comparison of methods for constructing confidence intervals of standardized indirect effects

Mediation models are often used as a means to explain the psychological mechanisms between an independent and a dependent variable in the behavioral and social sciences. A major limitation of the unstandardized indirect effect calculated from raw scores is that it cannot be interpreted as an effect-size measure. In contrast, the standardized indirect effect calculated from standardized scores can be a good candidate as a measure of effect size because it is scale invariant. In the present article, 11 methods for constructing the confidence intervals (CIs) of the standardized indirect effects were evaluated via a computer simulation. These included six Wald CIs, three bootstrap CIs, one likelihood-based CI, and the PRODCLIN CI. The results consistently showed that the percentile bootstrap, the bias-corrected bootstrap, and the likelihood-based approaches had the best coverage probability. Mplus, LISREL, and Mx syntax were included to facilitate the use of these preferred methods in applied settings. Future issues on the use of the standardized indirect effects are discussed.     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

Three-level meta-analysis of dependent effect sizes

Although dependence in effect sizes is ubiquitous, commonly used meta-analytic methods assume independent effect sizes. We describe and illustrate three-level extensions of a mixed effects meta-analytic model that accounts for various sources of dependence within and across studies, because multilevel extensions of meta-analytic models still are not well known. We also present a three-level model for the common case where, within studies, multiple effect sizes are calculated using the same sample. Whereas this approach is relatively simple and does not require imputing values for the unknown sampling covariances, it has hardly been used, and its performance has not been empirically investigated. Therefore, we set up a simulation study, showing that also in this situation, a three-level approach yields valid results: Estimates of the treatment effects and the corresponding standard errors are unbiased.     

ORMA: A strategy to reduce Psychology's replication problems

Because of their historical reliance upon null hypothesis statistical tests (NHST), the human sciences have developed a number of potentially problematic research literatures. While aware of the file drawer effect since the 1970s, scientists have been largely unsuccessful at addressing its pernicious effects. Because significant results have a greater likelihood of being published than do nonsignificant effects, many of our research literatures might currently be constructed upon a series of Type I errors and inflated effect sizes. A method (called Original Replication of Meta-Analyses or ORMA) has recently been developed for identifying problematic research literatures and offering a method to address the problems due to publication bias. Philosophers of science have long argued that a chief reason for science's preeminence as a source of knowledge rests in its ability to self-correct. Researchers in the human sciences are now able to empirically test their research literatures to ascertain which are in need of repair. The use of ORMA serves to lessen the problems that led to the recent calls for bans on significant/nonsignificant statistics in human science research. ORMA will also improve psychology's ability to successfully replicate its research findings.     

Effect Size Strengths in Subjective Well-Being Research

Subjective well-being (SWB) research is characterized by many large samples, which often results in virtually all variables being significantly related to well-being, even if the associations are small. In this article we explore the strengths of associations between various predictors and SWB outcomes. In addition to standard effect-size statistics, we also examined the range of the SWB scale covered in the distribution of the predictor, allowing us to estimate the strength of influence of each variable, independent of variability in the sample. We analyzed just a few variables to illustrate what our approach reveals. Our analyses included a representative sample of both the world and the United States, and our data included three types of SWB (life satisfaction (LS), positive affect (PA), and negative affect (NA)). The largest effect sizes emerged for societal characteristics, such as between-nations differences, as well as personal characteristics, such as perceived social support. Small or very small effect sizes were consistently found for demographic characteristics, such as sex, age, and marital status. Other effect sizes varied by the type of SWB being considered. For example, income resulted in a large effect size for LS, but small to medium effect sizes for PA and NA. We suggest that when scholars report and interpret the associations of predictor variables with SWB, they consider the strengths of their significant associations.

     

A generalized least squares regression approach for computing effect sizes in single-case research: application examples

A new method for deriving effect sizes from single-case designs is proposed. The strategy is applicable to small-sample time-series data with autoregressive errors. The method uses Generalized Least Squares (GLS) to model the autocorrelation of the data and estimate regression parameters to produce an effect size that represents the magnitude of treatment effect from baseline to treatment phases in standard deviation units. In this paper, the method is applied to two published examples using common single case designs (i.e., withdrawal and multiple-baseline). The results from these studies are described, and the method is compared to ten desirable criteria for single-case effect sizes. Based on the results of this application, we conclude with observations about the use of GLS as a support to visual analysis, provide recommendations for future research, and describe implications for practice.     

Ingroup role models and underrepresented students’ performance and interest in STEM: A meta-analysis of lab and field studies

This meta-analysis synthesizes research on using ingroup role models to improve the performance and interest of underrepresented students in science, technology, engineering, and math (STEM). A systematic literature search resulted in forty-five studies that met the selection criteria, including the presence of a comparison group. Both lab and field studies suffered from small sample bias, with smaller sample sizes predicting larger effect sizes among lab studies, but smaller effect sizes among field studies. Correcting for small sample bias, ingroup role models had a small, but significant positive overall effect (d?=?0.20) among field studies and a non-significant overall effect (d?=?0.04) among lab studies. The only significant moderator was level of interaction, with in-person role models having smaller effects among lab studies (p?=?.008). Implications for interventions to increase the representation of female and underrepresented minority students in STEM and future directions for research are discussed.

     

Current misuses of multiple regression for investigating bivariate hypotheses: an example from the organizational domain

By definition, multiple regression (MR) considers more than one predictor variable, and each variable’s beta will depend on both its correlation with the criterion and its correlation with the other predictor(s). Despite ad nauseam coverage of this characteristic in organizational psychology and statistical texts, researchers’ applications of MR in bivariate hypothesis testing has been the subject of recent and renewed interest. Accordingly, we conducted a targeted survey of the literature by coding articles, covering a five-year span from two top-tier organizational journals, that employed MR for testing bivariate relations. The results suggest that MR coefficients, rather than correlation coefficients, were most common for testing hypotheses of bivariate relations, yet supporting theoretical rationales were rarely offered. Regarding the potential impact on scientific advancement, in almost half of the articles reviewed (44 %), at least one conclusion of each study (i.e., that the hypothesis was or was not supported) would have been different, depending on the author’s use of correlation or beta to test the bivariate hypothesis. It follows that inappropriate decisions to interpret the correlation versus the beta will affect the accumulation of consistent and replicable scientific evidence. We conclude with recommendations for improving bivariate hypothesis testing.     

The use of the effect size in JCR Spanish journals of psychology: from theory to fact

In 1999, Wilkinson and the Task Force on Statistical Inference published "Statistical Methods and Psychology: Guidelines and Explanation." The authors made several recommendations about how to improve the quality of Psychology research papers. One of these was to report some effect-size index in the results of the research. In 2001, the fifth edition of the Publication Manual of the American Psychological Association included this recommendation. In Spain, in 2003, scientific journals like Psicothema or the International Journal of Clinical and Health Psychology (IJCHP) published editorials and papers expressing the need to calculate the effect size in the research papers. The aim of this study is to determine whether the papers published from 2003 to 2008 in the four Spanish journals indexed in the Journal Citation Reports have reported some effect-size index of their results. The findings indicate that, in general, the followup of the norm has been scanty, though the evolution over the analyzed period is different depending on the journal.