On robust estimation of effect size under semiparametric models

Estimation of effect size is of interest in many applied fields such as Psychology, Sociology and Education. However there are few nonparametric estimators of effect size proposed in the existing literature, and little is known about the distributional characteristics of these estimators. In this article, two estimators based on the sample quantiles are proposed and studied. The first one is the estimator suggested by Hedges and Olkin (see page 93 of Hedges & Olkin, 1985) for the situation where a treatment effect is evaluated against a control group (Case A). A modified version of the robust estimator by Hedges and Olkin is also proposed for the situation where two parallel treatments are compared (Case B). Large sample distributions of both estimators are derived. Their asymptotic relative efficiencies with respect to the normal maximum likelihood estimators under several common distributions are evaluated. The robust properties of the proposed estimators are discussed with respect to the sample-wise breakdown points proposed by Akritas (1991). Simulation studies are provided in which the performing characteristics of the proposed estimator are compared to that of the nonparametric estimators by Kraemer and Andrews (1982). Interval estimation of the effect sizes is also discussed. In an example, interval estimates for the data set in Kraemer and Andrews (1982) are calculated for both cases A and B.     

Measures of effect size

Two different approaches have been used to derive measures of effect size. One approach is based on the comparison of treatment means. The standardized mean difference is an appropriate measure of effect size when one is merely comparing two treatments, but there is no satisfactory analogue for comparing more than two treatments. The second approach is based on the proportion of variance in the dependent variable that is explained by the independent variable. Estimates have been proposed for both fixed-factor and random-factor designs, but their sampling properties are not well understood. Nevertheless, measures of effect size can allow quantitative comparisons to be made across different studies, and they can be a useful adjunct to more traditional outcome measures such as test statistics and significance levels.     

Group size and the framing effect:

Past research provides conflicting evidence for the role of value in the appearance of framing effects. In this study, the effects of frame and group size were examined using scenarios about less valuable and more valuable groups (animal vs. human). In addition, two picture manipulations, intended to increase the value of the group, were presented. Choice patterns differed for the human and animal groups, with participants exhibiting greater risk seeking overall for the human scenario and showing a framing effect for humans but not animals when no pictures were presented. A small group size increased the proportion of risky choices for both the animal and human scenarios. Presenting pictures with names did lead to framing effects for animals, but providing pictures or pictures and names eliminated framing effects for the human scenario. These findings suggest that the relationship between value and framing effects is a matter of degree.     

Dissociation of the distance effect and size effect in one-digit numbers

Magnitude comparison of single digits is robustly characterized by a distance effect (close numbers are more difficult to compare than numbers further apart) and a size effect (for a given distance, comparison difficulty increases with increasing size). The distance effect indicates access to the mental number line (Dehaene, 1997), and the size effect is usually interpreted as indicating that the mental number line represents larger numbers more vaguely than smaller ones. In contrast, we have argued earlier (Verguts, Fias, & Stevens, 2005) that for symbolic numbers (Arabic or verbal notation), the size effect does not originate from the mental number line but, instead, originates from mappings to relevant output components that are specific for magnitude comparison. If the latter is true, it should be possible to dissociate the distance effect from the size effect in tasks other than magnitude comparison. In two experiments, we observed a robust distance effect insame/different judgments, which implies access to the mental number line. Yet the size effect was absent. Consistent with our prediction, this finding establishes a dissociation between the size effect and the distance effect.     

Useful effect size interpretations for single case research

An obstacle to broader acceptability of effect sizes in single case research is their lack of intuitive and useful interpretations. Interpreting Cohen's d as "standard deviation units difference" and R2 as "percent of variance accounted for" do not resound with most visual analysts. In fact, the only comparative analysis widely supported in single case research (SCR) is "percent of nonoverlapping data." This article explores five alternative interpretations of Cohen's d and R2 effect sizes that may be more acceptable to the SCR field. They are: (a) Cohen's (Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum) "Percent of Nonoverlapping Data" (CPND), (b) Parker et al.'s (Parker, R.I., Cryer, J., Byrns, G., 2006. Controlling trend in single case research. School Psychology Quarterly, 21, 418-440, Parker, R.I., Hagan-Burke, S., Vannest, K., in press. Percent of all non-overlapping data (PAND): An alternative to PND. Journal of Special Education) "Percent of All Nonoverlapping Data" (PAND), (c) Rosenthal et al. (Rosenthal, R., Rosnow, R., & Rubin, D. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge: Cambridge University Press.) "Binomial Effect Size Display" (BESD), (d) "Percentile Rank in Control Group" (PR), and (e) McGraw and Wong's (McGraw, K. O., & Wong, S. P. (1992). A common language effect-size statistic. Psychological Bulletin, 111, 361-365) "Common Language Effect Size" (CLES). Each of the five interpretation schemes are applied to a published data set and are evaluated according to (a) intuitive appeal, (b) relevance to visual analysis, (c) ease of calculation, and (d) technical adequacy. Three of the five appear to be improvements over prevailing practice.     

Recommended effect size statistics for repeated measures designs

Investigators, who are increasingly implored to present and discuss effect size statistics, might comply more often if they understood more clearly what is required. When investigators wish to report effect sizes derived from analyses of variance that include repeated measures, past advice has been problematic. Only recently has a generally useful effect size statistic been proposed for such designs: generalized eta squared (η_G²; Olejnik & Algina, 2003). Here, we present this method, explain that η_G² is preferred to eta squared and partial eta squared because it provides comparability across between-subjects and within-subjects designs, show that it can easily be computed from information provided by standard statistical packages, and recommend that investigators provide it routinely in their research reports when appropriate.     

Required sample size to detect the mediated effect

Mediation models are widely used, and there are many tests of the mediated effect. One of the most common questions that researchers have when planning mediation studies is, "How many subjects do I need to achieve adequate power when testing for mediation?" This article presents the necessary sample sizes for six of the most common and the most recommended tests of mediation for various combinations of parameters, to provide a guide for researchers when designing studies or applying for grants.     

r equivalent: A simple effect size indicator

The purpose of this article is to propose a simple effect size estimate (obtained from the sample size, N, and a p value) that can be used (a) in meta-analytic research where only sample sizes and p values have been reported by the original investigator, (b) where no generally accepted effect size estimate exists, or (c) where directly computed effect size estimates are likely to be misleading. This effect size estimate is called r(equivalent) because it equals the sample point-biserial correlation between the treatment indicator and an exactly normally distributed outcome in a two-treatment experiment with N/2 units in each group and the obtained p value. As part of placing r(equivalent) into a broader context, the authors also address limitations of r(equivalent).     

Multiple trials may yield exaggerated effect size estimates

Published psychological research attempting to support the existence of small and medium effect sizes may not have enough participants to do so accurately, and thus, repeated trials or the use of multiple items may be used in an attempt to obtain significance. Through a series of Monte-Carlo simulations, this article describes the results of multiple trials or items on effect size estimates when the averages and aggregates of a dependent measure are analyzed. The simulations revealed a large increase in observed effect size estimates when the numbers of trials or items in an experiment were increased. Overestimation effects are mitigated by correlations between trials or items, but remain substantial in some cases. Some concepts, such as a P300 wave or a test score, are best defined as a composite of measures. Troubles may arise in more exploratory research where the interrelations among trials or items may not be well described.     

An explanation of the indentation size effect in ceramics

Abstract

A quantitative model is proposed to explain the indentation size effect (ISE) often observed in the hardness response of hard brittle materials, namely that hardness is observed to increase with decreasing indentation size. The model is based on a mixed elastic/plastic materials deformation response whereby plastic deformation occurs in a discrete manner progressively to relieve stresses created by elastic flexure of the surface at the edges of the indentation. During unloading of the indenter, recovery of the elastic increment of deformation, which precedes each new band of plastic deformation, results in the indentation appearing smaller than expected, particularly as the indentation sizes decrease to approach the scale of the plastic deformation band spacing. The model fits observed experimental data well and analysis of hardness/size data in this way is shown to allow both a bulk hardness value and a characteristic deformation band scale to be calculated for a given sample.     

Accuracy of effect size estimates from published psychological research

A Monte-Carlo simulation was used to model the biasing of effect sizes in published studies. The findings from the simulation indicate that, when a predominant bias to publish studies with statistically significant results is coupled with inadequate statistical power, there will be an overestimation of effect sizes. The consequences such an effect size overestimation will then have on meta-analyses and power analyses are highlighted and discussed along with measures which can be taken to reduce the problem.     

An effect size measure and Bayesian analysis of single-case designs

This article describes a linear modeling approach for the analysis of single-case designs (SCDs). Effect size measures in SCDs have been defined and studied for the situation where there is a level change without a time trend. However, when there are level and trend changes, effect size measures are either defined in terms of changes in R² or defined separately for changes in slopes and intercept coefficients. We propose an alternate effect size measure that takes into account changes in slopes and intercepts in the presence of serial dependence and provides an integrated procedure for the analysis of SCDs through estimation and inference based directly on the effect size measure. A Bayesian procedure is described to analyze the data and draw inferences in SCDs. A multilevel model that is appropriate when several subjects are available is integrated into the Bayesian procedure to provide a standardized effect size measure comparable to effect size measures in a between-subjects design. The applicability of the Bayesian approach for the analysis of SCDs is demonstrated through an example.     

An effect size index for comparing two independent alpha coefficients

Since Cronbach proposed the α coefficient in 1951, researchers have contributed to the derivation of its sampling distribution and the testing of related statistical hypotheses. Yet, there has been no research on effect size index relevant to coefficient α to our knowledge. Considering the importance of effect size in understanding quantitative research findings, we therefore developed an effect size index Δ for the comparison of two independent αs with equal test length based on the asymptotic distribution of under the assumptions of normality and compound symmetry. Simulations indicated that the index was applicable when the sample size was at least 100. The robustness of the derived index when the required assumptions were violated was also explored. It is suggested that the index should be applicable in most cases of unequal test lengths and could be extended to non‐normally distributed component scores. Moreover, a small simulation was conducted to explore the behaviour of Δ with correlated errors, a frequently studied situation violating the assumption of compound symmetry. The proposed index was found to be robust unless a large number of highly correlated errors were present in the data.     

The effect of unequal group size on Tukey's HSD procedure

A limitation of the Tukey HSD procedure for multiple comparison has been the requirement of equal number of observations for each group. Three approximation techniques have been suggested when the group sizes are unequal. Each of these techniques was empirically analyzed to determine its effect on TypeI error. Two of the considered variables, average group size and differences in group size, caused differing actual probabilities of TypeI error. One of the three techniques (Kramer's) consistently provided actual probabilities in close agreement with corresponding nominal probabilities.