When effect sizes disagree: the case of r and d

The increased use of effect sizes in single studies and meta-analyses raises new questions about statistical inference. Choice of an effect-size index can have a substantial impact on the interpretation of findings. The authors demonstrate the issue by focusing on two popular effect-size measures, the correlation coefficient and the standardized mean difference (e.g., Cohen's d or Hedges's g), both of which can be used when one variable is dichotomous and the other is quantitative. Although the indices are often practically interchangeable, differences in sensitivity to the base rate or variance of the dichotomous variable can alter conclusions about the magnitude of an effect depending on which statistic is used. Because neither statistic is universally superior, researchers should explicitly consider the importance of base rates to formulate correct inferences and justify the selection of a primary effect-size statistic.     

Enantiomorphy through the looking glass: literacy effects on mirror-image discrimination

To examine whether enantiomorphy (i.e., the ability to discriminate lateral mirror images) is influenced by the acquisition of a written system that incorporates mirrored letters (e.g., b and d), unschooled illiterate adults were compared with people reading the Latin alphabet, namely, both schooled literate adults and unschooled adults alphabetized in adulthood. In various sorting and same-different comparison tasks with nonlinguistic materials, illiterate participants displayed some sensitivity to enantiomorphic contrasts but performed far worse than all the other participant groups when the task required paying attention to such contrasts. The difficulties of illiterate participants were more severe with enantiomorphs than with rotations in the plane or shape contrasts. Learning a written system that incorporates enantiomorphic letters thus pushes the beginning reader to break the mirror invariance characteristic of the visual system, and this process generalizes beyond the realm of symbolic characters.     

A probability-based measure of effect size: robustness to base rates and other factors

Calculating and reporting appropriate measures of effect size are becoming standard practice in psychological research. One of the most common scenarios encountered involves the comparison of 2 groups, which includes research designs that are experimental (e.g., random assignment to treatment vs. placebo conditions) and nonexperimental (e.g., testing for gender differences). Familiar measures such as the standardized mean difference (d) or the point-biserial correlation (rpb) characterize the magnitude of the difference between groups, but these effect size measures are sensitive to a number of additional influences. For example, R. E. McGrath and G. J. Meyer (2006) showed that rpb is sensitive to sample base rates, and extending their analysis to situations of unequal variances reveals that d is, too. The probability-based measure A, the nonparametric generalization of what K. O. McGraw and S. P. Wong (1992) called the common language effect size statistic, is insensitive to base rates and more robust to several other factors (e.g., extreme scores, nonlinear transformations). In addition to its excellent generalizability across contexts, A is easy to understand and can be obtained from standard computer output or through simple hand calculations.     

A program to compute McGraw and wong’s common language effect size indicator

McGraw and Wong (1992) have described a very appealing index of effect size that requires no prior knowledge of statistics to understand, which they termed CL, the common language effect size indicator. CL is the probability that a score randomly sampled from one distribution will be larger than a randomly sampled score from a second distribution. McGraw and Wong describe how to compute CL from the means and standard deviations of two groups, using tables of normal curve probability values. The program described herein computes CL without lookup tables but also permits the user to compute CL from Cohen’s d, from a t test for independent groups, or from point-biserial r, the correlation between a dichotomous and a continuous variable. A table giving CL for various values of d is also provided, as are the equations for converting t and r to d.     

The Developmental Course of Distance,Time, and Velocity Concepts:A Generative Connectionist Model

Connectionist simulations of children's acquisition of distance (d), time (t), and velocity (v) concepts using a generative algorithm-cascade-correlation-are reported. Rules that correlated most highly with network responses during training were consistent with the developmental course of children's concepts. Networks integrated the defining dimensions of the concepts first by identity rules (e.g., v = d), then additive rules (e.g., v = d - t), and finally multiplicative rules (e.g., v = d ÷ t. The results are discussed in terms of similarity to children's development, the contribution of connectionism to the study of cognitive development, contrasts with alternative models, and directions for future research.     

Effect-size indices for dichotomized outcomes in meta-analysis

It is very common to find meta-analyses in which some of the studies compare 2 groups on continuous dependent variables and others compare groups on dichotomized variables. Integrating all of them in a meta-analysis requires an effect-size index in the same metric that can be applied to both types of outcomes. In this article, the performance in terms of bias and sampling variance of 7 different effect-size indices for estimating the population standardized mean difference from a 2 x 2 table is examined by Monte Carlo simulation, assuming normal and nonnormal distributions. The results show good performance for 2 indices, one based on the probit transformation and the other based on the logistic distribution.     

Comparing Personality Test Formats and Warnings: Effects on criterion‐related validity and test‐taker reactions

The potential for applicant response distortion on personality measures remains a major concern in high‐stakes testing situations. Many approaches to understanding response distortion are too transparent (e.g., instructed faking studies) – or are too subtle (e.g., correlations with social desirability measures as indices of faking). Recent research reveals more promising approaches in two methods: using forced‐choice (FC) personality test items and warning against faking. The present study examined effects of these two methods on criterion‐related validity and test‐taker reactions. Results supported incremental validity for an FC and Likert‐scale measure in warning and no‐warning conditions, above and beyond cognitive ability. No clear differences emerged between the FC vs Likert measures or warning vs no‐warning conditions in terms of validity. However, some evidence suggested that FC measures and warnings may produce negative test‐taker reactions. We conclude with implications for implementation in selection settings.     

How meta-analysis increases statistical power

One of the most frequently cited reasons for conducting a meta-analysis is the increase in statistical power that it affords a reviewer. This article demonstrates that fixed-effects meta-analysis increases statistical power by reducing the standard error of the weighted average effect size (T.) and, in so doing, shrinks the confidence interval around T.. Small confidence intervals make it more likely for reviewers to detect nonzero population effects, thereby increasing statistical power. Smaller confidence intervals also represent increased precision of the estimated population effect size. Computational examples are provided for 3 effect-size indices: d (standardized mean difference), Pearson's r, and odds ratios. Random-effects meta-analyses also may show increased statistical power and a smaller standard error of the weighted average effect size. However, the authors demonstrate that increasing the number of studies in a random-effects meta-analysis does not always increase statistical power.     

Searching for certainty: Religious beliefs and intolerance toward value-violating groups

Religiosity has been consistently linked to prejudice toward a variety of outgroups. This article proposes that this is the case only when religiosity reflects a specific aspect of seeking guidance and security in daily practices and complex sociocultural norms. Outgroups that challenge the epistemic certainty that belief in God provides are rejected in an effort to protect this certainty. The results from two studies found that uncertainty avoidance was related to belief in God and this belief mediated the relationship between uncertainty avoidance and intolerance within the context of general human rights (Study 1), and the derogation of value-violating groups (e.g., homosexuals or followers of other religions) but not of groups that pose no threat to religious values (old or poor people) (Study 2). The interpretative dimension of religiosity (i.e., the way in which people process religious content) is not connected to security seeking, as reflected in the lack of a correlation with uncertainty avoidance and with different prejudice measures. The results are discussed in relation to past research on religiosity and prejudice, and suggest that for people who avoid uncertainty, only those types of religious beliefs that provide a sense of certainty are linked with intolerance toward value-violating groups.     

III. WISC-R scatter indices: Useful information for differential diagnosis?

This study analyzed the WISC-R verbal-performance, full scale IQ, verbal IQ, and performance IQ scatter indices of 101 verified special education students. The results indicated that none of these measures discriminated among emotionally disturbed, learning disabled, minimally brain injured, and educable mentally retarded children. These scatter indices were also compared with analogous data collected from a nationwide, representative sample of normal children (Kaufman, 1976a, 1976b). Statistically significant differences were obtained between the two groups on verbal-performance, full scale IQ, and performance IQ scatter dimensions. Further analyses revealed, however, that 40% of the normal children exhibited as much or more scatter than the average special education student on each of these three scatter indices. There were no significant differences between the two groups on the verbal IQ scatter index. The results are discussed in terms of the limited utility of WISC-R scatter indices for the purpose of differential diagnosis.     

When variability matters more than meaning: the effect of lexical forms on use of phonemic contrasts

During the first half of the 2nd year of life, infants struggle to use phonemic distinctions in label-object association tasks. Prior experiments have demonstrated that exposure to the phonemes in distinct lexical forms (e.g., /d/ and /t/ in daddy and tiger, respectively) facilitates infants' use of phonemic contrasts but also that they struggle to generalize the use of phonemic contrasts to novel syllabic contexts (Thiessen, 2007; Thiessen & Yee, 2010). Further, in prior research, infants have been provided only with experience in lexical forms that refer to novel objects, while many lexical forms in the natural environment do not have easily identified visual referents. The experiments in this article show that even lexical forms without referents can facilitate use of phonemic contrasts. Additionally, the results indicate that when lexical forms provide infants with enough variability (for example, a consonant followed by multiple different vowels), infants are able to generalize to novel contexts.     

Discrimination and production of English vowels by bilingual speakers of Spanish and English

The goal of this study was to examine whether listeners bilingual in Spanish and English would have difficulty in the discrimination of English vowel contrasts. An additional goal was to estimate the correlation between their discrimination and production of these vowels. Participants (40 bilingual Spanish- and English-speaking and 40 native monolingual English-speaking college students, 23-36 years of age) participated (M age = 25.3 yr., Mdn = 25.0). The discrimination and production of English vowels in real and novel words by adult participants bilingual in Spanish and English were examined and their discrimination was compared with that of 40 native monolingual English-speaking participants. Stimuli were presented within triads in an ABX paradigm. Novel words were chosen to represent new words when learning a new language and to provide a more valid test of discrimination. Bilingual participants' productions of vowels were judged by two independent listeners to estimate the correlation between discrimination and production. Discrimination accuracy was significantly greater for native English-speaking participants than for bilingual participants for vowel contrasts and novel words. Significant errors also appeared in the bilingual participants' productions of certain vowels. Earlier age of acquisition, absence of communication problems, and greater percentage of time devoted to communication contributed to greater accuracy in discrimination and production.     

Thinking About One’s Subjective Well-Being: Average Trends and Individual Differences

In two studies, participants reported what they had been thinking about while completing measures of subjective well-being (SWB). These thought reports were analyzed with respect to life domain, valence, and how strongly they were related to actual levels of SWB. Most people focused on their life circumstances (e.g., career) rather than on dispositional predictors (e.g., personality) of SWB. The domains mentioned most frequently (career, family, romantic life) were also the ones that were most strongly related to actual SWB, indicating that most of people think about things that actually contribute to their SWB. Some domains are predominantly mentioned in positive contexts (e.g., family) whereas others are predominantly mentioned in negative contexts (e.g., money). On average, people thought more about positive than about negative things, a result that is magnified for respondents high in extraversion or emotional stability. In sum, these findings provide insight into what people think contributes to their SWB; beliefs that may guide them as they make important decisions.     

Cultural variations in achievement motivation despite equivalent motivational strength: Motivational concerns among Indian and American corporate professionals

We explored the possibility that persons from two cultures, Indian and American, might be similar in overall levels of motivational strength yet differ significantly in their experience and expression of achievement motivation. We anticipated that Indian, more than American, corporate professionals would incorporate not only self-based but also other-oriented concerns (e.g., for the welfare of co-workers and community members) into their experience of achievement motivation in the workplace. American and Indian corporate professionals responded to an online survey that included a novel measure tapping interpersonal concerns in achievement motivation. Despite the groups being equivalent on multiple demographic indicators and on traditional indices of both motivational strength and motivational orientation, Americans and Indians differed substantially in motivational concerns, with Indians grounding their sense of achievement motivation more strongly in concerns for extended family, co-workers and community. Implications for the study of cultural variations in personality processes are discussed.     

A Comparative Investigation of Confidence Intervals for IndependentVariables in Linear Regression

In linear regression, the most appropriate standardized effect size for individual independent variables having an arbitrary metric remains open to debate, despite researchers typically reporting a standardized regression coefficient. Alternative standardized measures include the semipartial correlation, the improvement in the squared multiple correlation, and the squared partial correlation. No arguments based on either theoretical or statistical grounds for preferring one of these standardized measures have been mounted in the literature. Using a Monte Carlo simulation, the performance of interval estimators for these effect-size measures was compared in a 5-way factorial design. Formal statistical design methods assessed both the accuracy and robustness of the four interval estimators. The coverage probability of a large-sample confidence interval for the semipartial correlation coefficient derived from Aloe and Becker was highly accurate and robust in 98% of instances. It was better in small samples than the Yuan-Chan large-sample confidence interval for a standardized regression coefficient. It was also consistently better than both a bootstrap confidence interval for the improvement in the squared multiple correlation and a noncentral interval for the squared partial correlation.     

Moral conviction: another contributor to attitude strength or something more?

Attitudes held with strong moral conviction (moral mandates) were predicted to have different interpersonal consequences than strong but nonmoral attitudes. After controlling for indices of attitude strength, the authors explored the unique effect of moral conviction on the degree that people preferred greater social (Studies 1 and 2) and physical (Study 3) distance from attitudinally dissimilar others and the effects of moral conviction on group interaction and decision making in attitudinally homogeneous versus heterogeneous groups (Study 4). Results supported the moral mandate hypothesis: Stronger moral conviction led to (a) greater preferred social and physical distance from attitudinally dissimilar others, (b) intolerance of attitudinally dissimilar others in both intimate (e.g., friend) and distant relationships (e.g., owner of a store one frequents), (c) lower levels of good will and cooperativeness in attitudinally heterogeneous groups, and (d) a greater inability to generate procedural solutions to resolve disagreements.     

Adding versus averaging: Evaluability theory applied to job choice decisions

While the unique roles of individual job attributes (e.g., salary and benefits) in job and organizational attraction have received extensive research attention, research examining the mechanisms through which an overall evaluation of a job option is made by combining evaluations of individual attributes is scarce. The current study examined the process through which job choice decisions are made under three conditions: when evaluating a single job offer, when comparing two job offers, and when evaluating more than two job offers. In Study 1, it was found that when a single job offer is evaluated, the average of perceived values of attributes in an offer (e.g., the perceived attractiveness of a salary) drives the choice, whereas the difference between jobs is what matters when two jobs are evaluated simultaneously, potentially leading to a preference reversal between conditions when the same two jobs are evaluated. In Study 2, it was found that average values of attributes across options (e.g., average salary in all job offers received) influence job choice when more than two job offers are evaluated simultaneously. These findings indicate that in all three conditions, job choice decisions are influenced by the evaluability of the choice set, which becomes low when a single job offer is evaluated without any context, or when more than two job offers are evaluated simultaneously, and becomes high when two jobs are compared with each other. When evaluability is low, candidates resort to averaging as the decision rule, whereas adding is used when evaluability is high.     

A measure of effect size for experimental designs with heterogeneous variances

A recent trend in the psychological literature has been to include measures of effect size when reporting probability values. The several measures of effect size associated with the Student t test for two independent samples are appropriate only when the variances are homogeneous. In this paper, commonly used measures of effect size are considered and compared, using four data sets. A chance-corrected measure of effect size is provided for two or more treatment groups characterized by either homogeneous or heterogeneous variances.