USING RANDOM RATHER THAN FIXED EFFECTS MODELS IN META-ANALYSIS: IMPLICATIONS FOR SITUATIONAL SPECIFICITY AND VALIDITY GENERALIZATION

Combining statistical information across studies (i.e., meta-analysis) is a standard research tool in applied psychology. The most common meta-analytic approach in applied psychology, the fixed effects approach, assumes that individual studies are homogeneous and are sampled from the same population. This model assumes that sampling error alone explains the majority of observed differences in study effect sizes and its use has lead some to challenge the notion of situational specificity in favor of validity generalization. We critique the fixed effects methodology and propose an advancement–the random effects model (RE) which provides estimates of how between-study differences influence the relationships under study. RE models assume that studies are heterogeneous since they are often conducted by different investigators under different settings. Parameter estimates of both models are compared and evidence in favor of the random effects approach is presented. We argue against use of the fixed effects model because it may lead to misleading conclusions about situational specificity.     

Interpreting statistical testing: Process and propensity, not population and random sampling

The standard textbook treatment of conventional statistical tests assumes random sampling from a population and interprets the outcome of the statistical testing as being about a population. Problems with this interpretation include that (1) experimenters rarely make any attempt to randomly sample, (2) if random sampling occurred, conventional statistical tests would not precisely describe the population, and (3) experimenters do not use statistical testing to generalize to a population. The assumption of random sampling can be replaced with the assumption that scores were produced by a process. Rejecting the null hypothesis then leads to a conclusion about process, applying to only the subjects in the experiment (e.g., that some difference in the treatment of two groups caused the difference in average scores). This interpretation avoids the problems noted and fits how statistical testing is used in psychology.     

领悟社会支持性别差异的元分析

研究采用元分析技术, 对采用Zimet“多维度领悟社会支持量表”调查中国人群领悟社会支持的58篇文献进行了元分析, 涉及样本总量为31286人(男性15279), 调查的时间跨度为2002~2014年。研究结果显示:(1)领悟社会支持有显著的性别差异, 女性在领悟社会支持总分、家庭内(外)支持、朋友支持及重要他人支持的得分均显著高于男性, 效果量d的取值范围为[-0.266,-0.183];(2)大学生领悟社会支持总分和领悟家庭支持的性别差异显著大于中学生。本研究结果对于探明中国人群中领悟社会支持的性别差异及其发展趋势具有一定的理论意义。     

Model Selection with the Linear Mixed Model for Longitudinal Data

Model building or model selection with linear mixed models (LMMs) is complicated by the presence of both fixed effects and random effects. The fixed effects structure and random effects structure are codependent, so selection of one influences the other. Most presentations of LMM in psychology and education are based on a multilevel or hierarchical approach in which the variance-covariance matrix of the random effects is assumed to be positive definite with nonzero values for the variances. When the number of fixed effects and random effects is unknown, the predominant approach to model building is a step-up method in which one starts with a limited model (e.g., few fixed and random intercepts) and then additional fixed effects and random effects are added based on statistical tests. A model building approach that has received less attention in psychology and education is a top-down method. In the top-down method, the initial model has a single random intercept but is loaded with fixed effects (also known as an “overelaborate” model). Based on the overelaborate fixed effects model, the need for additional random effects is determined. There has been little if any examination of the ability of these methods to identify a true population model (i.e., identifying the model that generated the data). The purpose of this article is to examine the performance of the step-up and top-down model building approaches for exploratory longitudinal data analysis. Student achievement data sets from the Chicago longitudinal study serve as the populations in the simulations.     

Moderated multiple regression for interactions involving categorical variables: a statistical control for heterogeneous variance across two groups

Moderated multiple regression (MMR) arguably is the most popular statistical technique for investigating regression slope differences (interactions) across groups (e.g., aptitude-treatment interactions in training and differential test score-job performance prediction in selection testing). However, heterogeneous error variances can greatly bias the typical MMR analysis, and the conditions that cause heterogeneity are not uncommon. Statistical corrections that have been developed require special calculations and are not conducive to follow-up analyses that describe an interaction effect in depth. A weighted least squares (WLS) approach is recommended for 2-group studies. For 2-group studies, WLS is statistically accurate, is readily executed through popular software packages (e.g., SAS Institute, 1999; SPSS, 1999), and allows follow-up tests.     

The Relationship Between Perceived and Actual Physical Fitness: A Meta-Analysis

The purpose of our study was to meta-analytically estimate the relationship between actual and perceived fitness, and variables that may moderate this relationship (i.e., gender, age, and perceived fitness measure). Computer, hand, and journal literature searches identified 53 potential studies. On review, 28 of these studies met the inclusion criteria of having a perceived and actual (i.e., objective) measure of physical fitness. Using random effects modeling procedures, we found a medium mean effect size of .38 (N = 33 independent effect sizes, p < .001, Fail-Safe N = 4,654). Moderator analyses revealed that younger people had more accurate estimates of their fitness, and standardized perceived fitness measures had significantly larger effect sizes than unstandardized perceived fitness measures. In summary, men and women of all ages accurately perceived their fitness level, with larger effects evidenced for younger people and for studies that used standardized perceived fitness measures. Unfortunately, because most Americans are either low active or sedentary, their perceptions reveal that they are not fit. Research limitations and future directions are discussed.     

What is the pronunciation for -ough and the spelling for /u/? A database for computing feedforward and feedback consistency in English

Recent studies suggest that performance attendant on visual word perception is affected not only by feedforward inconsistency (i.e., multiple ways to pronounce a spelling) but also by feedback inconsistency (i.e., multiple ways to spell a pronunciation). In the present study, we provide a statistical analysis of these types of inconsistency for all monosyllabic English words. This database can be used as a tool for controlling, selecting, and constructing stimulus materials for psycholinguistic and neuropsychological research. Such large-scale statistical analyses are necessary devices for developing metrics of inconsistency, for generating hypotheses for psycholinguistic experiments, and for building models of word perception, speech perception, and spelling.     

Problematic standard errors and confidence intervals for skewness and kurtosis

Many statistics packages print skewness and kurtosis statistics with estimates of their standard errors. The function most often used for the standard errors (e.g., in SPSS) assumes that the data are drawn from a normal distribution, an unlikely situation. Some textbooks suggest that if the statistic is more than about 2 standard errors from the hypothesized value (i.e., an approximate value for the critical value from the t distribution for moderate or large sample sizes when α = 5%), the hypothesized value can be rejected. This is an inappropriate practice unless the standard error estimate is accurate and the sampling distribution is approximately normal. We show distributions where the traditional standard errors provided by the function underestimate the actual values, often being 5 times too small, and distributions where the function overestimates the true values. Bootstrap standard errors and confidence intervals are more accurate than the traditional approach, although still imperfect. The reasons for this are discussed. We recommend that if you are using skewness and kurtosis statistics based on the 3^rd and 4^th moments, bootstrapping should be used to calculate standard errors and confidence intervals, rather than using the traditional standard. Software in the freeware R for this article provides these estimates.     

Testing mediation effects in cross-classified multilevel data

In this article, we propose an approach to test mediation effects in cross-classified multilevel data in which the initial cause is associated with one crossed factor, the mediator is associated with the other crossed factor, and the outcome is associated with Level-1 units (i.e., the 2^(A)?2^(B)?1 design). Multiple-membership models and cross-classified random effects models are used to estimate the indirect effects. The method is illustrated using real data from the Early Childhood Longitudinal Study–Kindergarten Cohort (1998). The results from the simulation study show that the proposed method can produce a consistent estimate of the indirect effect and reliable statistical inferences, given an adequate sample size.     

A survey of residual analysis and a new test of residual trend

A survey of residual analysis in behavior‐analytic research reveals that existing methods are problematic in one way or another. A new test for residual trends is proposed that avoids the problematic features of the existing methods. It entails fitting cubic polynomials to sets of residuals and comparing their effect sizes to those that would be expected if the sets of residuals were random. To this end, sampling distributions of effect sizes for fits of a cubic polynomial to random data were obtained by generating sets of random standardized residuals of various sizes, n. A cubic polynomial was then fitted to each set of residuals and its effect size was calculated. This yielded a sampling distribution of effect sizes for each n. To test for a residual trend in experimental data, the median effect size of cubic‐polynomial fits to sets of experimental residuals can be compared to the median of the corresponding sampling distribution of effect sizes for random residuals using a sign test. An example from the literature, which entailed comparing mathematical and computational models of continuous choice, is used to illustrate the utility of the test.     

Aggregate and individual replication probability within an explicit model of the research process

We study a model of the research process in which the true effect size, the replication jitter due to changes in experimental procedure, and the statistical error of effect size measurement are all normally distributed random variables. Within this model, we analyze the probability of successfully replicating an initial experimental result by obtaining either a statistically significant result in the same direction or any effect in that direction. We analyze both the probability of successfully replicating a particular experimental effect (i.e., the individual replication probability) and the average probability of successful replication across different studies within some research context (i.e., the aggregate replication probability), and we identify the conditions under which the latter can be approximated using the formulas of Killeen (2005a, 2007). We show how both of these probabilities depend on parameters of the research context that would rarely be known in practice. In addition, we show that the statistical uncertainty associated with the size of an initial observed effect would often prevent accurate estimation of the desired individual replication probability even if these research context parameters were known exactly. We conclude that accurate estimates of replication probability are generally unattainable.     

SYLLABARIUM: An online application for deriving complete statistics for Basque and Spanish orthographic syllables

The present article introduces SYLLABARIUM, a new Web tool addressing the needs of linguists, psycholinguists, and cognitive scientists who work with Spanish and/or Basque and are interested in retrieving information about several syllable-related parameters. This new online syllabic database allows the user to generate complete lists of Spanish and Basque syllables with information about the syllable frequency. Among other measures, for a given orthographic syllable, SYLLABARIUM provides its number of occurrences (i.e., the type frequency), the summed lexical frequency of the words that contain this syllable (i.e., the token frequency), and the positional distribution of type and token frequencies. The cross-language feature of SYLLABARIUM is of special interest to researchers aiming to explore the influence of the syllable in bilingualism. The Web tool is available at www.bcbl.eu/syllabarium.     

Treating stimuli as a random factor in social psychology: a new and comprehensive solution to a pervasive but largely ignored problem

Throughout social and cognitive psychology, participants are routinely asked to respond in some way to experimental stimuli that are thought to represent categories of theoretical interest. For instance, in measures of implicit attitudes, participants are primed with pictures of specific African American and White stimulus persons sampled in some way from possible stimuli that might have been used. Yet seldom is the sampling of stimuli taken into account in the analysis of the resulting data, in spite of numerous warnings about the perils of ignoring stimulus variation (Clark, 1973; Kenny, 1985; Wells & Windschitl, 1999). Part of this failure to attend to stimulus variation is due to the demands imposed by traditional analysis of variance procedures for the analysis of data when both participants and stimuli are treated as random factors. In this article, we present a comprehensive solution using mixed models for the analysis of data with crossed random factors (e.g., participants and stimuli). We show the substantial biases inherent in analyses that ignore one or the other of the random factors, and we illustrate the substantial advantages of the mixed models approach with both hypothetical and actual, well-known data sets in social psychology (Bem, 2011; Blair, Chapleau, & Judd, 2005; Correll, Park, Judd, & Wittenbrink, 2002).     

Fixed Effects vs. Random Effects Meta-Analysis Models: Implications for Cumulative Research Knowledge

Research conclusions in the social sciences are increasingly based on meta-analysis, making questions of the accuracy of meta-analysis critical to the integrity of the base of cumulative knowledge. Both fixed effects (FE) and random effects (RE) meta-analysis models have been used widely in published meta-analyses. This article shows that FE models typically manifest a substantial Type I bias in significance tests for mean effect sizes and for moderator variables (interactions), while RE models do not. Likewise, FE models, but not RE models, yield confidence intervals for mean effect sizes that are narrower than their nominal width, thereby overstating the degree of precision in meta-analysis findings. This article demonstrates analytically that these biases in FE procedures are large enough to create serious distortions in conclusions about cumulative knowledge in the research literature. We therefore recommend that RE methods routinely be employed in meta-analysis in preference to FE methods.     

Five-factor personality domains and job performance: A second order meta-analysis

Several meta-analyses have investigated the job-related validities of the traits associated with the Five-Factor Model (FFM). The presence of second-order sampling error, however, might complicate the interpretation of these meta-analyses (i.e., random error across different meta-analyses). The current paper therefore evaluates variability across different meta-analyses and tests for key moderators (i.e., performance criteria, sources of ratings, and context). Results suggested that the variation in the predictive validity of several FFM traits and facets across multiple performance criteria was attributable to second-order sampling error. However, true variation existed for some moderators such as sources of personality information (self-reports versus informant reports). The current analyses therefore enhance knowledge about the usefulness and generalizability of personality traits in work contexts.     

Working memory and second language comprehension and production: A meta-analysis

Although working memory (WM) figures centrally in many theories of second language (L2) proficiency development and processing, some have argued that the importance of WM is overstated (e.g., Juffs, Transactions of the Philological Society, 102, 199–225, 2004). Despite many studies over the past two decades, the literature lacks a quantitative synthesis of the extant results. In this article, we report a meta-analysis of data from 79 samples involving 3,707 participants providing 748 effect sizes. The results indicate that WM is positively associated with both L2 processing and proficiency outcomes, with an estimated population effect size (ρ) of .255. In additional analyses, we assessed whether the WM–criterion relationship was modulated by potential covariates identified in the literature search (i.e., participant characteristics, WM measure features, criterion measure factors, and publication status). The results of the covariate analyses indicated larger effect sizes for the executive control (vs. storage) component of WM, and for verbal (vs. nonverbal) measures of WM. Minimal publication bias was detected, suggesting that WM has a robust, positive relationship with L2 outcomes. We discuss the implications of these results for models of WM and theories of L2 processing and L2 proficiency development.     

有中介的调节模型的拓展及其效应量

传统的有中介的调节(mediated moderation, meMO)模型关于误差方差齐性的假设经常被违背, 应用研究中也缺乏测量meMO效应大小的指标。对于单层数据, 本文借助于两层建模的思想, 提出了一种可用于处理方差非齐性的两层有中介的调节(2meMO)模型; 给出了用于测量meMO分析中总调节效应、直接调节效应和有中介调节效应大小的效应量。通过Monte Carlo模拟研究, 比较了meMO和2meMO模型在参数和效应量估计上的表现。并通过实际案例解释了2meMO模型的应用以及效应量的计算和解释。     

A memory-based account of automatic numerosity processing

We investigated the mechanisms responsible for the automatic processing of the numerosities represented by digits in the size congruity effect (Henik & Tzelgov, 1982). The algorithmic model assumes that relational comparisons of digit magnitudes (e.g., larger than {8,2}) create this effect. If so, congruity effects ought to require two digits. Memory-based models assume that associations between individual digits and the attributes "small" and "large" create this effect. If so, congruity effects ought only to require one digit. Contrary to the algorithmic model and consistent with memory-based models, congruity effects were just as large when subjects judged the relative physical sizes of small digits paired with letters as when they judged the relative physical sizes of two digits. This finding suggests that size congruity effects can be produced without comparison algorithms.     

A Bayesian Approach to More Stable Estimates of Group-Level Effects in Contextual Studies

Multilevel analyses are often used to estimate the effects of group-level constructs. However, when using aggregated individual data (e.g., student ratings) to assess a group-level construct (e.g., classroom climate), the observed group mean might not provide a reliable measure of the unobserved latent group mean. In the present article, we propose a Bayesian approach that can be used to estimate a multilevel latent covariate model, which corrects for the unreliable assessment of the latent group mean when estimating the group-level effect. A simulation study was conducted to evaluate the choice of different priors for the group-level variance of the predictor variable and to compare the Bayesian approach with the maximum likelihood approach implemented in the software Mplus. Results showed that, under problematic conditions (i.e., small number of groups, predictor variable with a small ICC), the Bayesian approach produced more accurate estimates of the group-level effect than the maximum likelihood approach did.     

A phase transition model for the speed-accuracy trade-off in response time experiments

Most models of response time (RT) in elementary cognitive tasks implicitly assume that the speed-accuracy trade-off is continuous: When payoffs or instructions gradually increase the level of speed stress, people are assumed to gradually sacrifice response accuracy in exchange for gradual increases in response speed. This trade-off presumably operates over the entire range from accurate but slow responding to fast but chance-level responding (i.e., guessing). In this article, we challenge the assumption of continuity and propose a phase transition model for RTs and accuracy. Analogous to the fast guess model (Ollman, 1966), our model postulates two modes of processing: a guess mode and a stimulus-controlled mode. From catastrophe theory, we derive two important predictions that allow us to test our model against the fast guess model and against the popular class of sequential sampling models. The first prediction--hysteresis in the transitions between guessing and stimulus-controlled behavior--was confirmed in an experiment that gradually changed the reward for speed versus accuracy. The second prediction--bimodal RT distributions--was confirmed in an experiment that required participants to respond in a way that is intermediate between guessing and accurate responding.