Meta-analytic interval estimation for bivariate correlations

The currently available meta-analytic methods for correlations have restrictive assumptions. The fixed-effects methods assume equal population correlations and exhibit poor performance under correlation heterogeneity. The random-effects methods do not assume correlation homogeneity but are based on an equally unrealistic assumption that the selected studies are a random sample from a well-defined superpopulation of study populations. The random-effects methods can accommodate correlation heterogeneity, but these methods do not perform properly in typical applications where the studies are nonrandomly selected. A new fixed-effects meta-analytic confidence interval for bivariate correlations is proposed that is easy to compute and performs well under correlation heterogeneity and nonrandomly selected studies.     

单维测验合成信度元分析

元分析是根据现有研究对感兴趣的主题得出比较准确和有代表性结论的一种重要方法,在心理、教育、管理、医学等社会科学研究中得到广泛应用。信度是衡量测验质量的重要指标,用合成信度能比较准确的估计测验信度。未见有文献提供合成信度元分析方法。本研究在比较对参数进行元分析的三种模型优劣的基础上,在变化系数模型下推出合成信度元分析点估计及区间估计的方法;以区间覆盖率为衡量指标,模拟研究表明本研究提出的合成信度元分析区间估计的方法得当;举例说明如何对单维测验的合成信度进行元分析。     

Should we rely on the Kenward–Roger approximation when using linear mixed models if the groups have different distributions?

The study explores the robustness to violations of normality and sphericity of linear mixed models when they are used with the Kenward–Roger procedure (KR) in split‐plot designs in which the groups have different distributions and sample sizes are small. The focus is on examining the effect of skewness and kurtosis. To this end, a Monte Carlo simulation study was carried out, involving a split‐plot design with three levels of the between‐subjects grouping factor and four levels of the within‐subjects factor. The results show that: (1) the violation of the sphericity assumption did not affect KR robustness when the assumption of normality was not fulfilled; (2) the robustness of the KR procedure decreased as skewness in the distributions increased, there being no strong effect of kurtosis; and (3) the type of pairing between kurtosis and group size was shown to be a relevant variable to consider when using this procedure, especially when pairing is positive (i.e., when the largest group is associated with the largest value of the kurtosis coefficient and the smallest group with its smallest value). The KR procedure can be a good option for analysing repeated‐measures data when the groups have different distributions, provided the total sample sizes are 45 or larger and the data are not highly or extremely skewed.     

On sample size calculation for 2×2 fixed‐effect ANOVA when variances are unknown and possibly unequal

The factorial 2 × 2 fixed‐effect ANOVA is a procedure used frequently in scientific research to test mean differences between‐subjects in all of the groups. But if the assumption of homogeneity is violated, the test for the row, column, and the interaction effect might be invalid or less powerful. Therefore, for planning research in the case of unknown and possibly unequal variances, it is worth developing a sample size formula to obtain the desired power. This article suggests a simple formula to determine the sample size for 2 × 2 fixed‐effect ANOVA for heterogeneous variances across groups. We use the approximate Welch t test and consider the variance ratio to derive the formula. The sample size determination requires two‐step iterations but the approximate sample sizes needed for the main effect and the interaction effect can be determined separately with the specified power. The present study also provides an example and a SAS program to facilitate the calculation process.     

Some recommended statistical analytic practices when reliability generalization studies are conducted

Precursors of the reliability generalization (RG) meta‐analytic approach have not established a single preferred analytic method. By means of five real RG examples, we examine how using different statistical methods to integrate coefficients alpha can influence results in RG studies. Specifically, we compare thirteen different statistical models for averaging reliability coefficients and searching for moderator variables that differ in terms of: (a) whether to transform or not the coefficients alpha, and (b) the statistical model assumed, distinguishing between ordinary least squares methods, the fixed‐effect (FE) model, the varying coefficient (VC) model, and several versions of the random‐effects (RE) model. The results obtained with the different methods exhibited important discrepancies, especially regarding moderator analyses. The main criterion for the model choice should be the extent to which the meta‐analyst intends to generalize the results. RE models are the most appropriate when the meta‐analyst aims to generalize to a hypothetical population of past or future studies, while FE and VC models are the most appropriate when the interest focuses on generalizing the results to a population of studies identical to those included in the meta‐analysis. Finally, some guidelines are proposed for selecting the statistical model when conducting an RG study.     

Gender Differences in Social Dominance Orientation: Gender Invariance May Be Situation Invariance

Most studies of gender differences in social dominance orientation (SDO) have investigated settings more or less hierarchy-enhancing. The aim of this study was to explore gender differences in SDO (1) within social structures varying in equality-enhancement, i.e., communities differing in political equality between men and women, and (2) settings where equality was maximized and held constant, i.e., democratic, voluntary associations, but varying in gender composition, using survey data from a random sample of 831 Swedish adults (median age?=?47 years). There was a significant interaction effect between gender and membership in voluntary associations dominated by women, with men and women displaying equally low levels of SDO. In all other instances the main effect of gender remained statistically significant.     

Point-biserial correlation: Interval estimation,hypothesis testing,meta-analysis,and sample size determination

The point-biserial correlation is a commonly used measure of effect size in two-group designs. New estimators of point-biserial correlation are derived from different forms of a standardized mean difference. Point-biserial correlations are defined for designs with either fixed or random group sample sizes and can accommodate unequal variances. Confidence intervals and standard errors for the point-biserial correlation estimators are derived from the sampling distributions for pooled-variance and separate-variance versions of a standardized mean difference. The proposed point-biserial confidence intervals can be used to conduct directional two-sided tests, equivalence tests, directional non-equivalence tests, and non-inferiority tests. A confidence interval for an average point-biserial correlation in meta-analysis applications performs substantially better than the currently used methods. Sample size formulas for estimating a point-biserial correlation with desired precision and testing a point-biserial correlation with desired power are proposed. R functions are provided that can be used to compute the proposed confidence intervals and sample size formulas.     

Interpreting statistical testing: Process and propensity, not population and random sampling

The standard textbook treatment of conventional statistical tests assumes random sampling from a population and interprets the outcome of the statistical testing as being about a population. Problems with this interpretation include that (1) experimenters rarely make any attempt to randomly sample, (2) if random sampling occurred, conventional statistical tests would not precisely describe the population, and (3) experimenters do not use statistical testing to generalize to a population. The assumption of random sampling can be replaced with the assumption that scores were produced by a process. Rejecting the null hypothesis then leads to a conclusion about process, applying to only the subjects in the experiment (e.g., that some difference in the treatment of two groups caused the difference in average scores). This interpretation avoids the problems noted and fits how statistical testing is used in psychology.     

Generalization gradients and stimulus control in delayed matching-to-sample

Neurological patients were subjects in delayed visual matching-to-sample. The sample and choice stimuli were ellipses of varying size. By measuring the difference in size between the sample on a given trial and the ellipse the subject chose on that trial, gradients of differences between samples and choice stimuli could be plotted. These difference gradients broadened with increasing delays. Sharp gradients were controlled by the samples. Flat gradients were controlled by features of the choice display, independently of the samples. Intermediate gradients reflected combined control by the samples and by the choice displays.     

How Many Subjects Does It Take To Do A Regression Analysis

Numerous rules-of-thumb have been suggested for determining the minimum number of subjects required to conduct multiple regression analyses. These rules-of-thumb are evaluated by comparing their results against those based on power analyses for tests of hypotheses of multiple and partial correlations. The results did not support the use of rules-of-thumb that simply specify some constant (e.g., 100 subjects) as the minimum number of subjects or a minimum ratio of number of subjects (N) to number of predictors (m). Some support was obtained for a rule-of-thumb that N ≥ 50 + 8 m for the multiple correlation and N ≥104 + m for the partial correlation. However, the rule-of-thumb for the multiple correlation yields values too large for N when m ≥ 7, and both rules-of-thumb assume all studies have a medium-size relationship between criterion and predictors. Accordingly, a slightly more complex rule-of thumb is introduced that estimates minimum sample size as function of effect size as well as the number of predictors. It is argued that researchers should use methods to determine sample size that incorporate effect size.     

Meta‐CART: A tool to identify interactions between moderators in meta‐analysis

In the framework of meta‐analysis, moderator analysis is usually performed only univariately. When several study characteristics are available that may account for treatment effect, standard meta‐regression has difficulties in identifying interactions between them. To overcome this problem, meta‐CART has been proposed: an approach that applies classification and regression trees (CART) to identify interactions, and then subgroup meta‐analysis to test the significance of moderator effects. The previous version of meta‐CART has its shortcomings: when applying CART, the sample sizes of studies are not taken into account, and the effect sizes are dichotomized around the median value. Therefore, this article proposes new meta‐CART extensions, weighting study effect sizes by their accuracy, and using a regression tree to avoid dichotomization. In addition, new pruning rules are proposed. The performance of all versions of meta‐CART was evaluated via a Monte Carlo simulation study. The simulation results revealed that meta‐regression trees with random‐effects weights and a 0.5‐standard‐error pruning rule perform best. The required sample size for meta‐CART to achieve satisfactory performance depends on the number of study characteristics, the magnitude of the interactions, and the residual heterogeneity.     

Optimal sample sizes for Welch’s test under various allocation and cost considerations

The issue of the sample size necessary to ensure adequate statistical power has been the focus of considerableattention in scientific research. Conventional presentations of sample size determination do not consider budgetary and participant allocation scheme constraints, although there is some discussion in the literature. The introduction of additional allocation and cost concerns complicates study design, although the resulting procedure permits a practical treatment of sample size planning. This article presents exact techniques for optimizing sample size determinations in the context of Welch (Biometrika, 29, 350–362, 1938) test of the difference between two means under various design and cost considerations. The allocation schemes include cases in which (1) the ratio of group sizes is given and (2) one sample size is specified. The cost implications suggest optimally assigning subjects (1) to attain maximum power performance for a fixed cost and (2) to meet adesignated power level for the least cost. The proposed methods provide useful alternatives to the conventional procedures and can be readily implemented with the developed R and SAS programs that are available as supplemental materials from brm.psychonomic-journals.org/content/supplemental.     

The effect of covariate mean differences on the standard error and confidence interval for the comparison of treatment means

The use of covariates is commonly believed to reduce the unexplained error variance and the standard error for the comparison of treatment means, but the reduction in the standard error is neither guaranteed nor uniform over different sample sizes. The covariate mean differences between the treatment conditions can inflate the standard error of the covariate‐adjusted mean difference and can actually produce a larger standard error for the adjusted mean difference than that for the unadjusted mean difference. When the covariate observations are conceived of as randomly varying from one study to another, the covariate mean differences can be related to a Hotelling's T². Using this Hotelling's T² statistic, one can always find a minimum sample size to achieve a high probability of reducing the standard error and confidence interval width for the adjusted mean difference.     

Individual Differences in Risky Decision Making: A Meta‐analysis of Sensation Seeking and Impulsivity with the Balloon Analogue Risk Task

To represent the state‐of‐the‐art in an effort to understand the relation between personality and risk taking, we selected a popular decision task with characteristics that parallel risk taking in the real world and two personality traits commonly believed to influence risk taking. A meta‐analysis is presented based on 22 studies of the Balloon Analogue Risk Task from which correlations with sensation seeking and impulsivity assessments could be obtained. Results calculated on a total of 2120 participants showed that effect size for the relation of sensation seeking with risk taking was in the small–moderate range ( = .14), whereas the effect size for impulsivity was just around the small effect size threshold ( = .10). Although we considered participants' demographics as moderators, we found only significantly larger effect sizes for the older adolescents and young adults compared with other ages. The findings of the present review supported the view that inconsistencies in personality–risk research were mostly due to random fluctuations of specific effect sizes, rather than to lack of theoretical ties or to measurement unreliability. It is also concluded that studies aimed at relating individual differences in personality to performance in experimental decision tasks need an appropriate sample size to achieve the power to produce significant results. Copyright © 2013 John Wiley & Sons, Ltd.     

Sample size requirements for the design of reliability studies: precision consideration

In multilevel modeling, the intraclass correlation coefficient based on the one-way random-effects model is routinely employed to measure the reliability or degree of resemblance among group members. To facilitate the advocated practice of reporting confidence intervals in future reliability studies, this article presents exact sample size procedures for precise interval estimation of the intraclass correlation coefficient under various allocation and cost structures. Although the suggested approaches do not admit explicit sample size formulas and require special algorithms for carrying out iterative computations, they are more accurate than the closed-form formulas constructed from large-sample approximations with respect to the expected width and assurance probability criteria. This investigation notes the deficiency of existing methods and expands the sample size methodology for the design of reliability studies that have not previously been discussed in the literature.     

An alternative approach to relapse analysis: Using Monte Carlo methods and proportional rates of response

Relapse is the recovery of a previously suppressed response. Animal models have been useful in examining the mechanisms underlying relapse (e.g., reinstatement, renewal, reacquisition, resurgence). However, there are several challenges to analyzing relapse data using traditional approaches. For example, null hypothesis significance testing is commonly used to determine whether relapse has occurred. However, this method requires several a priori assumptions about the data, as well as a large sample size for between‐subjects comparisons or repeated testing for within‐subjects comparisons. Monte Carlo methods may represent an improved analytic technique, because these methods require no prior assumptions, permit smaller sample sizes, and can be tailored to account for all of the data from an experiment instead of some limited set. In the present study, we conducted reanalyses of three studies of relapse (Berry, Sweeney, & Odum, 2014 ; Galizio et al., 2018 ; Odum & Shahan, 2004 ) using Monte Carlo techniques to determine if relapse occurred and if there were differences in rate of response based on relevant independent variables (such as group membership or schedule of reinforcement). These reanalyses supported the previous findings. Finally, we provide general recommendations for using Monte Carlo methods in studies of relapse.     

Low‐price guarantees as advertisement strategy and compensation policy: The more,the better?

Companies sometimes employ a “lowest price or more than the difference back” policy (i.e., a price‐beating guarantee). We investigated whether such a policy is more effective to attract and retain customers than when the exact price difference is promised (i.e., a price‐matching guarantee). The first study revealed that about 60% of the marketers and shop owners in our sample thought that beating price differences is a more effective strategy than matching price differences. However, the four subsequent studies challenged this assumption. Specifically, the advertisement as well as the provision of price‐beating refunds did not have an incremental positive effect on customers' general attitudes in terms of trust, brand perception, loyalty, and shopping intentions beyond the level that was already reached by price‐matching refunds. Moreover, our mediation analyses revealed that the null effect of price‐matching versus price‐beating was mediated by fairness perceptions. From a theoretical perspective, these results are in line with a fairness account, which holds that people do not only evaluate the economic value of an outcome, but also take equality considerations into account. Because price‐beating is literally more expensive than price‐matching, from a practical point of view, companies should be informed that the employment of a price‐beating guarantee is a cost‐ineffective advertisement strategy and compensation policy.     

A dual-task analysis of detection accuracy for the case of high target-distractor similarity: Further evidence for independent processing

Previous studies of visual detection performance indicate that perceptual span has a negligible effect on detection accuracy; hence, previous results suggest that individual characters within a multielement array are perceived independently. In this study, additional experimental factors were examined in order to determine whether conditions exist in which perceptual span influences detection performance: targets and distractors were chosen to be maximally similar, and detection accuracy was studied in relation to display size, target redundancy, and homogeneity of distractor characters. To test for perceptual span effects, a three-state model was proposed which incorporates the assumption that individual characters are perceived independently. The model may be applied to (a) both yes-no and forced-choice tasks, and (b) the case when effects of display characteristics differ for response alternatives. Predictions based on the three-state model agree satisfactorily with observed proportions. Moreover, interactions between display characteristics and response alternatives accord well with model predictions, and accuracy differences between subjects are consistent with individual differences in model parameters. The results provide further evidence for the independence assumption and illustrate additional conditions for which perceptual span has a negligible effect on detection accuracy.     

Confidence intervals for standardized linear contrasts of means

Most psychology journals now require authors to report a sample value of effect size along with hypothesis testing results. The sample effect size value can be misleading because it contains sampling error. Authors often incorrectly interpret the sample effect size as if it were the population effect size. A simple solution to this problem is to report a confidence interval for the population value of the effect size. Standardized linear contrasts of means are useful measures of effect size in a wide variety of research applications. New confidence intervals for standardized linear contrasts of means are developed and may be applied to between-subjects designs, within-subjects designs, or mixed designs. The proposed confidence interval methods are easy to compute, do not require equal population variances, and perform better than the currently available methods when the population variances are not equal.     

Methods for averaging alpha coefficients in reliability generalization studies

The reliability generalization (RG) approach is a kind of meta-analysis that aims to statistically integrate a set of independent reliability coefficients obtained in several applications of a test, with the purpose of characterizing the measurement error and determining which factors related to the studies and samples can explain its variability. Diverse procedures have been proposed in the literature for averaging a set of independent alpha coefficients, and there is no consensus about which methods are best. Here, we present the results of a Monte Carlo simulation study, comparing the performance of twelve procedures proposed in Feldt and Charter, in terms of bias and mean square error. These procedures differ from each other in the transformation (or not) of the coefficients, and in the application or not of a weighting scheme based on sample size. Our results recommend using weighted methods in contrast to unweighted ones, and transforming the coefficients by the Hakstian and Whalen's proposal or by the proposal based on the square root of the inverse alpha coefficient. Lastly, we discuss the relations between the diverse procedures for averaging alpha coefficients with fixed-effects, random-effects, and varying coefficients models.