International comparison of group differences in general mental ability for immigrants versus non‐immigrants

Globalization has led to increased migration and labor mobility over the past several decades and immigrants generally seek jobs in their new countries. Tests of general mental ability (GMA) are common in personnel selection systems throughout the world. Unfortunately, GMA test scores often display differences between majority groups and ethnic subgroups that may represent a barrier to employment for immigrants. The purpose of this study was to examine differences in GMA based on immigrant status in 29 countries (or jurisdictions of countries) throughout the world using an existing database that employs high‐quality measurement and sampling methodologies with large sample sizes. The primary findings were that across countries, non‐immigrants (n = 139,464) scored approximately half of a standard deviation (d = .53) higher than first‐generation immigrants (n = 22,162) but only one‐tenth of a standard deviation (d = .12) higher than second‐generation immigrants (n = 6,428). Considerable variability in effect sizes was found across countries as Nordic European and Germanic European countries evidenced the highest non‐immigrant/first‐generation immigrant mean differences and Anglo countries the smallest. Countries with the lowest income inequality tended to evidence the highest differences in GMA between non‐immigrants and first‐generation immigrants. Implications for GMA testing as a potential barrier to immigrant employment success and the field's current understanding of group differences in GMA test scores will be discussed.     

The impact of sample non‐normality on ANOVA and alternative methods

In this journal, Zimmerman (2004, 2011) has discussed preliminary tests that researchers often use to choose an appropriate method for comparing locations when the assumption of normality is doubtful. The conceptual problem with this approach is that such a two‐stage process makes both the power and the significance of the entire procedure uncertain, as type I and type II errors are possible at both stages. A type I error at the first stage, for example, will obviously increase the probability of a type II error at the second stage. Based on the idea of Schmider et al. (2010) , which proposes that simulated sets of sample data be ranked with respect to their degree of normality, this paper investigates the relationship between population non‐normality and sample non‐normality with respect to the performance of the ANOVA, Brown–Forsythe test, Welch test, and Kruskal–Wallis test when used with different distributions, sample sizes, and effect sizes. The overall conclusion is that the Kruskal–Wallis test is considerably less sensitive to the degree of sample normality when populations are distinctly non‐normal and should therefore be the primary tool used to compare locations when it is known that populations are not at least approximately normal.     

The meta‐analysis of partial effect sizes

This paper discusses the synthesis of partial effect sizes derived from multivariate settings. The general statistical properties of the d‐effect size are derived, extending Hedges's statement of zero‐order properties. These general properties have direct relevance in the synthesis of a set of independent effect sizes arising from empirical studies predicated on a common theoretical model. We discuss possible solutions to the problem of comparing effect sizes arising from models employing differing sets of covariates. We apply the general statistical properties in the synthesis of gender performance differences in first‐level economics courses at three New Zealand universities. The model of academic performance is based on Spearman's conception of general academic ability and specific ability in economics.     

Hypothesis tests for population heterogeneity in meta‐analysis

Choice of the appropriate model in meta‐analysis is often treated as an empirical question which is answered by examining the amount of variability in the effect sizes. When all of the observed variability in the effect sizes can be accounted for based on sampling error alone, a set of effect sizes is said to be homogeneous and a fixed‐effects model is typically adopted. Whether a set of effect sizes is homogeneous or not is usually tested with the so‐called Q test. In this paper, a variety of alternative homogeneity tests – the likelihood ratio, Wald and score tests – are compared with the Q test in terms of their Type I error rate and power for four different effect size measures. Monte Carlo simulations show that the Q test kept the tightest control of the Type I error rate, although the results emphasize the importance of large sample sizes within the set of studies. The results also suggest under what conditions the power of the tests can be considered adequate.     

Meta‐CART: A tool to identify interactions between moderators in meta‐analysis

In the framework of meta‐analysis, moderator analysis is usually performed only univariately. When several study characteristics are available that may account for treatment effect, standard meta‐regression has difficulties in identifying interactions between them. To overcome this problem, meta‐CART has been proposed: an approach that applies classification and regression trees (CART) to identify interactions, and then subgroup meta‐analysis to test the significance of moderator effects. The previous version of meta‐CART has its shortcomings: when applying CART, the sample sizes of studies are not taken into account, and the effect sizes are dichotomized around the median value. Therefore, this article proposes new meta‐CART extensions, weighting study effect sizes by their accuracy, and using a regression tree to avoid dichotomization. In addition, new pruning rules are proposed. The performance of all versions of meta‐CART was evaluated via a Monte Carlo simulation study. The simulation results revealed that meta‐regression trees with random‐effects weights and a 0.5‐standard‐error pruning rule perform best. The required sample size for meta‐CART to achieve satisfactory performance depends on the number of study characteristics, the magnitude of the interactions, and the residual heterogeneity.     

Significance tests to determine the direction of effects in linear regression models

Previous studies have discussed asymmetric interpretations of the Pearson correlation coefficient and have shown that higher moments can be used to decide on the direction of dependence in the bivariate linear regression setting. The current study extends this approach by illustrating that the third moment of regression residuals may also be used to derive conclusions concerning the direction of effects. Assuming non‐normally distributed variables, it is shown that the distribution of residuals of the correctly specified regression model (e.g., Y is regressed on X) is more symmetric than the distribution of residuals of the competing model (i.e., X is regressed on Y). Based on this result, 4 one‐sample tests are discussed which can be used to decide which variable is more likely to be the response and which one is more likely to be the explanatory variable. A fifth significance test is proposed based on the differences of skewness estimates, which leads to a more direct test of a hypothesis that is compatible with direction of dependence. A Monte Carlo simulation study was performed to examine the behaviour of the procedures under various degrees of associations, sample sizes, and distributional properties of the underlying population. An empirical example is given which illustrates the application of the tests in practice.     

Detecting a lack of association: An equivalence testing approach

Researchers often test for a lack of association between variables. A lack of association is usually established by demonstrating a non‐significant relationship with a traditional test (e.g., Pearson's r). However, for logical as well as statistical reasons, such conclusions are problematic. In this paper, we discuss and compare the empirical Type I error and power rates of three lack of association tests. The results indicate that large, sometimes very large, sample sizes are required for the test statistics to be appropriate. What is especially problematic is that the required sample sizes may exceed what is practically feasible for the conditions that are expected to be common among researchers in psychology. This paper highlights the importance of using available lack of association tests, instead of traditional tests of association, for demonstrating the independence of variables, and qualifies the conditions under which these tests are appropriate.     

Effects of cultural background on WAIS‐III and WMS‐III performances after moderate‐severe traumatic brain injury

The assessment of cognitive function in individuals of culturally and linguistically diverse background poses considerable challenges for the psychologist, particularly when English proficiency is limited. This study explored the effects of diverse cultural background and non‐Western educational background on Wechsler Adult Intelligence Scale‐Third edition (WAIS‐III) and Wechsler Memory Scale‐Third Edition (WMS‐III) performances in moderate‐severe traumatic brain injury within an outpatient rehabilitation setting. Participants were aged 16–65 years and met careful selection criteria. IQ, index and age‐scaled subtest scores were compared across three groups: (a) English‐speaking background (n = 130), (b) culturally and linguistically diverse background and education completed in English (n = 33), and (c) culturally and linguistically diverse background and non‐English education (n = 33). Cultural backgrounds included people of Asian, European, Middle Eastern, African and Oceania origin. Results were that the English‐educated culturally and linguistically diverse group performed lower than the English‐speaking background group on some verbal WAIS‐III measures; effect sizes were small‐moderate. The non‐English‐educated culturally and linguistically diverse group performed lower than both groups on several WAIS‐III and one WMS‐III measure, with large effect sizes. Clinical implications included the need for caution in interpreting test scores to avoid diagnostic errors and the need for further development of valid assessment tools.     

Worry in imagery and verbal form: effect on residual working memory capacity

Worry-prone individuals have less residual working memory capacity during worry compared to low-worriers (Hayes, Hirsch, & Mathews, 2008). People typically worry in verbal form, and the present study investigated whether verbal worry depletes working memory capacity more than worry in imagery-based form. High and low-worriers performed a working memory task, random interval generation, whilst thinking about a worry in verbal or imagery form. High (but not low) worriers had less available working memory capacity when worrying in verbal compared to imagery-based form. The findings could not be accounted for by general attentional control, amount of negatively-valenced thought, or appraisals participants made about worry topics. The findings indicate that the verbal nature of worry is implicated in the depletion of working memory resources during worry among high-worriers, and point to the potential value of imagery-based techniques in cognitive-behavioural treatments for problematic worry.     

Multilevel mediation analysis: The effects of omitted variables in the 1–1–1 model

Multilevel mediation analysis examines the indirect effect of an independent variable on an outcome achieved by targeting and changing an intervening variable in clustered data. We study analytically and through simulation the effects of an omitted variable at level 2 on a 1–1–1 mediation model for a randomized experiment conducted within clusters in which the treatment, mediator, and outcome are all measured at level 1. When the residuals in the equations for the mediator and the outcome variables are fully orthogonal, the two methods of calculating the indirect effect (ab, c – c′) are equivalent at the between‐ and within‐cluster levels. Omitting a variable at level 2 changes the interpretation of the indirect effect and will induce correlations between the random intercepts or random slopes. The equality of within‐cluster ab and c – c′ no longer holds. Correlation between random slopes implies that the within‐cluster indirect effect is conditional, interpretable at the grand mean level of the omitted variable.     

Tests of Homoscedasticity,Normality, and Missing Completely at Random for Incomplete Multivariate Data

Test of homogeneity of covariances (or homoscedasticity) among several groups has many applications in statistical analysis. In the context of incomplete data analysis, tests of homoscedasticity among groups of cases with identical missing data patterns have been proposed to test whether data are missing completely at random (MCAR). These tests of MCAR require large sample sizes n and/or large group sample sizes n _i , and they usually fail when applied to nonnormal data. Hawkins (Technometrics 23:105–110, 1981) proposed a test of multivariate normality and homoscedasticity that is an exact test for complete data when n _i are small. This paper proposes a modification of this test for complete data to improve its performance, and extends its application to test of homoscedasticity and MCAR when data are multivariate normal and incomplete. Moreover, it is shown that the statistic used in the Hawkins test in conjunction with a nonparametric k-sample test can be used to obtain a nonparametric test of homoscedasticity that works well for both normal and nonnormal data. It is explained how a combination of the proposed normal-theory Hawkins test and the nonparametric test can be employed to test for homoscedasticity, MCAR, and multivariate normality. Simulation studies show that the newly proposed tests generally outperform their existing competitors in terms of Type I error rejection rates. Also, a power study of the proposed tests indicates good power. The proposed methods use appropriate missing data imputations to impute missing data. Methods of multiple imputation are described and one of the methods is employed to confirm the result of our single imputation methods. Examples are provided where multiple imputation enables one to identify a group or groups whose covariance matrices differ from the majority of other groups.     

Informative sources of math‐related self‐efficacy expectations and their relationship with math‐related self‐efficacy,interest, and preference

In this study, by using the hypotheses of social‐cognitive career theory (SCCT), the relationship between informative sources of math‐related self‐efficacy expectations and self‐efficacy, interest, and math‐weighted major preferences were investigated. The participants in this study were students (N = 590) from high schools in Southern Turkey. Participants completed measures of sources of math‐related self‐efficacy, self‐efficacy, interest, and choice consideration related to math‐weighted majors. While running analyses, sampling was split in half at random and psychometric analyses of the scales were run with the first half (n = 298); the second half (n = 292) was used to test the hypotheses. By doing this, the testing of hypotheses would not be affected by just one specific sampling characteristic. A structural equation model was tested and findings indicated general support for hypotheses that these sources predict self‐efficacy beliefs and these beliefs also predict interest. However, contrary to SCCT's predictions, math‐weighted major preferences are not predicted by math‐related self‐efficacy expectations and interest. In the Discussion, first, the reasons why the measurement of vicarious learning or modelling did not significantly predict self‐efficacy expectations focus on the relationships between personal accomplishments and persuasion and reported physiological arousal and personal accomplishments. Some information is given as to how counsellors and math teachers can work collaboratively to raise perceptions about informative sources. Second, explanations are given about why self‐efficacy and interest did not predict math‐weighted preferences. It is mentioned that this could be the reason for the constrained and complex system of the university entrance exams and placement in Turkey. The importance of investigating environmental variables on SCCT in developing countries was emphasized. Suggestions are also given for further research. It is concluded that this theory should be tested in developing countries like Turkey.     

Enhancing the language development of toddlers in foster care by promoting foster parents’ sensitivity: Results from a randomized controlled trial

Young children in foster care are at increased risk for problematic language development, making early intervention a critical tool in enhancing these children's foundational language abilities. This study examined the efficacy of an early preventative intervention, Attachment and Biobehavioral Catch‐up for Toddlers (ABC‐T), in improving the receptive vocabulary abilities of toddlers placed in foster care. All the children had been removed from their biological parents’ care and placed into foster care. When children were between 24 and 36 months old, foster parents were contacted by research staff and consented to participate. Parents were randomly assigned using a random number generator to receive either ABC‐T (n = 45), which aimed to promote sensitive parenting for children who have experienced early adversity, or a control intervention (n = 43). Foster children's receptive vocabulary skills were assessed post‐intervention using the Peabody Picture Vocabulary Test, Third Edition, when children were between 36 and 60 months old. Children whose foster parents received ABC‐T demonstrated more advanced receptive vocabulary abilities than children whose foster parents received the control intervention. The positive effect of ABC‐T on foster children's receptive vocabulary was mediated by increases in foster parents’ sensitivity during parent–child interactions. Trial registration: ClinicalTrials.gov NCT01261806.     

Assessing Item Fit for Unidimensional Item Response Theory Models Using Residuals from Estimated Item Response Functions

Residual analysis (e.g. Hambleton & Swaminathan, Item response theory: principles and applications, Kluwer Academic, Boston, 1985; Hambleton, Swaminathan, & Rogers, Fundamentals of item response theory, Sage, Newbury Park, 1991) is a popular method to assess fit of item response theory (IRT) models. We suggest a form of residual analysis that may be applied to assess item fit for unidimensional IRT models. The residual analysis consists of a comparison of the maximum-likelihood estimate of the item characteristic curve with an alternative ratio estimate of the item characteristic curve. The large sample distribution of the residual is proved to be standardized normal when the IRT model fits the data. We compare the performance of our suggested residual to the standardized residual of Hambleton et al. (Fundamentals of item response theory, Sage, Newbury Park, 1991) in a detailed simulation study. We then calculate our suggested residuals using data from an operational test. The residuals appear to be useful in assessing the item fit for unidimensional IRT models.     

Analysis of categorical moderators in mixed‐effects meta‐analysis: Consequences of using pooled versus separate estimates of the residual between‐studies variances

Subgroup analyses allow us to examine the influence of a categorical moderator on the effect size in meta‐analysis. We conducted a simulation study using a dichotomous moderator, and compared the impact of pooled versus separate estimates of the residual between‐studies variance on the statistical performance of the Q _{B (P)} and Q _{B (S)} tests for subgroup analyses assuming a mixed‐effects model. Our results suggested that similar performance can be expected as long as there are at least 20 studies and these are approximately balanced across categories. Conversely, when subgroups were unbalanced, the practical consequences of having heterogeneous residual between‐studies variances were more evident, with both tests leading to the wrong statistical conclusion more often than in the conditions with balanced subgroups. A pooled estimate should be preferred for most scenarios, unless the residual between‐studies variances are clearly different and there are enough studies in each category to obtain precise separate estimates.     

Equivalence tests for comparing correlation and regression coefficients

Equivalence tests are an alternative to traditional difference‐based tests for demonstrating a lack of association between two variables. While there are several recent studies investigating equivalence tests for comparing means, little research has been conducted on equivalence methods for evaluating the equivalence or similarity of two correlation coefficients or two regression coefficients. The current project proposes novel tests for evaluating the equivalence of two regression or correlation coefficients derived from the two one‐sided tests (TOST) method (Schuirmann, 1987, J. Pharmacokinet. Biopharm, 15, 657) and an equivalence test by Anderson and Hauck (1983, Stat. Commun., 12, 2663). A simulation study was used to evaluate the performance of these tests and compare them with the common, yet inappropriate, method of assessing equivalence using non‐rejection of the null hypothesis in difference‐based tests. Results demonstrate that equivalence tests have more accurate probabilities of declaring equivalence than difference‐based tests. However, equivalence tests require large sample sizes to ensure adequate power. We recommend the Anderson–Hauck equivalence test over the TOST method for comparing correlation or regression coefficients.     

Detecting Changes in Simulated Events Using Partial‐Interval Recording and Momentary Time Sampling III: Evaluating Sensitivity as a Function of Session Length

In a series of two studies, we graphed simulated data representing continuous duration recording and continuous frequency recording into ABAB reversal designs depicting small, moderate, and large behavior changes during 10‐min, 30‐min, and 60‐min sessions. Data sets were re‐scored using partial‐interval recording and momentary time sampling with interval sizes set at 10 s, 20 s, 30 s, 1 min, and 2 min. In study 1, we visually inspected converted data for experimental control and compared the conclusion with those from the respective continuous duration recording or continuous frequency recording data to test for false negatives. In study 2, we evaluated the extent to which interval methods that were sensitive to changes in study 1 produced false positives. In part, the results show that momentary time sampling with interval sizes up to 30 s detected a wide range of changes in duration events and frequency events during lengthier observation periods. The practical implications of the findings are briefly discussed. Copyright © 2011 John Wiley & Sons, Ltd.     

Effectiveness of Seeking Safety for Co‐Occurring Posttraumatic Stress Disorder and Substance Use

The authors evaluated the Seeking Safety program's effectiveness for treating posttraumatic stress disorder (PTSD) and substance use symptoms across 12 between‐groups studies (N = 1,997 participants). Separate meta‐analytic procedures for studies implementing wait list/no treatment (n = 1,042) or alternative treatments (n = 1,801) yielded medium effect sizes for Seeking Safety for decreasing symptoms of PTSD and modest effects for decreasing symptoms of substance use. Limitations of the findings and implications for counselors are discussed.     

Administrative Assumptions in Top‐Down Selection: A Test in Graduate School Admission Decisions

Although top‐down selection is the gold standard for making personnel decisions, several administrative assumptions must be met for it to be effective. We discuss three of these assumptions and test two of them: (1) top applicants will accept an offer, and (2) the time organisations give applicants to consider an offer will not influence the availability of next‐tier applicants. We also examine the effectiveness of top‐down selection by comparing it to an administratively simpler procedure, random selection above a threshold. Using archival admissions data from three university graduate psychology programs, we found that top applicants were less likely to accept an offer; however, waiting time did not influence applicant availability. In comparing the quality of applicants actually selected (with a top‐down procedure) with the quality of applicants selected at random (from above five progressively stringent thresholds), we found that at higher admission thresholds, random selection resulted in better or equal quality applicants as top‐down selection, depending on the criteria. We discuss implications for future research and practice.     

Methods for second order meta-analysis and illustrative applications

This paper presents methods for second order meta-analysis along with several illustrative applications. A second order meta-analysis is a meta-analysis of a number of statistically independent and methodologically comparable first order meta-analyses examining ostensibly the same relationship in different contexts. First order meta-analysis greatly reduces sampling error variance but does not eliminate it. The residual sampling error is called second order sampling error. The purpose of a second order meta-analysis is to estimate the proportion of the variance in mean meta-analytic effect sizes across multiple first order meta-analyses attributable to second order sampling error and to use this information to improve accuracy of estimation for each first order meta-analytic estimate. We present equations and methods based on the random effects model for second order meta-analysis for three situations and three empirical applications of second order meta-analysis to illustrate the potential value of these methods to the pursuit of cumulative knowledge.