INCREASING THE ACCURACY OF CORRECTIONS FOR RANGE RESTRICTION: IMPLICATIONS FOR SELECTION PROCEDURE VALIDITIES AND OTHER RESEARCH RESULTS

The common practice in meta-analyses and in individual studies of correcting for direct range restriction even though range restriction is actually indirect has long been known to lead to undercorrection, but this error has been assumed to be small. Using validity generalization data sets for 4 jobs, this study calibrated this error by comparing meta-analysis results based on corrections for direct range restriction with the more accurate results from a recently developed method of correcting for indirect range restriction. It was found that, on average, correction for direct range restriction resulted in substantial underestimation of operational validities for both job performance measures (21%) and training performance measures (28%). In addition, 90% credibility values were on average underestimated by 38%–40%. In addition to the implications for personnel selection, these findings suggest that similar underestimation of important relationships has occurred in other areas of research, with potential implications for theory development.     

Sampling behaviour in estimating predictive validity in the context of selection and latent variable modelling: A Monte Carlo study

While the effect of selection in predictive validity studies has long been recognized and discussed in psychometric studies, little consideration has been given to this problem in the context of latent variable models. In a recent paper, Muthén & Hsu (1993) proposed and compared estimators of predictive validity of a multifactorial test. Both selectivity and measurement error were considered in the estimation of predictive validity. The purpose of the present paper is to expand on Muthén & Hsu (1993) by examining and comparing the sampling behaviour of three estimators for predictive validity, LQL (listwise, quasi-likelihood estimator), FQL (full, quasi-likelihood estimator) and FS (factor score estimator), using a Monte Carlo approach. Effects of selection procedures, selection ratios and sample sizes on the sampling behaviours of the estimators are also investigated. The results show that FQL and FS are the two preferred estimators and each has different strengths and weaknesses. A real data application is presented to illustrate the practical implementation of the estimators.     

Error-related psychophysiology and negative affect

The error-related negativity (ERN/Ne) and error positivity (Pe) have been associated with error detection and response monitoring. More recently, heart rate (HR) and skin conductance (SC) have also been shown to be sensitive to the internal detection of errors. An enhanced ERN has consistently been observed in anxious subjects and there is some suggestion that the ERN is related to general negative affective experience (NA). The ERN has been source localized to the anterior cingulate cortex-a structure implicated in the regulation of affective, response selection, and autonomic resources. Thus, the findings that autonomic measures and affective distress are related to response monitoring are consistent with anterior cingulate cortex function. In the present experiment, we sought to evaluate more comprehensively the relationship between self-reported negative affect and error-related physiology in a between-groups design. Results indicate that high NA was associated with significantly greater ERN and error-related SCR, and smaller Pe. These results are discussed in terms of anterior cingulate cortex function, psychopathology, and response monitoring.     

On the fate of distractor representations

Research on the topic of distractor inhibition has used different empirical approaches to study how the human mind selects relevant information from the environment, and the results are controversially discussed. One key question that typically arises is how selection deals with the irrelevant information. We used a new selection task, in which participants sometimes had to respond to the distractors instead of the target. Importantly, we varied the time interval between stimuli onset and the cue that signaled participants to respond to the distractors. We analyzed RTs and error rates from responses to distractors as a function of how long the target had been processed (and the distractor ignored) before the cue required a response to the distractor (i.e., stimulus-cue SOA). The data are compatible with selection models assuming that distractor stimuli are initially activated and then deactivated. Thus, we argue for selection models assuming top down deactivation of distractor representations that work in parallel with top down activation of target representations.     

The diagnosticity of individual data for model selection: Comparing signal-detection models of recognition memory

We tested whether the unequal-variance signal-detection (UVSD) and dual-process signal-detection (DPSD) models of recognition memory mimic the behavior of each other when applied to individual data. Replicating previous results, there was no mimicry for an analysis that fit each individual, summed the goodness-of-fit values over individuals, and compared the two sums (i.e., a single model selection). However, when the models were compared separately for each individual (i.e., multiple model selections), mimicry was substantial. To quantify the diagnosticity of the individual data, we used mimicry to calculate the probability of making a model selection error for each individual. For nondiagnostic data (high model selection error), the results were compatible with equal-variance signal-detection theory. Although neither model was justified in this situation, a forced-choice between the UVSD and DPSD models favored the DPSD model for being less flexible. For diagnostic data (low model selection error), the UVSD model was selected more often.     

Pairwise Likelihood Ratio Tests and Model Selection Criteria for Structural Equation Models with Ordinal Variables

Correlated multivariate ordinal data can be analysed with structural equation models. Parameter estimation has been tackled in the literature using limited-information methods including three-stage least squares and pseudo-likelihood estimation methods such as pairwise maximum likelihood estimation. In this paper, two likelihood ratio test statistics and their asymptotic distributions are derived for testing overall goodness-of-fit and nested models, respectively, under the estimation framework of pairwise maximum likelihood estimation. Simulation results show a satisfactory performance of type I error and power for the proposed test statistics and also suggest that the performance of the proposed test statistics is similar to that of the test statistics derived under the three-stage diagonally weighted and unweighted least squares. Furthermore, the corresponding, under the pairwise framework, model selection criteria, AIC and BIC, show satisfactory results in selecting the right model in our simulation examples. The derivation of the likelihood ratio test statistics and model selection criteria under the pairwise framework together with pairwise estimation provide a flexible framework for fitting and testing structural equation models for ordinal as well as for other types of data. The test statistics derived and the model selection criteria are used on data on ‘trust in the police’ selected from the 2010 European Social Survey. The proposed test statistics and the model selection criteria have been implemented in the R package lavaan.     

Selection in touch: negative priming with tactile stimuli

Abundant experimental evidence has demonstrated attentional selection within the visual modality. Furthermore, the consensus view is currently that two processes contribute to selection: the amplification of the target stimuli and the ignoring (or suppression) of any distractor information. However, at present it is less clear how selection is achieved within the tactile modality. In the present study, we analyze the aftereffects of ignoring tactile distractors, in a novel tactile variant of the negative priming paradigm. In the typical negative priming paradigm, repeating an ignored distractor stimulus as the target on the following trial usually leads to a cost in terms of reaction times (RTs) or error rates, thereby indicating that selection is achieved in part by the ignoring of distractors. In two experiments, we observed significant RT costs when a previously ignored vibrotactile stimulus constituted the target in the next trial. This result shows that tactile selection is in part achieved by active ignoring of distractor representations, as has been shown previously in both the visual and auditory modalities.     

Effects of kindergarten retention on children's social-emotional development: an application of propensity score method to multivariate, multilevel data

This study examines the effects of kindergarten retention on children's social-emotional development in the early, middle, and late elementary years. Previous studies have generated mixed results partly due to some major methodological challenges, including selection bias, measurement error, and divergent perceptions of multiple respondents in different domains of child development. The authors address these challenges by using propensity score stratification to contend with selection bias and by embedding measurement models in hierarchical models to account for measurement error and to model dependence among observations. The authors' analyses of a series of multivariate models enable them to compare the retention effects across different respondents over different time points. In general, the results show no evidence suggesting that kindergarten retention does harm to children's social-emotional development. Rather, the findings suggest that, had the retained kindergartners been promoted to the first grade instead, they would possibly have developed a lower level of self-confidence and interest in reading and all school subjects 2 years later and would have displayed a higher level of internalizing problem behaviors at the end of the treatment year and 2 years later.     

Attenuated hypothesis or attenuated test of hypothesis?

After discussing several of the criticisms made by Professor Jensen, it is concluded that the Spearman hypothesis, as currently worded, has been disconfirmed. The near-zero correlation between blacks and low socio-economic status (SES) whites is not an artifact of the methodology. Neither is it a sampling fluke. The logic of sampling error does not apply to the arbitrary selection of intercorrelated tests from an undefined population. It is important also that low and high SES white differences are highly correlate with general factor loadings in a very heterogeneous selection of tests. Research attention should be turned to the difference in outcome of the race and SES comparisons.     

Using Lancaster's mid-P correction to the Fisher's exact test for adverse impact analyses

Adverse impact is often assessed by evaluating whether the success rates for 2 groups on a selection procedure are significantly different. Although various statistical methods have been used to analyze adverse impact data, Fisher's exact test (FET) has been widely adopted, especially when sample sizes are small. In recent years, however, the statistical field has expressed concern regarding the default use of the FET and has proposed several alternative tests. This article reviews Lancaster's mid-P (LMP) test (Lancaster, 1961), an adjustment to the FET that tends to have increased power while maintaining a Type I error rate close to the nominal level. On the basis of Monte Carlo simulation results, the LMP test was found to outperform the FET across a wide range of conditions typical of adverse impact analyses. The LMP test was also found to provide better control over Type I errors than the large-sample Z-test when sample size was very small, but it tended to have slightly lower power than the Z-test under some conditions.     

Incremental validity of the frame-of-reference effect in personality scale scores: a replication and extension

Context-specific personality items provide respondents with a common frame of reference unlike more traditional, noncontextual personality items. The common frame of reference standardizes item interpretation and has been shown to reduce measurement error while increasing validity in comparison to noncontextual items (M. J. Schmit, A. M. Ryan. S. L. Stierwalt. & S. L. Powell, 1995). Although the frame-of-reference effect on personality scales scores has been well investigated (e.g., M. J. Schmit et al., 1995), the ability of this innovation to obtain incremental validity above and beyond the well-established, noncontextual personality scale scores has yet to be examined. The current study replicates and extends work by M. J. Schmit et al. (1995) to determine the incremental validity of the frame-of-reference effect. The results indicate that context-specific personality items do indeed obtain incremental validity above and beyond both noncontextual items and cognitive ability, and in spite of socially desirable responding induced by applicant instructions. The implications of these findings for personnel selection are discussed.     

Pairwise multiple comparisons: A model comparison approach versus stepwise procedures

Researchers in the behavioural sciences have been presented with a host of pairwise multiple comparison procedures that attempt to obtain an optimal combination of Type I error control, power, and ease of application. However, these procedures share one important limitation: intransitive decisions. Moreover, they can be characterized as a piecemeal approach to the problem rather than a holistic approach. Dayton has recently proposed a new approach to pairwise multiple comparisons testing that eliminates intransitivity through a model selection procedure. The present study compared the model selection approach (and a protected version) with three powerful and easy‐to‐use stepwise multiple comparison procedures in terms of the proportion of times that the procedure identified the true pattern of differences among a set of means across several one‐way layouts. The protected version of the model selection approach selected the true model a significantly greater proportion of times than the stepwise procedures and, in most cases, was not affected by variance heterogeneity and non‐normality.     

Distracted by danger: Temporal and spatial dynamics of visual selection in the presence of threat

Threatening stimuli are known to influence attentional and visual processes in order to prioritize selection. For example, previous research showed faster detection of threatening relative to nonthreatening stimuli. This has led to the proposal that threatening stimuli are prioritized automatically via a rapid subcortical route. However, in most studies, the threatening stimulus is always to some extent task relevant. Therefore, it is still unclear if threatening stimuli are automatically prioritized by the visual system. We used the additional singleton paradigm with task-irrelevant fear-conditioned distractors (CS+ and CS-) and indexed the time course of eye movement behavior. The results demonstrate automatic prioritization of threat. First, mean latency of saccades directed to the neutral target was increased in the presence of a threatening (CS+) relative to a nonthreatening distractor (CS-), indicating exogenous attentional capture and delayed disengagement of covert attention. Second, more error saccades were directed to the threatening than to the nonthreatening distractor, indicating a modulation of automatically driven saccades. Nevertheless, cumulative distributions of the saccade latencies showed no modulation of threat for the fastest goal-driven saccades, and threat did not affect the latency of the error saccades to the distractors. Together these results suggest that threatening stimuli are automatically prioritized in attentional and visual selection but not via faster processing. Rather, we suggest that prioritization results from an enhanced representation of the threatening stimulus in the oculomotor system, which drives attentional and visual selection. The current findings are interpreted in terms of a neurobiological model of saccade programming.     

How Do Propensity Score Methods Measure Up in the Presence of Measurement Error? A Monte Carlo Study

Considering that the absence of measurement error in research is a rare phenomenon and its effects can be dramatic, we examine the impact of measurement error on propensity score (PS) analysis used to minimize selection bias in behavioral and social observational studies. A Monte Carlo study was conducted to explore the effects of measurement error on the treatment effect and balance estimates in PS analysis across seven different PS conditioning methods. In general, the results indicate that even low levels of measurement error in the covariates lead to substantial bias in estimates of treatment effects and concomitant reduction in confidence interval coverage across all methods of conditioning on the PS.     

Human food preferences and cultural identity: The case of Aragón (Spain)

This research aims to analyze the relationship between sociocultural values and human food preferences. The latter, as shown in this paper, are greatly influenced by cultural identity. This work stems from a theoretical context that originated in Europe and the United States towards the mid‐twentieth century, within the field of the anthropology of food. A qualitative and quantitative analysis has been performed in the Comunidad Autónoma de Aragón (Spain). Research methods include focus groups, in‐depth interviews, participant observation, and a questionnaire that was handed out to a representative sample of the Aragonese population (816 people over 21 years of age; confidence level of 95.5% and error margin of ±3.5). Regarding the research outcome, a highly significant qualitative and quantitative connection has been found between food selection and cultural identity. In other words, people prefer to consume foods that are symbolically associated with their own culture, in order to reinforce their sense of belonging. Although this study has been carried out in Aragón, it is our belief that the results can be generalized to other areas. The originality and interest of our findings are notable considering that, to date, few works have analyzed the sociocultural factors motivating food behavior. Moreover, these results could be used by public and private organizations to meet objectives such as health promotion and product marketing.     

Informational Darwinism

The Theory of Evolution has, since Darwin, been sustained by contributions from many sciences, most especially from molecular biology. Philosophers, like biologists and the man in the street, have accepted the idea that the contemporary form of evolutionary theory has arrived at a convincing and final structure. As it now stands, natural selection is thought to work through the information-handling mechanism of the DNA molecule. Variation in the genome?s constructive message is achieved through random errors of processing called mutations. How that mechanism and its revision works, and how much information it can hold are fundamental questions for the Neo-Darwinian theory to face. It is argued here that neither the operation nor the data content of the genome, as science understand them, can underpin the role Darwinism assigns to natural selection. It follows that we cannot put our confidence in the explanatory force of Darwinian reasoning, but neither is there an alternative to it.     

Bootstrap standard error and confidence intervals for the correlations corrected for indirect range restriction

The standard Pearson correlation coefficient, r, is a biased estimator of the population correlation coefficient, ρ(XY) , when predictor X and criterion Y are indirectly range-restricted by a third variable Z (or S). Two correction algorithms, Thorndike's (1949) Case III, and Schmidt, Oh, and Le's (2006) Case IV, have been proposed to correct for the bias. However, to our knowledge, the two algorithms did not provide a procedure to estimate the associated standard error and confidence intervals. This paper suggests using the bootstrap procedure as an alternative. Two Monte Carlo simulations were conducted to systematically evaluate the empirical performance of the proposed bootstrap procedure. The results indicated that the bootstrap standard error and confidence intervals were generally accurate across simulation conditions (e.g., selection ratio, sample size). The proposed bootstrap procedure can provide a useful alternative for the estimation of the standard error and confidence intervals for the correlation corrected for indirect range restriction.     

Action sets and decisions in the medial frontal cortex

Activations in human dorsomedial frontal and cingulate cortices are often present in neuroimaging studies of decision making and action selection. Interpretations have emphasized executive control, movement sequencing, error detection and conflict monitoring. Recently, however, experimental approaches, using lesions, inactivation, and cell recording, have suggested that these are just components of the areas' functions. Here we review these results and integrate them with those from neuroimaging. A medial superior frontal gyrus (SFG) region centred on the pre-supplementary motor area (pre-SMA) is involved in the selection of action sets whereas the anterior cingulate cortex (ACC) has a fundamental role in relating actions to their consequences, both positive reinforcement outcomes and errors, and in guiding decisions about which actions are worth making.     

Perspectives of probabilistic inferences: Reinforcement learning and an adaptive network compared

The assumption that people possess a strategy repertoire for inferences has been raised repeatedly. The strategy selection learning theory specifies how people select strategies from this repertoire. The theory assumes that individuals select strategies proportional to their subjective expectations of how well the strategies solve particular problems; such expectations are assumed to be updated by reinforcement learning. The theory is compared with an adaptive network model that assumes people make inferences by integrating information according to a connectionist network. The network's weights are modified by error correction learning. The theories were tested against each other in 2 experimental studies. Study 1 showed that people substantially improved their inferences through feedback, which was appropriately predicted by the strategy selection learning theory. Study 2 examined a dynamic environment in which the strategies' performances changed. In this situation a quick adaptation to the new situation was not observed; rather, individuals got stuck on the strategy they had successfully applied previously. This "inertia effect" was most strongly predicted by the strategy selection learning theory.     

Standard errors and confidence intervals for correlations corrected for indirect range restriction: A simulation study comparing analytic and bootstrap methods

A frequent topic of psychological research is the estimation of the correlation between two variables from a sample that underwent a selection process based on a third variable. Due to indirect range restriction, the sample correlation is a biased estimator of the population correlation, and a correction formula is used. In the past, bootstrap standard error and confidence intervals for the corrected correlations were examined with normal data. The present study proposes a large-sample estimate (an analytic method) for the standard error, and a corresponding confidence interval for the corrected correlation. Monte Carlo simulation studies involving both normal and non-normal data were conducted to examine the empirical performance of the bootstrap and analytic methods. Results indicated that with both normal and non-normal data, the bootstrap standard error and confidence interval were generally accurate across simulation conditions (restricted sample size, selection ratio, and population correlations) and outperformed estimates of the analytic method. However, with certain combinations of distribution type and model conditions, the analytic method has an advantage, offering reasonable estimates of the standard error and confidence interval without resorting to the bootstrap procedure's computer-intensive approach. We provide SAS code for the simulation studies.