Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

Previous studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta-analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta-analysis of the Hispanic–White mean difference (d-value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d-values for indirect range restriction and combine them with other meta-analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion-related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.     

The Performance of Methods to Test Upper-Level Mediation in the Presence of Nonnormal Data

A Monte Carlo study compared the statistical performance of standard and robust multilevel mediation analysis methods to test indirect effects for a cluster randomized experimental design under various departures from normality. The performance of these methods was examined for an upper-level mediation process, where the indirect effect is a fixed effect and a group-implemented treatment is hypothesized to impact a person-level outcome via a person-level mediator. Two methods—the bias-corrected parametric percentile bootstrap and the empirical-M test—had the best overall performance. Methods designed for nonnormal score distributions exhibited elevated Type I error rates and poorer confidence interval coverage under some conditions. Although preliminary, the findings suggest that new mediation analysis methods may provide for robust tests of indirect effects.     

“Statistical,” “Practical,” and “Clinical”: How Many Kinds of Significance Do Counselors Need to Consider?

The present article reviews and distinguishes 3 related but different types of significance: “statistical,” “practical,” and “clinical.” A framework for conceptualizing the many “practical” effect size indices is described. Several effect size indices that counseling researchers can use, or that counselors reading the literature may encounter, are summarized. A way of estimating “corrected” intervention effects is proposed. It is suggested that readers should expect authors to report indices of “practical” or “clinical” significance, or both, within their research reports; and it is noted that indeed some journals now require such reports.     

Maximal force production requires OPTIMAL conditions

The OPTIMAL theory of motor learning identifies several motivational and attentional factors that draw out latent motor performance capabilities. One implication of the OPTIMAL theory of motor learning (Wulf & Lewthwaite, 2016) is that standardized motor performance assessments likely do not reflect maximal capabilities unless they are “optimized” with appropriate testing conditions. The present study examined the effects of three key motivational (enhanced expectancies, EE, and autonomy support, AS) and attentional (external focus, EF) variables in the OPTIMAL theory on maximum force production. In Experiment 1, a handgrip strength task was used. EE, AS, and EF were implemented, in a counterbalanced order, on consecutive trial blocks in an optimized group. A control group performed all blocks under neutral conditions. While there were no group differences on Block 1 (baseline), the optimized group outperformed the control group on all other blocks. In Experiment 2, participants performed two one-repetition maximum (1-RM) squat lift tests, separated by one week. Two groups, an optimized group and control group, had similar 1-RM values on the first test performed under neutral conditions. However, on the second test, a group performing under optimized conditions (EE, AS, EF) showed an increase in 1-RM, while there was no change from the first to the second test for a control group. We argue that standard test conditions may not produce true maximal performance. The findings corroborate the importance of key factors in the OPTIMAL theory and should be applied to ensure accurate strength performance assessment.     

SOME COMMENTS ON CULTURE FAIR TESTS1

An attempt has been made in this paper to show that culture fair tests have some problems associated with them. These tests should be examined and reviewed closely before being used and should not be regarded as the answer to testing the culturally disadvantaged. The following points were made in this paper: Culture fair tests measure different psychological functions. Culture fair tests today measure such functions as spatial visualization, abstract reasoning, perceptual speed, etc. Culture fair tests vary considerably in format. Some are pencil and paper tests, some are performance tests. Some use verbal instructions, others do not. There are many test parameters along which culture fair tests now vary. Some evidence suggests that culture fair tests possibly increase the differential between the culturally disadvantaged and the more advantaged population. Use of these tests may not be in the best interests of minority groups. It is not yet definite about the kind of items on which culturally disadvantaged people perform poorer. Some evidence suggests that they do better on verbal items and worst on perceptual items, which is in contrast to the assumption of most proponents of culture-fair tests. The validity of culture fair tests has not been shown to be better than more traditional tests. In contrast, some research even indicates that they do not show relationships as high. What is to be done, if anything, about the test differentials between the culturally disadvantaged and the majority population? Some individuals (Lorge, 1964; Coffman, 1964) agree that the elimination of group differences on tests is futile and argue that the real task at hand is a realistic attempt to study the behavioral significance of test differences. In essence, this is an all out attempt to collect validation information. Does a particular score for a black have the same behavioral implications of a higher (or lower) test score for a white? Are there criterion differences that are related to test differences? Do differential validities exist for various subgroups? Are the standard errors of estimates different for different groups? This approach is essentially what has been pursued by individuals investigating the “moderating” effects of subgrouping by race and/or socio-economic factors. The investigation of test differences within and between subgroups is called for. In my opinion, attempting to mask test differentials by using culture fair tests may in actuality have a reverse effect than what was intended. Test differentials may actually increase, making it more difficult for culturally disadvantaged individuals to be selected into schools, jobs, etc. Clearly, the construction of culture fair tests is not the only answer to testing the disadvantaged.     

The accessibility of the term “contempt” and the meaning of the unilateral lip curl

Four studies are reported investigating the conditions under which various proposed facial expressions of contempt are labelled “contempt”. Only under forced-choice conditions are any of these expressions labelled “contempt” above chance; free responses are at or below chance. Contrary to predictions from Rosenberg and Ekman's (1995) explanation of poor free-response performance, participants demonstrating the best understanding of “contempt”, and those primed by prior tasks to have the concept readily accessible did not do better than other subjects. Using the forced-choice paradigm, supposedly neutral expressions were labelled “contempt” by 70% of respondents. It is concluded that poor performance in free-response studies is not due to inaccessibility or unfamiliarity of “contempt”, that the unilateral lip curl included in the JACFEE set of expressions of basic emotions (Matsumoto & Ekman, 1988) is not decoded as contempt, and that good performance in forced-choice studies results from artifacts of the method.     

Evaluation of Multi-parameter Test Statistics for Multiple Imputation

In Ordinary Least Square regression, researchers often are interested in knowing whether a set of parameters is different from zero. With complete data, this could be achieved using the gain in prediction test, hierarchical multiple regression, or an omnibus F test. However, in substantive research scenarios, missing data often exist. In the context of multiple imputation, one of the current state-of-art missing data strategies, there are several different analogous multi-parameter tests of the joint significance of a set of parameters, and these multi-parameter test statistics can be referenced to various distributions to make statistical inferences. However, little is known about the performance of these tests, and virtually no research study has compared the Type 1 error rates and statistical power of these tests in scenarios that are typical of behavioral science data (e.g., small to moderate samples, etc.). This paper uses Monte Carlo simulation techniques to examine the performance of these multi-parameter test statistics for multiple imputation under a variety of realistic conditions. We provide a number of practical recommendations for substantive researchers based on the simulation results, and illustrate the calculation of these test statistics with an empirical example.     

Inhibition of Threshold Excitation in the Human Eye

Sixty–seven albino rats were tested in a maze situation under five different conditions in an attempt to investigate the presence of “social influences.” One condition was comprised of naive subjects, each run singly. A second condition was comprised of naive subjects run in pairs. A third condition consisted of naive subjects run in squads of three each. A fourth condition consisted of one naive subject run with one trained subject; and the remaining condition consisted of one naive subject run with two trained subjects. The results showed that performance was facilitated by increasing the number of naive subjects in the presence of other naive subjects (“numbers effect”), by the presence of a performing, trained S (“trained effect”), and by increasing the number of trained subjects (“interaction effect”). Inasmuch as these findings are in apparent contradiction to those reported by some previous researchers, an attempt was made to provide an explanation for the differences in experimental findings.     

Inclined to see it your way: Do altercentric intrusion effects in visual perspective taking reflect an intrinsically social process?

It has been suggested that some aspects of mental state understanding recruit a rudimentary, but fast and efficient, processing system, demonstrated by the obligatory slowing down of judgements about what the self can see when this is incongruent with what another can see. We tested the social nature of this system by investigating to what extent these altercentric intrusions are elicited under conditions that differed in their social relevance and, further, how these related to self-reported social perspective taking and empathy. In Experiment 1, adult participants were asked to make “self” or “other” perspective-taking judgements during congruent (“self” and “other” can see the same items) or incongruent conditions (“self” and “other” cannot see the same items) in conditions that were social (i.e., involving a social agent), semisocial (an arrow), or nonsocial (a dual-coloured block). Reaction time indices of altercentric intrusion effects were present across all conditions, but were significantly stronger for the social than for the less social conditions. Self-reported perspective taking and empathy correlated with altercentric intrusion effects in the social condition only. In Experiment 2, the significant correlations for the social condition were replicated, but this time with gaze duration indices of altercentric intrusion effects. Findings are discussed with regard to the degree to which this rudimentary system is socially specialized and how it is linked to more conceptual understanding.     

Modeling the Practical Effects of Applicant Reactions: Subgroup Differences in Test‐Taking Motivation,Test Performance,and Selection Rates

Research suggests that Black–White differences in test‐taking motivation may be related to subgroup test score differences, but this research has not shown the extent to which minimizing subgroup motivation differences will reduce subgroup differences in selection rates and adverse impact. This Monte Carlo study examined how enhancing Blacks' test‐taking motivation for cognitive ability tests might reduce adverse impact across a range of (a) subgroup test differences, (b) selection ratios, (c) subgroup differences in test‐taking motivation, and (d) relationships between motivation and test scores. The results suggest that although enhancing test‐taking motivation will consistently reduce subgroup differences in test performance and adverse impact, the effect is often small and will not eliminate adverse impact for any condition we examine. However, under some conditions the reduction may be important, and the discussion considers conditions where even these minimal reductions may be practically helpful.     

UNDER WHAT CIRCUMSTANCES CAN A UNIONIZED COMPANY USE TESTING FOR PROMOTION?1

A job-related, “fair test” of ability, when used as one tool and not as the sole determining factor, has been established by arbitrators to be an appropriate selection instrument. If a test has been determined as being job-related (i.e., related to the actual performance of the job), has been administered and scored both fairly and consistently, it will be considered a “fair test.” Generally, in unioncompany contracts that mention testing, a “fair test” contains the above qualifications. Arbitrators have indicated that the union involved should be afforded the opportunity to see the test; it was made quite clear, however, that the union should not have the test. In all cases reviewed, tests were upheld by arbitrators when they (1) were “fair tests,” and (2) did not conflict with the contract language. Past testing practice was not a determining factor when these two conditions existed. A total of 69 cases between 1953 and 1967 have been found relevant–27 from 1953 to 1962 and 42 from 1963 to 1967. In the first study, 13 cases were lost by a company; in six of these cases a company violated the union-company contract and in seven cases the test used did not qualify as a “fair test.” A total of twelve cases were lost by a company in the second study; in two cases a company violated the union-company contract and in ten cases the test used did not qualify as a “fair test.”     

The Impact of Stimuli Color in Lexical Decision and Semantic Word Categorization Tasks

In two experiments, we examined the impact of color on cognitive performance by asking participants to categorize stimuli presented in three different colors: red, green, and gray (baseline). Participants were either asked to categorize the meaning of words as related to the concepts of “go” or “stop” (Experiment 1) or to indicate if a neutral verbal stimulus was a word or not (lexical decision task, Experiment 2). Overall, we observed performance facilitation in response to go stimuli presented in green (vs. red or gray) and performance inhibition in response to go stimuli presented in red. The opposite pattern was observed for stop‐related stimuli. Importantly, results also indicated that color might also be used to categorize neutral stimuli. Overall, these findings provide support to the green‐go and red‐stop color associations and test the potential functional autonomy acquired by these colors and the boundary conditions to their effects on stimuli categorization.     

Being In Front Is Good—But Where Is In Front? Preferences for Spatial Referencing Affect Evaluation

Speakers of English frequently associate location in space with valence, as in moving up and down the “social ladder.” If such an association also holds for the sagittal axis, an object “in front of” another object would be evaluated more positively than the one “behind.” Yet how people conceptualize relative locations depends on which frame of reference (FoR) they adopt—and hence on cross-linguistically diverging preferences. What is conceptualized as “in front” in one variant of the relative FoR (e.g., translation) is “behind” under another variant (reflection), and vice versa. Do such diverging conceptualizations of an object's location also lead to diverging evaluations? In two studies employing an implicit association test, we demonstrate, first, that speakers of German, Chinese, and Japanese indeed evaluate the object “in front of” another object more positively than the one “behind.” Second, and crucially, the reversal of which object is conceptualized as “in front” involves a corresponding reversal of valence, suggesting an impact of linguistically imparted FoR preferences on evaluative processes.     

A General Method of Empirical Q-matrix Validation

In contrast to unidimensional item response models that postulate a single underlying proficiency, cognitive diagnosis models (CDMs) posit multiple, discrete skills or attributes, thus allowing CDMs to provide a finer-grained assessment of examinees’ test performance. A common component of CDMs for specifying the attributes required for each item is the Q-matrix. Although construction of Q-matrix is typically performed by domain experts, it nonetheless, to a large extent, remains a subjective process, and misspecifications in the Q-matrix, if left unchecked, can have important practical implications. To address this concern, this paper proposes a discrimination index that can be used with a wide class of CDM subsumed by the generalized deterministic input, noisy “and” gate model to empirically validate the Q-matrix specifications by identifying and replacing misspecified entries in the Q-matrix. The rationale for using the index as the basis for a proposed validation method is provided in the form of mathematical proofs to several relevant lemmas and a theorem. The feasibility of the proposed method was examined using simulated data generated under various conditions. The proposed method is illustrated using fraction subtraction data.     

The Effects of Subject-and Experimenter-Induced Defensive Response Sets on Picture-Frustration Test Reactions

Thirty male college students, half scoring high and half low on the MMPI K-scale, were administered a multiple-choice (M-C) modification of the Rosenzweig P-F Study under three instructional sets. Two of these sets were induced by E and compared with the third which represented S's normal test-taking set. The two induced sets included (1) a “should” set in which concern for the social consequences of behavior was accentuated, and (2) a “wish” set in which concern for social consequences was minimized, and freedom of self-expression was accentuated.

It was hypothesized that Ss making extremely high or low scores on the K-scale would likewise score differently on the M-C, P-F test. High K, defined by its authors as a measure of test defensiveness, and low K, as test-taking candidness, would be associated with P-F score patterns reflecting these test-taking attitudes: High K Ss showing Impunitive (M) dominant reaction patterns and low K showing Extrapunitive (E) or Intropunitive (I) dominant patterns. These hypothèses were confirmed except that intropunitive responses were extremely rare in the entire subject sample.

It was further hypothesized that the typical high K pattern would be induced in low K Ss under the “should” instructional set and the low K pattern would be elicited from high K Ss under “wish” instructions. This expectation was confirmed. It was suggested that the technique of manipulating responses via test taking instructions can generate correction factors useful in projective as well as structured personality tests.     

A Constrained Linear Estimator for Multiple Regression

“Improper linear models” (see Dawes, Am. Psychol. 34:571–582, 1979), such as equal weighting, have garnered interest as alternatives to standard regression models. We analyze the general circumstances under which these models perform well by recasting a class of “improper” linear models as “proper” statistical models with a single predictor. We derive the upper bound on the mean squared error of this estimator and demonstrate that it has less variance than ordinary least squares estimates. We examine common choices of the weighting vector used in the literature, e.g., single variable heuristics and equal weighting, and illustrate their performance in various test cases.     

Frontal tests and models for cognitive ageing

Results of investigations of how specific behavioural changes in old age may be related to local changes in the brain have been discrepant because of neglect of methodological issues. Among these are neglect of the reliability and validity of frontal and executive tests; the operational definition and construct validity of hypothetical functional processes such as “inhibition”; the nature and origins of mutual relationships between tests; the relationships of test performance to higher order constructs of general mental ability such as “gf” and, most importantly, problems of cohort selection arising from imprecise definitions of the models of the ageing process. These problems are illustrated by a discussion of the results of several experiments in which younger and older groups of people are compared on frontal and executive tests.     

NÉCESSITÉ ET SIGNIFICATION DES RECHERCHES COMPARATIVES EN PSYCHOLOGIE GÉNÉTIQUE

Need and meaning of comparative studies in genetic psychology. — The comparative studies in the field of genetic psychology are indispensable for Psychology in general and also for Sociology, because only such studies allow us to separate the effects of biological or mental factors from those of social and cultural influences on the formation and the socialization of individuals. Relevant to this discussion is the well-known issue between culturalistic psychoanalysts like Fromm, Horney, etc., and classical freudian psychoanalysts who reduce the whole individual development to an endogenous evolution of « instinct ». In the field of cognitive functions to which this paper is devoted, at least four kinds of various factors must be distinguished, the respective influences of which can be separated through comparative studies : I. Biological factors depending on the “epigenetic” system (maturation of nervous system, etc.). These factors probably explain the sequential aspects (constant and necessary order) of the stages in the development of operative intelligence. However, if only these factors were acting, the stages would not only appear in a sequence, but at the same ages, whereas in fact, the ages where a stage appears differ from one environment to another. 2. Equilibration or autoregulation factors, determining behavior and thought in their various specific activities. They correspond to the sequential forms in general coordination of the actions of individuals as interacting with their physical environment; such intervening regulations are probably at the origin of the mental operations themselves, especially logical - mathematical operations. 3. General socialization factors, which are identical for all societies : cooperations - discussions - oppositions - exchanges, etc., between children or between adults or between adults and children. These factors 3 are closely related with the factors 2, because the general coordination of actions concerns inter-individual as so as intra-individual actions. 4. Factors related to educational and cultural transmission, which differ from one society to another; they are those we usually have in mind when we say briefly “social factors”. To discuss the influence of these four factors, Mohseni's study on Teheran school children and illiterate rural children is given as an example. Three results were obtained : (1) For “conservation” tasks, the same stages of intellectual development are observed for urban and rural children, in Iran as well as in Europe. (2) For these same tasks, a systematic delay of two to three years is observed among rural children with respect to Teheran children. (3) For performance tests (i.e. Porteus, Goodenough, etc.) there is a delay of four and primarily five years. These data seem to show that there is some sequential order of the stages which depends partially on the factors I. But, the ages for each stage are not constant, showing the factors 2 and 3 are intervening in a probably inseparable way. On the other hand, the difference between the actual operative results and the performance tests seems to indicate that a distinction might be made between the factors of general coordination (factors 2 and 3) and the educational transmissions (factors 4). About these last factors, the three and four year delays observed by Canadian psychologists working in Martinique with school children (French curriculum) in operative tasks (like conservations) seem to indicate that the general operations depend less on school than on the activities themselves of children or in general on the adult stimulations in the environment. We must pay special attention to the problems raised by language. In some precise and systematic experiments in Geneva, Sinclair shows children on the “preoperative” level (i.e. with no conservation…) do not spontaneously use the same language as operative children : the former use chiefly “scale” words (“much” or “many”, “little” or “few”, “large”, “small”) and the latter use chiefly “vector” words (“more”, “less”). If the younger subjects are submitted to a verbal training, they do learn to use the older children's language, but it only results in very little operative improvement (about one case out of 10 subjects on the average). Therefore, replicating the same experiments in countries speaking different languages would be very interesting. For instance, in the Turkish language, there is only one “vector” word, i.e. “again”; so that the Turk says “again much, again many” to say “more”. Then, the question is to know whether a change will be observed for the stages of logical-mathematical operations or whether these operations will be found everywhere with their identical common background; it seems that this last conclusion has already been observed in Aden (Hyde), Hong Kong (Goodnow), South Africa (Price-Williams). In conclusion, the kind of psychology we develop in our social environments, remains conjectural as long as comparative extensive and systematical research is not available; a great effort is still to be made in this direction.     

Définir ou détecter des pathologies ? Utilisation et interprétation des scores seuils à la lumière du débat dimensions/catégories

A number of psychometric tests are aimed at locating the performance of an individual along a continuous range that goes from normal to worrying. In a different paradigm, there are tests whose goal is to assess whether or not an individual is healthy with respect to a psychological, psychiatric or neurological pathology that is defined independently from the test itself. The former rely on a dimensional concept of psychopathologies; the latter imply a categorical concept. The former require to compare the performance of the individual to a standard sample that is representative of the general population; the latter involve a standard sample representative of the healthy population only. The former might imply the use of a cut-off score to indicate that a certain degree of rarity of the performance has been reached, while the latter must involve a cut-off score to make a decision (whether the individual belongs to the “healthy” or “pathological” category), a score that is most often set so as to control the false-positive rate in making this decision. In the former paradigm, the cut-off score defines or contributes to define the pathology. In the latter, it only helps detecting it. This article aims at making clear the difference between these two realms and highlight their practical as well as theoretical impact on psychological practice.     

THE GOOD PRODUCTIVITY GAME: INCREASING WORK PERFORMANCE IN A REHABILITATION SETTING1

Simple reinforcement systems have been used to improve performance in a broad range of settings. For example, in classrooms, the “Good Behavior Game” has been shown to be very effective (Barrish, Saunders, and Wolf, Journal of Applied Behavior Analysis, 1969, 2 , 119–124). In industry, small bonuses were used to increase the punctuality of workers (Hermann, deMontes, Dominquez, Montes, and Hopkins, Journal of Applied Behavior Analysis, 1973, 6 , 503–572). In a sheltered workshop setting, Shroeder (Journal of Applied Behavior Analysis, 1972, 5 , 45–52) examined work rates under varying frequencies and amounts of reinforcement and response force. The present study involved the utilization of simple group contingencies to increase productivity in a rehabilitation industry. Four state hospital residents who were trainees at a rehabilitation industry participated in the study which examined the effects of feedback, and feedback plus the “Good Productivity Game” to improve work output. The task, for which the employees were paid a wage, involved sorting boards by size. When the employees were provided with feedback on the number of boards sorted during the observation period, productivity increased slightly over baseline. After a return to baseline, the “Good Productivity Game” was played. For performance, the game afforded the employees pseudo-competition (in that teams were paired against each other, but both teams always “won”) and simple rewards such as candy and early work termination. The game improved performance by 104% over the second baseline and by 64% over the third baseline. Data gathered on rates of on-task behavior by the employees correlate with the productivity rates. Data gathered on rates of staff attention paid to employees show little difference across conditions, thus corroborating the function of the “Good Productivity Game” in increasing work output. Although no formal data were collected, the staff continued to use the game with considerable success after the formal termination of the study. The “Good Productivity Game” appears useful in increasing work output in a rehabilitation setting. Further research should concentrate on the utility of the game throughout longer periods of the workday and over extended periods of time.