The critical role of the research question, inclusion criteria, and transparency in meta-analyses of integrity test research: a reply to Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012)

We clear up a number of misconceptions from the critiques of our meta-analysis (Van Iddekinge, Roth, Raymark, & Odle-Dusseau, 2012). We reiterate that our research question focused on the criterion-related validity of integrity tests for predicting individual work behavior and that our inclusion criteria flowed from this question. We also reviewed the primary studies we could access from Ones, Viswesvaran, and Schmidt's (1993) meta-analysis of integrity tests and found that only about 30% of the studies met our inclusion criteria. Further, analyses of some of the types of studies we had to exclude revealed potentially inflated validity estimates (e.g., corrected validities as high as .80 for polygraph studies). We also discuss our experience trying to obtain primary studies and other information from authors of Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012). In addition, we address concerns raised about certain decisions we made and values we used, and we demonstrate how such concerns would have little or no effect on our results or conclusions. Finally, we discuss some other misconceptions about our meta-analysis, as well as some divergent views about the integrity test literature in general. Overall, we stand by our research question, methods, and results, which suggest that the validity of integrity tests for criteria such as job performance and counterproductive work behavior is weaker than the authors of the critiques appear to believe.     

On reconciling conflicting meta-analytic findings regarding integrity test validity

We react to the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012a) meta-analysis of the relationship between integrity test scores and work-related criteria, the earlier Ones, Viswesvaran, and Schmidt (1993) meta-analysis of those relationships, the Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012) responses, and the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012b) rebuttal. We highlight differences between the findings of the 2 meta-analyses by focusing on studies that used predictive designs, applicant samples, and non-self-report criteria. We conclude that study exclusion criteria, correction for artifacts, and second order sampling error are not likely explanations for the differences in findings. The lack of detailed documentation of all effect size estimates used in either meta-analysis makes it impossible to ascertain the bases for the differences in findings. We call for increased detail in meta-analytic reporting and for better information sharing among the parties producing and meta-analytically integrating validity evidence.     

Test publishers' perspective on "an updated meta-analysis": comment on Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012)

Van Iddekinge, Roth, Raymark, and Odle-Dusseau's (2012) meta-analysis of pre-employment integrity test results confirmed that such tests are meaningfully related to counterproductive work behavior. The article also offered some cautionary conclusions, which appear to stem from the limited scope of the authors' focus and the specific research procedures used. Issues discussed in this commentary include the following: (a) test publishers' provision of studies for meta-analytic consideration; (b) errors and questions in the coding of statistics from past studies; (c) debatable corrections for unreliable criterion measures; (d) exclusion of laboratory, contrasted-groups, unit-level, and time-series studies of counterproductive behavior; (e) under-emphasis on the prediction of counterproductive workplace behaviors compared with job performance, training outcomes, and turnover; (f) overlooking the industry practice of deploying integrity scales with other valid predictors of employee outcomes; (g) implication that integrity test publishers produce biased research results; (h) incomplete presentation of integrity tests' resistance to faking; and (i) omission of data indicating applicants' favorable response to integrity tests, the tests' lack of adverse impact, and the positive business impact of integrity testing. This commentary, therefore, offers an alternate perspective, addresses omissions and apparent inaccuracies, and urges a return to the use of diverse methodologies to evaluate the validity of integrity tests and other psychometric instruments.     

The criterion-related validity of integrity tests: an updated meta-analysis

Integrity tests have become a prominent predictor within the selection literature over the past few decades. However, some researchers have expressed concerns about the criterion-related validity evidence for such tests because of a perceived lack of methodological rigor within this literature, as well as a heavy reliance on unpublished data from test publishers. In response to these concerns, we meta-analyzed 104 studies (representing 134 independent samples), which were authored by a similar proportion of test publishers and non-publishers, whose conduct was consistent with professional standards for test validation, and whose results were relevant to the validity of integrity-specific scales for predicting individual work behavior. Overall mean observed validity estimates and validity estimates corrected for unreliability in the criterion (respectively) were .12 and .15 for job performance, .13 and .16 for training performance, .26 and .32 for counterproductive work behavior, and .07 and .09 for turnover. Although data on restriction of range were sparse, illustrative corrections for indirect range restriction did increase validities slightly (e.g., from .15 to .18 for job performance). Several variables appeared to moderate relations between integrity tests and the criteria. For example, corrected validities for job performance criteria were larger when based on studies authored by integrity test publishers (.27) than when based on studies from non-publishers (.12). In addition, corrected validities for counterproductive work behavior criteria were larger when based on self-reports (.42) than when based on other-reports (.11) or employee records (.15).     

No indication that the ego depletion manipulation can affect insight: a comment on DeCaro and Van Stockum (2018)

Abstract

Recently, DeCaro and Van Stockum have suggested that ego depletion following intensive self-control can improve insight problem-solving; this finding was interpreted in terms of insight relying on decreased control over attention and memory. However, DeCaro and Van Stockum used three variants of the single matchstick arithmetic problem. Experiment 1 involved low sample and non-standard problem application, while the more powered Experiment 2 yielded a surprisingly low solution rate. These facts made both studies problematic and called for their replication. In the two present studies, the DeCaro and Van Stockum ego-depletion manipulation and their matchstick problems were administered to a total of 316 people. Furthermore, various other insight problems, subjective ratings of insight experience, analytical problems and executive control tests were applied. The key result was that no reliable effect of ego depletion could be found for any of these measures.     

TOWARDS AN UNDERSTANDING OF INTEGRITY TEST SIMILARITIES AND DIFFERENCES: AN ITEM-LEVEL ANALYSIS OF SEVEN TESTS

A judgmental sort of 798 items from 7 paper-and-pencil integrity tests produced 23 thematic composites. Patterns of correlations between these composites and the 7 integrity measures and 2 Big Five measures shed light on similarities and differences between different integrity tests. Principal components analysis of 23 composites indicated 4 principal components that further illuminate the content domain of integrity tests. The relationships between 4 integrity principal components and integrity test scores as well as measures of the Big Five dimensions of personality are reported. The findings suggest that integrity tests can differ in their emphasis on various thematic composites, and, yet, be very similar in terms of their standing on the 4 integrity principal components. Different integrity tests can be quite different in terms of surface content, and, yet, assess the same underlying constructs.     

Fakability of a commercially produced pre-employment integrity test

Three groups of male inmates were given the PSC Survey ADT, a commercially available integrity test, under differing instructional conditions. Subjects told to fake good were able to provide more favorable test profiles than those who were told to respond truthfully or who were given no specific response set to follow. However, no differences were observed among groups on the Drinking and Drugs subscale. Using only recommended cut scores as criteria, subjects in the fake good condition would have been hired in greater numbers than subjects in the other two conditions. Practical implications of this study include the use of social desirability scales along with integrity tests, and the avoidance of inducing inappropriate response sets by strictly adhering to test instructions.This study was supported by a grant from Auburn University at Montgomery.We gratefully acknowledge the support and assistance of Warden Leoneal Davis and Dr. Paul Van Wyk of the Bullock County Correctional Facility and the Alabama Department of Corrections in conducting this study.     

Empirical research on the identification of singular,multiple and subordinate argumentation

To examine to what degree argumentation skills are mastered by pupils who attend the vocational, general and academic streams in Dutch secondary education various subtests were constructed. The theoretical study of argumentation as exposed by Van Eemeren and Grootendorst was the basis for this test construction. In this article tests for the identification of singular, multiple and subordinate argumentation are described. Also an account is given of a pretest of these three subtests.     

THE SUSCEPTIBILITY OF OVERT AND COVERT INTEGRITY TESTS TO COACHING AND FAKING

Abstract— Although previous research has indicated that faking can affect integrity test scores, the effects of coaching on integrity test scores have never been examined We conducted a between-subjects experiment to assess the effects of coaching and faking instructions on an overt and a covert integrity test Coaching provided simple rules to follow when answering test items and instructions on how to avoid elevated validity scale scores There were five instruction conditions "just take,""fake good,""coach overt,""coach covert," and coach both All subjects completed both overt and covert tests and a measure of intelligence Results provided strong evidence for the coachability of the overt integrity test, over and above the much smaller elevation in the faking condition The covert test apparently could be neither coached nor faked successfully Scores on both integrity tests tended to be positively correlated with intelligence in the coaching and faking conditions We discuss the generalizability of these results to other samples and other integrity tests, and the relevance of the coachability of integrity tests to the ongoing debate concerning the prediction of counterproductive behavior     

Appropriateness measurement: Review,critique and validating studies

The test-taking behaviour of some examinees may be so unusual that their test scores cannot be regarded as appropriate measures of their ability. Appropriateness measurement is a model-based approach to the problem of identifying these test scores. The intuitions and basic theory supporting appropriateness measurement are presented together with a critical review of earlier work and a series of interrelated experiments. We conclude that appropriateness measurement techniques are robust to errors in parameter estimation and robust to the presence of unidentified aberrant examinees in the test norming sample. In addition, the frequently criticized ‘three-parameter logistic’ latent trait model was found to be adequate for the detection of spuriously low scores in actual test data.     

PERSONALITY DIMENSIONS EXPLAINING RELATIONSHIPS BETWEEN INTEGRITY TESTS AND COUNTERPRODUCTIVE BEHAVIOR: BIG FIVE,OR ONE IN ADDITION?

Although the criterion-related validity of integrity tests is well established, there has not been enough research examining which personality constructs contribute to their criterion-related validity. Moreover, evidence of how well findings on integrity tests in North America generalize to non-English speaking countries is virtually absent. This research addressed these issues with data obtained from employees and students in Canada and Germany (total N = 853). Specifically, we tested the hypotheses that (a) Honesty–Humility, as specified in the HEXACO model of personality, is relatively more important than the Big 5 dimensions of personality in accounting for the criterion-related validity of overt integrity tests, whereas (b) the Big 5 are relatively more important in explaining the validity of personality-based integrity tests. These predictions were tested using 2 criteria (counterproductive work behavior and counterproductive academic behavior) as well as 2 overt and 2 personality-based integrity tests. We found evidence of the expected differences between types of integrity tests largely regardless of culture of the sample, specific test, criterion, or population under research, pointing to some degree of generalizability of findings in integrity testing research. Implications include theoretical refinements in research on integrity testing and encouragement of practical applications beyond North America.     

A Research Note on the Incremental Validity of Job Knowledge and Integrity Tests for Predicting Maximal Performance

In a sample of industrial job applicants, relationships among scores on an integrity test, a job knowledge measure, and maximal performance as assessed by a work sample measure were investigated. The observed correlation between the personality-based integrity test and maximal performance was .27, indicating that integrity tests can be predictive of maximal performance. Furthermore, integrity test scores correlated .14 with job knowledge and job knowledge scores had a validity of .36 for maximal performance. Theoretical implications for the maximal/typical performance distinction are discussed.     

Connectionist and Memory-Array Models of Artificial Grammar Learning

Subjects exposed to strings of letters generated by a finite state grammar can later classify grammatical and nongrammatical test strings, even though they cannot adequately say what the rules of the grammar are (e.g., Reber, 1989). The MINERVA 2 (Hintzman, 1986) and Medin and Schaffer (1978) memory-array models and a number of connectionist outoassociator models are tested against experimental data by deriving mainly parameter-free predictions from the models of the rank order of classification difficulty of test strings. The importance of different assumptions regarding the coding of features (How should the absence of a feature be coded? Should single letters or digrams be coded?), the learning rule used (Hebb rule vs. delta rule), and the connectivity (Should features be predicted only by previous features in the string, or by all features simultaneously?) is investigated by determining the performance of the models with and without each assumption. Only one class of connectionist model (the simultaneous delta rule) passes all the tests. It is shown that this class of model can be regarded by abstracting a set of representative but incomplete rules of the grammar.     

On the Construct Validity of Integrity Tests: Individual and Situational Factors as Predictors of Test Performance

Although integrity tests are widely applied in screening job applicants, there is a need for research for examining the construct validity of these tests. In the present study, a theoretical model examining the causes of destructive behavior in organizational settings was used to develop background data measures of individual and situational variables that might be related to integrity test scores. Subsequently, 692 undergraduates were asked to complete these background data scales along with (a) two overt integrity tests – the Reid Report and the Personnel Selection Inventory, and (b) two personality‐based measures – the delinquency and socialization scales of the California Psychological Inventory. When scores of these measures were correlated with and regressed on the background data scales, it was found that relevant individual variables, such as narcissism and power motives, and relevant situational variables, such as alienation and exposure to negative peer groups, were related to scores on both types of integrity tests. However, a stronger pattern of validity evidence was obtained for the personality‐based measures and, in all cases, situational variables were found to be better predictors than individual variables. The implications of these findings for the validity of inferences drawn from overt and personality‐based integrity tests are discussed.     

ADHD Symptoms and Attachment Representations: Considering the Role of Conduct Problems,Cognitive Deficits and Narrative Responses in Non-Attachment-Related Story Stems

The overall aim of the present study was to investigate ADHD symptoms in relation to attachment representations. We used both attachment- and non-attachment-related story stems, which allowed us to investigate whether problems with narrative production can explain the relation between ADHD symptoms and attachment representations. We also investigated the role of cognitive deficits and conduct problems in these relations. The sample consisted of 89 children (27 % girls) between 6 and 10 years old, with an oversampling of children with high levels of ADHD symptoms. ADHD symptoms and conduct problems were rated by parents and teachers. Cognitive functioning was investigated using laboratory tests of inhibition, working memory and sustained attention. Attachment representations were coded as secure, organized insecure and disorganized categories. Narrative responses to non-attachment-related story stems were coded for incoherence and negative content. Results showed that children in the disorganized attachment category had significantly higher levels of ADHD symptoms compared to those in the secure category. Both ADHD symptoms and disorganized attachment were related to incoherence and negative content. Attachment representations were not associated with ADHD symptoms when controlling for negative content in response to non-attachment-related story stems. These results suggest that the associations between attachment security and ADHD are yet to be fully understood. Importantly, a propensity to envisage negative events seems to characterize children with high levels of ADHD symptoms.     

Conceptual Knowledge,Procedural Knowledge,and Metacognition in Routine and Nonroutine Problem Solving

When, how, and why students use conceptual knowledge during math problem solving is not well understood. We propose that when solving routine problems, students are more likely to recruit conceptual knowledge if their procedural knowledge is weak than if it is strong, and that in this context, metacognitive processes, specifically feelings of doubt, mediate interactions between procedural and conceptual knowledge. To test these hypotheses, in two studies (Ns = 64 and 138), university students solved fraction and decimal arithmetic problems while thinking aloud; verbal protocols and written work were coded for overt uses of conceptual knowledge and displays of doubt. Consistent with the hypotheses, use of conceptual knowledge during calculation was not significantly positively associated with accuracy, but was positively associated with displays of doubt, which were negatively associated with accuracy. In Study 1, participants also explained solutions to rational arithmetic problems; using conceptual knowledge in this context was positively correlated with calculation accuracy, but only among participants who did not use conceptual knowledge during calculation, suggesting that the correlation did not reflect “online” effects of using conceptual knowledge. In Study 2, participants also completed a nonroutine problem-solving task; displays of doubt on this task were positively associated with accuracy, suggesting that metacognitive processes play different roles when solving routine and nonroutine problems. We discuss implications of the results regarding interactions between procedural knowledge, conceptual knowledge, and metacognitive processes in math problem solving.     

Integrity, conscientiousness, and honesty

Undergraduate volunteers from the Psychology participant pool (N=86, age M=22.7, SD=4.8 yr.; 72 of 86 were female) took two tests, a commercially available integrity test (the Personnel Selection Inventory) and the NEO-Five Factor Inventory. Later in the study, each participant was given an opportunity to report dishonestly the amount of time spent in the laboratory and thus to receive more extra-credit points than earned. An observer recorded participants' actual times in the laboratory to provide an accurate assessment of participants' honesty. Analysis indicated that the Personnel Selection Inventory did not predict whether the participant was honest or dishonest in reporting time spent. Conscientiousness as measured by the NEO-Five Factor Inventory, however, not only predicted behavior, but did so significantly better than did the Personnel Selection Inventory. The study provides new information about the value of integrity tests by comparing integrity and personality tests in the simultaneous prediction of a specific criterion.     

Perceptions of Overt and Personality-based Integrity Tests

Differences in test-taker perceptions between overt and personality-based integrity tests were examined. Following administration of both types of integrity tests, 255 undergraduate students provided ratings of perceived face validity and perceived predictive validity. Following receipt of actual test scores, 126 test takers participated in a second phase of the study in which they reported perceptions of distributive justice. Test takers perceived overt integrity tests as having greater face validity and predictive validity than personality-based integrity tests. Perceptions of job-relatedness were not strongly related to test performance on either test type. Distributive justice perceptions were related to test performance, but not type of integrity test.     

微表情的特征、识别、训练和影响因素

微表情是人压抑的真实情绪的泄露, 时间约为1/25~1/2 s, 常出现在说谎情境, 可能源于椎体束运动系统控制的自主面部运动和椎体束外运动系统控制的非自主面部运动的“拔河比赛”。已有研究构建了微表情数据库, 开发了BART和JACBART等微表情识别测验以及METT训练, 并探讨了微表情识别及训练与认知、人格、文化、职业等的关系。但这些研究考察的中性表情背景下微表情只是镶嵌在各表情背景下真实微表情的特例, 生态程度不高; 少量研究探索了表情背景对微表情识别的影响但未建立标准测验。未来应开发生态化微表情识别标准测验和训练, 尝试跨越微表情发生和识别的界限, 并采用生理−心理、电生理和脑成像技术探索其机制。     

Bodily Integrity and Conceptions of Subjectivity

This paper examines two different ways of understanding the concept of bodily integrity and their political implications. In Drucilla Cornell's use of the concept, the body cannot be separated from the mind. Protecting bodily integrity means protecting possibilities of imagining the self as whole. Martha Nussbaum's theorizing is based on a liberal way of conceptualizing subjectivity, in which the mind and the body are separate, and bodily integrity is used to refer to physical inviolability.