The Content Validity of Cognitively Oriented Tests: Commentary on Schmidt ()

Schmidt (International Journal of Selection and Assessment, 20, 1–13 (2012)) argues that it is possible for scores based on measures of general cognitive ability (GCA) to have content validity evidence. This commentary examines this argument further. I first decompose the various lines of validity evidence that may exist for GCA scores. Next, I consider whether GCA scores can have content validity evidence and whether they cannot. I conclude with several observations about the meaning of content validity within GCA research and practice. The bottom line is that although I agree with Schmidt that GCA scores can have content validity evidence, I am not sure such evidence tells us much about the overall validity of GCA.     

More than g: Evidence for the Incremental Validity of Performance-Based Assessments for Predicting Training Performance

Past research has consistently shown that tests measuring specific cognitive abilities provide little if any incremental validity over tests of general mental ability when predicting performance on the job. In this study, we suggest that the seeming lack of incremental validity may have been due to the type of content that has traditionally been assessed. Therefore, we hypothesised that incremental validity can be obtained using specific cognitive abilities that are less highly correlated with g and are matched to the tasks performed on the job. To test this, we examined a recently developed performance-based measure that assesses a number of cognitive abilities related to training performance. In a sample of 310 US Navy student pilots, results indicated that performance-based scores added sizeable incremental validity to a measure of g. The significant increases in R² ranged from .08 to .10 across criteria. Similar results were obtained after correcting correlations for range restriction, though the magnitude of incremental validity was slightly smaller (ΔR² ranged from .05 to .07).     

Content Validity and Cognitive Tests: Response to Kehoe (), Ployhart (), and Sackett ()

The most important fact emerging from the combination of my article and the three commentaries is the consensus judgment that content validity is appropriate scientifically and professionally for use with tests of specific cognitive skills used in job performance. This is important because the 1978 Uniform Guidelines on Employee Selection Procedures have typically been interpreted as not permitting such usage, and this is particularly the case in the interpretation given to the Guidelines by federal government enforcement agencies. Although the Society for Industrial and Organizational Psychology Principles and the Standards do not prohibit such usage, many industrial–organizational psychologists believe that it is not professionally or scientifically appropriate to employ content validity methods with cognitive measures. The hope is that this series will convince them otherwise. On this point, all four authors in the series are in agreement. The major disagreement among us concerns whether specific cognitive skills used in content valid tests must be considered constructs or not. My position, and apparently that of Kehoe, is that they need not be so considered. I argue that constructs must be invoked only in the context of a substantive theory. Sackett and Ployhart, on the other hand, argue that all measures taken on people must be viewed as constructs, regardless of whether any theoretical propositions and assumptions are involved. In this response, I present reasons why this need not be the case.     

Validity of Interpretation: A user validity perspective beyond the test score

This paper introduces the concept of user validity and provides a new perspective on the validity of interpretations from tests. Test interpretation is based on outputs such as test scores, profiles, reports, spreadsheets of multiple candidates' scores, etc. The user validity perspective focuses on the interpretations a test user makes given the purpose of the test and the information provided in the test output. This innovative perspective focuses on how user validity can be extended to content, criterion, and to some extent construct‐related validity. It provides a basis for researching the validity of interpretations and an improved understanding of the appropriateness of different approaches to score interpretation, as well as how to design test outputs and assessments that are pragmatic and optimal.     

Effects of quantity and quality of home proximal processes on Dutch,Surinamese–Dutch and Turkish–Dutch pre‐schoolers’ cognitive development

This study examines the effects of mother–child literacy and problem solving interactions on 3–4‐year‐old children's cognitive development in a sample of Dutch middle‐class, Dutch lower‐class, Surinamese and Turkish immigrant families (n=130). Following the bioecological model of Bronfenbrenner and Ceci, mother–child book reading and problem solving interactions are viewed as proximal processes actualizing genetic cognitive potential. Interactions are studied from a quantitative perspective, based on questionnaire data, concerning frequency of exposure to these interactions, and a qualitative perspective, based on observational data, concerning affective quality and cognitive content. The study was designed as a 1 year longitudinal study. It was found that effects of mother–child interactions on cognitive development differed across the groups, controlling for earlier cognitive competence and family's SES. The findings are interpreted within the frame of the bioecological model and the behavioral genetic research paradigm. Copyright © 1999 John Wiley & Sons, Ltd.     

Cognitive consequences of timeframe bias. On the link between working memory,cognitive switching,and time perspective

The present study examined how time perspective is associated with working memory updating and cognitive switching. Additionally, stress states and mood as potential mediators of the relationship between time perspective and cognitive performance were analysed. During two sessions participants (n = 200) completed a set of questionnaires measuring time perspective, task-related stress states, and mood. Moreover, in two separate sessions they performed working memory updating and switching tasks. The results indicated that two time perspectives, i.e. Present Fatalism and Past Positive, were associated with updating. Furthermore, mediation analysis showed that positive mood accounted for these relationships. Specifically, Present Fatalism was correlated with low positive mood and in turn, worse working memory scores, whereas Past Positive was associated with high positive mood leading to better performance on the working memory task. None of the time perspective dimensions correlated with cognitive switching. These findings shed more light on the cognitive consequences of timeframe bias and suggest new approaches in research on time perspective and cognitive functioning.

     

Emotional Intelligence in Selection Contexts: Measurement method,criterion‐related validity,and vulnerability to response distortion

Two studies examined aspects of the validity of self‐report and performance‐based measures of emotional intelligence (EI) relevant to their use in personnel selection. In Study 1, structural equation modeling indicated that a two‐factor model with separate factors for the two types of EI measures fit better than a one‐factor solution. The performance‐based EI factor was more related to cognitive ability (R=.38) than personality (R=.26), whereas the self‐report EI factor was more related to personality (R=.85) than cognitive ability (R=.09). Although the performance‐based EI factor correlated more strongly with job performance (ρ=.24) than did that of the self‐report (ρ=.05), it provided little incremental validity beyond cognitive ability and conscientiousness. In Study 2, participants were asked to complete the measures as if applying for a job, and mean scores were then compared with those of Study 1. Results indicated that self‐report EI measures were more vulnerable to distortion than were the performance‐based measures. Implications for the assessment of EI in personnel selection contexts are discussed.     

Cognitive ability,cognitive aptitudes,job knowledge,and job performance

This paper reviews the hundreds of studies showing that general cognitive ability predicts job performance in all jobs. The first section shows that general cognitive ability predicts supervisor ratings and training success. The second section shows that general cognitive ability predicts objective, rigorously content valid work sample performance with even higher validity. Path analysis shows that much of this predictive power stems from the fact that general cognitive ability predicts job knowledge (r = .80 for civilian jobs) and job knowledge predicts job performance (r = .80). However, cognitive ability predicts performance beyond this value (r = .75 versus r = [.80][.80] = .64) verifying job analyses showing that most major cognitive skills are used in everyday work. The third section of the paper briefly reviews evidence showing that it is general cognitive ability and not specific cognitive aptitudes that predict performance.     

Consensus and Cognitivism in Habermas’s Discourse Ethics

Abstract

Habermas asserts that his discourse ethics rests on two main commitments:

1. Moral judgements have cognitive content analogous to truth value; and

2. moral justification requires real-life discourse. Habermas elaborates on the second claim by making actual consensus a necessary condition of normative validity. I argue that Habermas’s two commitments sit uneasily together. The second entails that his cognitivism is revisionist in the sense that it must reject the law of the excluded middle. Moreover, Habermas’s argument in defence of the need for real-life discourse is unconvincing and his derivation of the principle which requires consensus is fallacious.

     

Cognitive Tests Used in Selection Can Have Content Validity as Well as Criterion Validity: A broader research review and implications for practice

Many industrial/organizational (I/O) psychologists, both academics and practitioners, believe that the content validity model is not appropriate for cognitive ability measures used in personnel selection. They believe that cognitive tests can have criterion validity and construct validity but not content validity. Based on a review of the broader differential psychology research literature on cognitive skills, aptitudes, and abilities, this article demonstrates that with the proper content validity procedures, cognitive ability measures, including, ultimately, some de facto measures of general cognitive ability, can have content validity in addition to criterion and construct validity. Finally, the article considers, critiques, and refutes the specific arguments contending that content validity is inappropriate for use with cognitive skills and abilities. These research facts have implications for I/O practice, professional standards, and legal defensibility of selection programs.     

Concepts,Constructs, Cognitive Psychology,and Personal Construct Theory

In this article, I have addressed the efforts to locate and integrate personal construct psychology with other perspectives in psychology (Neimeyer, 1985). The general domain is cognitive psychology, but with a focus on the basic building block of the cognitive perspective, the concept. Personal construct psychology is particularly interesting in this regard at this time because Kelly (1955) was adamant that his construct encompassed more than concept. Now, 36 years after his deliberations, a discussion of concepts and constructs contributes to an ongoing theoretical debate in psychology that can only serve to better reinforce the discipline.     

Development and content validity of a screening instrument for gaming addiction in adolescents: The Gaming Addiction Identification Test (GAIT)

This study describes the development of a screening tool for gaming addiction in adolescents – the Gaming Addiction Identification Test (GAIT). Its development was based on the research literature on gaming and addiction. An expert panel comprising professional raters (n = 7), experiential adolescent raters (n = 10), and parent raters (n = 10) estimated the content validity of each item (I‐CVI) as well as of the whole scale (S‐CVI/Ave), and participated in a cognitive interview about the GAIT scale. The mean scores for both I‐CVI and S‐CVI/Ave ranged between 0.97 and 0.99 compared with the lowest recommended I‐CVI value of 0.78 and the S‐CVI/Ave value of 0.90. There were no sex differences and no differences between expert groups regarding ratings in content validity. No differences in the overall evaluation of the scale emerged in the cognitive interviews. Our conclusions were that GAIT showed good content validity in capturing gaming addiction. The GAIT needs further investigation into its psychometric properties of construct validity (convergent and divergent validity) and criterion‐related validity, as well as its reliability in both clinical settings and in community settings with adolescents.     

Test anxiety and the validity of cognitive tests: A confirmatory factor analysis perspective and some empirical findings

The validity of cognitive ability tests is often interpreted solely as a function of the cognitive abilities that these tests are supposed to measure, but other factors may be at play. The effects of test anxiety on the criterion related validity (CRV) of tests was the topic of a recent study by Reeve, Heggestad, and Lievens (2009) (Reeve, C. L., Heggestad, E. D., & Lievens, F. (2009). Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence, 37, 34?41.). They proposed a model on the basis of classical test theory, and concluded on the basis of data simulations that test anxiety typically decreases the CRV. In this paper, we view the effects of test anxiety on cognitive ability test scores and its implications for validity coefficients from the perspective of confirmatory factor analysis. We argue that CRV will be increased above the effect of targeted constructs if test anxiety affects both predictor and criterion performance. This prediction is tested empirically by considering convergent validity of subtests in five experimental studies of the effect of stereotype threat on test performance. Results show that the effects of test anxiety on cognitive test performance may actually enhance the validity of tests.     

THE CROSS-CULTURAL STUDY OF INTELLIGENCE: PIAGET AND THE BAOULÉ*

There are two aspects of Piaget's theory that can be at least partly distinguished: (1) The stage theory, or the development of particular concepts through a series of hierarchical stages; (2) The metatheory, or the interactionist model of adaptation (assimilation and accommodation) that explains the mechanisms of cognitive development. Most cross-cultural research has been based on the first of these aspects, using and adapting “Piagetian tasks ”in various conceptual areas. Some findings of this line of enquiry, and the methodological problems encountered, are briefly reviewed. It is argued that no specific task, nor a combination of them, can be taken to measure a general cognitive level; the tasks measure the attainment of particular concepts rather than “intelligence”. New evidence is presented on the reliability and validity of Piagetian tasks used in a cross-cultural setting. In the second model, intelligence is broadly defined as adaptation to the environment; in this conception it would be reasonable to expect divergent paths of cognitive development in different cultures. In other words, one might need a different Piagetian psychology in each culture, but based on universal deep mechanisms. As a modest contribution to a more “emic ”study of intelligence, the concept of intelligence as defined by the Baoulé of Ivory Coast, n'glouèlě is presented and discussed. Ratings on n'glouèlě and its different components are related to performance on Piagetian tasks in a sample of 8- to 9-year-old rural Baoulé children. The thrust of the paper is to point out once more the cultural relativity of any conception of intelligence, be it from a Piagetian or any other perspective.     

心理健康素质测评系统·中国成年人认知风格问卷的编制

通过对认知风格既往文献的回顾和分析,从临床心理学的角度将认知风格定义为个体在解释生活事件时所偏爱的和习惯化的特征性认知方式,并确定了4个维度：过度概括化、选择性提取、个体化和灾难化。通过开放式问卷调查、参考国内外已有量表、自编项目及专家评定,形成27个项目的认知风格问卷初测版,并分别在大学生样本和非学生样本中进行测试,筛选出在两个样本中都符合心理测量学要求的16个项目,最终形成认知风格问卷正式版。对认知风格问卷正式版进行心理测量学指标考察,结果发现问卷的内部一致性信度、重测信度、结构效度、内容效度、聚合效     

The heterogeneity of attention‐deficit/hyperactivity disorder symptoms and conduct problems: Cognitive inhibition,emotion regulation,emotionality, and disorganized attachment

This study examined the contributions of several important domains of functioning to attention‐deficit/hyperactivity disorder (ADHD) symptoms and conduct problems. Specifically, we investigated whether cognitive inhibition, emotion regulation, emotionality, and disorganized attachment made independent and specific contributions to these externalizing behaviour problems from a multiple pathways perspective. The study included laboratory measures of cognitive inhibition and disorganized attachment in 184 typically developing children (M age = 6 years, 10 months, SD = 1.7). Parental ratings provided measures of emotion regulation, emotionality, and externalizing behaviour problems. Results revealed that cognitive inhibition, regulation of positive emotion, and positive emotionality were independently and specifically related to ADHD symptoms. Disorganized attachment and negative emotionality formed independent and specific relations to conduct problems. Our findings support the multiple pathways perspective on ADHD, with poor regulation of positive emotion and high positive emotionality making distinct contributions to ADHD symptoms. More specifically, our results support the proposal of a temperamentally based pathway to ADHD symptoms. The findings also indicate that disorganized attachment and negative emotionality constitute pathways specific to conduct problems rather than to ADHD symptoms.     

FUTURE TIME PERSPECTIVE AND HUMAN COGNITION: AN EVOLUTIONAL VIEW

The creation of future time perspective is one of the culminating accomplishments of the human cognitive system. For the purpose of uncovering the implications of this achievement, a hypothetical retracing of the evolutional development of biological information processing systems is attempted. The retracing goes through four stages concerning the software architecture of such a system; fixed-I/O (reflex), flexible-I/O (adaptive learning), motivational subprogram (typically emotions), and cognitive information processing. Through the retracing it is shown that most of the information processing modes of biological systems already emerge as early as the flexible-I/O stage, which are subsequently developed and refined till reaching the stage of creating future time perspective as a context-free cognitive background upon which survival-oriented behaviors can be planned ahead of time.     

SITUATIONAL JUDGMENT TESTS, RESPONSE INSTRUCTIONS, AND VALIDITY: A META-ANALYSIS

Situational judgment tests (SJTs) are personnel selection instruments that present job applicants with work-related situations and possible responses to the situations. There are typically 2 types of instructions: behavioral tendency and knowledge. Behavioral tendency instructions ask respondents to identify how they would likely behave in a given situation. Knowledge instructions ask respondents to evaluate the effectiveness of possible responses to a given situation. Results showed that response instructions influenced the constructs measured by the tests. Tests with knowledge instructions had higher correlations with cognitive ability. Tests with behavioral tendency instructions showed higher correlations with personality constructs. Results also showed that response instructions had little moderating effect on criterion-related validity. Supplemental analyses showed that the moderating effect of response instructions on construct validity was not due to systematic differences in item content. SJTs have incremental validity over cognitive ability, the Big 5, and over a composite of cognitive ability and the Big 5.     

Salesperson Selection,Training, and Development: Trends,Implications, and Research Opportunities

Building on the forces of change in selling identified in the introductory paper of this Special Issue of JPSSM, we identify and explicate trends, managerial implications, and research opportunities in two important domains of the selling field: training and development (TD) and selection. Part One focuses on TD of salespeople, and argues why a fresh perspective on salesperson TD is essential. Then, taking a knowledge, skills, and ability (KSA) approach, a framework for future TD content is proposed around three levels of KSAs: task-related KSAs, growth-related KSAs, and meta-KSAs. Part Two, on selection, focuses on several important predictors of salesperson success: cognitive ability tests, personality inventories, and biodata. Then, assessment methods are reviewed. Finally, we summarize with a discussion of emerging trends and research opportunities. The focus of the discussion is on both enhancing salesperson performance and affording scholars new research opportunities.