Psychometric and Validity Issues in Machine Learning Approaches to Personality Assessment: A Focus on Social Media Text Mining

In the age of big data, substantial research is now moving toward using digital footprints like social media text data to assess personality. Nevertheless, there are concerns and questions regarding the psychometric and validity evidence of such approaches. We seek to address this issue by focusing on social media text data and (i) conducting a review of psychometric validation efforts in social media text mining (SMTM) for personality assessment and discussing additional work that needs to be done; (ii) considering additional validity issues from the standpoint of reference (i.e. ‘ground truth’) and causality (i.e. how personality determines variations in scores derived from SMTM); and (iii) discussing the unique issues of generalizability when validating SMTM for personality assessment across different social media platforms and populations. In doing so, we explicate the key validity and validation issues that need to be considered as a field to advance SMTM for personality assessment, and, more generally, machine learning personality assessment methods. © 2020 European Association of Personality Psychology     

Applying theory to the black box: A model for empirically scoring biodata

Despite being an effective predictor of job performance, empirically keyed biodata assessments have been criticized as black box empiricism unlikely to generalize to new contexts. This paper introduces a model that challenges this perspective, explicating how biodata content, job demands, and criterion variables collectively influence the construct validity, and generalizability of empirically scored biodata. Across two field studies, expected changes in scale correlations with external measures were found that coincided with changes in the contextual similarity between calibration and holdout contexts, the criteria used, and the content validity of biodata items. Collectively, this paper offers a framework that helps understand and optimize empirical biodata keying in practice, furthering confidence for their use in applied settings.     

简式元认知问卷中文版的心理计量学特性

以大学生为被试,考察简式元认知问卷(BQM)在中国文化背景下的心理计量学特性。采用探索性和验证性因素分析方法对BQM的结构效度进行交叉检验,结果支持了元记忆和元注意的双因子结构; 问卷及其因子的内部一致性和重测信度良好; 项目分析的结果显示,问卷的条目具有良好的分数分布、区分度和内部一致性; BQM分数与负性情绪、执行功能和睡眠质量的关联性也为效标关联效度提供了支持性证据。上述结果表明BQM中文版具有良好的心理计量学特性,可作为青年群体元认知水平的测量工具。     

The Importance of Construct Validity in Consumer Research

Vosgerau, Scopelliti, and Huh (2020) question the construct validity of the most commonly used measures of self‐control and argue that many studies that claim to be examining self‐control may in fact be examining something else. In this comment, we generalize this argument and propose that this criticism could be leveraged against many other areas of consumer research. Indeed, many studies fail to properly establish the construct validity of their measures. Consequently, they may not be examining the construct they purport to be studying, which can lead to unreliable and conflicting findings.     

Statistical learning of multiple speech streams: A challenge for monolingual infants

Language acquisition depends on the ability to detect and track the distributional properties of speech. Successful acquisition also necessitates detecting changes in those properties, which can occur when the learner encounters different speakers, topics, dialects, or languages. When encountering multiple speech streams with different underlying statistics but overlapping features, how do infants keep track of the properties of each speech stream separately? In four experiments, we tested whether 8‐month‐old monolingual infants (N = 144) can track the underlying statistics of two artificial speech streams that share a portion of their syllables. We first presented each stream individually. We then presented the two speech streams in sequence, without contextual cues signaling the different speech streams, and subsequently added pitch and accent cues to help learners track each stream separately. The results reveal that monolingual infants experience difficulty tracking the statistical regularities in two speech streams presented sequentially, even when provided with contextual cues intended to facilitate separation of the speech streams. We discuss the implications of our findings for understanding how infants learn and separate the input when confronted with multiple statistical structures.     

Psychometric adaption of the impulsive driver behavior scale in a Chinese sample

Impulsivity plays an important role in aberrant driving behavior and crash involvement. The purpose of this study was to evaluate the reliability and validity of the Impulsive Driver Behavior Scale (IDBS, Bıçaksız & Özkan, 2016a) with a Chinese sample. Two hundred and ninety-nine drivers completed the Chinese version of the Impulsive Driver Behavior Scale, the Driver Behavior Questionnaire, the Big Five Inventory and some social-demographic and traffic violation items. A confirmatory factor analysis indicated that a four-factor solution with 20 items yielded a better fit to the data than other solutions. The reduced IDBS showed good reliability and a stable structure. Drivers’ functional impulsivity was positively correlated with positive driver behavior and some ordinary violations. Meanwhile, the other three dimensions of dysfunctional impulsivity showed negative correlations with positive driver behavior and positive associations with aberrant driving behavior and penalty points and fines. Impulsive driver behavior is also associated with agreeableness and conscientiousness. These results provide evidence supporting the IDBS as a reliable and valuable instrument for measuring driving impulsivity in the Chinese traffic environment.     

统计学习的认知神经机制及其与语言的关系

统计学习是指个体在连续刺激流中发现转移概率等统计规律的过程, 在Saffran等(1996)的经典婴儿语音切分研究中首次被提出。大量研究证实了统计学习的普遍存在, 近期学界开始关注统计学习的特异性及其对认知的影响, 尤其是从学习过程及其特异性两个方面阐述统计学习的认知神经机制并揭示其和语言的交互作用。未来应从脑和行为的多模态数据视角, 丰富统计学习结果的行为和神经指标, 考察不同类型统计学习过程的动态神经活动模式, 建立统计学习行为和脑的关联, 深化对统计学习认知神经机制的认识, 在统计学习与语言交互作用的基础上, 从成人二语学习切入结合音乐统计学习训练探讨促进语言学习的统计学习干预手段。     

矛盾年龄偏见量表在中国大学生群体中的修订及信效度检验

目的：修订矛盾年龄偏见量表(the Ambivalent Ageism Scale, AAS),并在中国大学生群体中进行信、效度检验。方法：对1182名大学生施测AAS中文翻译版,探索并验证其因子结构,分析信、效度和测量等值性。其中347人还完成老化知识量表(FAQ)和容纳他人量表(AOS),用以检验AAS中文版的校标关联效度。另有74人在4周后重测AAS中文版。结果：AAS中文修订版包含认知帮助/保护、不想要的帮助和敌意年龄偏见三因子,总量表和各维度的Cronbach’s α系数依次为0.83、0.80、0.85、0.76,重测信度依次为0.78、0.62、0.78、0.71。AAS中文修订版及各维度与FAQ和AOS负相关(r=-0.23～-0.48,ps<0.001)。验证因素分析表明AAS中文修订版的拟合良好(χ²/df=3.51,CFI=0.96,TLI=0.94,RMSEA=0.07,SRMR=0.05)。且该量表跨性别形态、弱等值、部分强等值和部分严格测量等值成立。结论：修订后的矛盾年龄偏见量表(AAS)中文版的信、效度良好,具备跨性别的测量部分...     

An empirical look at the Defense Mechanism Test (DMT): reliability and construct validity

Although the Defense Mechanism Test (DMT) has been in use for almost half a century, there are still quite contradictory views about whether it is a reliable instrument, and if so, what it really measures. Thus, based on data from 39 female students, we first examined DMT inter-coder reliability by analyzing the agreement among trained judges in their coding of the same DMT protocols. Second, we constructed a "parallel" photographic picture that retained all structural characteristic of the original and analyzed DMT parallel-test reliability. Third, we examined the construct validity of the DMT by (a) employing three self-report defense-mechanism inventories and analyzing the intercorrelations between DMT defense scores and corresponding defenses in these instruments, (b) studying the relationships between DMT responses and scores on trait and state anxiety, and (c) relating DMT-defense scores to measures of self-esteem. The main results showed that the DMT can be coded with high reliability by trained coders, that the parallel-test reliability is unsatisfactory compared to traditional psychometric standards, that there is a certain generalizability in the number of perceptual distortions that people display from one picture to another, and that the construct validation provided meager empirical evidence for the conclusion that the DMT measures what it purports to measure, that is, psychological defense mechanisms.     

A Multitrait–Multisource Confirmatory Factor Analytic Approach to the Construct Validity of ADHD and ODD Rating Scales with Malaysian Children

Confirmatory factor analysis (CFA) was used to model a multitrait by multisource matrix to determine the convergent and discriminant validity of measures of attention-deficit hyperactivity disorder (ADHD)-inattention (IN), ADHD-hyperactivity/impulsivity (HI), and oppositional defiant disorder (ODD) in 917 Malaysian elementary school children. The three trait factors were ADHD-IN, ADHD-HI, and ODD. The two source factors were parents and teachers. Similar to earlier studies with Australian and Brazilian children, the parent and teacher measures failed to show convergent and discriminant validity with Malaysian children. The study outlines the implications of such strong source effects in ADHD-IN, ADHD-HI, and ODD measures for the use of such parent and teacher scales to study the symptom dimensions.