Psychometric and Validity Issues in Machine Learning Approaches to Personality Assessment: A Focus on Social Media Text Mining

In the age of big data, substantial research is now moving toward using digital footprints like social media text data to assess personality. Nevertheless, there are concerns and questions regarding the psychometric and validity evidence of such approaches. We seek to address this issue by focusing on social media text data and (i) conducting a review of psychometric validation efforts in social media text mining (SMTM) for personality assessment and discussing additional work that needs to be done; (ii) considering additional validity issues from the standpoint of reference (i.e. ‘ground truth’) and causality (i.e. how personality determines variations in scores derived from SMTM); and (iii) discussing the unique issues of generalizability when validating SMTM for personality assessment across different social media platforms and populations. In doing so, we explicate the key validity and validation issues that need to be considered as a field to advance SMTM for personality assessment, and, more generally, machine learning personality assessment methods. © 2020 European Association of Personality Psychology     

Applying theory to the black box: A model for empirically scoring biodata

Despite being an effective predictor of job performance, empirically keyed biodata assessments have been criticized as black box empiricism unlikely to generalize to new contexts. This paper introduces a model that challenges this perspective, explicating how biodata content, job demands, and criterion variables collectively influence the construct validity, and generalizability of empirically scored biodata. Across two field studies, expected changes in scale correlations with external measures were found that coincided with changes in the contextual similarity between calibration and holdout contexts, the criteria used, and the content validity of biodata items. Collectively, this paper offers a framework that helps understand and optimize empirical biodata keying in practice, furthering confidence for their use in applied settings.     

A comparison of teaching tacts with and without background stimuli on acquisition and generality

We evaluated whether teaching tacts of images with or without backgrounds would affect acquisition during teaching and generality across untaught targets with four children diagnosed with autism spectrum disorder. Including backgrounds could slow acquisition but bolster generality across other media containing backgrounds, whereas removing backgrounds may improve acquisition but result in poorer generality. Overall, acquisition rates were similar across conditions for three children, although one child displayed consistently slower acquisition during the background condition. We rarely observed generality to untaught targets when programming extinction, but embedding differential reinforcement or teaching additional images with backgrounds led to equally successful outcomes irrespective of initial teaching conditions. Teaching with or without backgrounds may result in marginal differences in acquisition of teaching and untaught targets, especially when arranging supplemental procedures during tests for generality. We discuss the importance of intrasubject replication, considerations when testing for generality, and implications for practice.     

The Importance of Construct Validity in Consumer Research

Vosgerau, Scopelliti, and Huh (2020) question the construct validity of the most commonly used measures of self‐control and argue that many studies that claim to be examining self‐control may in fact be examining something else. In this comment, we generalize this argument and propose that this criticism could be leveraged against many other areas of consumer research. Indeed, many studies fail to properly establish the construct validity of their measures. Consequently, they may not be examining the construct they purport to be studying, which can lead to unreliable and conflicting findings.     

Actions speak louder than words: Differences in memory flexibility between monolingual and bilingual 18‐month‐olds

Bilingual infants from 6‐ to 24‐months of age are more likely to generalize, flexibly reproducing actions on novel objects significantly more often than age‐matched monolingual infants are. In the current study, we examine whether the addition of novel verbal labels enhances memory generalization in a perceptually complex imitation task. We hypothesized that labels would provide an additional retrieval cue and aid memory generalization for bilingual infants. Specifically, we hypothesized that bilinguals might be more likely than monolinguals to map multiple perceptual features onto a novel label and therefore show enhanced generalization. Eighty‐seven 18‐month‐old monolingual and bilingual infants were randomly assigned to one of two experimental conditions or a baseline control condition. In the experimental conditions, either no label or a novel label was added during demonstration and again at the beginning of the test session. After a 24‐hr delay, infants were tested with the same stimulus set to test cued recall and with a perceptually different but functionally equivalent stimulus set to test memory generalization. Bilinguals performed significantly above baseline on both cued recall and memory generalization in both experimental conditions, whereas monolinguals performed significantly above baseline only on cued recall in both experimental conditions. These findings show a difference between monolinguals and bilinguals in memory generalization and suggest that generalization differences between groups may arise from visual perceptual processing rather than linguistic processing. A video abstract of this article can be viewed at https://youtu.be/yXB4pM3fF2k     

Psychometric adaption of the impulsive driver behavior scale in a Chinese sample

Impulsivity plays an important role in aberrant driving behavior and crash involvement. The purpose of this study was to evaluate the reliability and validity of the Impulsive Driver Behavior Scale (IDBS, Bıçaksız & Özkan, 2016a) with a Chinese sample. Two hundred and ninety-nine drivers completed the Chinese version of the Impulsive Driver Behavior Scale, the Driver Behavior Questionnaire, the Big Five Inventory and some social-demographic and traffic violation items. A confirmatory factor analysis indicated that a four-factor solution with 20 items yielded a better fit to the data than other solutions. The reduced IDBS showed good reliability and a stable structure. Drivers’ functional impulsivity was positively correlated with positive driver behavior and some ordinary violations. Meanwhile, the other three dimensions of dysfunctional impulsivity showed negative correlations with positive driver behavior and positive associations with aberrant driving behavior and penalty points and fines. Impulsive driver behavior is also associated with agreeableness and conscientiousness. These results provide evidence supporting the IDBS as a reliable and valuable instrument for measuring driving impulsivity in the Chinese traffic environment.     

Virtual reality: A new track in psychological research

One major challenge of social interaction research is to achieve high experimental control over social interactions to allow for rigorous scientific reasoning. Virtual reality (VR) promises this level of control. Pan and Hamilton guide us with a detailed review on existing and future possibilities and challenges of using VR for social interaction research. Here, we extend the discussion to methodological and practical implications when using VR.     

An empirical look at the Defense Mechanism Test (DMT): reliability and construct validity

Although the Defense Mechanism Test (DMT) has been in use for almost half a century, there are still quite contradictory views about whether it is a reliable instrument, and if so, what it really measures. Thus, based on data from 39 female students, we first examined DMT inter-coder reliability by analyzing the agreement among trained judges in their coding of the same DMT protocols. Second, we constructed a "parallel" photographic picture that retained all structural characteristic of the original and analyzed DMT parallel-test reliability. Third, we examined the construct validity of the DMT by (a) employing three self-report defense-mechanism inventories and analyzing the intercorrelations between DMT defense scores and corresponding defenses in these instruments, (b) studying the relationships between DMT responses and scores on trait and state anxiety, and (c) relating DMT-defense scores to measures of self-esteem. The main results showed that the DMT can be coded with high reliability by trained coders, that the parallel-test reliability is unsatisfactory compared to traditional psychometric standards, that there is a certain generalizability in the number of perceptual distortions that people display from one picture to another, and that the construct validation provided meager empirical evidence for the conclusion that the DMT measures what it purports to measure, that is, psychological defense mechanisms.     

A Multitrait–Multisource Confirmatory Factor Analytic Approach to the Construct Validity of ADHD and ODD Rating Scales with Malaysian Children

Confirmatory factor analysis (CFA) was used to model a multitrait by multisource matrix to determine the convergent and discriminant validity of measures of attention-deficit hyperactivity disorder (ADHD)-inattention (IN), ADHD-hyperactivity/impulsivity (HI), and oppositional defiant disorder (ODD) in 917 Malaysian elementary school children. The three trait factors were ADHD-IN, ADHD-HI, and ODD. The two source factors were parents and teachers. Similar to earlier studies with Australian and Brazilian children, the parent and teacher measures failed to show convergent and discriminant validity with Malaysian children. The study outlines the implications of such strong source effects in ADHD-IN, ADHD-HI, and ODD measures for the use of such parent and teacher scales to study the symptom dimensions.     

Validity of the Eating Disorder Examination Questionnaire (EDE-Q) in screening for eating disorders in community samples

In order to examine the concurrent and criterion validity of the questionnaire version of the Eating Disorders Examination (EDE-Q), self-report and interview formats were administered to a community sample of women aged 18-45 (n = 208). Correlations between EDE-Q and EDE subscales ranged from 0.68 for Eating Concern to 0.78 for Shape Concern. Scores on the EDE-Q were significantly higher than those of the EDE for all subscales, with the mean difference ranging from 0.25 for Restraint to 0.85 for Shape Concern. Frequency of both objective bulimic episodes (OBEs) and subjective bulimic episodes (SBEs) was significantly correlated between measures. Chance-corrected agreement between EDE-Q and EDE ratings of the presence of OBEs was fair, while that for SBEs was poor. Receiver operating characteristic (ROC) analysis, based on a sample of 13 cases, indicated that a score of 2.3 on the global scale of the EDE-Q in conjunction with the occurrence of any OBEs and/or use of exercise as a means of weight control, yielded optimal validity coefficients (sensitivity = 0.83, specificity = 0.96, positive predictive value = 0.56). A stepwise discriminant function analysis yielded eight EDE-Q items which best distinguished cases from non-cases, including frequency of OBEs, use of exercise as a means of weight control, use of self-induced vomiting, use of laxatives and guilt about eating. The EDE-Q has good concurrent validity and acceptable criterion validity. The measure appears well-suited to use in prospective epidemiological studies.