The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)

The cognitive reflection test (CRT) is a short measure of a person's ability to resist intuitive response tendencies and to produce a normatively correct response, which is based on effortful reasoning. Although the CRT is a very popular measure, its psychometric properties have not been extensively investigated. A major limitation of the CRT is the difficulty of the items, which can lead to floor effects in populations other than highly educated adults. The present study aimed at investigating the psychometric properties of the CRT applying item response theory analyses (a two‐parameter logistic model) and at developing a new version of the scale (the CRT‐long), which is appropriate for participants with both lower and higher levels of cognitive reflection. The results demonstrated the good psychometric properties of the original, as well as the new scale. The validity of the new scale was also assessed by measuring correlations with various indicators of intelligence, numeracy, reasoning and decision‐making skills, and thinking dispositions. Moreover, we present evidence for the suitability of the new scale to be used with developmental samples. Finally, by comparing the performance of adolescents and young adults on the CRT and CRT‐long, we report the first investigation into the development of cognitive reflection. Copyright © 2015 John Wiley & Sons, Ltd.     

Test of memory malingering with children: The utility of Trial 1 and TOMMe10 as screeners of test validity

There is a growing body of research suggesting that the shorter versions of the Test of Memory Malingering (TOMM) may provide an accurate assessment of effort in children. During neuropsychological evaluations, some circumstances result in only one completed trial of the TOMM or partial completion of a trial. Research suggests that a cut-score of 40 or 41 on Trial1 is highly predictive of passing the TOMM overall. In the current study, 194 school-age children with academic and/or behavioral problems were used to compare the accuracy of TOMM1 and TOMMe10 (errors on the first 10 items of TOMM1) in predicting passing/failing of TOMM2. For the children in this sample, a score of < 40 items correct (≥ 10 errors) on TOMM1 was highly accurate in predicting a passing performance on the TOMM2 (sensitivity = .80, specificity = .91) with a Negative Predictive Value = .98 at the malingering base rate of 7% (TOMM2 failure in our sample). A score of 2 errors (8 items correct) on the TOMMe10 was slightly less sensitive than that of the TOMM1 (specificity = .96, sensitivity = .53) but with a similar Negative Predictive Value (.96). Consistent with the research from adult populations, TOMM1 and TOMMe10 appear to be quite accurate in predicting performance on the standard administration of the TOMM and may be useful screeners. However, compared to that found in adult samples, slight differences in suggested cutoffs for TOMM1 and TOMMe10 may be warranted for children.     

The International Test Commission Guidelines on the Security of Tests,Examinations, and Other Assessments

The amount and severity of security threats have increased considerably over the past two decades, calling into question the validity of assessments administered around the world. These threats have increased for a number of reasons, including the popular use of computerized and online technologies for test administration and the use of almost undetectable technologies for capturing test content and illegally sharing it instantly across borders and cultures. No assessment program, large or small, is immune to this potential damage. The International Test Commission has recognized the critical need for every organization with an important assessment program to be aware of these and be prepared to counter them. It was for this purpose that these guidelines were developed. Knowing the threats and the guidelines will lead to effective measures to protect the program and its assets, maintaining the value of the tests and assessments to the international community. The guidelines listed in this article provide recommendations on planning for better security, maintaining security during the development of tests and while they are administered, and responding well when a security breach occurs. Following these guidelines will create a significant protective barrier between those who willingly commit test fraud and the valuable assets a program has spent time and money to build.     

Psychometrics in Support of a Valid Assessment of Linguistic Minorities: Implications for the Test and Sampling Designs

In this study, we propose that the unique needs and characteristics of linguistic minorities should be considered throughout the test development process. Unlike most measurement invariance investigations in the assessment of linguistic minorities, which typically are conducted after test administration, we propose strategies that focus on the early stages of test development. Our approaches involve considering linguistic minorities in the selection of the test and sampling designs. We posit that joint consideration of these issues preemptively can strengthen the claims derived from tests used to assess linguistically diverse populations. This also will contribute to more psychometrically robust assessments, which can yield accurate and valid score-based inferences for linguistic minorities. To this end, we provide various examples and methodological approaches that can be used in the selection of the test and sampling designs that take these issues into consideration.     

Disentangling proactive and reactive aggression in children using self-report

The distinction between proactive and reactive functions of aggression is one of the most common divisions when investigating aggression among children and adolescents. To date, self-report is the least used measurement, despite existing literature supporting the view that the best informant regarding internal processes and motives are children themselves. The main aim of this study was to examine the construct and concurrent validity of a new self-report questionnaire, which aims to disentangle acts of reactive vs. proactive aggression that are most common within the daily lives of children. We examined the self-report measure among 578 children (313 girls, 265 boys, mean age 11 years, range 9–13 years). Most children (90% boys; 85% girls) reported at least one act of aggression over the last four weeks. Furthermore, the outcomes support the two-factor structure (reactive and proactive aggression) and the questionnaire showed good concurrent and discriminant validity with measures for emotional and social functioning. This study validates the use of the self-report instrument for reactive and proactive aggression and demonstrates that children can successfully distinguish between their own motives for reactive and proactive forms of aggressive behaviours.     

Construct validity and construct equivalence of work values across boys and girls

This study investigated the structural validity and construct comparability, across boys and girls, of a newly developed measure of work values. The total sample was split into two equal parts, and based on an item-level factor analysis of the first sub-sample, fit statistics provided support for a six-factor first-order structural model with the following constructs: Power, Money, Tidy environment, Teamwork, Work–Family Balance and Ability Utilization. This structure was confirmed in the second sub-sample, and strict measurement invariance for the measurement model was obtained in this sample. Additionally, the factor variances/covariances were invariant. Five non-invariant latent means were found, with girls scoring higher on the Balance and Personal Challenge values, and boys scoring higher on the Power, Money and Teamwork values.     

A multitrait-multimethod model with minimal assumptions

A new model of confirmatory factor analysis (CFA) for multitrait-multimethod (MTMM) data sets is presented. It is shown that this model can be defined by only three assumptions in the framework of classical psychometric test theory (CTT). All other properties of the model, particularly the uncorrelated-ness of the trait with the method factors are logical consequences of the definition of the model. In the model proposed there are as many trait factors as different traits considered, but the number of method factors is one fewer than the number of methods included in an MTMM study. The covariance structure implied by this model is derived, and it is shown that this model is identified even under conditions under which other CFA-MTMM models are not. The model is illustrated by two empirical applications. Furthermore, its advantages and limitations are discussed with respect to previously developed CFA models for MTMM data.     

艾森克人格问卷简式量表中国版(EPQ-RSC)的修订

该文引进艾森克人格问卷简式量表（EPQ-RShortScale,EPQ-RS）,首次在国内加以修订。根据来自30个省市56个地区的8637人（含汉族7725人）的样本数据,修订形成了艾森克人格问卷简式量表中国版（EPQ-RSC）,共包括4个分量表,各12个项目,这一形式和EPQ-RS相同。作者并基于本研究的样本提供了EPQ-RSC的中国常模。研究表明该问卷的E、N和L三个分量表具有令人满意的信度和效度,P量表的信度和效度基本达到有关的心理测量学标准。作者最后对此问卷的信效度结果以及问卷结果反映的当代中国人人格特点进行了讨论。     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

Rethinking Epistemic Relativism

“Relativism” is often treated as a dirty word in philosophy, but relativistic theories are not entirely unappealing—they have features that might be tempting if they weren't thought to be outweighed by problematic consequences. The aim of this paper is to rethink both our attitude to epistemic relativism and the basic features of the view itself. The paper discusses four objections and uses them to isolate five constraints on a more plausible epistemic relativism. It then sketches out a view that meets all of these constraints. This stratified epistemic relativism offers a complex, socially informed picture of justification that accounts for the many different kinds of roles that epistemic agents act, and think, in accordance with each day, and can serve as a starting point for constructing a more detailed epistemic relativism, which could secure its appealing benefits without incurring the costs traditionally associated with relativist views.