The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)

The cognitive reflection test (CRT) is a short measure of a person's ability to resist intuitive response tendencies and to produce a normatively correct response, which is based on effortful reasoning. Although the CRT is a very popular measure, its psychometric properties have not been extensively investigated. A major limitation of the CRT is the difficulty of the items, which can lead to floor effects in populations other than highly educated adults. The present study aimed at investigating the psychometric properties of the CRT applying item response theory analyses (a two‐parameter logistic model) and at developing a new version of the scale (the CRT‐long), which is appropriate for participants with both lower and higher levels of cognitive reflection. The results demonstrated the good psychometric properties of the original, as well as the new scale. The validity of the new scale was also assessed by measuring correlations with various indicators of intelligence, numeracy, reasoning and decision‐making skills, and thinking dispositions. Moreover, we present evidence for the suitability of the new scale to be used with developmental samples. Finally, by comparing the performance of adolescents and young adults on the CRT and CRT‐long, we report the first investigation into the development of cognitive reflection. Copyright © 2015 John Wiley & Sons, Ltd.     

Test of memory malingering with children: The utility of Trial 1 and TOMMe10 as screeners of test validity

There is a growing body of research suggesting that the shorter versions of the Test of Memory Malingering (TOMM) may provide an accurate assessment of effort in children. During neuropsychological evaluations, some circumstances result in only one completed trial of the TOMM or partial completion of a trial. Research suggests that a cut-score of 40 or 41 on Trial1 is highly predictive of passing the TOMM overall. In the current study, 194 school-age children with academic and/or behavioral problems were used to compare the accuracy of TOMM1 and TOMMe10 (errors on the first 10 items of TOMM1) in predicting passing/failing of TOMM2. For the children in this sample, a score of < 40 items correct (≥ 10 errors) on TOMM1 was highly accurate in predicting a passing performance on the TOMM2 (sensitivity = .80, specificity = .91) with a Negative Predictive Value = .98 at the malingering base rate of 7% (TOMM2 failure in our sample). A score of 2 errors (8 items correct) on the TOMMe10 was slightly less sensitive than that of the TOMM1 (specificity = .96, sensitivity = .53) but with a similar Negative Predictive Value (.96). Consistent with the research from adult populations, TOMM1 and TOMMe10 appear to be quite accurate in predicting performance on the standard administration of the TOMM and may be useful screeners. However, compared to that found in adult samples, slight differences in suggested cutoffs for TOMM1 and TOMMe10 may be warranted for children.     

Unique designs,errors and strategies in the Five-Point Test: The contribution of age,phonemic fluency and visuospatial abilities in Italian children aged 6–11 years

Performances on the Five-Point Test of 161 Italian children aged 6 to 11 years were investigated, along with phonemic fluency, visual-motor integration, visual perception, motor coordination, visuospatial memory, and fluid intelligence. Five-Point Test accuracy was significantly related to phonemic fluency and visual-motor integration, while phonemic fluency was linked to motor coordination. The two fluency measures increased linearly with age, but the developmental progression of Five-Point Test accuracy was less influenced by age. Different age effects were also found on the relationship between fluid intelligence and the two fluency measures. The inspection of qualitative aspects of Five-Point Test performance (errors and strategies) suggested that strategy usage enhanced both productivity and accuracy in children; age-related changes were observed in the relationship between the number of errors and the total number of designs produced.     

The International Test Commission Guidelines on the Security of Tests,Examinations, and Other Assessments

The amount and severity of security threats have increased considerably over the past two decades, calling into question the validity of assessments administered around the world. These threats have increased for a number of reasons, including the popular use of computerized and online technologies for test administration and the use of almost undetectable technologies for capturing test content and illegally sharing it instantly across borders and cultures. No assessment program, large or small, is immune to this potential damage. The International Test Commission has recognized the critical need for every organization with an important assessment program to be aware of these and be prepared to counter them. It was for this purpose that these guidelines were developed. Knowing the threats and the guidelines will lead to effective measures to protect the program and its assets, maintaining the value of the tests and assessments to the international community. The guidelines listed in this article provide recommendations on planning for better security, maintaining security during the development of tests and while they are administered, and responding well when a security breach occurs. Following these guidelines will create a significant protective barrier between those who willingly commit test fraud and the valuable assets a program has spent time and money to build.     

Developing Pairwise Preference-Based Personality Test and Experimental Investigation of Its Resistance to Faking Effect by Item Response Model

Recent years have shown increased awareness of the importance of personality tests in educational, clinical, and occupational settings, and developing faking-resistant personality tests is a very pragmatic issue for achieving more precise measurement. Inspired by Stark (2002) and Stark, Chernyshenko, and Drasgow (2005), we develop a pairwise preference-based personality test that aims to measure multidimensional personality traits using a large-scale statement bank. An experiment compares the resistance of the developed personality test to faking with that of rating scale-based personality tests in the item response theory model framework. Results show that latent traits estimated from the personality test based on the rating scale method are severely biased, and that faking effect can be pragmatically ignored in the personality test developed based on the pairwise preference method.     

Psychometrics in Support of a Valid Assessment of Linguistic Minorities: Implications for the Test and Sampling Designs

In this study, we propose that the unique needs and characteristics of linguistic minorities should be considered throughout the test development process. Unlike most measurement invariance investigations in the assessment of linguistic minorities, which typically are conducted after test administration, we propose strategies that focus on the early stages of test development. Our approaches involve considering linguistic minorities in the selection of the test and sampling designs. We posit that joint consideration of these issues preemptively can strengthen the claims derived from tests used to assess linguistically diverse populations. This also will contribute to more psychometrically robust assessments, which can yield accurate and valid score-based inferences for linguistic minorities. To this end, we provide various examples and methodological approaches that can be used in the selection of the test and sampling designs that take these issues into consideration.     

PDP启动型实验程序与Richardson-Klavehn模型的比较研究

2个实验对PDP的启动型实验程序与Richardson-Klavehn的修正模型进行了比较研究,通过对指示语的操纵,考察提取策略是否完全受意志控制,以及注意水平、刺激材料、测验类型、加工水平对被试作业的影响.结果发现刺激材料对有意识控制记忆的实验结果影响很大,对PDP的无意识自动加工参数无影响;启动型实验程序的结果与PDP理论相当一致;提取策略不一定是受意志精确控制的.     

A multitrait-multimethod model with minimal assumptions

A new model of confirmatory factor analysis (CFA) for multitrait-multimethod (MTMM) data sets is presented. It is shown that this model can be defined by only three assumptions in the framework of classical psychometric test theory (CTT). All other properties of the model, particularly the uncorrelated-ness of the trait with the method factors are logical consequences of the definition of the model. In the model proposed there are as many trait factors as different traits considered, but the number of method factors is one fewer than the number of methods included in an MTMM study. The covariance structure implied by this model is derived, and it is shown that this model is identified even under conditions under which other CFA-MTMM models are not. The model is illustrated by two empirical applications. Furthermore, its advantages and limitations are discussed with respect to previously developed CFA models for MTMM data.     

心理学视野下的社会变迁研究：研究设计与分析方法

近年来,社会变迁导致的文化、心理与行为变化及其相互建构已经成为心理学的研究热点。社会变迁研究通常涉及和变迁相关的时间效应、时代效应以及和变迁无关的年龄效应。在考察变迁趋势及其影响因素时,常用的研究设计有跨时间比较、跨代际比较、跨地区比较(历史重构)等方法;常用的分析方法有传统的相关和回归分析和现代的时间序列分析(比如交叉滞后相关和格兰杰因果检验)等。由于每一种设计都有其优点和不足,在具体研究中,研究者要根据研究的问题和可操作性选择合适的方法;如果可能,最好同时采用多种方法,以寻求基于不同方法的聚合证据。     

不同铆测验设计下多维IRT等值方法的比较

实际应用中测验往往具有多维结构, 如果仍采用单维IRT方法进行等值, 会得到不准确的结果。因此对于多维结构的测验, 需要使用多维IRT等值方法来实现参数的转换。基于共同题设计, 文章通过模拟研究的方法, 考察了不同铆测验设计下几种多维IRT等值方法的表现, 同时考虑了测验长度、两个维度题目数量的比例、铆测验长度、铆测验的选择策略、两个维度之间的相关和等值群体的能力水平差异六个因素的影响。所比较的多维IRT等值方法有：均值/均值(MM)方法, 均值/标准差(MS)方法, Stoking-Lord (SL)方法, Haebara (HB)方法, 最小平方(LS)方法。结果显示：(1) SL, HB和LS方法得到的等值误差均方根最小, 且在各条件下表现较为稳定。(2) MM和MS方法在非等组条件下呈现出很大的误差均方根。(3)铆测验设计对SL, HB和LS方法的等值结果没有显著影响。(4)在两个维度之间的相关较高, 测验长度和铆测验长度较长, 等值群体的能力水平没有差异的条件下, SL, HB和LS方法得到的等值误差均方根最小。