Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not?

When examinees' test-taking motivation is questionable, practitioners must determine whether careless responding is of practical concern and if so, decide on the best approach to filter such responses. As there has been insufficient research on these topics, the objectives of this study were to: a) evaluate the degree of underestimation in the true mean when careless responses are present, and b) compare the effectiveness of two filtering procedures in purifying biased aggregated-scores. Results demonstrated that: a) the true mean was underestimated by around 0.20 SDs if the total amount of careless responses exceeded 6.25%, 12.5%, and 12.5% for easy, moderately difficult, and difficult tests, respectively, and b) listwise deleting data from unmotivated examinees artificially inflated the true mean by as much as .42 SDs when ability was related to careless responding. Findings from this study have implications for when and how practitioners should handle careless responses for group-based low-stakes assessments.     

心理与教育测验中反应时模型应用的优势与举例

基于计算机形式的测验使得收集作答反应时信息成为可能,这些信息的有效利用对心理与教育测验的理论研究和实际应用产生了重大影响。首先,归纳并总结了测验中使用反应时信息的五大优势。其次,分别介绍了4种不同取向下较典型的反应时模型与模型特征,并分别进行评价。再次,较系统地梳理了反应时模型在实践中的应用,使读者了解反应时信息在测验中所发挥的作用。最后,探讨了未来将反应时应用于心理与教育测量领域的几个研究方向。     

Executive dysfunction affects word list recall performance: Evidence from amyotrophic lateral sclerosis and other neurodegenerative diseases

The Rey Auditory Verbal Learning Test (RAVLT) is widely used in clinical practice to evaluate verbal episodic memory. While there is evidence that RAVLT performance can be influenced by executive dysfunction, the way executive disorders affect the serial position curve (SPC) has not been yet explored. To this aim, we analysed immediate and delayed recall performances of 13 non‐demented amyotrophic lateral sclerosis (ALS) patients with a specific mild executive dysfunction (ALSci) and compared their performances to those of 48 healthy controls (HC) and 13 cognitively normal patients with ALS. Moreover, to control for the impact of a severe dysexecutive syndrome and a genuine episodic memory deficit on the SPC, we enrolled 15 patients with a diagnosis of behavioural variant of frontotemporal dementia (bvFTD) and 18 patients with probable Alzheimer's disease (AD). Results documented that, compared to cognitively normal subjects, ALSci patients had a selective mid‐list impairment for immediate recall scores. The bvFTD group obtained low performances with a selectively increased forgetting rate for terminal items, whereas the AD group showed a disproportionately large memory loss on the primary and middle part of the SPC for immediate recall scores and were severely impaired in the delayed recall trial. These results suggested that subtle executive dysfunctions might influence the recall of mid‐list items, possibly reflecting deficiency in control strategies at retrieval of word lists, whereas severer dysexecutive syndrome might also affect the recall of terminal items possibly due to attention deficit or retroactive interference.     

Developing new online calibration methods for multidimensional computerized adaptive testing

Multidimensional computerized adaptive testing (MCAT) has received increasing attention over the past few years in educational measurement. Like all other formats of CAT, item replenishment is an essential part of MCAT for its item bank maintenance and management, which governs retiring overexposed or obsolete items over time and replacing them with new ones. Moreover, calibration precision of the new items will directly affect the estimation accuracy of examinees’ ability vectors. In unidimensional CAT (UCAT) and cognitive diagnostic CAT, online calibration techniques have been developed to effectively calibrate new items. However, there has been very little discussion of online calibration in MCAT in the literature. Thus, this paper proposes new online calibration methods for MCAT based upon some popular methods used in UCAT. Three representative methods, Method A, the ‘one EM cycle’ method and the ‘multiple EM cycles’ method, are generalized to MCAT. Three simulation studies were conducted to compare the three new methods by manipulating three factors (test length, item bank design, and level of correlation between coordinate dimensions). The results showed that all the new methods were able to recover the item parameters accurately, and the adaptive online calibration designs showed some improvements compared to the random design under most conditions.     

Introducing a forced-choice recognition task to the California Verbal Learning Test – Children’s Version

The importance of performance validity tests (PVTs) is increasingly recognized in pediatric neuropsychology. To date, research has focused on investigating whether PVTs designed for adults function similarly in children. The downward extension of adult cutoffs is counter-intuitive considering the robust effect of age-related changes in basic cognitive skills in children and adolescents. The purpose of this study was to examine the signal detection properties of a forced-choice recognition trial (FCR-C) for the California Verbal Learning Test – Children’s Version. A total of 72 children aged 6–15 years (M = 11.1 , SD = 2.6) completed the FCR-C as part of a larger neuropsychological assessment battery. Cross-validation analyses revealed that the FCR-C had good signal detection performance against reference PVTs. The first level of failure (≤14/15) produced the best combination of overall sensitivity (.31) and specificity (.87). A more conservative FCR-C cutoff (≤13) resulted in a predictable trade-off between sensitivity (.15) and specificity (.94), but also a net loss in discriminant power. Lowering the cutoff to ≤12 resulted in a slight improvement in specificity (.97) but further deterioration in sensitivity (.14). These preliminary findings suggest that the FCR-C has the potential to become the newest addition to a growing arsenal of pediatric PVTs.     

Does mere exposure enhance positive evaluation,independent of stimulus recognition? A replication study in Japan and the USA1

Abstract: In order to examine whether the mere exposure effect, the finding that exposure to a stimulus enhances a positive evaluation of it, occurs in Japan as well as in the USA, a test was carried out using a novel auditory stimulus. Japanese and American people were asked to listen to repeatedly presented Tagalog (the native Filipino language) utterances. They were then asked to estimate their liking for each utterance and whether they recognized it as one presented during the first exposure stage. The mere exposure effect was found in both cultures. Further, it resulted from only exposure frequency, not recognition. Implications for preference and the role of culture are discussed.     

Taking theoretical risks in a world of directional predictions

As Meehl (1978) observed, one major reason for the painfully slow progress of soft psychology is our field’s heavy reliance on directional predictions, which leads to feeble tests of theories. In the present commentary, I outline six ways of taking theoretical risks in a world of directional predictions: (1) constructive replication of findings, (2) generating multiple maximally independent predictions, (3) adherence to the total evidence rule, (4) discriminant validation of measures, (5) incremental validation of measures, and (6) according attention to higher-order personality dimensions.     

Wofür sprechen die Daten?

What Do the Data Tell Us? Justification of scientific theories is a three-place relation between data, theories, and background knowledge. Though this should be a commonplace, many methodologies in science neglect it. The article will elucidate the significance and function of our background knowledge in epistemic justification and their consequences for different scientific methodologies. It is argued that there is no simple and at the same time acceptable statistical algorithm that justifies a given theory merely on the basis of certain data. And even if we think to know the probability of a theory, that does not decide whether we should accept it or not.     

人员选拔中的不准确性来源及对策

在组织人员选拔中存在着一些不准确性来源．这些不准确性直接影响组织人员招募与配置的效果。该文从主观和客观两个维度探讨了人员选拔中的不准确性来源,并提出了减少人员选拔中不准确性来源的对策。