Prediction of true test scores from observed item scores and ancillary data

In many educational tests which involve constructed responses, a traditional test score is obtained by adding together item scores obtained through holistic scoring by trained human raters. For example, this practice was used until 2008 in the case of GRE^®General Analytical Writing and until 2009 in the case of TOEFL^® iBT Writing. With use of natural language processing, it is possible to obtain additional information concerning item responses from computer programs such as e‐rater^®. In addition, available information relevant to examinee performance may include scores on related tests. We suggest application of standard results from classical test theory to the available data to obtain best linear predictors of true traditional test scores. In performing such analysis, we require estimation of variances and covariances of measurement errors, a task which can be quite difficult in the case of tests with limited numbers of items and with multiple measurements per item. As a consequence, a new estimation method is suggested based on samples of examinees who have taken an assessment more than once. Such samples are typically not random samples of the general population of examinees, so that we apply statistical adjustment methods to obtain the needed estimated variances and covariances of measurement errors. To examine practical implications of the suggested methods of analysis, applications are made to GRE General Analytical Writing and TOEFL iBT Writing. Results obtained indicate that substantial improvements are possible both in terms of reliability of scoring and in terms of assessment reliability.     

Failure of further learning: Activities,structure, and meaning

Previous research has shown that little benefit is achieved through spaced study and recall of text passages after the first recall attempt, an effect that we term the failure‐of‐further‐learning. We hypothesized that the effect occurs because a situation model of the text's gist is formed when the text is first comprehended and is consolidated when recalled; it dominates later recall after verbatim memories of more recent study episodes have been lost. Experiments 1 and 2 attempted to circumvent the effect by varying the activities of participants and requiring interactive exploration. In both experiments, recall after four, weekly sessions showed little benefit beyond performance on the first recall. Experiment 3 interfered with the formation of an immediate situation model by introducing passages that were hard to comprehend without a title. Performance improved substantially across four sessions when titles were not supplied, but the standard effect was replicated when titles were given. Experiment 4 made verbatim memories available by incorporating all re‐presentations and tests into one session; as predicted, recall improved over successive tests.     

Development of Japanese version of the long‐term mating orientation scale (LTMO‐J)

The purpose of this study was to validate the 7‐item long‐term mating orientation scale (LTMO) as translated into Japanese. Two samples of Japanese adults (N = 2000; 50.0% male; Mage = 40.9 for the first survey; N = 300; 54.7% male; Mage = 42.4 for the second survey) completed a web‐based questionnaire, including the Japanese translation of the LTMO. The results showed that the psychometric properties of the Japanese LTMO scale were comparable to those of the original English version. The scale had adequate reliability based on Cronbach's α and McDonald's ω. Convergent validity was demonstrated by the correlation between the LTMO scores and related variables: human life history strategies, short‐term sociosexual orientation, attitude to infidelity, romantic attachment style, and so on. The translated scale provides a valid and reliable instrument in Japanese that measures human mating strategy.     

The effects of linearity on sentence comprehension in oral and silent reading

This study examined the effects of linearity of language (e.g., de Saussure, 1916/1972) on performance on a sentence comprehension task, manipulating (a) presentation method (simultaneous, phrase, or flow) and (b) reading method (oral or silent reading). The study also examined the predictive powers of the digit span test (DST) and the reading span test (RST) for task performance. Fifty‐four undergraduate and graduate students participated in the experiment. The results showed that the DST score predicted performance on both garden‐path (GP) and non‐garden‐path (NGP) sentences, whereas the RST score did so only for GP sentences. This suggests that the sentence comprehension task requires not only comprehension but also temporarily storing verbal information, and that the RST score reflects the degree of tolerance for deviation from linearity. The results also revealed that silent reading has an advantage over oral reading in simultaneous presentation and vice versa in flow presentation, whereas the difference between reading methods disappeared in phrase presentation. This suggests that oral and silent reading require different strategies to facilitate sentence comprehension.     

适用于多维迫选测验的IRT计分模型

迫选(forced-choice,FC)测验由于可以控制传统李克特方法带来的反应偏差,被广泛应用于非认知测验中,而迫选测验的传统计分方式会产生自模式数据,这种数据由于不适合于个体间的比较,一直备受批评。近年来,多种迫选IRT模型的发展使研究者能够从迫选测验中获得接近常模性的数据,再次引起了研究者与实践人员对迫选IRT模型的兴趣。首先,依据所采纳的决策模型和题目反应模型对6种较为主流的迫选IRT模型进行分类和介绍。然后,从模型构建思路、参数估计方法两个角度对各模型进行比较与总结。其次,从参数不变性检验、计算机化自适应测验(computerized adaptive testing, CAT)和效度研究3个应用研究方面进行述评。最后提出未来研究可以在模型拓展、参数不变性检验、迫选CAT测验和效度研究4个方向深入。     

问题解决测验中过程数据的特征抽取与能力评估

基于计算机的问题解决测验可以实时记录被试探索环境和解决问题时的详细行动痕迹,并保存为过程数据。首先介绍了过程数据的分析流程,然后从问题解决测验入手,分别对过程数据的特征抽取和能力估计建模两方面的研究进行了梳理和评价。未来研究应注意：提高分析结果的可解释性;特征提取时纳入更多信息;实现更复杂问题情景下的能力评估;注重方法的实用性;以及融合与借鉴不同领域的分析方法。     

Assessing the Multi-faceted Nature of Test Anxiety Among Secondary School Students: An English Version of the German Test Anxiety Questionnaire: PAF-E

The current study concerns the validation of an English version of the German Test Anxiety Inventory, namely the PAF-E. This questionnaire is a multi-faceted measure of test anxiety designed to detect normative test anxiety levels and in consequence meet the need of consultancy. Construct and criterion validity of (PAF-E) were examined with a sample of 96 secondary students (Mage = 12.8, SD = 0.67; 55% girls) from an international school in Berlin (Germany) and 399 secondary students (Mage = 13.4, SD = 0.80; 56% girls) from Montréal (Canada). Both samples completed the PAF-E and related constructs, such as school-related self-efficacy, inhibitory test anxiety, achievement motivation, and the Big Five. Exploratory and confirmatory factor analyses confirmed the four-factor-structure (worry, emotionality, interfering thoughts, lack of confidence) of the original German Test Anxiety Inventory (PAF). Each subscale consists of five items with a total of 20 questions. Cronbach's alpha, ranging from.71 to.82 among Germans and.77 to.87 among Canadians as well as the re-test reliability (from.80 to.85 among Canadians) were sufficient. The differential patterns of correlations between other constructs and the indices of test anxiety indicate good construct validity.     

Measuring the automatic negative self-schema: New evidence for the construct and criterion validity of the Depression Implicit Association

The construct and criterion validity of the Depression Implicit Association Test (Depression IAT) as a marker of an automatic negative self-schema was investigated. The Depression IAT and other measures were administered to a sample of 116 participants (72 females) aged 37.28 (SD = 15.69) that was composed by 56 patients with an history of suicide ideation (SI) and by 60 university students. Combining students and patients’ sub-samples, results revealed that the Depression IAT was significantly and positively correlated with self-report scales of depression and hopelessness as well as with implicit and explicit measures of death-life identification. On the contrary, it was negatively correlated with life satisfaction, optimism, and self-esteem scales. These results give new evidence for the construct validity of the Depression IAT. Moreover, considering only the patients’ sub-sample, significant correlations between Depression IAT scores and SI in the last year, month, and week, as well as in a follow-up observation two months later, were found. These correlations remain unchanged even when the Death IAT was controlled for, supporting the incremental validity of the Depression IAT. Overall, these results provide new evidence for the construct and criterion validity of the Depression IAT as a marker of an automatic negative self-schema.     

No evidence of risk-taking or impulsive behaviour in a person with episodic amnesia: Implications for the role of the hippocampus in future-regarding decision-making

Does advantageous decision-making require one to explicitly remember the outcome of a series of past decisions or to imagine future personal consequences of one's choices? Findings that amnesic people with hippocampal damage cannot form a clear preference for advantageous decks over many learning trials on the Iowa Gambling Task (IGT) have been taken to suggest that complex decision-making on the IGT depends on declarative (episodic) memory and hippocampal integrity. Alternatively, impaired IGT performance in amnesic individuals could be secondary to risk-taking and/or impulsive behaviour resulting from impaired episodic future thinking (i.e. prospection) known to accompany amnesia. We tested this possibility in the amnesic individual K.C. using the IGT and the Toronto Gambling Task (TGT), a novel task that dissociates impulsivity from risk-taking without placing demands on declarative memory. K.C. did not develop a preference for advantageous over disadvantageous decks on the IGT and, instead, showed a slight preference for short-term gains and an inability to acquire a more adaptive appreciation of longer-term losses. He also did not display impulsive or risk-taking behaviour on the TGT, despite a profound inability to imagine personal future experiences. These findings suggest that impaired decision-making on the IGT in amnesia is unlikely to reflect a predilection to act in the moment or failure to take future consequences into account. Instead, some forms of future-regarding decision-making may be dissociable, with performance on tasks relying on declarative learning or on episodic-constructive processes more likely to be impaired.     

Adaptation transculturelle de tests et échelles de mesure psychologiques : guide pratique basé sur les Recommandations de la Commission Internationale des Tests et les Standards de pratique du testing de l’APA

Thirty years ago, Vallerand (1989) wrote a pioneering article entitled “Toward a methodology for the transcultural validation of psychological questionnaires: Implications for research in the French language”, whose success has never failed, as well as its usefulness. However, considerable progress has been made in the field of test development and test adaptation since the publication of this article, notably with the publications (and their updates) of the ITC Guidelines for Translating and Adapting Tests (2017) by the International Test Commission, and of the Standards for Educational and Psychological Testing (2014) by the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME). Based on these two key references, the present article aimed to update the procedure proposed by Vallerand. Without claiming to replace these major references, on the contrary, our objective here is to propose a 10-step procedure for adapting tests and psychological measures originally developed for other cultures and/or languages, a sort of framework guiding in step-by-step manner the adaptation of tests and the validation of their scores.