Automatic grading and hinting in open-ended text questions

Open-ended text questions provide better assessment of learner’s knowledge, but analysing answers for this kind of questions, checking their correctness, and generating of detailed formative feedback about errors for the learner are more difficult and complex tasks than for closed-ended questions like multiple-choice.The analysis of answers for open-ended questions can be performed on different levels. Analysis on character level allows to find errors in characters’ placement inside a word or a token; it is typically used to detect and correct typos, allowing to differ typos from actual errors in the learner’s answer. The word-level or token-level analysis allows finding misplaced, extraneous, or missing words in the sentence. The semantic-level analysis is used to capture formally the meaning of the learner’s answer and compare it with the meaning of the correct answer that can be provided in a natural or formal language. Some systems and approaches use analysis on several levels.The variability of the answers for open-ended questions significantly increases the complexity of the error search and formative feedback generation tasks. Different types of patterns including regular expressions and their use in questions with patterned answers are discussed. The types of formative feedback and modern approaches and their capabilities to generate feedback on different levels are discussed too.Statistical approaches or loosely defined template rules are inclined to false-positive grading. They are generally lowering the workload of creating questions, but provide low feedback. Approaches based on strictly-defined sets of correct answers perform better in providing hinting and answer-until-correct feedback. They are characterised by a higher workload of creating questions because of the need to account for every possible correct answer by the teacher and fewer types of detected errors.The optimal choice for creating automatised e-learning courses are template-based open-ended question systems like OntoPeFeGe, Preg, METEOR, and CorrectWriting which allows answer-until-correct feedback and are able to find and report various types of errors. This approach requires more time to create questions, but less time to manage the learning process in the courses once they are run.     

Consistency measures for pairwise comparison matrices

We propose new measures of consistency of additive and multiplicative pairwise comparison matrices. These measures, the relative consistency and relative error, are easy to compute and have clear and simple algebraic and geometric meaning, interpretation and properties. The correspondence between these measures in the additive and multiplicative cases reflects the same correspondence which underpins the algebraic structure of the problem and relates naturally to the corresponding optimization models and axiom systems. The relative consistency and relative error are related to one another by the theorem of Pythagoras through the decomposition of comparison matrices into their consistent and error components. One of the conclusions of our analysis is that inconsistency is not a sufficient reason for revision of judgements. © 1998 John Wiley & Sons, Ltd.     

On Being Happy and Mistaken on a Good Day: Revisiting Forgas’s (1998) Mood-Bias Result

Forgas (1998) reported “on being happy and mistaken,” finding that a happy mood increased the fundamental attribution error (FAE) compared to a sad mood. However, the standard attitude-attribution paradigm used by Forgas might contain demand characteristics, to which happy people might be especially susceptible. In addition, Goldenberg and Forgas (2012) showed that a happy mood decreased a form of the FAE. Using the questioner-contestant paradigm, we replicated Forgas’s result, but only when participants were already having a “good day.”     

The relationship between cognitive errors,coping strategies,and clients' experiences in session: An exploratory study

Aim: Cognitive errors (CE) and coping strategies (CS) can bear weight on how individuals relate to others and perceive interpersonal relationships. However, there is little research into how clients' erroneous beliefs and maladaptive coping strategies can interfere with the therapeutic process. This study utilised a sample of healthy clients to explore the relationship between their CEs and CSs and their evaluation of therapy. Method: Therapy sessions of undergraduate student clients (N =26) were rated using the Cognitive Error Rating Scale (CERS – 3rd edition), the Coping Patterns Rating Scale (CPRS;), the Session Evaluation Questionnaire (SEQ) and the Session Impact Scale (SIS). Results: Clients who engaged in dichotomous thinking endorsed problem solving less and were more likely to feel unsupported and misunderstood by the therapist. Clients who discounted the positive tended to feel more pressured and judged by therapists. Conversely, those who engaged in problem solving were more likely to find sessions deeper and more valuable as compared to those who reacted to stressful events by submission, escape, or opposition. Implications: Better understanding how and when a client's cognitive errors and coping mechanisms are at play during therapy can help clinicians to address them and intervene appropriately.     

新手考官的工作记忆和人格对其评分有效性的影响——以无领导小组讨论为例

相对于其它评价中心技术而言,在无领导小组讨论中考官因素对评分结果的影响尤为重要.本研究主要探讨无领导小组讨论中新手考官的工作记忆与人格对其评分有效性的影响.结果发现,首先,新手考官的评分者一致性较低,评分准确度较差.其次,工作记忆和人格的部分因素分别从不同方面影响新手考官的评分有效性,具体表现在：（1）利他性越强,新手考官评分总均值的准确性越高,且评分结果越宽松;（2）新手考官的决断性越强,对所有应聘者做出有效区分的准确性越高;（3）新手考官的沉稳性越高,对各维度的区分越有效;（4）注意转换和抑制能力对新手考官的晕轮效应及其在各个维度上进行区分的准确度有抑制作用.     

Prediction of true test scores from observed item scores and ancillary data

In many educational tests which involve constructed responses, a traditional test score is obtained by adding together item scores obtained through holistic scoring by trained human raters. For example, this practice was used until 2008 in the case of GRE^®General Analytical Writing and until 2009 in the case of TOEFL^® iBT Writing. With use of natural language processing, it is possible to obtain additional information concerning item responses from computer programs such as e‐rater^®. In addition, available information relevant to examinee performance may include scores on related tests. We suggest application of standard results from classical test theory to the available data to obtain best linear predictors of true traditional test scores. In performing such analysis, we require estimation of variances and covariances of measurement errors, a task which can be quite difficult in the case of tests with limited numbers of items and with multiple measurements per item. As a consequence, a new estimation method is suggested based on samples of examinees who have taken an assessment more than once. Such samples are typically not random samples of the general population of examinees, so that we apply statistical adjustment methods to obtain the needed estimated variances and covariances of measurement errors. To examine practical implications of the suggested methods of analysis, applications are made to GRE General Analytical Writing and TOEFL iBT Writing. Results obtained indicate that substantial improvements are possible both in terms of reliability of scoring and in terms of assessment reliability.     

Children’s scale errors and object processing: Early evidence for cross-cultural differences

Scale errors are observed when young children make mistakes by attempting to put their bodies into miniature versions of everyday objects. Such errors have been argued to arise from children’s insufficient integration of size into their object representations. The current study investigated whether Japanese and UK children’s (18–24 months old, N = 80) visual exploration in a categorization task related to their scale error production. UK children who showed greater local processing made more scale errors, whereas Japanese children, who overall showed greater global processing, showed no such relationship. These results raise the possibility that children’s suppression of scale errors emerges not from attention to size per se, but from a critical integration of global (i.e., size) and local (i.e., object features) information during object processing, and provide evidence that this mechanism differs cross-culturally.     

Tactile order memory: evidence for sequence learning phenomena found with other stimulus types

We examine serial order memory for sequences of tactile stimuli and investigate whether established characteristics of order memory, namely serial position effects, error distributions, and Hebb repetition learning, are observed with tactile memory. Visually obscured participants received six tactile stimulations: one to each of six fingers. At test, participants lifted the six fingers in the order of stimulation. For every third trial participants received the same order of stimulation (i.e. the Hebb sequence). Serial recall accuracy produced the canonical bowed serial position function found for immediate serial recall. In addition, recall for the Hebb sequence improved relative to the filler sequences, providing the first demonstration of the Hebb repetition effect with tactile stimuli. Analysis of errors revealed close similarities to that reported with verbal and visual stimuli. This experiment further generalises established features of order memory to tactile memory, supporting the utilisation of an analogous order memory mechanism across stimuli.