Cognitive Evaluation of Machine Learning Agents

Advances in applying statistical Machine Learning (ML) led to several claims of human-level or near-human performance in tasks such as image classification & speech recognition. Such claims are unscientific primarily for two reasons, (1) They incorrectly enforce the notion that task-specific performance can be treated as manifestation of General Intelligence and (2) They are not verifiable as currently there is no set benchmark for measuring human-like cognition in a machine learning agent. Moreover, ML agent’s performance is influenced by knowledge ingested in it by its human designers. Therefore, agent’s performance may not necessarily reflect its true cognition. In this paper, we propose a framework that draws parallels from human cognition to measure machine’s cognition. Human cognitive learning is quite well studied in developmental psychology with frameworks and metrics in place to measure actual learning. To either believe or refute the claims of human-level performance of machine learning agent, we need scientific methodology to measure its cognition. Our framework formalizes incremental implementation of human-like cognitive processes in ML agents with an implicit goal to measure it. The framework offers guiding principles for measuring, (1) Task-specific machine cognition and (2) General machine cognition that spans across tasks. The framework also provides guidelines for building domain-specific task taxonomies to cognitively profile tasks. We demonstrate application of the framework with a case study where two ML agents that perform Vision and NLP tasks are cognitively evaluated.     

A two-part evaluation approach for measuring the usability and user experience of an Augmented Reality-based assistance system to support the temporal coordination of spatially dispersed teams

One key predictor of team performance and productivity is the teams’ ability to coordinate its subtasks as precisely as possible over time. Thereby, the quality of the temporal coordination in teams is highly dependent on several cognitive team skills, like for instance the ability to build a precise and stable mental model of the situation (shared situation awareness) and of the team task process (task state awareness/ TSA). To support these cognitive skills in teams, various methods and trainings already exist. However, considering teams that work de-located, the prerequisites and therefore the requirements for team supportive methods change. This work presents the results of an empirical usability and user experience (UX) study of an interface for an Augmented Reality-based assistance system for spatially dispersed teams, called Ambient Awareness Tool (AAT). The AAT interface consists of graphical information about the teamwork process and aims at enhancing the TSA of the team, thereby supporting its temporal coordination. Within the framework of a participatory design process we conducted a two-part expert-user-study, comprising a laboratory experiment for the evaluation of the interface usability and a follow-up online survey to additionally investigate the UX. By means of the usability scores we first inferred three interface configuration clusters. A subsequent UX evaluation then allowed us to designate the configuration cluster with the highest usability and UX scores. As a result, we defined one interface configuration that will be investigated concerning its actual impact on the temporal coordination of spatially dispersed teams in a further study.     

An expanded model for perceptual visual single object recognition system using expectation priming following neuroscientific evidence

Under numerous circumstances, humans recognize visual objects in their environment with remarkable response times and accuracy. Existing artificial visual object recognition systems have not yet surpassed human vision, especially in its universality of application. We argue that modeling the recognition process in an exclusive feedforward manner hinders those systems’ performance. To bridge that performance gap between them and human vision, we present a brief review of neuroscientific data, which suggests that considering an agent’s internal influences (from cognitive systems that peripherally interact with visual-perceptual processes) recognition can be improved. Then, we propose a model for visual object recognition which uses these systems’ information, such as affection, for generating expectation to prime the object recognition system, thus reducing its execution times. Later, an implementation of the model is described. Finally, we present and discuss an experiment and its results.     

Learning algorithm for an intelligent decision making system based on multi-agent neurocognitive architectures

The paper presents the formalism of an intelligent decision-making system based on multi-agent neurocognitive architectures, which has an architectural similarity to the human brain. An invariant of the organizational and functional structure of the intellectual decision-making process based on the multi-agent neurocognitive architecture is developed. An algorithm for teaching intelligent decision-making systems based on the self-organization of the invariant of multi-agent neurocognitive architectures is presented. Using this algorithm, an intelligent agent was trained and the architecture of the learning process was built on the basis of an invariant of neurocognitive architecture. Further research is related to training an intelligent agent in more complex behavior and expanding the capabilities of an intelligent decision-making system based on multi-agent neurocognitive architectures.     

Declarative working memory: A bio-inspired cognitive architecture proposal

Memory is considered one of the most important functions since it allows us to code, store and retrieve knowledge. These qualities make it an indispensable function for a virtual creature. In general, memory can be classified based on the durability of the stored data in working memory and long-term memory. Working memory refers to the capacity to maintain temporarily a limited amount of information in mind, which can then be used to support various abilities, including learning, reasoning, planning and decision-making. Unlike short-term memory, working memory is not only a storage site, but it is also a framework of interacting processes that involve the temporary storage and manipulation of information in the service of performing complex cognitive activities. Declarative memory is a type of long-term memory related with the storage of facts and events. This research focuses on the development of a cognitive architecture for the type of working memory that maintains and manipulates declarative information. The construction of the model was grounded in theoretical evidence taken from cognitive sciences such as neuroscience and psychology, which gave us the components and their processes. The model was evaluated through a case study that covers the encoding, storing, and retrieval stages. Our hypothesis is that a virtual creature endowed with our working memory model will provide faster access to the information needed for the ongoing task. Therefore, it improves the planning and decision-making processes.     

A study on understanding cognitive states through gait analysis

In this work, we attempted to find out the relationship between different gait patterns and their corresponding cognitive states by using different statistical and machine learning approaches. This paper strongly focusses on the simulations followed by implementation of the proposed cognitive states i.e. (i) EmotionOriented State (EOS) (ii) Thinking Oriented State (TOS) (iii) Memory Oriented State(MOS) (iv) Simple Regular Oriented State (SROS). A novel approach was implemented by creating different environmental contexts for different gaits in our lab. An experimental method was performed to isolate movement artifact using Independent Component Analysis from recorded EEG(Electroencephalogram) signals. Measurement of joint angles from joint positions captured using Kinect V2 sensors was done with the help of OpenSim software. The relationship between different gaits and mental states was established using Pearsons Correlation Coefficient, ANOVA(Analysis of variance) and SVM(Support Vector Machine) classifier respectively. A strong relationship was found between them. The SVM classifier for the EOS and the non-EOS states based on joint angles inferred an accuracy of 81.08%. The ROC Curve for SVM classification depicted an AUC (area under the curve) of 0.9724.     

多级评分认知诊断模型述评

认知诊断评估旨在探讨个体内部的知识掌握结构,并提供关于学生优缺点的详细诊断信息,以促进个体的全面发展。当前研究者已开发了大量0-1评分的认知诊断模型,但对于多级评分认知诊断模型的研究还比较少。本文对已有的多级评分认知诊断模型进行了归纳,介绍了模型的假设,计量特征以及适用范围,为实际应用者和研究者在多级评分认知诊断模型的比较和选用上提供借鉴和参考。最后,对未来关于多级评分诊断模型的研究方向进行了展望。     

寻找镜中的自我：精神分裂症患者的视觉观点采择

精神分裂症患者的视觉观点采择影响其必要的社交活动和生活质量。本文通过对精神分裂症患者视觉观点采择的研究进行梳理,总结出相关的心理特点：精神分裂症患者的视觉观点采择过程受损,表现出自我中心性和他人中心性。在神经生理上表现出与视觉观点采择相关的脑区激活减弱。可能的原因是患者认知功能中反应抑制能力和心理旋转能力的受损所致,建议采用认知干预疗法可能会改善患者的视觉观点采择能力。目前的研究论证了精神分裂症患者视觉观点采择过程的损伤,对于不同水平的视觉观点采择下精神分裂症患者的认知模式还未有明确的定论,未来的研究可以从该角度入手,更加明确精神分裂症患者不同水平视觉观点采择的特点和神经生理活动。     

Creation and humanity: Veli-Matti Kärkäinen's answer to the ultimate question

This article focuses on the third volume of Vel-Matti Kärkkäinen's systematic theology, titled Creation and Humanity. After a brief overview of the volume the article offers some clarification on the tenth chapter, especially on its interaction with paleontology. After that, two constructive observations follow. The first is related to Kärkkäinen's notion of the embeddedness of human nature, and the second is to further employ systems thinking in his constructive work on creation and human evolution.     

Exploring Processes in Day-to-Day Parent–Adolescent Conflict and Angry Mood: Evidence for Circular Causality

Parent–adolescent conflict has been studied both as a precursor of long-term macrolevel developmental risks and as an outcome of microlevel, moment-to-moment interaction patterns. However, the family-level processes underlying the maintenance or regulation of conflict in daily life are largely overlooked. A meso-level understanding of parent–adolescent conflict offers important practical insights that have direct implications for interventions. The present study explores day-to-day reciprocal processes and carryover in parents’ and adolescents’ experiences of anger and conflict. Daily diary data provided by parent–adolescent dyads (N = 151) from two-caregiver households (adolescents: 61.59% female, mean age = 14.60 years) over 21 days were examined using a multivariate Poisson multilevel model to evaluate the circular causality principle in parents’ and adolescents’ daily conflict and anger. Findings offer empirical support for the theory, suggesting that parents’ and adolescents’ anger and conflict exist together in a feedback loop wherein conflict is both a consequence of past anger and also an antecedent of future anger, both within and across persons. Increased understanding of the daily interaction patterns and maintenance of parent–adolescent conflict can guide more informed, targeted, and well-timed interventions intended to ameliorate the consequences of problematic parent–adolescent conflict sequences.