Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective

Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that prevent instrumental goals for two different types of reward tampering (reward function tampering and RF-input tampering). Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations.

     

Security of multi-agent systems: A case study on comparison shopping

The multi-agent-systems paradigm is becoming more and more popular as a basis for realizing net-based solutions. This development is accompanied by an increasing relevance of security issues. For instance, the potential loss of privacy and other assets is a major concern for, both merchants and customers, in Internet-based commerce and, without being properly addressed, such very legitimate concerns hamper the growth of e-commerce.This article uses a comparison-shopping scenario to introduce a general methodology for formally verifying the security of multi-agent systems. Following the approach of possibilistic information flow security, the flow of information between and within agents is restricted in order to ensure that secrets will not be disclosed to unauthorized meddlers. The security requirements for the overall system are then decomposed into requirements for the individual agents that can be verified independently from each other. Exploiting the modular structure of a multi-agent system considerably reduces the complexity of the overall security analysis. The techniques for decomposing security requirements, for verifying individual agents, and for deriving global security guarantees for the entire system from locally verified properties are all generic in the sense that they apply also to many other systems and security requirements than the ones that appear in the example scenario.     

Resolving conflicting social categories: the role of age-related executive ability

We investigated the effect of age-related decline in executive ability on the application of emergent features to incongruent social category conjunctions (e.g., male midwife). When forming an impression of an incongruent conjunction, older adults used more emergent attributes (attributes associated exclusively with the category conjunction and not the constituents), relative to younger adults. Moreover, this relationship was mediated by a reduction in inhibitory ability (measured using a Stroop task) and processing speed (measured using a Digit Symbol Substitution Test, DSST). These findings are consistent with the notion that executive ability is pivotal in understanding social functioning in older adults. We discuss the implications of these findings for the continuing development of models outlining the processes and stages involved in perceiving social category conjunctions.     

Aiming at a far target under different viewing conditions: visual control in basketball jump shooting

Most research on visual search in aiming at far targets assumes preprogrammed motor control implying that relevant visual information is detected prior to the final shooting or throwing movements. Eye movement data indirectly support this claim for stationary tasks. Using the basketball jump shot as experimental task we investigated whether in dynamic tasks in which the target can be seen until ball release, continuous, instead of preprogrammed, motor control is possible. We tested this with the temporal occlusion paradigm: 10 expert shooters took shots under four viewing conditions, namely, no vision, full vision, early vision (vision occluded during the final +/-350 ms before ball release), and late vision (vision occluded until these final +/-350 ms). Late-vision shooting appeared to be as good as shooting with full vision while early-vision performance was severely impaired. The results imply that the final shooting movements were controlled by continuous detection and use of visual information until ball release. The data further suggest that visual and movement control of aiming at a far target develop in close correspondence with the style of execution.     

The formation of novel social category conjunctions in working memory: A possible role for the episodic buffer?

Recent research (e.g., Hutter, Crisp, Humphreys, Waters, &; Moffit; Siebler) has confirmed that combining novel social categories involves two stages (e.g., Hampton; Hastie, Schroeder, &; Weber). Furthermore, it is also evident that following stage 1 (constituent additivity), the second stage in these models involves cognitively effortful complex reasoning. However, while current theory and research has addressed how category conjunctions are initially represented to some degree, it is not clear precisely where we first combine or bind existing social constituent categories. For example, how and where do we compose and temporarily store a coherent representation of an individual who shares membership of “female” and “blacksmith” categories? In this article, we consider how the revised multi-component model of working memory (Baddeley) can assist in resolving the representational limitations in the extant two-stage theoretical models. This is a new approach to understanding how novel conjunctions form new bound “composite” representations.     

Math weaknesses in survivors of acute lymphoblastic leukemia compared to healthy children.

Difficulties in math are the most frequently reported area of academic deficit in survivors of acute lymphoblastic leukemia (ALL) and the most frequent academic complaint among parents of ALL survivors. However, previous studies that included measures of math skills have been limited by the use of only a single measure of math skills, most often a measure of written calculations, without any assessment of math reasoning or math application skills. Further, the nature of these math difficulties has not been adequately investigated. The purpose of this study was to examine the performance of ALL survivors using multiple measures of math skills. Performance was compared to a group of healthy controls matched for age and sex as well as to normative levels. Other measures of neuropsychological function were also administered, and the relationships between these measures and the math measures were explored. Converging evidence for math difficulties in ALL survivors compared to healthy controls and normative levels was found. While ALL survivors generally performed within the average range on measures of math skills, math performance was mostly related to memory function and dominant-hand psychomotor speed. By contrast, math performance of healthy children was mostly related to basic reading skills and visual-motor integration. These findings shed light on the nature of math difficulties in ALL survivors and have implications for intervention.