Adversarial policy training against deep reinforcement learning

Xian Wu, Wenbo Guo, Hua Wei, Xinyu Xing

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Reinforcement learning is a set of goal-oriented learning algorithms, through which an agent could learn to behave in an environment, by performing certain actions and observing the reward which it gets from those actions. Integrated with deep neural networks, it becomes deep reinforcement learning, a new paradigm of learning methods. Recently, deep reinforcement learning demonstrates great potential in many applications such as playing video games, mastering GO competition, and even performing autonomous pilot. However, coming together with these great successes is adversarial attacks, in which an adversary could force a well-trained agent to behave abnormally by tampering the input to the agent's policy network or training an adversarial agent to exploit the weakness of the victim. In this work, we show existing adversarial attacks against reinforcement learning either work in an impractical setting or perform less effectively when being launched in a two-agent competitive game. Motivated by this, we propose a new method to train adversarial agents. Technically speaking, our approach extends the Proximal Policy Optimization (PPO) algorithm and then utilizes an explainable AI technique to guide an attacker to train an adversarial agent. In comparison with the adversarial agent trained by the state-of-the-art technique, we show that our adversarial agent exhibits a much stronger capability in exploiting the weakness of victim agents. Besides, we demonstrate that our adversarial attack introduces less variation in the training process and exhibits less sensitivity to the selection of initial states.

Original languageEnglish (US)
Title of host publicationProceedings of the 30th USENIX Security Symposium
PublisherUSENIX Association
Pages1883-1900
Number of pages18
ISBN (Electronic)9781939133243
StatePublished - 2021
Event30th USENIX Security Symposium, USENIX Security 2021 - Virtual, Online
Duration: Aug 11 2021Aug 13 2021

Publication series

NameProceedings of the 30th USENIX Security Symposium

Conference

Conference30th USENIX Security Symposium, USENIX Security 2021
CityVirtual, Online
Period8/11/218/13/21

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'Adversarial policy training against deep reinforcement learning'. Together they form a unique fingerprint.

Cite this