Deep reinforcement learning for transfer of control policies

James D. Cunningham, Simon W. Miller, Michael A. Yukish, Timothy W. Simpson, Conrad S. Tucker

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a form-aware reinforcement learning (RL) method to extend control knowledge from one design form to another, without losing the ability to control the original design. A major challenge in developing control knowledge is the creation of generalized control policies across designs of varying form. Our presented RL policy is form-aware because in addition to receiving dynamic state information about the environment, it also receives states that encode information about the form of the design that is being controlled. In this paper, we investigate the impact of this mixed state space on transfer learning. We present a transfer learning method for extending a control policy to a different design form, while continuing to expose the agent to the original design during the training of the new design. To demonstrate this concept, we present a case study of a multi-rotor aircraft simulation, wherein the designated task is to achieve a stable hover. We show that by introducing form states, an RL agent is able to learn a control policy to achieve the hovering task with both a four rotor and three rotor design at once, whereas without the form states it can only hover with the four rotor design. We also benchmark our method against a test case that removes the transfer learning component, as well as a test case that removes the continued exposure to the original design to show the value of each of these components. We find that form states, transfer learning, and parallel learning all contribute to a more robust control policy for the new design, and that parallel learning is especially important for maintaining control knowledge of the original design.

Original languageEnglish (US)
Title of host publication45th Design Automation Conference
PublisherAmerican Society of Mechanical Engineers (ASME)
ISBN (Electronic)9780791859186
DOIs
StatePublished - Jan 1 2019
EventASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC-CIE 2019 - Anaheim, United States
Duration: Aug 18 2019Aug 21 2019

Publication series

NameProceedings of the ASME Design Engineering Technical Conference
Volume2A-2019

Conference

ConferenceASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC-CIE 2019
CountryUnited States
CityAnaheim
Period8/18/198/21/19

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Mechanical Engineering
  • Computer Graphics and Computer-Aided Design
  • Computer Science Applications
  • Modeling and Simulation

Cite this

Cunningham, J. D., Miller, S. W., Yukish, M. A., Simpson, T. W., & Tucker, C. S. (2019). Deep reinforcement learning for transfer of control policies. In 45th Design Automation Conference (Proceedings of the ASME Design Engineering Technical Conference; Vol. 2A-2019). American Society of Mechanical Engineers (ASME). https://doi.org/10.1115/DETC2019-97689