Managing engineering systems with large state and action spaces through deep reinforcement learning

C. P. Andriotis, K. G. Papakonstantinou

Research output: Contribution to journalArticle

7 Scopus citations

Abstract

Decision-making for engineering systems management can be efficiently formulated using Markov Decision Processes (MDPs) or Partially Observable MDPs (POMDPs). Typical MDP/POMDP solution procedures utilize offline knowledge about the environment and provide detailed policies for relatively small systems with tractable state and action spaces. However, in large multi-component systems the dimensions of these spaces easily explode, as system states and actions scale exponentially with the number of components, whereas environment dynamics are difficult to be described explicitly for the entire system and may, often, only be accessible through computationally expensive numerical simulators. In this work, to address these issues, an integrated Deep Reinforcement Learning (DRL) framework is introduced. The Deep Centralized Multi-agent Actor Critic (DCMAC) is developed, an off-policy actor-critic DRL algorithm that directly probes the state/belief space of the underlying MDP/POMDP, providing efficient life-cycle policies for large multi-component systems operating in high-dimensional spaces. Apart from deep network approximators parametrizing complex functions with vast state spaces, DCMAC also adopts a factorized representation of the system actions, thus being able to designate individualized component- and subsystem-level decisions, while maintaining a centralized value function for the entire system. DCMAC compares well against Deep Q-Network and exact solutions, where applicable, and outperforms optimized baseline policies that are based, on time-based, condition-based, and periodic inspection and maintenance considerations.

Original languageEnglish (US)
Article number106483
JournalReliability Engineering and System Safety
Volume191
DOIs
StatePublished - Nov 2019

All Science Journal Classification (ASJC) codes

  • Safety, Risk, Reliability and Quality
  • Industrial and Manufacturing Engineering

Fingerprint Dive into the research topics of 'Managing engineering systems with large state and action spaces through deep reinforcement learning'. Together they form a unique fingerprint.

  • Cite this