Context matters: Using reinforcement learning to develop human-readable, state-dependent outbreak response policies

W. J.M. Probert, S. Lakkur, C. J. Fonnesbeck, Katriona Shea, M. C. Runge, M. J. Tildesley, Matthew Joseph Ferrari

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

The number of all possible epidemics of a given infectious disease that could occur on a given landscape is large for systems of real-world complexity. Furthermore, there is no guarantee that the control actions that are optimal, on average, over all possible epidemics are also best for each possible epidemic. Reinforcement learning (RL) and Monte Carlo control have been used to develop machine-readable context-dependent solutions for complex problems with many possible realizations ranging from video-games to the game of Go. RL could be a valuable tool to generate context-dependent policies for outbreak response, though translating the resulting policies into simple rules that can be read and interpreted by human decision-makers remains a challenge. Here we illustrate the application of RL to the development of context-dependent outbreak response policies to minimize outbreaks of foot-and-mouth disease. We show that control based on the resulting context-dependent policies, which adapt interventions to the specific outbreak, result in smaller outbreaks than static policies. We further illustrate two approaches for translating the complex machine-readable policies into simple heuristics that can be evaluated by human decision-makers. This article is part of the theme issue 'Modelling infectious disease outbreaks in humans, animals and plants: epidemic forecasting and control'. This theme issue is linked with the earlier issue 'Modelling infectious disease outbreaks in humans, animals and plants: approaches and important themes'.

Original languageEnglish (US)
Article number20180277
JournalPhilosophical Transactions of the Royal Society B: Biological Sciences
Volume374
Issue number1776
DOIs
StatePublished - Jan 1 2019

Fingerprint

Reinforcement learning
Disease Outbreaks
learning
Learning
infectious diseases
Animals
Foot-and-Mouth Disease
Video Games
foot-and-mouth disease
Reinforcement (Psychology)
animals
Communicable Diseases

All Science Journal Classification (ASJC) codes

  • Biochemistry, Genetics and Molecular Biology(all)
  • Agricultural and Biological Sciences(all)

Cite this

@article{389c1efad586429c80e17fb7af407104,
title = "Context matters: Using reinforcement learning to develop human-readable, state-dependent outbreak response policies",
abstract = "The number of all possible epidemics of a given infectious disease that could occur on a given landscape is large for systems of real-world complexity. Furthermore, there is no guarantee that the control actions that are optimal, on average, over all possible epidemics are also best for each possible epidemic. Reinforcement learning (RL) and Monte Carlo control have been used to develop machine-readable context-dependent solutions for complex problems with many possible realizations ranging from video-games to the game of Go. RL could be a valuable tool to generate context-dependent policies for outbreak response, though translating the resulting policies into simple rules that can be read and interpreted by human decision-makers remains a challenge. Here we illustrate the application of RL to the development of context-dependent outbreak response policies to minimize outbreaks of foot-and-mouth disease. We show that control based on the resulting context-dependent policies, which adapt interventions to the specific outbreak, result in smaller outbreaks than static policies. We further illustrate two approaches for translating the complex machine-readable policies into simple heuristics that can be evaluated by human decision-makers. This article is part of the theme issue 'Modelling infectious disease outbreaks in humans, animals and plants: epidemic forecasting and control'. This theme issue is linked with the earlier issue 'Modelling infectious disease outbreaks in humans, animals and plants: approaches and important themes'.",
author = "Probert, {W. J.M.} and S. Lakkur and Fonnesbeck, {C. J.} and Katriona Shea and Runge, {M. C.} and Tildesley, {M. J.} and Ferrari, {Matthew Joseph}",
year = "2019",
month = "1",
day = "1",
doi = "10.1098/rstb.2018.0277",
language = "English (US)",
volume = "374",
journal = "Philosophical Transactions of the Royal Society B: Biological Sciences",
issn = "0962-8436",
publisher = "Royal Society of London",
number = "1776",

}

Context matters : Using reinforcement learning to develop human-readable, state-dependent outbreak response policies. / Probert, W. J.M.; Lakkur, S.; Fonnesbeck, C. J.; Shea, Katriona; Runge, M. C.; Tildesley, M. J.; Ferrari, Matthew Joseph.

In: Philosophical Transactions of the Royal Society B: Biological Sciences, Vol. 374, No. 1776, 20180277, 01.01.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Context matters

T2 - Using reinforcement learning to develop human-readable, state-dependent outbreak response policies

AU - Probert, W. J.M.

AU - Lakkur, S.

AU - Fonnesbeck, C. J.

AU - Shea, Katriona

AU - Runge, M. C.

AU - Tildesley, M. J.

AU - Ferrari, Matthew Joseph

PY - 2019/1/1

Y1 - 2019/1/1

N2 - The number of all possible epidemics of a given infectious disease that could occur on a given landscape is large for systems of real-world complexity. Furthermore, there is no guarantee that the control actions that are optimal, on average, over all possible epidemics are also best for each possible epidemic. Reinforcement learning (RL) and Monte Carlo control have been used to develop machine-readable context-dependent solutions for complex problems with many possible realizations ranging from video-games to the game of Go. RL could be a valuable tool to generate context-dependent policies for outbreak response, though translating the resulting policies into simple rules that can be read and interpreted by human decision-makers remains a challenge. Here we illustrate the application of RL to the development of context-dependent outbreak response policies to minimize outbreaks of foot-and-mouth disease. We show that control based on the resulting context-dependent policies, which adapt interventions to the specific outbreak, result in smaller outbreaks than static policies. We further illustrate two approaches for translating the complex machine-readable policies into simple heuristics that can be evaluated by human decision-makers. This article is part of the theme issue 'Modelling infectious disease outbreaks in humans, animals and plants: epidemic forecasting and control'. This theme issue is linked with the earlier issue 'Modelling infectious disease outbreaks in humans, animals and plants: approaches and important themes'.

AB - The number of all possible epidemics of a given infectious disease that could occur on a given landscape is large for systems of real-world complexity. Furthermore, there is no guarantee that the control actions that are optimal, on average, over all possible epidemics are also best for each possible epidemic. Reinforcement learning (RL) and Monte Carlo control have been used to develop machine-readable context-dependent solutions for complex problems with many possible realizations ranging from video-games to the game of Go. RL could be a valuable tool to generate context-dependent policies for outbreak response, though translating the resulting policies into simple rules that can be read and interpreted by human decision-makers remains a challenge. Here we illustrate the application of RL to the development of context-dependent outbreak response policies to minimize outbreaks of foot-and-mouth disease. We show that control based on the resulting context-dependent policies, which adapt interventions to the specific outbreak, result in smaller outbreaks than static policies. We further illustrate two approaches for translating the complex machine-readable policies into simple heuristics that can be evaluated by human decision-makers. This article is part of the theme issue 'Modelling infectious disease outbreaks in humans, animals and plants: epidemic forecasting and control'. This theme issue is linked with the earlier issue 'Modelling infectious disease outbreaks in humans, animals and plants: approaches and important themes'.

UR - http://www.scopus.com/inward/record.url?scp=85065400140&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065400140&partnerID=8YFLogxK

U2 - 10.1098/rstb.2018.0277

DO - 10.1098/rstb.2018.0277

M3 - Article

C2 - 31104604

AN - SCOPUS:85065400140

VL - 374

JO - Philosophical Transactions of the Royal Society B: Biological Sciences

JF - Philosophical Transactions of the Royal Society B: Biological Sciences

SN - 0962-8436

IS - 1776

M1 - 20180277

ER -