Learning contextual bandits in a non-stationary environment

Qingyun Wu, Naveen Iyer, Hongning Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Citations (SciVal)

Abstract

Multi-armed bandit algorithms have become a reference solution for handling the explore/exploit dilemma in recommender systems, and many other important real-world problems, such as display advertisement. However, such algorithms usually assume a stationary reward distribution, which hardly holds in practice as users' preferences are dynamic. This inevitably costs a recommender system consistent suboptimal performance. In this paper, we consider the situation where the underlying distribution of reward remains unchanged over (possibly short) epochs and shifts at unknown time instants. In accordance, we propose a contextual bandit algorithm that detects possible changes of environment based on its reward estimation confidence and updates its arm selection strategy respectively. Rigorous upper regret bound analysis of the proposed algorithm demonstrates its learning effectiveness in such a non-trivial environment. Extensive empirical evaluations on both synthetic and real-world datasets for recommendation confirm its practical utility in a changing environment.

Original languageEnglish (US)
Title of host publication41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018
PublisherAssociation for Computing Machinery, Inc
Pages495-504
Number of pages10
ISBN (Electronic)9781450356572
DOIs
StatePublished - Jun 27 2018
Event41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018 - Ann Arbor, United States
Duration: Jul 8 2018Jul 12 2018

Publication series

Name41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018

Other

Other41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018
Country/TerritoryUnited States
CityAnn Arbor
Period7/8/187/12/18

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design
  • Information Systems

Cite this