Scheduling distributed resources in heterogeneous private clouds

George Kesidis, Yuquan Shan, Aman Jain, Bhuvan Urgaonkar, Jalal Khamse-Ashari, Ioannis Lambadaris

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

We first consider the static problem of allocating resources to (i.e., scheduling) multiple distributed application frameworks, possibly with different priorities and server preferences, in a private cloud with heterogeneous servers. Several fair scheduling mechanisms have been proposed for this purpose. We extend prior results on max-min fair (MMF) and proportional fair (PF) scheduling to this constrained multiresource and multiserver case for generic fair scheduling criteria. The task efficiencies (a metric related to proportional fairness) of max-min fair allocations found by progressive filling are compared by illustrative examples. In the second part of this paper, we consider the online problem (with framework churn) by implementing variants of these schedulers in Apache Mesos using progressive filling to dynamically approximate max-min fair allocations. We evaluate the implemented schedulers in terms of overall execution time of realistic distributed Spark workloads. Our experiments show that resource efficiency is improved and execution times are reduced when the scheduler is 'server specific' or when it leverages characterized required resources of the workloads (when known).

Original languageEnglish (US)
Title of host publicationProceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages102-108
Number of pages7
ISBN (Electronic)9781538668863
DOIs
StatePublished - Nov 7 2018
Event26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018 - Milwaukee, United States
Duration: Sep 25 2018Sep 28 2018

Publication series

NameProceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018

Other

Other26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018
CountryUnited States
CityMilwaukee
Period9/25/189/28/18

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Modeling and Simulation

Cite this

Kesidis, G., Shan, Y., Jain, A., Urgaonkar, B., Khamse-Ashari, J., & Lambadaris, I. (2018). Scheduling distributed resources in heterogeneous private clouds. In Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018 (pp. 102-108). [8526876] (Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MASCOTS.2018.00018