Improving multi-job mapreduce scheduling in an opportunistic environment

Yuting Ji, Lang Tong, Ting He, Jian Tan, Kang Won Lee, Li Zhang

Research output: Contribution to journalConference article

7 Scopus citations

Abstract

As a state-of-the-art programming model for big data analytics, MapReduce is well suited for parallel processing of large data sets in opportunistic environments. Existing research on MapReduce in opportunistic environment has focused on improving single job performance, the issue of fairness that is critical in the more dominant scenario of multiple concurrent jobs remains unexplored. We address this problem by proposing an opportunistic fair scheduling algorithm, which extends the broadly adopted Fair Scheduler to an environment where nodes are intermittently available with possibly different availability patterns. The proposed scheduler maintains statistics specific to the opportunistic environment, e.g., node availability rates and pairwise availability correlations, and utilizes this information in scheduling decisions to improve fairness. Using a Hadoop-based implementation, we compare our scheduler with the current Hadoop Fair Scheduler on representative benchmarks. Our experiments verify that our scheduler can significantly reduce the variability in job completion times.

Original languageEnglish (US)
Article number6676672
Pages (from-to)9-16
Number of pages8
JournalIEEE International Conference on Cloud Computing, CLOUD
DOIs
StatePublished - Dec 1 2013
Event2013 IEEE 6th International Conference on Cloud Computing, CLOUD 2013 - Santa Clara, CA, United States
Duration: Jun 27 2013Jul 2 2013

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Information Systems
  • Software

Cite this