MROrchestrator

A fine-grained resource orchestration framework for MapReduce clusters

Bikash Sharma, Ramya Prabhakar, Seung Hwan Lim, Mahmut Kandemir, Chitaranjan Das

Research output: Chapter in Book/Report/Conference proceedingConference contribution

31 Citations (Scopus)

Abstract

Efficient resource management in data centers and clouds running large distributed data processing frameworks like MapReduce is crucial for enhancing the performance of hosted applications and increasing resource utilization. However, existing resource scheduling schemes in Hadoop MapReduce allocate resources at the granularity of fixed-size, static portions of nodes, called slots. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixed-size slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across nodes prevents dynamic slot reconfiguration, and leads to resource contention. Motivated by this, we propose MROrchestrator, a MapReduce resource Orchestrator framework, which can dynamically identify resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with a suite of representative MapReduce benchmarks demonstrate up to 38% reduction in job completion times, and up to 25% increase in resource utilization. We further demonstrate the performance boost in existing resource managers like NGM and Mesos, when augmented with MROrchestrator.

Original languageEnglish (US)
Title of host publicationProceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012
Pages1-8
Number of pages8
DOIs
StatePublished - Oct 2 2012
Event2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012 - Honolulu, HI, United States
Duration: Jun 24 2012Jun 29 2012

Publication series

NameProceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012

Other

Other2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012
CountryUnited States
CityHonolulu, HI
Period6/24/126/29/12

Fingerprint

Resource allocation
Managers
Scheduling

All Science Journal Classification (ASJC) codes

  • Software

Cite this

Sharma, B., Prabhakar, R., Lim, S. H., Kandemir, M., & Das, C. (2012). MROrchestrator: A fine-grained resource orchestration framework for MapReduce clusters. In Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012 (pp. 1-8). [6253482] (Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012). https://doi.org/10.1109/CLOUD.2012.37
Sharma, Bikash ; Prabhakar, Ramya ; Lim, Seung Hwan ; Kandemir, Mahmut ; Das, Chitaranjan. / MROrchestrator : A fine-grained resource orchestration framework for MapReduce clusters. Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012. 2012. pp. 1-8 (Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012).
@inproceedings{75eb649dc1de431390d3f1b29c8661d1,
title = "MROrchestrator: A fine-grained resource orchestration framework for MapReduce clusters",
abstract = "Efficient resource management in data centers and clouds running large distributed data processing frameworks like MapReduce is crucial for enhancing the performance of hosted applications and increasing resource utilization. However, existing resource scheduling schemes in Hadoop MapReduce allocate resources at the granularity of fixed-size, static portions of nodes, called slots. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixed-size slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across nodes prevents dynamic slot reconfiguration, and leads to resource contention. Motivated by this, we propose MROrchestrator, a MapReduce resource Orchestrator framework, which can dynamically identify resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with a suite of representative MapReduce benchmarks demonstrate up to 38{\%} reduction in job completion times, and up to 25{\%} increase in resource utilization. We further demonstrate the performance boost in existing resource managers like NGM and Mesos, when augmented with MROrchestrator.",
author = "Bikash Sharma and Ramya Prabhakar and Lim, {Seung Hwan} and Mahmut Kandemir and Chitaranjan Das",
year = "2012",
month = "10",
day = "2",
doi = "10.1109/CLOUD.2012.37",
language = "English (US)",
isbn = "9780769547558",
series = "Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012",
pages = "1--8",
booktitle = "Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012",

}

Sharma, B, Prabhakar, R, Lim, SH, Kandemir, M & Das, C 2012, MROrchestrator: A fine-grained resource orchestration framework for MapReduce clusters. in Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012., 6253482, Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012, pp. 1-8, 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012, Honolulu, HI, United States, 6/24/12. https://doi.org/10.1109/CLOUD.2012.37

MROrchestrator : A fine-grained resource orchestration framework for MapReduce clusters. / Sharma, Bikash; Prabhakar, Ramya; Lim, Seung Hwan; Kandemir, Mahmut; Das, Chitaranjan.

Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012. 2012. p. 1-8 6253482 (Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - MROrchestrator

T2 - A fine-grained resource orchestration framework for MapReduce clusters

AU - Sharma, Bikash

AU - Prabhakar, Ramya

AU - Lim, Seung Hwan

AU - Kandemir, Mahmut

AU - Das, Chitaranjan

PY - 2012/10/2

Y1 - 2012/10/2

N2 - Efficient resource management in data centers and clouds running large distributed data processing frameworks like MapReduce is crucial for enhancing the performance of hosted applications and increasing resource utilization. However, existing resource scheduling schemes in Hadoop MapReduce allocate resources at the granularity of fixed-size, static portions of nodes, called slots. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixed-size slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across nodes prevents dynamic slot reconfiguration, and leads to resource contention. Motivated by this, we propose MROrchestrator, a MapReduce resource Orchestrator framework, which can dynamically identify resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with a suite of representative MapReduce benchmarks demonstrate up to 38% reduction in job completion times, and up to 25% increase in resource utilization. We further demonstrate the performance boost in existing resource managers like NGM and Mesos, when augmented with MROrchestrator.

AB - Efficient resource management in data centers and clouds running large distributed data processing frameworks like MapReduce is crucial for enhancing the performance of hosted applications and increasing resource utilization. However, existing resource scheduling schemes in Hadoop MapReduce allocate resources at the granularity of fixed-size, static portions of nodes, called slots. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixed-size slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across nodes prevents dynamic slot reconfiguration, and leads to resource contention. Motivated by this, we propose MROrchestrator, a MapReduce resource Orchestrator framework, which can dynamically identify resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with a suite of representative MapReduce benchmarks demonstrate up to 38% reduction in job completion times, and up to 25% increase in resource utilization. We further demonstrate the performance boost in existing resource managers like NGM and Mesos, when augmented with MROrchestrator.

UR - http://www.scopus.com/inward/record.url?scp=84866744799&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84866744799&partnerID=8YFLogxK

U2 - 10.1109/CLOUD.2012.37

DO - 10.1109/CLOUD.2012.37

M3 - Conference contribution

SN - 9780769547558

T3 - Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012

SP - 1

EP - 8

BT - Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012

ER -

Sharma B, Prabhakar R, Lim SH, Kandemir M, Das C. MROrchestrator: A fine-grained resource orchestration framework for MapReduce clusters. In Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012. 2012. p. 1-8. 6253482. (Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012). https://doi.org/10.1109/CLOUD.2012.37