TY - GEN
T1 - Network scheduling aware task placement in datacenters
AU - Munir, Ali
AU - He, Ting
AU - Raghavendra, Ramya
AU - Le, Franck
AU - Liu, Alex X.
N1 - Funding Information:
This work is partially supported by the National Science Foundation under Grant Numbers CNS-1318563, CNS-152- 4698, and CNS-1421407, and the National Natural Science Foundation of China under Grant Numbers 61472184 and 61321491, and the Jiangsu High-level Innovation and En- trepreneurship (Shuangchuang) Program. Part of the work was done while Ali Munir was at IBM Research Watson. The authors would like to thank anonymous reviewers and shepherd Mosharaf Chowdhury for their useful feedback.
PY - 2016/12/6
Y1 - 2016/12/6
N2 - To improve the performance of data-intensive applications, existing datacenter schedulers optimize either the placement of tasks or the scheduling of network flows. The task scheduler strives to place tasks close to their input data (i.e., maximize data locality) to minimize network traffic, while assuming fair sharing of the network. The network scheduler strives to finish flows as quickly as possible based on their sources and destinations determined by the task scheduler, while the scheduling is based on flow properties (e.g., size, deadline, and correlation) and not bound to fair sharing. Inconsistent assumptions of the two schedulers can compromise the overall application performance. In this paper, we propose NEAT, a task scheduling framework that leverages information from the underlying network scheduler to make task placement decisions. The core of NEAT is a task completion time predictor that estimates the completion tijme of a task under given network condition and a given network scheduling policy. NEAT leverages the predicted task completion times to minimize the average completion time of active tasks. Evaluation using ns2 simulations and real-testbed shows that NEAT improves application performance by up to 3.7x for the suboptimal network scheduling policies and up to 30% for the optimal network scheduling policy.
AB - To improve the performance of data-intensive applications, existing datacenter schedulers optimize either the placement of tasks or the scheduling of network flows. The task scheduler strives to place tasks close to their input data (i.e., maximize data locality) to minimize network traffic, while assuming fair sharing of the network. The network scheduler strives to finish flows as quickly as possible based on their sources and destinations determined by the task scheduler, while the scheduling is based on flow properties (e.g., size, deadline, and correlation) and not bound to fair sharing. Inconsistent assumptions of the two schedulers can compromise the overall application performance. In this paper, we propose NEAT, a task scheduling framework that leverages information from the underlying network scheduler to make task placement decisions. The core of NEAT is a task completion time predictor that estimates the completion tijme of a task under given network condition and a given network scheduling policy. NEAT leverages the predicted task completion times to minimize the average completion time of active tasks. Evaluation using ns2 simulations and real-testbed shows that NEAT improves application performance by up to 3.7x for the suboptimal network scheduling policies and up to 30% for the optimal network scheduling policy.
UR - http://www.scopus.com/inward/record.url?scp=85009804095&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85009804095&partnerID=8YFLogxK
U2 - 10.1145/2999572.2999588
DO - 10.1145/2999572.2999588
M3 - Conference contribution
AN - SCOPUS:85009804095
T3 - CoNEXT 2016 - Proceedings of the 12th International Conference on Emerging Networking EXperiments and Technologies
SP - 221
EP - 235
BT - CoNEXT 2016 - Proceedings of the 12th International Conference on Emerging Networking EXperiments and Technologies
PB - Association for Computing Machinery, Inc
T2 - 12th ACM Conference on Emerging Networking Experiments and Technologies, ACM CoNEXT 2016
Y2 - 12 December 2016 through 15 December 2016
ER -