Partitioning Trillion-Edge Graphs in Minutes

George M. Slota, Sivasankaran Rajamanickam, Karen Devine, Kamesh Madduri

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Citations (Scopus)

Abstract

We introduce XtraPuLP, a new distributed-memory graph partitioner designed to process trillion-edge graphs. XtraPuLP is based on the scalable label propagation community detection technique, which has been demonstrated as a viable means to produce high quality partitions with minimal computation time. On a collection of large sparse graphs, we show that XtraPuLP partitioning quality is comparable to state-of-the-art partitioning methods. We also demonstrate that XtraPuLP can produce partitions of real-world graphs with billion+ vertices in minutes. Further, we show that using XtraPuLP partitions for distributed-memory graph analytics leads to significant end-to-end execution time reduction.

Original languageEnglish (US)
Title of host publicationProceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages646-655
Number of pages10
ISBN (Electronic)9781538639146
DOIs
StatePublished - Jun 30 2017
Event31st IEEE International Parallel and Distributed Processing Symposium, IPDPS 2017 - Orlando, United States
Duration: May 29 2017Jun 2 2017

Publication series

NameProceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017

Other

Other31st IEEE International Parallel and Distributed Processing Symposium, IPDPS 2017
CountryUnited States
CityOrlando
Period5/29/176/2/17

Fingerprint

Data storage equipment
Labels

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Networks and Communications
  • Hardware and Architecture

Cite this

Slota, G. M., Rajamanickam, S., Devine, K., & Madduri, K. (2017). Partitioning Trillion-Edge Graphs in Minutes. In Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017 (pp. 646-655). [7967155] (Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2017.95
Slota, George M. ; Rajamanickam, Sivasankaran ; Devine, Karen ; Madduri, Kamesh. / Partitioning Trillion-Edge Graphs in Minutes. Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 646-655 (Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017).
@inproceedings{a0a45d9326b0484997775b24b7c8e146,
title = "Partitioning Trillion-Edge Graphs in Minutes",
abstract = "We introduce XtraPuLP, a new distributed-memory graph partitioner designed to process trillion-edge graphs. XtraPuLP is based on the scalable label propagation community detection technique, which has been demonstrated as a viable means to produce high quality partitions with minimal computation time. On a collection of large sparse graphs, we show that XtraPuLP partitioning quality is comparable to state-of-the-art partitioning methods. We also demonstrate that XtraPuLP can produce partitions of real-world graphs with billion+ vertices in minutes. Further, we show that using XtraPuLP partitions for distributed-memory graph analytics leads to significant end-to-end execution time reduction.",
author = "Slota, {George M.} and Sivasankaran Rajamanickam and Karen Devine and Kamesh Madduri",
year = "2017",
month = "6",
day = "30",
doi = "10.1109/IPDPS.2017.95",
language = "English (US)",
series = "Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "646--655",
booktitle = "Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017",
address = "United States",

}

Slota, GM, Rajamanickam, S, Devine, K & Madduri, K 2017, Partitioning Trillion-Edge Graphs in Minutes. in Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017., 7967155, Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017, Institute of Electrical and Electronics Engineers Inc., pp. 646-655, 31st IEEE International Parallel and Distributed Processing Symposium, IPDPS 2017, Orlando, United States, 5/29/17. https://doi.org/10.1109/IPDPS.2017.95

Partitioning Trillion-Edge Graphs in Minutes. / Slota, George M.; Rajamanickam, Sivasankaran; Devine, Karen; Madduri, Kamesh.

Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 646-655 7967155 (Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Partitioning Trillion-Edge Graphs in Minutes

AU - Slota, George M.

AU - Rajamanickam, Sivasankaran

AU - Devine, Karen

AU - Madduri, Kamesh

PY - 2017/6/30

Y1 - 2017/6/30

N2 - We introduce XtraPuLP, a new distributed-memory graph partitioner designed to process trillion-edge graphs. XtraPuLP is based on the scalable label propagation community detection technique, which has been demonstrated as a viable means to produce high quality partitions with minimal computation time. On a collection of large sparse graphs, we show that XtraPuLP partitioning quality is comparable to state-of-the-art partitioning methods. We also demonstrate that XtraPuLP can produce partitions of real-world graphs with billion+ vertices in minutes. Further, we show that using XtraPuLP partitions for distributed-memory graph analytics leads to significant end-to-end execution time reduction.

AB - We introduce XtraPuLP, a new distributed-memory graph partitioner designed to process trillion-edge graphs. XtraPuLP is based on the scalable label propagation community detection technique, which has been demonstrated as a viable means to produce high quality partitions with minimal computation time. On a collection of large sparse graphs, we show that XtraPuLP partitioning quality is comparable to state-of-the-art partitioning methods. We also demonstrate that XtraPuLP can produce partitions of real-world graphs with billion+ vertices in minutes. Further, we show that using XtraPuLP partitions for distributed-memory graph analytics leads to significant end-to-end execution time reduction.

UR - http://www.scopus.com/inward/record.url?scp=85027726099&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85027726099&partnerID=8YFLogxK

U2 - 10.1109/IPDPS.2017.95

DO - 10.1109/IPDPS.2017.95

M3 - Conference contribution

AN - SCOPUS:85027726099

T3 - Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017

SP - 646

EP - 655

BT - Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Slota GM, Rajamanickam S, Devine K, Madduri K. Partitioning Trillion-Edge Graphs in Minutes. In Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 646-655. 7967155. (Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017). https://doi.org/10.1109/IPDPS.2017.95