A Case Study of Complex Graph Analysis in Distributed Memory: Implementation and Optimization

George M. Slota, Sivasankaran Rajamanickam, Kamesh Madduri

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Citations (Scopus)

Abstract

In recent years, a large number of graph processing frameworks have been introduced, with their goal to simplify analysis of real-world graphs on commodity hardware. Additionally, the Graph500 benchmark has motivated extensive optimization of fundamental graph computations such as breadth-first search and shortest paths on leading high-performance computing systems. The purpose of this current work is to bridge the gap between these two research areas: we introduce a methodology for graph processing that is simple to implement, and yet offers high performance when scaling up from a single compute node up to several thousand nodes. We develop a compact and efficient graph representation, implement several graph analytics, and describe a number of optimizations that can be applied to these analytics. We test our implementations on the 2012 Web Data Commons hyperlink graph with 3.56 billion vertices and 128.7 billion edges, and perform scalability studies up to 4096 nodes of the Blue Waters supercomputer. On 256 nodes of Blue Waters, we demonstrate execution of six graph analytics on this large hyperlink graph in about 20 minutes.

Original languageEnglish (US)
Title of host publicationProceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages293-302
Number of pages10
ISBN (Electronic)9781509021406
DOIs
StatePublished - Jul 18 2016
Event30th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2016 - Chicago, United States
Duration: May 23 2016May 27 2016

Publication series

NameProceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016

Other

Other30th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2016
CountryUnited States
CityChicago
Period5/23/165/27/16

Fingerprint

Data storage equipment
Supercomputers
Processing
Scalability
Water
Hardware

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications

Cite this

Slota, G. M., Rajamanickam, S., & Madduri, K. (2016). A Case Study of Complex Graph Analysis in Distributed Memory: Implementation and Optimization. In Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016 (pp. 293-302). [7516025] (Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2016.93
Slota, George M. ; Rajamanickam, Sivasankaran ; Madduri, Kamesh. / A Case Study of Complex Graph Analysis in Distributed Memory : Implementation and Optimization. Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 293-302 (Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016).
@inproceedings{bc0876b6b9984a38807926e7bd767ebb,
title = "A Case Study of Complex Graph Analysis in Distributed Memory: Implementation and Optimization",
abstract = "In recent years, a large number of graph processing frameworks have been introduced, with their goal to simplify analysis of real-world graphs on commodity hardware. Additionally, the Graph500 benchmark has motivated extensive optimization of fundamental graph computations such as breadth-first search and shortest paths on leading high-performance computing systems. The purpose of this current work is to bridge the gap between these two research areas: we introduce a methodology for graph processing that is simple to implement, and yet offers high performance when scaling up from a single compute node up to several thousand nodes. We develop a compact and efficient graph representation, implement several graph analytics, and describe a number of optimizations that can be applied to these analytics. We test our implementations on the 2012 Web Data Commons hyperlink graph with 3.56 billion vertices and 128.7 billion edges, and perform scalability studies up to 4096 nodes of the Blue Waters supercomputer. On 256 nodes of Blue Waters, we demonstrate execution of six graph analytics on this large hyperlink graph in about 20 minutes.",
author = "Slota, {George M.} and Sivasankaran Rajamanickam and Kamesh Madduri",
year = "2016",
month = "7",
day = "18",
doi = "10.1109/IPDPS.2016.93",
language = "English (US)",
series = "Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "293--302",
booktitle = "Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016",
address = "United States",

}

Slota, GM, Rajamanickam, S & Madduri, K 2016, A Case Study of Complex Graph Analysis in Distributed Memory: Implementation and Optimization. in Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016., 7516025, Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016, Institute of Electrical and Electronics Engineers Inc., pp. 293-302, 30th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2016, Chicago, United States, 5/23/16. https://doi.org/10.1109/IPDPS.2016.93

A Case Study of Complex Graph Analysis in Distributed Memory : Implementation and Optimization. / Slota, George M.; Rajamanickam, Sivasankaran; Madduri, Kamesh.

Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 293-302 7516025 (Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A Case Study of Complex Graph Analysis in Distributed Memory

T2 - Implementation and Optimization

AU - Slota, George M.

AU - Rajamanickam, Sivasankaran

AU - Madduri, Kamesh

PY - 2016/7/18

Y1 - 2016/7/18

N2 - In recent years, a large number of graph processing frameworks have been introduced, with their goal to simplify analysis of real-world graphs on commodity hardware. Additionally, the Graph500 benchmark has motivated extensive optimization of fundamental graph computations such as breadth-first search and shortest paths on leading high-performance computing systems. The purpose of this current work is to bridge the gap between these two research areas: we introduce a methodology for graph processing that is simple to implement, and yet offers high performance when scaling up from a single compute node up to several thousand nodes. We develop a compact and efficient graph representation, implement several graph analytics, and describe a number of optimizations that can be applied to these analytics. We test our implementations on the 2012 Web Data Commons hyperlink graph with 3.56 billion vertices and 128.7 billion edges, and perform scalability studies up to 4096 nodes of the Blue Waters supercomputer. On 256 nodes of Blue Waters, we demonstrate execution of six graph analytics on this large hyperlink graph in about 20 minutes.

AB - In recent years, a large number of graph processing frameworks have been introduced, with their goal to simplify analysis of real-world graphs on commodity hardware. Additionally, the Graph500 benchmark has motivated extensive optimization of fundamental graph computations such as breadth-first search and shortest paths on leading high-performance computing systems. The purpose of this current work is to bridge the gap between these two research areas: we introduce a methodology for graph processing that is simple to implement, and yet offers high performance when scaling up from a single compute node up to several thousand nodes. We develop a compact and efficient graph representation, implement several graph analytics, and describe a number of optimizations that can be applied to these analytics. We test our implementations on the 2012 Web Data Commons hyperlink graph with 3.56 billion vertices and 128.7 billion edges, and perform scalability studies up to 4096 nodes of the Blue Waters supercomputer. On 256 nodes of Blue Waters, we demonstrate execution of six graph analytics on this large hyperlink graph in about 20 minutes.

UR - http://www.scopus.com/inward/record.url?scp=84983234050&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84983234050&partnerID=8YFLogxK

U2 - 10.1109/IPDPS.2016.93

DO - 10.1109/IPDPS.2016.93

M3 - Conference contribution

AN - SCOPUS:84983234050

T3 - Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016

SP - 293

EP - 302

BT - Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Slota GM, Rajamanickam S, Madduri K. A Case Study of Complex Graph Analysis in Distributed Memory: Implementation and Optimization. In Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 293-302. 7516025. (Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016). https://doi.org/10.1109/IPDPS.2016.93