Parallel Read Partitioning for Concurrent Assembly of Metagenomic Data

Vasudevan Rengasamy, Mahmut T. Kandemir, Paul Medvedev, Kamesh Madduri

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present MetaPartMin and MetaPart, two new lightweight parallel metagenomic read partitioning strategies. Metagenomic data partitioning can aid the concurrent de novo assembly of partitions. Prior read partitioning methods tend to create a giant component of reads. We avoid this problem with new heuristics amenable to statically load-balanced parallelization. Our strategies require enumerating and sorting k-mers and minimizers from the input read sequences, and traversing an implicit graph to identify components. MetaPartMin uses minimizers to significantly lower aggregate main memory use, thereby enabling the processing of massive datasets on a modest number of compute nodes. All steps in our strategies exploit hybrid multicore and distributed-memory parallelism. We demonstrate scaling and efficiency on a collection of large-scale datasets. MetaPartMin can process a 1.25 terabase soil metagenome in 6 minutes on just 32 Intel Skylake nodes (48 cores each) of the Stampede2 supercomputer, and a 252 gigabase soil metagenome in 54 seconds on 16 Stampede2 Skylake nodes. The source code is available at https://github.com/vasupsu/MetaPart.

Original languageEnglish (US)
Title of host publicationProceedings - 25th IEEE International Conference on High Performance Computing, HiPC 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages324-333
Number of pages10
ISBN (Electronic)9781538683866
DOIs
StatePublished - Feb 8 2019
Event25th IEEE International Conference on High Performance Computing, HiPC 2018 - Bengaluru, India
Duration: Dec 17 2018Dec 20 2018

Publication series

NameProceedings - 25th IEEE International Conference on High Performance Computing, HiPC 2018

Conference

Conference25th IEEE International Conference on High Performance Computing, HiPC 2018
CountryIndia
CityBengaluru
Period12/17/1812/20/18

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Theoretical Computer Science

Fingerprint Dive into the research topics of 'Parallel Read Partitioning for Concurrent Assembly of Metagenomic Data'. Together they form a unique fingerprint.

  • Cite this

    Rengasamy, V., Kandemir, M. T., Medvedev, P., & Madduri, K. (2019). Parallel Read Partitioning for Concurrent Assembly of Metagenomic Data. In Proceedings - 25th IEEE International Conference on High Performance Computing, HiPC 2018 (pp. 324-333). [8638083] (Proceedings - 25th IEEE International Conference on High Performance Computing, HiPC 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HiPC.2018.00044