Data access reorganizations in compiling out-of-core data parallel programs on distributed memory machines

Mahmut Kandemir, Rajesh Bordawekar, Alok Choudhary

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

This paper describes optimization techniques for translating out-of-core programs written in a data parallel language to message passing node programs with explicit parallel I/O. We demonstrate that straightforward extension of in-core compilation techniques does not work well for out-of-core programs. We then describe how the compiler can optimize the code by (1) determining appropriate file layouts for out-of-core arrays, (2) permuting the loops in the nest(s) to allow efficient file access, and (3) partitioning the available node memory among references based on I/O cost estimation. Our experimental results indicate that these optimizations can reduce the amount of time spent in I/O by as much as an order of magnitude.

Original languageEnglish (US)
Pages (from-to)559-564
Number of pages6
JournalProceedings of the International Parallel Processing Symposium, IPPS
StatePublished - 1997

Fingerprint

Data storage equipment
Message passing
Costs

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture

Cite this

@article{d3b727281dfc4fb4967d3932973ae3ff,
title = "Data access reorganizations in compiling out-of-core data parallel programs on distributed memory machines",
abstract = "This paper describes optimization techniques for translating out-of-core programs written in a data parallel language to message passing node programs with explicit parallel I/O. We demonstrate that straightforward extension of in-core compilation techniques does not work well for out-of-core programs. We then describe how the compiler can optimize the code by (1) determining appropriate file layouts for out-of-core arrays, (2) permuting the loops in the nest(s) to allow efficient file access, and (3) partitioning the available node memory among references based on I/O cost estimation. Our experimental results indicate that these optimizations can reduce the amount of time spent in I/O by as much as an order of magnitude.",
author = "Mahmut Kandemir and Rajesh Bordawekar and Alok Choudhary",
year = "1997",
language = "English (US)",
pages = "559--564",
journal = "Proceedings of the International Parallel Processing Symposium, IPPS",
issn = "1063-7133",

}

TY - JOUR

T1 - Data access reorganizations in compiling out-of-core data parallel programs on distributed memory machines

AU - Kandemir, Mahmut

AU - Bordawekar, Rajesh

AU - Choudhary, Alok

PY - 1997

Y1 - 1997

N2 - This paper describes optimization techniques for translating out-of-core programs written in a data parallel language to message passing node programs with explicit parallel I/O. We demonstrate that straightforward extension of in-core compilation techniques does not work well for out-of-core programs. We then describe how the compiler can optimize the code by (1) determining appropriate file layouts for out-of-core arrays, (2) permuting the loops in the nest(s) to allow efficient file access, and (3) partitioning the available node memory among references based on I/O cost estimation. Our experimental results indicate that these optimizations can reduce the amount of time spent in I/O by as much as an order of magnitude.

AB - This paper describes optimization techniques for translating out-of-core programs written in a data parallel language to message passing node programs with explicit parallel I/O. We demonstrate that straightforward extension of in-core compilation techniques does not work well for out-of-core programs. We then describe how the compiler can optimize the code by (1) determining appropriate file layouts for out-of-core arrays, (2) permuting the loops in the nest(s) to allow efficient file access, and (3) partitioning the available node memory among references based on I/O cost estimation. Our experimental results indicate that these optimizations can reduce the amount of time spent in I/O by as much as an order of magnitude.

UR - http://www.scopus.com/inward/record.url?scp=0030710004&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030710004&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:0030710004

SP - 559

EP - 564

JO - Proceedings of the International Parallel Processing Symposium, IPPS

JF - Proceedings of the International Parallel Processing Symposium, IPPS

SN - 1063-7133

ER -