Design and evaluation of a compiler-directed collective I/O technique

Gokhan Memik, Mahmut T. Kandemir, Alok Choudhary

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Current approaches to parallel I/O demand extensive user effort to obtain acceptable performance. This is in part due to difficulties in understanding the characteristics of a wide variety of I/O devices and in part due to inherent complexity of I/O software. While parallel I/O systems provide users with environments where large datasets can be shared between parallel processors, the ultimate performance of I/O-intensive codes depends largely on the relation between data access patterns and storage patterns of data in files and on disks. Collective I/O is one of the most popular methods to access the data when the storage and access patterns do not match. In this strategy, each processor does I/O on behalf of other processors if doing so improves the overall performance. While it is generally accepted that collective I/O and its variants can bring impressive improvements as far as the I/O performance is concerned, it is difficult for the programmer to use collective I/O effectively. In this paper, we propose and evaluate a compiler-directed collective I/O approach which detects the opportunities for collective I/O and inserts the necessary I/O calls in the code automatically. An important characteristic of the approach is that instead of applying collective I/O indiscriminately, it uses collective I/O selectively, only in cases where independent parallel I/O would not be possible. We have conducted several experiments using an IBM SP-2di stributed-memory message-passing machine with 128 nodes. Our compiler directed collective I/O scheme was able to perform 18% better in average than an indiscriminate collective I/O scheme in our base configuration.

Original languageEnglish (US)
Title of host publicationEuro-Par 2000 Parallel Processing - 6th International Euro-Par Conference, Proceedings
EditorsArndt Bode, Thomas Ludwig, Wolfgang Karl, Roland Wismüller
PublisherSpringer Verlag
Pages1263-1272
Number of pages10
ISBN (Electronic)9783540679561
StatePublished - Jan 1 2000
Event6th International European Conference on Parallel Computing, Euro-Par 2000 - Munich, Germany
Duration: Aug 29 2000Sep 1 2000

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume1900
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other6th International European Conference on Parallel Computing, Euro-Par 2000
CountryGermany
CityMunich
Period8/29/009/1/00

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Memik, G., Kandemir, M. T., & Choudhary, A. (2000). Design and evaluation of a compiler-directed collective I/O technique. In A. Bode, T. Ludwig, W. Karl, & R. Wismüller (Eds.), Euro-Par 2000 Parallel Processing - 6th International Euro-Par Conference, Proceedings (pp. 1263-1272). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 1900). Springer Verlag.