Multicollective I/O: A technique for exploiting inter-file access patterns

Gokhan Memik, Mahmut T. Kandemir, Wei Keng Liao, Alok Choudhary

Research output: Contribution to journalArticle

5 Scopus citations

Abstract

The increasing gap between processor cycle times and access times to storage devices makes it necessary to use powerful optimizations. This is especially true for applications in the parallel computing domain that frequently perform large amounts of file I/O. Collective I/O strategy that coordinates the processes to perform I/O on each other's behalf has demonstrated a significant performance improvement. This article proposes a new concept called Multicollective I/O (MCIO) that expands the collective I/O to allow data from multiple files to be requested in a single I/O request, in contrast to allowing only multiple segments for a single file to be specified together. MCIO considers multiple arrays simultaneously by having a more global view of the overall I/O behavior exhibited by parallel applications. This article shows that determining the optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem, also called the assignment problem. Both heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our results show that MCIO outperforms collective I/O by as much as 87%. Our runtime library-based implementation can be used by application users as well as by optimizing compilers. Based on our results, we recommend that future library designers for I/O-intensive applications include MCIO in their suite of optimizations.

Original languageEnglish (US)
Pages (from-to)349-369
Number of pages21
JournalACM Transactions on Storage
Volume2
Issue number3
DOIs
StatePublished - Aug 1 2006

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Multicollective I/O: A technique for exploiting inter-file access patterns'. Together they form a unique fingerprint.

  • Cite this