An experimental evaluation of I/O optimizations on different applications

Meenakshi A. Kandaswamy, Mahmut Kandemir, Alok Choudhary, David Bernholdt

Research output: Contribution to journalArticle

3 Scopus citations

Abstract

Many large scale applications have significant I/O requirements as well as computational and memory requirements. Unfortunately, the limited number of I/O nodes provided in a typical configuration of the modern message-passing distributed-memory architectures such as Intel Paragon and IBM SP-2 limits the I/O performance of these applications severely. In this paper, we examine some software optimization techniques and evaluate their effects in five different I/O-intensive codes from both small and large application domains. Our goals in this study are twofold. First, we want to understand the behavior of large-scale data-intensive applications and the impact of I/O subsystems on their performance and vice versa. Second, and more importantly, we strive to determine the solutions for improving the applications' performance by a mix of software techniques. Our results reveal that different applications can benefit from different optimizations. For example, we found that some applications benefit from file layout optimizations whereas others take advantage of collective I/O. A combination of architectural and software solutions is normally needed to obtain good I/O performance. For example, we show that with a limited number of I/O resources, it is possible to obtain good performance by using appropriate software optimizations. We also show that beyond a certain level, imbalance in the architecture results in performance degradation even when using optimized software, thereby indicating the necessity of an increase in I/O resources.

Original languageEnglish (US)
Pages (from-to)728-744
Number of pages17
JournalIEEE Transactions on Parallel and Distributed Systems
Volume13
Issue number7
DOIs
StatePublished - Jul 2002

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'An experimental evaluation of I/O optimizations on different applications'. Together they form a unique fingerprint.

  • Cite this