IOPro: a parallel I/O profiling and visualization framework for high-performance storage systems

Seong Jo Kim, Yuanrui Zhang, Seung Woo Son, Mahmut Kandemir, Wei keng Liao, Rajeev Thakur, Alok Choudhary

Research output: Contribution to journalArticlepeer-review

Abstract

Efficient execution of large-scale scientific applications requires high-performance computing systems designed to meet the I/O requirements. To achieve high-performance, such data-intensive parallel applications use a multi-layer layer I/O software stack, which consists of high-level I/O libraries such as PnetCDF and HDF5, the MPI library, and parallel file systems. To design efficient parallel scientific applications, understanding the complicated flow of I/O operations and the involved interactions among the libraries is quintessential. Such comprehension helps identify I/O bottlenecks and thus exploits the potential performance in different layers of the storage hierarchy. To profile the performance of individual components in the I/O stack and to understand complex interactions among them, we have implemented a GUI-based integrated profiling and analysis framework, IOPro. IOPro automatically generates an instrumented I/O stack, runs applications on it, and visualizes detailed statistics based on the user-specified metrics of interest. We present experimental results from two different real-life applications and show how our framework can be used in practice. By generating an end-to-end trace of the whole I/O stack and pinpointing I/O interference, IOPro aids in understanding I/O behavior and improving the I/O performance significantly.

Original languageEnglish (US)
Pages (from-to)840-870
Number of pages31
JournalJournal of Supercomputing
Volume71
Issue number3
DOIs
StatePublished - Jan 1 2015

All Science Journal Classification (ASJC) codes

  • Software
  • Theoretical Computer Science
  • Information Systems
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'IOPro: a parallel I/O profiling and visualization framework for high-performance storage systems'. Together they form a unique fingerprint.

Cite this