IOPin: Runtime profiling of parallel I/O in HPC systems

Seong Jo Kim, Seung Woo Son, Wei Keng Liao, Mahmut Kandemir, Rajeev Thakur, Alok Choudhary

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)

Abstract

Many I/O- and data-intensive scientific applications use parallel I/O software to access files in high performance. On modern parallel machines, the I/O software consists of several layers, including high-level libraries such as Parallel netCDF and HDF, middleware such as MPI-IO, and low-level POSIX interface supported by the file systems. For the I/O software developers, ensuring data flow is important among these software layers with performance close to the hardware limits. This task requires understanding the design of individual libraries and the characteristics of data flow among them. In this paper, we propose a dynamic instrumentation framework that can be used to understand the complex interactions across different I/O layers from applications to the underlying parallel file systems. Our preliminary experience indicates that the costs of using the proposed dynamic instrumentation is about 7% of the application execution time.

Original languageEnglish (US)
Title of host publicationProceedings - 2012 SC Companion
Subtitle of host publicationHigh Performance Computing, Networking Storage and Analysis, SCC 2012
Pages18-23
Number of pages6
DOIs
StatePublished - Dec 1 2012
Event2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 - Salt Lake City, UT, United States
Duration: Nov 10 2012Nov 16 2012

Publication series

NameProceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Other

Other2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012
CountryUnited States
CitySalt Lake City, UT
Period11/10/1211/16/12

Fingerprint

Middleware
Hardware
Costs

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Cite this

Kim, S. J., Son, S. W., Liao, W. K., Kandemir, M., Thakur, R., & Choudhary, A. (2012). IOPin: Runtime profiling of parallel I/O in HPC systems. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 (pp. 18-23). [6495796] (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012). https://doi.org/10.1109/SC.Companion.2012.14
Kim, Seong Jo ; Son, Seung Woo ; Liao, Wei Keng ; Kandemir, Mahmut ; Thakur, Rajeev ; Choudhary, Alok. / IOPin : Runtime profiling of parallel I/O in HPC systems. Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. pp. 18-23 (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012).
@inproceedings{a17ecc8549e341bc9346f83e9a189b5a,
title = "IOPin: Runtime profiling of parallel I/O in HPC systems",
abstract = "Many I/O- and data-intensive scientific applications use parallel I/O software to access files in high performance. On modern parallel machines, the I/O software consists of several layers, including high-level libraries such as Parallel netCDF and HDF, middleware such as MPI-IO, and low-level POSIX interface supported by the file systems. For the I/O software developers, ensuring data flow is important among these software layers with performance close to the hardware limits. This task requires understanding the design of individual libraries and the characteristics of data flow among them. In this paper, we propose a dynamic instrumentation framework that can be used to understand the complex interactions across different I/O layers from applications to the underlying parallel file systems. Our preliminary experience indicates that the costs of using the proposed dynamic instrumentation is about 7{\%} of the application execution time.",
author = "Kim, {Seong Jo} and Son, {Seung Woo} and Liao, {Wei Keng} and Mahmut Kandemir and Rajeev Thakur and Alok Choudhary",
year = "2012",
month = "12",
day = "1",
doi = "10.1109/SC.Companion.2012.14",
language = "English (US)",
isbn = "9780769549569",
series = "Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012",
pages = "18--23",
booktitle = "Proceedings - 2012 SC Companion",

}

Kim, SJ, Son, SW, Liao, WK, Kandemir, M, Thakur, R & Choudhary, A 2012, IOPin: Runtime profiling of parallel I/O in HPC systems. in Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012., 6495796, Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, pp. 18-23, 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, Salt Lake City, UT, United States, 11/10/12. https://doi.org/10.1109/SC.Companion.2012.14

IOPin : Runtime profiling of parallel I/O in HPC systems. / Kim, Seong Jo; Son, Seung Woo; Liao, Wei Keng; Kandemir, Mahmut; Thakur, Rajeev; Choudhary, Alok.

Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. p. 18-23 6495796 (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - IOPin

T2 - Runtime profiling of parallel I/O in HPC systems

AU - Kim, Seong Jo

AU - Son, Seung Woo

AU - Liao, Wei Keng

AU - Kandemir, Mahmut

AU - Thakur, Rajeev

AU - Choudhary, Alok

PY - 2012/12/1

Y1 - 2012/12/1

N2 - Many I/O- and data-intensive scientific applications use parallel I/O software to access files in high performance. On modern parallel machines, the I/O software consists of several layers, including high-level libraries such as Parallel netCDF and HDF, middleware such as MPI-IO, and low-level POSIX interface supported by the file systems. For the I/O software developers, ensuring data flow is important among these software layers with performance close to the hardware limits. This task requires understanding the design of individual libraries and the characteristics of data flow among them. In this paper, we propose a dynamic instrumentation framework that can be used to understand the complex interactions across different I/O layers from applications to the underlying parallel file systems. Our preliminary experience indicates that the costs of using the proposed dynamic instrumentation is about 7% of the application execution time.

AB - Many I/O- and data-intensive scientific applications use parallel I/O software to access files in high performance. On modern parallel machines, the I/O software consists of several layers, including high-level libraries such as Parallel netCDF and HDF, middleware such as MPI-IO, and low-level POSIX interface supported by the file systems. For the I/O software developers, ensuring data flow is important among these software layers with performance close to the hardware limits. This task requires understanding the design of individual libraries and the characteristics of data flow among them. In this paper, we propose a dynamic instrumentation framework that can be used to understand the complex interactions across different I/O layers from applications to the underlying parallel file systems. Our preliminary experience indicates that the costs of using the proposed dynamic instrumentation is about 7% of the application execution time.

UR - http://www.scopus.com/inward/record.url?scp=84876582826&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84876582826&partnerID=8YFLogxK

U2 - 10.1109/SC.Companion.2012.14

DO - 10.1109/SC.Companion.2012.14

M3 - Conference contribution

AN - SCOPUS:84876582826

SN - 9780769549569

T3 - Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

SP - 18

EP - 23

BT - Proceedings - 2012 SC Companion

ER -

Kim SJ, Son SW, Liao WK, Kandemir M, Thakur R, Choudhary A. IOPin: Runtime profiling of parallel I/O in HPC systems. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. p. 18-23. 6495796. (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012). https://doi.org/10.1109/SC.Companion.2012.14