An evolutionary path to object storage access

David Goodell, Seong Jo Kim, Robert Latham, Mahmut Kandemir, Robert Ross

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations

Abstract

High-performance computing (HPC) storage sys- tems typically consist of an object storage system that is accessed via the POSIX file interface. However, rapid increases in system scales and storage system complexity have uncovered a number of limitations in this model. In particular, applications and libraries are limited in their ability to partition data into units with independent concurrency control, and mapping complex science data models into the POSIX file model is inconvenient at best. In this paper we propose an alternative interface for use by applications and libraries that provides direct access to underlying storage objects. This model allows applications and libraries to organize storage access around these objects in order to avoid lock contention without needing to create many separate files. Additionally, complex data models are more readily organized into multiple object data streams, simplifying the storage of variable-length data and allowing a choice of degree of parallelism related to access needs. Our approach provides for datasets stored in this new model to coexist with POSIX files, allowing evolution to the new model over time. We apply these concepts in the PVFS, PLFS, and Parallel netCDF packages to prototype the model and describe our experiences.

Original languageEnglish (US)
Title of host publicationProceedings - 2012 SC Companion
Subtitle of host publicationHigh Performance Computing, Networking Storage and Analysis, SCC 2012
Pages36-41
Number of pages6
DOIs
StatePublished - Dec 1 2012
Event2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 - Salt Lake City, UT, United States
Duration: Nov 10 2012Nov 16 2012

Publication series

NameProceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Other

Other2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012
CountryUnited States
CitySalt Lake City, UT
Period11/10/1211/16/12

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Cite this

Goodell, D., Kim, S. J., Latham, R., Kandemir, M., & Ross, R. (2012). An evolutionary path to object storage access. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 (pp. 36-41). [6495799] (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012). https://doi.org/10.1109/SC.Companion.2012.17