Exploiting inter-file access patterns using multi-collective I/O

Gokhan Memik, Mahmut Kandemir, Alok Choudhary

Research output: Contribution to conferencePaper

7 Citations (Scopus)

Abstract

This paper introduces a new concept called Multi-Collective I/O (MCIO) that extends conventional collective I/O to optimize I/O accesses to multiple arrays simultaneously. In this approach, as in collective I/O, multiple processors co-ordinate to perform I/O on behalf of each other if doing so improves overall I/O time. However, unlike collective I/O, MCIO considers multiple arrays simultaneously; that is, it has a more global view of the overall I/O behavior exhibited by application. This paper shows that determining optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem (also called the assignment problem). Both of the heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our preliminary results show that MCIO out-performs collective I/O by as much as . Our runtime library-based implementation can be used by users as well as optimizing compilers. Based on our results, we recommend future library designers for I/O-intensive applications to include MCIO in their suite of optimizations.

Original languageEnglish (US)
StatePublished - Jan 1 2002
Event2002 USENIX Conference on File and Storage Technologies, FAST 2002 - Monterey, United States
Duration: Jan 28 2002Jan 30 2002

Conference

Conference2002 USENIX Conference on File and Storage Technologies, FAST 2002
CountryUnited States
CityMonterey
Period1/28/021/30/02

Fingerprint

Computational complexity
Coprocessor

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Networks and Communications
  • Hardware and Architecture

Cite this

Memik, G., Kandemir, M., & Choudhary, A. (2002). Exploiting inter-file access patterns using multi-collective I/O. Paper presented at 2002 USENIX Conference on File and Storage Technologies, FAST 2002, Monterey, United States.
Memik, Gokhan ; Kandemir, Mahmut ; Choudhary, Alok. / Exploiting inter-file access patterns using multi-collective I/O. Paper presented at 2002 USENIX Conference on File and Storage Technologies, FAST 2002, Monterey, United States.
@conference{2d59d8758a0949d2bc4c2b7c779c76d4,
title = "Exploiting inter-file access patterns using multi-collective I/O",
abstract = "This paper introduces a new concept called Multi-Collective I/O (MCIO) that extends conventional collective I/O to optimize I/O accesses to multiple arrays simultaneously. In this approach, as in collective I/O, multiple processors co-ordinate to perform I/O on behalf of each other if doing so improves overall I/O time. However, unlike collective I/O, MCIO considers multiple arrays simultaneously; that is, it has a more global view of the overall I/O behavior exhibited by application. This paper shows that determining optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem (also called the assignment problem). Both of the heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our preliminary results show that MCIO out-performs collective I/O by as much as . Our runtime library-based implementation can be used by users as well as optimizing compilers. Based on our results, we recommend future library designers for I/O-intensive applications to include MCIO in their suite of optimizations.",
author = "Gokhan Memik and Mahmut Kandemir and Alok Choudhary",
year = "2002",
month = "1",
day = "1",
language = "English (US)",
note = "2002 USENIX Conference on File and Storage Technologies, FAST 2002 ; Conference date: 28-01-2002 Through 30-01-2002",

}

Memik, G, Kandemir, M & Choudhary, A 2002, 'Exploiting inter-file access patterns using multi-collective I/O', Paper presented at 2002 USENIX Conference on File and Storage Technologies, FAST 2002, Monterey, United States, 1/28/02 - 1/30/02.

Exploiting inter-file access patterns using multi-collective I/O. / Memik, Gokhan; Kandemir, Mahmut; Choudhary, Alok.

2002. Paper presented at 2002 USENIX Conference on File and Storage Technologies, FAST 2002, Monterey, United States.

Research output: Contribution to conferencePaper

TY - CONF

T1 - Exploiting inter-file access patterns using multi-collective I/O

AU - Memik, Gokhan

AU - Kandemir, Mahmut

AU - Choudhary, Alok

PY - 2002/1/1

Y1 - 2002/1/1

N2 - This paper introduces a new concept called Multi-Collective I/O (MCIO) that extends conventional collective I/O to optimize I/O accesses to multiple arrays simultaneously. In this approach, as in collective I/O, multiple processors co-ordinate to perform I/O on behalf of each other if doing so improves overall I/O time. However, unlike collective I/O, MCIO considers multiple arrays simultaneously; that is, it has a more global view of the overall I/O behavior exhibited by application. This paper shows that determining optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem (also called the assignment problem). Both of the heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our preliminary results show that MCIO out-performs collective I/O by as much as . Our runtime library-based implementation can be used by users as well as optimizing compilers. Based on our results, we recommend future library designers for I/O-intensive applications to include MCIO in their suite of optimizations.

AB - This paper introduces a new concept called Multi-Collective I/O (MCIO) that extends conventional collective I/O to optimize I/O accesses to multiple arrays simultaneously. In this approach, as in collective I/O, multiple processors co-ordinate to perform I/O on behalf of each other if doing so improves overall I/O time. However, unlike collective I/O, MCIO considers multiple arrays simultaneously; that is, it has a more global view of the overall I/O behavior exhibited by application. This paper shows that determining optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem (also called the assignment problem). Both of the heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our preliminary results show that MCIO out-performs collective I/O by as much as . Our runtime library-based implementation can be used by users as well as optimizing compilers. Based on our results, we recommend future library designers for I/O-intensive applications to include MCIO in their suite of optimizations.

UR - http://www.scopus.com/inward/record.url?scp=0012272933&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0012272933&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:0012272933

ER -

Memik G, Kandemir M, Choudhary A. Exploiting inter-file access patterns using multi-collective I/O. 2002. Paper presented at 2002 USENIX Conference on File and Storage Technologies, FAST 2002, Monterey, United States.