Locality optimization algorithms for compilation of out-of-core codes

Mahmut Kandemir, A. Choudhary, J. Ramanujam, M. Kandaswamy

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

The difficulty of handling out-of-core data limits the performance of supercomputers as well as the potential of parallel machines. Since writing an efficient out-of-core version of a program is a difficult tasks, and since virtual memory systems do not perform well on scientific computations, we believe that there is a clear need for a compiler-directed explicit I/O approach for out-of-core computations. In this paper, we present a compiler algorithm to optimize the locality of disk accesses in out-of-core codes by choosing a good combination of file layouts on disks and loop transformations. The transformations change the access order of array data. Our solution goes beyond the single-loop level (called global I/O optimization in this paper) for out-of-core computations. Since the general problem is NP-complete, we present efficient heuristics that produce near-optimal solutions for several programs encountered in practice. Experimental results obtained on an IBM SP-2 and Intel Paragon provide encouraging evidence that our approach is successful at optimizing programs which depend on disk-resident data in distributed-memory machines.

Original languageEnglish (US)
Pages (from-to)107-138
Number of pages32
JournalJournal of Information Science and Engineering
Volume14
Issue number1
StatePublished - Mar 1 1998

Fingerprint

Data storage equipment
Supercomputers
layout
heuristics
Computational complexity
resident
performance
evidence

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Hardware and Architecture
  • Library and Information Sciences
  • Computational Theory and Mathematics

Cite this

Kandemir, Mahmut ; Choudhary, A. ; Ramanujam, J. ; Kandaswamy, M. / Locality optimization algorithms for compilation of out-of-core codes. In: Journal of Information Science and Engineering. 1998 ; Vol. 14, No. 1. pp. 107-138.
@article{dec083cdf06b4882a17a80590fc083fe,
title = "Locality optimization algorithms for compilation of out-of-core codes",
abstract = "The difficulty of handling out-of-core data limits the performance of supercomputers as well as the potential of parallel machines. Since writing an efficient out-of-core version of a program is a difficult tasks, and since virtual memory systems do not perform well on scientific computations, we believe that there is a clear need for a compiler-directed explicit I/O approach for out-of-core computations. In this paper, we present a compiler algorithm to optimize the locality of disk accesses in out-of-core codes by choosing a good combination of file layouts on disks and loop transformations. The transformations change the access order of array data. Our solution goes beyond the single-loop level (called global I/O optimization in this paper) for out-of-core computations. Since the general problem is NP-complete, we present efficient heuristics that produce near-optimal solutions for several programs encountered in practice. Experimental results obtained on an IBM SP-2 and Intel Paragon provide encouraging evidence that our approach is successful at optimizing programs which depend on disk-resident data in distributed-memory machines.",
author = "Mahmut Kandemir and A. Choudhary and J. Ramanujam and M. Kandaswamy",
year = "1998",
month = "3",
day = "1",
language = "English (US)",
volume = "14",
pages = "107--138",
journal = "Journal of Information Science and Engineering",
issn = "1016-2364",
publisher = "Institute of Information Science",
number = "1",

}

Kandemir, M, Choudhary, A, Ramanujam, J & Kandaswamy, M 1998, 'Locality optimization algorithms for compilation of out-of-core codes', Journal of Information Science and Engineering, vol. 14, no. 1, pp. 107-138.

Locality optimization algorithms for compilation of out-of-core codes. / Kandemir, Mahmut; Choudhary, A.; Ramanujam, J.; Kandaswamy, M.

In: Journal of Information Science and Engineering, Vol. 14, No. 1, 01.03.1998, p. 107-138.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Locality optimization algorithms for compilation of out-of-core codes

AU - Kandemir, Mahmut

AU - Choudhary, A.

AU - Ramanujam, J.

AU - Kandaswamy, M.

PY - 1998/3/1

Y1 - 1998/3/1

N2 - The difficulty of handling out-of-core data limits the performance of supercomputers as well as the potential of parallel machines. Since writing an efficient out-of-core version of a program is a difficult tasks, and since virtual memory systems do not perform well on scientific computations, we believe that there is a clear need for a compiler-directed explicit I/O approach for out-of-core computations. In this paper, we present a compiler algorithm to optimize the locality of disk accesses in out-of-core codes by choosing a good combination of file layouts on disks and loop transformations. The transformations change the access order of array data. Our solution goes beyond the single-loop level (called global I/O optimization in this paper) for out-of-core computations. Since the general problem is NP-complete, we present efficient heuristics that produce near-optimal solutions for several programs encountered in practice. Experimental results obtained on an IBM SP-2 and Intel Paragon provide encouraging evidence that our approach is successful at optimizing programs which depend on disk-resident data in distributed-memory machines.

AB - The difficulty of handling out-of-core data limits the performance of supercomputers as well as the potential of parallel machines. Since writing an efficient out-of-core version of a program is a difficult tasks, and since virtual memory systems do not perform well on scientific computations, we believe that there is a clear need for a compiler-directed explicit I/O approach for out-of-core computations. In this paper, we present a compiler algorithm to optimize the locality of disk accesses in out-of-core codes by choosing a good combination of file layouts on disks and loop transformations. The transformations change the access order of array data. Our solution goes beyond the single-loop level (called global I/O optimization in this paper) for out-of-core computations. Since the general problem is NP-complete, we present efficient heuristics that produce near-optimal solutions for several programs encountered in practice. Experimental results obtained on an IBM SP-2 and Intel Paragon provide encouraging evidence that our approach is successful at optimizing programs which depend on disk-resident data in distributed-memory machines.

UR - http://www.scopus.com/inward/record.url?scp=0032025292&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0032025292&partnerID=8YFLogxK

M3 - Article

VL - 14

SP - 107

EP - 138

JO - Journal of Information Science and Engineering

JF - Journal of Information Science and Engineering

SN - 1016-2364

IS - 1

ER -