Cross-Iteration Coded Computing

Farzin Haddadpour, Yaoqing Yang, Viveck Ramesh Cadambe, Pulkit Grover

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

We introduce the idea of cross-iteration coded computing, an approach to reducing communication costs for a large class of distributed iterative algorithms involving linear operations, including gradient descent and accelerated gradient descent for quadratic loss functions. The state-of-the-art approach for these iterative algorithms involves performing one iteration of the algorithm per round of communication among the nodes. In contrast, our approach performs multiple iterations of the underlying algorithm in a single round of communication by incorporating some redundancy storage and computation. Our algorithm works in the master-worker setting with the workers storing carefully constructed linear transformations of input matrices and using these matrices in an iterative algorithm, with the master node inverting the effect of these linear transformations. In addition to reduced communication costs, a trivial generalization of our algorithm also includes resilience to stragglers and failures. The degree of redundancy of our algorithm can be tuned based on the amount of communication and straggler resilience required. Finally, we also describe a variant of our algorithm that can flexibly recover the results based on the degree of straggling in the worker nodes. The variant allows for the performance to degrade gracefully as the number of successful (non-straggling) workers is lowered.

Original languageEnglish (US)
Title of host publication2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages196-203
Number of pages8
ISBN (Electronic)9781538665961
DOIs
StatePublished - Feb 5 2019
Event56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018 - Monticello, United States
Duration: Oct 2 2018Oct 5 2018

Publication series

Name2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018

Conference

Conference56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018
CountryUnited States
CityMonticello
Period10/2/1810/5/18

Fingerprint

Iteration
Computing
Iterative Algorithm
Gradient Descent
Communication Cost
Resilience
Linear transformation
Communication
Redundancy
Linear transformations
Vertex of a graph
Quadratic Loss Function
Distributed Algorithms
Trivial
Costs

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Hardware and Architecture
  • Signal Processing
  • Energy Engineering and Power Technology
  • Control and Optimization

Cite this

Haddadpour, F., Yang, Y., Cadambe, V. R., & Grover, P. (2019). Cross-Iteration Coded Computing. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018 (pp. 196-203). [8635933] (2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ALLERTON.2018.8635933
Haddadpour, Farzin ; Yang, Yaoqing ; Cadambe, Viveck Ramesh ; Grover, Pulkit. / Cross-Iteration Coded Computing. 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 196-203 (2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018).
@inproceedings{5f422ee2e2eb487ab3fcbceb1b6d8057,
title = "Cross-Iteration Coded Computing",
abstract = "We introduce the idea of cross-iteration coded computing, an approach to reducing communication costs for a large class of distributed iterative algorithms involving linear operations, including gradient descent and accelerated gradient descent for quadratic loss functions. The state-of-the-art approach for these iterative algorithms involves performing one iteration of the algorithm per round of communication among the nodes. In contrast, our approach performs multiple iterations of the underlying algorithm in a single round of communication by incorporating some redundancy storage and computation. Our algorithm works in the master-worker setting with the workers storing carefully constructed linear transformations of input matrices and using these matrices in an iterative algorithm, with the master node inverting the effect of these linear transformations. In addition to reduced communication costs, a trivial generalization of our algorithm also includes resilience to stragglers and failures. The degree of redundancy of our algorithm can be tuned based on the amount of communication and straggler resilience required. Finally, we also describe a variant of our algorithm that can flexibly recover the results based on the degree of straggling in the worker nodes. The variant allows for the performance to degrade gracefully as the number of successful (non-straggling) workers is lowered.",
author = "Farzin Haddadpour and Yaoqing Yang and Cadambe, {Viveck Ramesh} and Pulkit Grover",
year = "2019",
month = "2",
day = "5",
doi = "10.1109/ALLERTON.2018.8635933",
language = "English (US)",
series = "2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "196--203",
booktitle = "2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018",
address = "United States",

}

Haddadpour, F, Yang, Y, Cadambe, VR & Grover, P 2019, Cross-Iteration Coded Computing. in 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018., 8635933, 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018, Institute of Electrical and Electronics Engineers Inc., pp. 196-203, 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018, Monticello, United States, 10/2/18. https://doi.org/10.1109/ALLERTON.2018.8635933

Cross-Iteration Coded Computing. / Haddadpour, Farzin; Yang, Yaoqing; Cadambe, Viveck Ramesh; Grover, Pulkit.

2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 196-203 8635933 (2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Cross-Iteration Coded Computing

AU - Haddadpour, Farzin

AU - Yang, Yaoqing

AU - Cadambe, Viveck Ramesh

AU - Grover, Pulkit

PY - 2019/2/5

Y1 - 2019/2/5

N2 - We introduce the idea of cross-iteration coded computing, an approach to reducing communication costs for a large class of distributed iterative algorithms involving linear operations, including gradient descent and accelerated gradient descent for quadratic loss functions. The state-of-the-art approach for these iterative algorithms involves performing one iteration of the algorithm per round of communication among the nodes. In contrast, our approach performs multiple iterations of the underlying algorithm in a single round of communication by incorporating some redundancy storage and computation. Our algorithm works in the master-worker setting with the workers storing carefully constructed linear transformations of input matrices and using these matrices in an iterative algorithm, with the master node inverting the effect of these linear transformations. In addition to reduced communication costs, a trivial generalization of our algorithm also includes resilience to stragglers and failures. The degree of redundancy of our algorithm can be tuned based on the amount of communication and straggler resilience required. Finally, we also describe a variant of our algorithm that can flexibly recover the results based on the degree of straggling in the worker nodes. The variant allows for the performance to degrade gracefully as the number of successful (non-straggling) workers is lowered.

AB - We introduce the idea of cross-iteration coded computing, an approach to reducing communication costs for a large class of distributed iterative algorithms involving linear operations, including gradient descent and accelerated gradient descent for quadratic loss functions. The state-of-the-art approach for these iterative algorithms involves performing one iteration of the algorithm per round of communication among the nodes. In contrast, our approach performs multiple iterations of the underlying algorithm in a single round of communication by incorporating some redundancy storage and computation. Our algorithm works in the master-worker setting with the workers storing carefully constructed linear transformations of input matrices and using these matrices in an iterative algorithm, with the master node inverting the effect of these linear transformations. In addition to reduced communication costs, a trivial generalization of our algorithm also includes resilience to stragglers and failures. The degree of redundancy of our algorithm can be tuned based on the amount of communication and straggler resilience required. Finally, we also describe a variant of our algorithm that can flexibly recover the results based on the degree of straggling in the worker nodes. The variant allows for the performance to degrade gracefully as the number of successful (non-straggling) workers is lowered.

UR - http://www.scopus.com/inward/record.url?scp=85062823050&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062823050&partnerID=8YFLogxK

U2 - 10.1109/ALLERTON.2018.8635933

DO - 10.1109/ALLERTON.2018.8635933

M3 - Conference contribution

AN - SCOPUS:85062823050

T3 - 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018

SP - 196

EP - 203

BT - 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Haddadpour F, Yang Y, Cadambe VR, Grover P. Cross-Iteration Coded Computing. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 196-203. 8635933. (2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018). https://doi.org/10.1109/ALLERTON.2018.8635933