Compositional Stochastic Average Gradient for Machine Learning and Related Applications

Tsung Yu Hsieh, Yasser EL-Manzalawy, Yiwei Sun, Vasant Honavar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.

Original languageEnglish (US)
Title of host publicationIntelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings
EditorsHujun Yin, Paulo Novais, David Camacho, Antonio J. Tallón-Ballesteros
PublisherSpringer Verlag
Pages740-752
Number of pages13
ISBN (Print)9783030034924
DOIs
StatePublished - Jan 1 2018
Event19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018 - Madrid, Spain
Duration: Nov 21 2018Nov 23 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11314 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018
CountrySpain
CityMadrid
Period11/21/1811/23/18

Fingerprint

Learning systems
Machine Learning
Gradient methods
Gradient
Gradient Descent
Chemical analysis
Gradient Method
Expected Value
Value Function
Data storage equipment
Gradient Descent Method
Query Complexity
Gradient Estimate
Linear Convergence
Statistical Inference
Experiments
Convex function
Convergence Rate
Objective function
Optimization Problem

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Hsieh, T. Y., EL-Manzalawy, Y., Sun, Y., & Honavar, V. (2018). Compositional Stochastic Average Gradient for Machine Learning and Related Applications. In H. Yin, P. Novais, D. Camacho, & A. J. Tallón-Ballesteros (Eds.), Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings (pp. 740-752). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11314 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-03493-1_77
Hsieh, Tsung Yu ; EL-Manzalawy, Yasser ; Sun, Yiwei ; Honavar, Vasant. / Compositional Stochastic Average Gradient for Machine Learning and Related Applications. Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings. editor / Hujun Yin ; Paulo Novais ; David Camacho ; Antonio J. Tallón-Ballesteros. Springer Verlag, 2018. pp. 740-752 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{3a6e9a48567148bc8e7ffc004bb13db7,
title = "Compositional Stochastic Average Gradient for Machine Learning and Related Applications",
abstract = "Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.",
author = "Hsieh, {Tsung Yu} and Yasser EL-Manzalawy and Yiwei Sun and Vasant Honavar",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-03493-1_77",
language = "English (US)",
isbn = "9783030034924",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "740--752",
editor = "Hujun Yin and Paulo Novais and David Camacho and Tall{\'o}n-Ballesteros, {Antonio J.}",
booktitle = "Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings",
address = "Germany",

}

Hsieh, TY, EL-Manzalawy, Y, Sun, Y & Honavar, V 2018, Compositional Stochastic Average Gradient for Machine Learning and Related Applications. in H Yin, P Novais, D Camacho & AJ Tallón-Ballesteros (eds), Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11314 LNCS, Springer Verlag, pp. 740-752, 19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018, Madrid, Spain, 11/21/18. https://doi.org/10.1007/978-3-030-03493-1_77

Compositional Stochastic Average Gradient for Machine Learning and Related Applications. / Hsieh, Tsung Yu; EL-Manzalawy, Yasser; Sun, Yiwei; Honavar, Vasant.

Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings. ed. / Hujun Yin; Paulo Novais; David Camacho; Antonio J. Tallón-Ballesteros. Springer Verlag, 2018. p. 740-752 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11314 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Compositional Stochastic Average Gradient for Machine Learning and Related Applications

AU - Hsieh, Tsung Yu

AU - EL-Manzalawy, Yasser

AU - Sun, Yiwei

AU - Honavar, Vasant

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.

AB - Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.

UR - http://www.scopus.com/inward/record.url?scp=85057092729&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057092729&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-03493-1_77

DO - 10.1007/978-3-030-03493-1_77

M3 - Conference contribution

AN - SCOPUS:85057092729

SN - 9783030034924

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 740

EP - 752

BT - Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings

A2 - Yin, Hujun

A2 - Novais, Paulo

A2 - Camacho, David

A2 - Tallón-Ballesteros, Antonio J.

PB - Springer Verlag

ER -

Hsieh TY, EL-Manzalawy Y, Sun Y, Honavar V. Compositional Stochastic Average Gradient for Machine Learning and Related Applications. In Yin H, Novais P, Camacho D, Tallón-Ballesteros AJ, editors, Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings. Springer Verlag. 2018. p. 740-752. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-03493-1_77