Abstract
Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.
| Original language | English (US) |
|---|---|
| Title of host publication | Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings |
| Editors | Hujun Yin, Paulo Novais, David Camacho, Antonio J. Tallón-Ballesteros |
| Publisher | Springer Verlag |
| Pages | 740-752 |
| Number of pages | 13 |
| ISBN (Print) | 9783030034924 |
| DOIs | |
| State | Published - Jan 1 2018 |
| Event | 19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018 - Madrid, Spain Duration: Nov 21 2018 → Nov 23 2018 |
Publication series
| Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 11314 LNCS |
| ISSN (Print) | 0302-9743 |
| ISSN (Electronic) | 1611-3349 |
Other
| Other | 19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018 |
|---|---|
| Country | Spain |
| City | Madrid |
| Period | 11/21/18 → 11/23/18 |
Fingerprint
All Science Journal Classification (ASJC) codes
- Theoretical Computer Science
- Computer Science(all)
Cite this
}
Compositional Stochastic Average Gradient for Machine Learning and Related Applications. / Hsieh, Tsung Yu; EL-Manzalawy, Yasser; Sun, Yiwei; Honavar, Vasant.
Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings. ed. / Hujun Yin; Paulo Novais; David Camacho; Antonio J. Tallón-Ballesteros. Springer Verlag, 2018. p. 740-752 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11314 LNCS).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
TY - GEN
T1 - Compositional Stochastic Average Gradient for Machine Learning and Related Applications
AU - Hsieh, Tsung Yu
AU - EL-Manzalawy, Yasser
AU - Sun, Yiwei
AU - Honavar, Vasant
PY - 2018/1/1
Y1 - 2018/1/1
N2 - Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.
AB - Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.
UR - http://www.scopus.com/inward/record.url?scp=85057092729&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85057092729&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-03493-1_77
DO - 10.1007/978-3-030-03493-1_77
M3 - Conference contribution
AN - SCOPUS:85057092729
SN - 9783030034924
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 740
EP - 752
BT - Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings
A2 - Yin, Hujun
A2 - Novais, Paulo
A2 - Camacho, David
A2 - Tallón-Ballesteros, Antonio J.
PB - Springer Verlag
ER -