### Abstract

Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.

Original language | English (US) |
---|---|

Title of host publication | Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings |

Editors | Hujun Yin, Paulo Novais, David Camacho, Antonio J. Tallón-Ballesteros |

Publisher | Springer Verlag |

Pages | 740-752 |

Number of pages | 13 |

ISBN (Print) | 9783030034924 |

DOIs | |

State | Published - Jan 1 2018 |

Event | 19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018 - Madrid, Spain Duration: Nov 21 2018 → Nov 23 2018 |

### Publication series

Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
---|---|

Volume | 11314 LNCS |

ISSN (Print) | 0302-9743 |

ISSN (Electronic) | 1611-3349 |

### Other

Other | 19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018 |
---|---|

Country | Spain |

City | Madrid |

Period | 11/21/18 → 11/23/18 |

### Fingerprint

### All Science Journal Classification (ASJC) codes

- Theoretical Computer Science
- Computer Science(all)

### Cite this

*Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings*(pp. 740-752). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11314 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-03493-1_77

}

*Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings.*Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11314 LNCS, Springer Verlag, pp. 740-752, 19th International Conference on Intelligent Data Engineering and Automated Learning, IDEAL 2018, Madrid, Spain, 11/21/18. https://doi.org/10.1007/978-3-030-03493-1_77

**Compositional Stochastic Average Gradient for Machine Learning and Related Applications.** / Hsieh, Tsung Yu; EL-Manzalawy, Yasser; Sun, Yiwei; Honavar, Vasant.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Compositional Stochastic Average Gradient for Machine Learning and Related Applications

AU - Hsieh, Tsung Yu

AU - EL-Manzalawy, Yasser

AU - Sun, Yiwei

AU - Honavar, Vasant

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.

AB - Many machine learning, and statistical inference problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG) methods that combine stochastic compositional gradient descent (SCGD) and stochastic variance reduced gradient descent (SVRG) methods are the state-of-the-art methods for FS-CEVF problems. We introduce compositional stochastic average gradient descent (C-SAG) a novel extension of the stochastic average gradient method (SAG) to minimize composition of finite-sum functions. C-SAG, like SAG, estimates gradient by incorporating memory of previous gradient information. We present theoretical analyses of C-SAG which show that C-SAG, like C-SVRG, achieves a linear convergence rate for strongly convex objective function; However, C-CAG achieves lower oracle query complexity per iteration than C-SVRG. Finally, we present results of experiments showing that C-SAG converges substantially faster than full gradient (FG), as well as C-SVRG.

UR - http://www.scopus.com/inward/record.url?scp=85057092729&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057092729&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-03493-1_77

DO - 10.1007/978-3-030-03493-1_77

M3 - Conference contribution

AN - SCOPUS:85057092729

SN - 9783030034924

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 740

EP - 752

BT - Intelligent Data Engineering and Automated Learning – IDEAL 2018 - 19th International Conference, Proceedings

A2 - Yin, Hujun

A2 - Novais, Paulo

A2 - Camacho, David

A2 - Tallón-Ballesteros, Antonio J.

PB - Springer Verlag

ER -