Mixed optimization for smooth functions

Mehrdad Mahdavi, Lijun Zhang, Rong Jin

Research output: Contribution to journalConference articlepeer-review

21 Scopus citations

Abstract

It is well known that the optimal convergence rate for stochastic optimization of smooth functions is O(1= √T), which is same as stochastic optimization of Lips-chitz continuous convex functions. This is in contrast to optimizing smooth functions using full gradients, which yields a convergence rate of O(1=T 2). In this work, we consider a new setup for optimizing smooth functions, termed as Mixed Optimization, which allows to access both a stochastic oracle and a full gradient oracle. Our goal is to significantly improve the convergence rate of stochastic optimization of smooth functions by having an additional small number of accesses to the full gradient oracle. We show that, with an O(ln T) calls to the full gradient oracle and an O(T) calls to the stochastic oracle, the proposed mixed optimization algorithm is able to achieve an optimization error of O(1=T ).

Original languageEnglish (US)
JournalAdvances in Neural Information Processing Systems
StatePublished - Jan 1 2013
Event27th Annual Conference on Neural Information Processing Systems, NIPS 2013 - Lake Tahoe, NV, United States
Duration: Dec 5 2013Dec 10 2013

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this