A simpler approach to accelerated optimization: iterative averaging meets optimism

Pooria Joulani, Anant Raj, Andras Gyorgy, Csaba Szepesvari
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4984-4993, 2020.

Abstract

Recently there have been several attempts to extend Nesterov’s accelerated algorithm to smooth stochastic and variance-reduced optimization. In this paper, we show that there is a simpler approach to acceleration: applying optimistic online learning algorithms and querying the gradient oracle at the online average of the intermediate optimization iterates. In particular, we tighten a recent result of Cutkosky (2019) to demonstrate theoretically that online iterate averaging results in a reduced optimization gap, independently of the algorithm involved. We show that carefully combining this technique with existing generic optimistic online learning algorithms yields the optimal accelerated rates for optimizing strongly-convex and non-strongly-convex, possibly composite objectives, with deterministic as well as stochastic first-order oracles. We further extend this idea to variance-reduced optimization. Finally, we also provide “universal” algorithms that achieve the optimal rate for smooth and non-smooth composite objectives simultaneously without further tuning, generalizing the results of Kavis et al. (2019) and solving a number of their open problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-joulani20a, title = {A simpler approach to accelerated optimization: iterative averaging meets optimism}, author = {Joulani, Pooria and Raj, Anant and Gyorgy, Andras and Szepesvari, Csaba}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {4984--4993}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/joulani20a/joulani20a.pdf}, url = {https://proceedings.mlr.press/v119/joulani20a.html}, abstract = {Recently there have been several attempts to extend Nesterov’s accelerated algorithm to smooth stochastic and variance-reduced optimization. In this paper, we show that there is a simpler approach to acceleration: applying optimistic online learning algorithms and querying the gradient oracle at the online average of the intermediate optimization iterates. In particular, we tighten a recent result of Cutkosky (2019) to demonstrate theoretically that online iterate averaging results in a reduced optimization gap, independently of the algorithm involved. We show that carefully combining this technique with existing generic optimistic online learning algorithms yields the optimal accelerated rates for optimizing strongly-convex and non-strongly-convex, possibly composite objectives, with deterministic as well as stochastic first-order oracles. We further extend this idea to variance-reduced optimization. Finally, we also provide “universal” algorithms that achieve the optimal rate for smooth and non-smooth composite objectives simultaneously without further tuning, generalizing the results of Kavis et al. (2019) and solving a number of their open problems.} }
Endnote
%0 Conference Paper %T A simpler approach to accelerated optimization: iterative averaging meets optimism %A Pooria Joulani %A Anant Raj %A Andras Gyorgy %A Csaba Szepesvari %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-joulani20a %I PMLR %P 4984--4993 %U https://proceedings.mlr.press/v119/joulani20a.html %V 119 %X Recently there have been several attempts to extend Nesterov’s accelerated algorithm to smooth stochastic and variance-reduced optimization. In this paper, we show that there is a simpler approach to acceleration: applying optimistic online learning algorithms and querying the gradient oracle at the online average of the intermediate optimization iterates. In particular, we tighten a recent result of Cutkosky (2019) to demonstrate theoretically that online iterate averaging results in a reduced optimization gap, independently of the algorithm involved. We show that carefully combining this technique with existing generic optimistic online learning algorithms yields the optimal accelerated rates for optimizing strongly-convex and non-strongly-convex, possibly composite objectives, with deterministic as well as stochastic first-order oracles. We further extend this idea to variance-reduced optimization. Finally, we also provide “universal” algorithms that achieve the optimal rate for smooth and non-smooth composite objectives simultaneously without further tuning, generalizing the results of Kavis et al. (2019) and solving a number of their open problems.
APA
Joulani, P., Raj, A., Gyorgy, A. & Szepesvari, C.. (2020). A simpler approach to accelerated optimization: iterative averaging meets optimism. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:4984-4993 Available from https://proceedings.mlr.press/v119/joulani20a.html.

Related Material