Accelerated Stochastic Gradient Method for Composite Regularization

Wenliang Zhong, James Kwok
Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, PMLR 33:1086-1094, 2014.

Abstract

Regularized risk minimization often involves nonsmooth optimization. This can be particularly challenging when the regularizer is a sum of simpler regularizers, as in the overlapping group lasso. Very recently, this is alleviated by using the proximal average, in which an implicitly nonsmooth function is employed to approximate the composite regularizer. In this paper, we propose a novel extension with accelerated gradient method for stochastic optimization. On both general convex and strongly convex problems, the resultant approximation errors reduce at a faster rate than methods based on stochastic smoothing and ADMM. This is also verified experimentally on a number of synthetic and real-world data sets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v33-zhong14, title = {{Accelerated Stochastic Gradient Method for Composite Regularization}}, author = {Zhong, Wenliang and Kwok, James}, booktitle = {Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics}, pages = {1086--1094}, year = {2014}, editor = {Kaski, Samuel and Corander, Jukka}, volume = {33}, series = {Proceedings of Machine Learning Research}, address = {Reykjavik, Iceland}, month = {22--25 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v33/zhong14.pdf}, url = {https://proceedings.mlr.press/v33/zhong14.html}, abstract = {Regularized risk minimization often involves nonsmooth optimization. This can be particularly challenging when the regularizer is a sum of simpler regularizers, as in the overlapping group lasso. Very recently, this is alleviated by using the proximal average, in which an implicitly nonsmooth function is employed to approximate the composite regularizer. In this paper, we propose a novel extension with accelerated gradient method for stochastic optimization. On both general convex and strongly convex problems, the resultant approximation errors reduce at a faster rate than methods based on stochastic smoothing and ADMM. This is also verified experimentally on a number of synthetic and real-world data sets.} }
Endnote
%0 Conference Paper %T Accelerated Stochastic Gradient Method for Composite Regularization %A Wenliang Zhong %A James Kwok %B Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2014 %E Samuel Kaski %E Jukka Corander %F pmlr-v33-zhong14 %I PMLR %P 1086--1094 %U https://proceedings.mlr.press/v33/zhong14.html %V 33 %X Regularized risk minimization often involves nonsmooth optimization. This can be particularly challenging when the regularizer is a sum of simpler regularizers, as in the overlapping group lasso. Very recently, this is alleviated by using the proximal average, in which an implicitly nonsmooth function is employed to approximate the composite regularizer. In this paper, we propose a novel extension with accelerated gradient method for stochastic optimization. On both general convex and strongly convex problems, the resultant approximation errors reduce at a faster rate than methods based on stochastic smoothing and ADMM. This is also verified experimentally on a number of synthetic and real-world data sets.
RIS
TY - CPAPER TI - Accelerated Stochastic Gradient Method for Composite Regularization AU - Wenliang Zhong AU - James Kwok BT - Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics DA - 2014/04/02 ED - Samuel Kaski ED - Jukka Corander ID - pmlr-v33-zhong14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 33 SP - 1086 EP - 1094 L1 - http://proceedings.mlr.press/v33/zhong14.pdf UR - https://proceedings.mlr.press/v33/zhong14.html AB - Regularized risk minimization often involves nonsmooth optimization. This can be particularly challenging when the regularizer is a sum of simpler regularizers, as in the overlapping group lasso. Very recently, this is alleviated by using the proximal average, in which an implicitly nonsmooth function is employed to approximate the composite regularizer. In this paper, we propose a novel extension with accelerated gradient method for stochastic optimization. On both general convex and strongly convex problems, the resultant approximation errors reduce at a faster rate than methods based on stochastic smoothing and ADMM. This is also verified experimentally on a number of synthetic and real-world data sets. ER -
APA
Zhong, W. & Kwok, J.. (2014). Accelerated Stochastic Gradient Method for Composite Regularization. Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 33:1086-1094 Available from https://proceedings.mlr.press/v33/zhong14.html.

Related Material