Accelerated Stochastic Gradient Method for Composite Regularization


Wenliang Zhong, James Kwok ;
Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, PMLR 33:1086-1094, 2014.


Regularized risk minimization often involves nonsmooth optimization. This can be particularly challenging when the regularizer is a sum of simpler regularizers, as in the overlapping group lasso. Very recently, this is alleviated by using the proximal average, in which an implicitly nonsmooth function is employed to approximate the composite regularizer. In this paper, we propose a novel extension with accelerated gradient method for stochastic optimization. On both general convex and strongly convex problems, the resultant approximation errors reduce at a faster rate than methods based on stochastic smoothing and ADMM. This is also verified experimentally on a number of synthetic and real-world data sets.

Related Material