Stochastic Dual Coordinate Ascent with Adaptive Probabilities

Dominik Csiba, Zheng Qu, Peter Richtarik
; Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:674-683, 2015.

Abstract

This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSDCA achieves provably better complexity bound than SDCA with the best fixed probability distribution, known as importance sampling. However, it is of a theoretical character as it is expensive to implement. We also propose AdaSDCA+: a practical variant which in our experiments outperforms existing non-adaptive methods.

Cite this Paper


BibTeX
@InProceedings{pmlr-v37-csiba15, title = {Stochastic Dual Coordinate Ascent with Adaptive Probabilities}, author = {Dominik Csiba and Zheng Qu and Peter Richtarik}, pages = {674--683}, year = {2015}, editor = {Francis Bach and David Blei}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/csiba15.pdf}, url = {http://proceedings.mlr.press/v37/csiba15.html}, abstract = {This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSDCA achieves provably better complexity bound than SDCA with the best fixed probability distribution, known as importance sampling. However, it is of a theoretical character as it is expensive to implement. We also propose AdaSDCA+: a practical variant which in our experiments outperforms existing non-adaptive methods.} }
Endnote
%0 Conference Paper %T Stochastic Dual Coordinate Ascent with Adaptive Probabilities %A Dominik Csiba %A Zheng Qu %A Peter Richtarik %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-csiba15 %I PMLR %J Proceedings of Machine Learning Research %P 674--683 %U http://proceedings.mlr.press %V 37 %W PMLR %X This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSDCA achieves provably better complexity bound than SDCA with the best fixed probability distribution, known as importance sampling. However, it is of a theoretical character as it is expensive to implement. We also propose AdaSDCA+: a practical variant which in our experiments outperforms existing non-adaptive methods.
RIS
TY - CPAPER TI - Stochastic Dual Coordinate Ascent with Adaptive Probabilities AU - Dominik Csiba AU - Zheng Qu AU - Peter Richtarik BT - Proceedings of the 32nd International Conference on Machine Learning PY - 2015/06/01 DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-csiba15 PB - PMLR SP - 674 DP - PMLR EP - 683 L1 - http://proceedings.mlr.press/v37/csiba15.pdf UR - http://proceedings.mlr.press/v37/csiba15.html AB - This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSDCA achieves provably better complexity bound than SDCA with the best fixed probability distribution, known as importance sampling. However, it is of a theoretical character as it is expensive to implement. We also propose AdaSDCA+: a practical variant which in our experiments outperforms existing non-adaptive methods. ER -
APA
Csiba, D., Qu, Z. & Richtarik, P.. (2015). Stochastic Dual Coordinate Ascent with Adaptive Probabilities. Proceedings of the 32nd International Conference on Machine Learning, in PMLR 37:674-683

Related Material