Stable Gradient Descent

Yingxue Zhou, Sheng Chen, Arindam Banerjee
Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, PMLR R16:765-774, 2018.

Abstract

The goal of many machine learning tasks is to learn a model that has small population risk. While mini-batch stochastic gradient descent (SGD) and variants are popular approaches for achieving this goal, it is hard to prescribe a clear stopping criterion and to establish high probability convergence bounds to the population risk. In this paper, we introduce Stable Gradient Descent which validates stochastic gradient computations by splitting data into training and validation sets and reuses samples using a differential private mechanism. StGD comes with a natural upper bound on the number of iterations and has high-probability convergence to the population risk. Experimental results illustrate that StGD is empirically competitive and often better than SGD and GD.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR16-zhou18a, title = {Stable Gradient Descent}, author = {Zhou, Yingxue and Chen, Sheng and Banerjee, Arindam}, booktitle = {Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence}, pages = {765--774}, year = {2018}, editor = {Globerson, Amir and Silva, Ricardo}, volume = {R16}, series = {Proceedings of Machine Learning Research}, month = {06--10 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r16/main/assets/zhou18a/zhou18a.pdf}, url = {https://proceedings.mlr.press/r16/zhou18a.html}, abstract = {The goal of many machine learning tasks is to learn a model that has small population risk. While mini-batch stochastic gradient descent (SGD) and variants are popular approaches for achieving this goal, it is hard to prescribe a clear stopping criterion and to establish high probability convergence bounds to the population risk. In this paper, we introduce Stable Gradient Descent which validates stochastic gradient computations by splitting data into training and validation sets and reuses samples using a differential private mechanism. StGD comes with a natural upper bound on the number of iterations and has high-probability convergence to the population risk. Experimental results illustrate that StGD is empirically competitive and often better than SGD and GD.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Stable Gradient Descent %A Yingxue Zhou %A Sheng Chen %A Arindam Banerjee %B Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2018 %E Amir Globerson %E Ricardo Silva %F pmlr-vR16-zhou18a %I PMLR %P 765--774 %U https://proceedings.mlr.press/r16/zhou18a.html %V R16 %X The goal of many machine learning tasks is to learn a model that has small population risk. While mini-batch stochastic gradient descent (SGD) and variants are popular approaches for achieving this goal, it is hard to prescribe a clear stopping criterion and to establish high probability convergence bounds to the population risk. In this paper, we introduce Stable Gradient Descent which validates stochastic gradient computations by splitting data into training and validation sets and reuses samples using a differential private mechanism. StGD comes with a natural upper bound on the number of iterations and has high-probability convergence to the population risk. Experimental results illustrate that StGD is empirically competitive and often better than SGD and GD. %Z Reissued by PMLR on 04 October 2026.
APA
Zhou, Y., Chen, S. & Banerjee, A.. (2018). Stable Gradient Descent. Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R16:765-774 Available from https://proceedings.mlr.press/r16/zhou18a.html. Reissued by PMLR on 04 October 2026.

Related Material