Towards Flatter Loss Surface via Nonmonotonic Learning Rate Scheduling

Sihyeon Seong, Yegang Lee, Youngwook Kee, Dongyoon Han, Junmo Kim
Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, PMLR R16:1019-1029, 2018.

Abstract

Whereas optimizing deep neural networks using stochastic gradient descent has shown great performances in practice, the rule for setting step size (i.e. learning rate) of gradient descent is not well studied. Although it appears that some intriguing learning rate rules such as ADAM (Kingma and Ba, 2014) have since been developed, they concentrated on improving convergence, not on improving generalization capabilities. Recently, the improved generalization property of the flat minima was revisited, and this research guides us towards promising solutions to many current optimization problems. In this paper, we analyze the flatness of loss surfaces through the lens of robustness to input perturbations and advocate that gradient descent should be guided to reach flatter region of loss surfaces to achieve generalization. Finally, we suggest a learning rate rule for escaping sharp regions of loss surfaces, and we demonstrate the capacity of our approach by performing numerous experiments.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR16-seong18a, title = {Towards Flatter Loss Surface via Nonmonotonic Learning Rate Scheduling}, author = {Seong, Sihyeon and Lee, Yegang and Kee, Youngwook and Han, Dongyoon and Kim, Junmo}, booktitle = {Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence}, pages = {1019--1029}, year = {2018}, editor = {Globerson, Amir and Silva, Ricardo}, volume = {R16}, series = {Proceedings of Machine Learning Research}, month = {06--10 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r16/main/assets/seong18a/seong18a.pdf}, url = {https://proceedings.mlr.press/r16/seong18a.html}, abstract = {Whereas optimizing deep neural networks using stochastic gradient descent has shown great performances in practice, the rule for setting step size (i.e. learning rate) of gradient descent is not well studied. Although it appears that some intriguing learning rate rules such as ADAM (Kingma and Ba, 2014) have since been developed, they concentrated on improving convergence, not on improving generalization capabilities. Recently, the improved generalization property of the flat minima was revisited, and this research guides us towards promising solutions to many current optimization problems. In this paper, we analyze the flatness of loss surfaces through the lens of robustness to input perturbations and advocate that gradient descent should be guided to reach flatter region of loss surfaces to achieve generalization. Finally, we suggest a learning rate rule for escaping sharp regions of loss surfaces, and we demonstrate the capacity of our approach by performing numerous experiments.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Towards Flatter Loss Surface via Nonmonotonic Learning Rate Scheduling %A Sihyeon Seong %A Yegang Lee %A Youngwook Kee %A Dongyoon Han %A Junmo Kim %B Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2018 %E Amir Globerson %E Ricardo Silva %F pmlr-vR16-seong18a %I PMLR %P 1019--1029 %U https://proceedings.mlr.press/r16/seong18a.html %V R16 %X Whereas optimizing deep neural networks using stochastic gradient descent has shown great performances in practice, the rule for setting step size (i.e. learning rate) of gradient descent is not well studied. Although it appears that some intriguing learning rate rules such as ADAM (Kingma and Ba, 2014) have since been developed, they concentrated on improving convergence, not on improving generalization capabilities. Recently, the improved generalization property of the flat minima was revisited, and this research guides us towards promising solutions to many current optimization problems. In this paper, we analyze the flatness of loss surfaces through the lens of robustness to input perturbations and advocate that gradient descent should be guided to reach flatter region of loss surfaces to achieve generalization. Finally, we suggest a learning rate rule for escaping sharp regions of loss surfaces, and we demonstrate the capacity of our approach by performing numerous experiments. %Z Reissued by PMLR on 04 October 2026.
APA
Seong, S., Lee, Y., Kee, Y., Han, D. & Kim, J.. (2018). Towards Flatter Loss Surface via Nonmonotonic Learning Rate Scheduling. Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R16:1019-1029 Available from https://proceedings.mlr.press/r16/seong18a.html. Reissued by PMLR on 04 October 2026.

Related Material