SAM operates far from home: eigenvalue regularization as a dynamical phenomenon

Atish Agarwala, Yann Dauphin
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:152-168, 2023.

Abstract

The Sharpness Aware Minimization (SAM) optimization algorithm has been shown to control large eigenvalues of the loss Hessian and provide generalization benefits in a variety of settings. The original motivation for SAM was a modified loss function which penalized sharp minima; subsequent analyses have also focused on the behavior near minima. However, our work reveals that SAM provides a strong regularization of the eigenvalues throughout the learning trajectory. We show that in a simplified setting, SAM dynamically induces a stabilization related to the edge of stability (EOS) phenomenon observed in large learning rate gradient descent. Our theory predicts the largest eigenvalue as a function of the learning rate and SAM radius parameters. Finally, we show that practical models can also exhibit this EOS stabilization, and that understanding SAM must account for these dynamics far away from any minima.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-agarwala23a, title = {{SAM} operates far from home: eigenvalue regularization as a dynamical phenomenon}, author = {Agarwala, Atish and Dauphin, Yann}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {152--168}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/agarwala23a/agarwala23a.pdf}, url = {https://proceedings.mlr.press/v202/agarwala23a.html}, abstract = {The Sharpness Aware Minimization (SAM) optimization algorithm has been shown to control large eigenvalues of the loss Hessian and provide generalization benefits in a variety of settings. The original motivation for SAM was a modified loss function which penalized sharp minima; subsequent analyses have also focused on the behavior near minima. However, our work reveals that SAM provides a strong regularization of the eigenvalues throughout the learning trajectory. We show that in a simplified setting, SAM dynamically induces a stabilization related to the edge of stability (EOS) phenomenon observed in large learning rate gradient descent. Our theory predicts the largest eigenvalue as a function of the learning rate and SAM radius parameters. Finally, we show that practical models can also exhibit this EOS stabilization, and that understanding SAM must account for these dynamics far away from any minima.} }
Endnote
%0 Conference Paper %T SAM operates far from home: eigenvalue regularization as a dynamical phenomenon %A Atish Agarwala %A Yann Dauphin %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-agarwala23a %I PMLR %P 152--168 %U https://proceedings.mlr.press/v202/agarwala23a.html %V 202 %X The Sharpness Aware Minimization (SAM) optimization algorithm has been shown to control large eigenvalues of the loss Hessian and provide generalization benefits in a variety of settings. The original motivation for SAM was a modified loss function which penalized sharp minima; subsequent analyses have also focused on the behavior near minima. However, our work reveals that SAM provides a strong regularization of the eigenvalues throughout the learning trajectory. We show that in a simplified setting, SAM dynamically induces a stabilization related to the edge of stability (EOS) phenomenon observed in large learning rate gradient descent. Our theory predicts the largest eigenvalue as a function of the learning rate and SAM radius parameters. Finally, we show that practical models can also exhibit this EOS stabilization, and that understanding SAM must account for these dynamics far away from any minima.
APA
Agarwala, A. & Dauphin, Y.. (2023). SAM operates far from home: eigenvalue regularization as a dynamical phenomenon. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:152-168 Available from https://proceedings.mlr.press/v202/agarwala23a.html.

Related Material