Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows

Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:34790-34821, 2024.

Abstract

Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles for non-Euclidean geometries and momentum-based dynamics remain largely unknown. In this paper, we characterize "all" conservation laws in this general setting. In stark contrast to the case of gradient flows, we prove that the conservation laws for momentum-based dynamics exhibit temporal dependence. Additionally, we often observe a "conservation loss" when transitioning from gradient flow to momentum dynamics. Specifically, for linear networks, our framework allows us to identify all momentum conservation laws, which are less numerous than in the gradient flow case except in sufficiently over-parameterized regimes. With ReLU networks, no conservation law remains. This phenomenon also manifests in non-Euclidean metrics, used e.g. for Nonnegative Matrix Factorization (NMF): all conservation laws can be determined in the gradient flow context, yet none persists in the momentum case.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-marcotte24a, title = {Keep the Momentum: Conservation Laws beyond {E}uclidean Gradient Flows}, author = {Marcotte, Sibylle and Gribonval, R\'{e}mi and Peyr\'{e}, Gabriel}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {34790--34821}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/marcotte24a/marcotte24a.pdf}, url = {https://proceedings.mlr.press/v235/marcotte24a.html}, abstract = {Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles for non-Euclidean geometries and momentum-based dynamics remain largely unknown. In this paper, we characterize "all" conservation laws in this general setting. In stark contrast to the case of gradient flows, we prove that the conservation laws for momentum-based dynamics exhibit temporal dependence. Additionally, we often observe a "conservation loss" when transitioning from gradient flow to momentum dynamics. Specifically, for linear networks, our framework allows us to identify all momentum conservation laws, which are less numerous than in the gradient flow case except in sufficiently over-parameterized regimes. With ReLU networks, no conservation law remains. This phenomenon also manifests in non-Euclidean metrics, used e.g. for Nonnegative Matrix Factorization (NMF): all conservation laws can be determined in the gradient flow context, yet none persists in the momentum case.} }
Endnote
%0 Conference Paper %T Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows %A Sibylle Marcotte %A Rémi Gribonval %A Gabriel Peyré %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-marcotte24a %I PMLR %P 34790--34821 %U https://proceedings.mlr.press/v235/marcotte24a.html %V 235 %X Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles for non-Euclidean geometries and momentum-based dynamics remain largely unknown. In this paper, we characterize "all" conservation laws in this general setting. In stark contrast to the case of gradient flows, we prove that the conservation laws for momentum-based dynamics exhibit temporal dependence. Additionally, we often observe a "conservation loss" when transitioning from gradient flow to momentum dynamics. Specifically, for linear networks, our framework allows us to identify all momentum conservation laws, which are less numerous than in the gradient flow case except in sufficiently over-parameterized regimes. With ReLU networks, no conservation law remains. This phenomenon also manifests in non-Euclidean metrics, used e.g. for Nonnegative Matrix Factorization (NMF): all conservation laws can be determined in the gradient flow context, yet none persists in the momentum case.
APA
Marcotte, S., Gribonval, R. & Peyré, G.. (2024). Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:34790-34821 Available from https://proceedings.mlr.press/v235/marcotte24a.html.

Related Material