Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization

Giuseppe Bruno De Luca, Eva Silverstein
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:4918-4936, 2022.

Abstract

We introduce a novel framework for optimization based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its key properties analytically and numerically. The prototype is a discretization of Born-Infeld dynamics, with a squared relativistic speed limit depending on the objective function. This class of frictionless, energy-conserving optimizers proceeds unobstructed until slowing naturally near the minimal loss, which dominates the phase space volume of the system. Building from studies of chaotic systems such as dynamical billiards, we formulate a specific algorithm with good performance on machine learning and PDE-solving tasks, including generalization. It cannot stop at a high local minimum, an advantage in non-convex loss functions, and proceeds faster than GD+momentum in shallow valleys.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-de-luca22a, title = {Born-Infeld ({BI}) for {AI}: Energy-Conserving Descent ({ECD}) for Optimization}, author = {De Luca, Giuseppe Bruno and Silverstein, Eva}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {4918--4936}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/de-luca22a/de-luca22a.pdf}, url = {https://proceedings.mlr.press/v162/de-luca22a.html}, abstract = {We introduce a novel framework for optimization based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its key properties analytically and numerically. The prototype is a discretization of Born-Infeld dynamics, with a squared relativistic speed limit depending on the objective function. This class of frictionless, energy-conserving optimizers proceeds unobstructed until slowing naturally near the minimal loss, which dominates the phase space volume of the system. Building from studies of chaotic systems such as dynamical billiards, we formulate a specific algorithm with good performance on machine learning and PDE-solving tasks, including generalization. It cannot stop at a high local minimum, an advantage in non-convex loss functions, and proceeds faster than GD+momentum in shallow valleys.} }
Endnote
%0 Conference Paper %T Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization %A Giuseppe Bruno De Luca %A Eva Silverstein %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-de-luca22a %I PMLR %P 4918--4936 %U https://proceedings.mlr.press/v162/de-luca22a.html %V 162 %X We introduce a novel framework for optimization based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its key properties analytically and numerically. The prototype is a discretization of Born-Infeld dynamics, with a squared relativistic speed limit depending on the objective function. This class of frictionless, energy-conserving optimizers proceeds unobstructed until slowing naturally near the minimal loss, which dominates the phase space volume of the system. Building from studies of chaotic systems such as dynamical billiards, we formulate a specific algorithm with good performance on machine learning and PDE-solving tasks, including generalization. It cannot stop at a high local minimum, an advantage in non-convex loss functions, and proceeds faster than GD+momentum in shallow valleys.
APA
De Luca, G.B. & Silverstein, E.. (2022). Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:4918-4936 Available from https://proceedings.mlr.press/v162/de-luca22a.html.

Related Material