From Learning to Optimize to Learning Optimization Algorithms

Camille Castera, Peter Ochs
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:1792-1800, 2025.

Abstract

Towards designing learned optimization algorithms that are usable beyond their training setting, we identify key principles that classical algorithms obey, but have up to now, not been used for Learning to Optimize (L2O). Following these principles, we provide a general design pipeline, taking into account data, architecture and learning strategy, and thereby enabling a synergy between classical optimization and L2O, resulting in a philosophy of Learning Optimization Algorithms. As a consequence our learned algorithms perform well far beyond problems from the training distribution. We demonstrate the success of these novel principles by designing a new learning-enhanced BFGS algorithm and provide numerical experiments evidencing its adaptation to many settings at test time.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-castera25a, title = {From Learning to Optimize to Learning Optimization Algorithms}, author = {Castera, Camille and Ochs, Peter}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {1792--1800}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/castera25a/castera25a.pdf}, url = {https://proceedings.mlr.press/v258/castera25a.html}, abstract = {Towards designing learned optimization algorithms that are usable beyond their training setting, we identify key principles that classical algorithms obey, but have up to now, not been used for Learning to Optimize (L2O). Following these principles, we provide a general design pipeline, taking into account data, architecture and learning strategy, and thereby enabling a synergy between classical optimization and L2O, resulting in a philosophy of Learning Optimization Algorithms. As a consequence our learned algorithms perform well far beyond problems from the training distribution. We demonstrate the success of these novel principles by designing a new learning-enhanced BFGS algorithm and provide numerical experiments evidencing its adaptation to many settings at test time.} }
Endnote
%0 Conference Paper %T From Learning to Optimize to Learning Optimization Algorithms %A Camille Castera %A Peter Ochs %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-castera25a %I PMLR %P 1792--1800 %U https://proceedings.mlr.press/v258/castera25a.html %V 258 %X Towards designing learned optimization algorithms that are usable beyond their training setting, we identify key principles that classical algorithms obey, but have up to now, not been used for Learning to Optimize (L2O). Following these principles, we provide a general design pipeline, taking into account data, architecture and learning strategy, and thereby enabling a synergy between classical optimization and L2O, resulting in a philosophy of Learning Optimization Algorithms. As a consequence our learned algorithms perform well far beyond problems from the training distribution. We demonstrate the success of these novel principles by designing a new learning-enhanced BFGS algorithm and provide numerical experiments evidencing its adaptation to many settings at test time.
APA
Castera, C. & Ochs, P.. (2025). From Learning to Optimize to Learning Optimization Algorithms. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:1792-1800 Available from https://proceedings.mlr.press/v258/castera25a.html.

Related Material