Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness

Konstantinos Oikonomidis, Jan Quan, Emanuel Laude, Panagiotis Patrinos
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:47132-47154, 2025.

Abstract

We analyze nonlinearly preconditioned gradient methods for solving smooth minimization problems. We introduce a generalized smoothness property, based on the notion of abstract convexity, that is broader than Lipschitz smoothness and provide sufficient first- and second-order conditions. Notably, our framework encapsulates algorithms associated with the gradient clipping method and brings out novel insights for the class of $(L_0,L_1)$-smooth functions that has received widespread interest recently, thus allowing us to extend beyond already established methods. We investigate the convergence of the proposed method in both the convex and nonconvex setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-oikonomidis25a, title = {Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness}, author = {Oikonomidis, Konstantinos and Quan, Jan and Laude, Emanuel and Patrinos, Panagiotis}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {47132--47154}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/oikonomidis25a/oikonomidis25a.pdf}, url = {https://proceedings.mlr.press/v267/oikonomidis25a.html}, abstract = {We analyze nonlinearly preconditioned gradient methods for solving smooth minimization problems. We introduce a generalized smoothness property, based on the notion of abstract convexity, that is broader than Lipschitz smoothness and provide sufficient first- and second-order conditions. Notably, our framework encapsulates algorithms associated with the gradient clipping method and brings out novel insights for the class of $(L_0,L_1)$-smooth functions that has received widespread interest recently, thus allowing us to extend beyond already established methods. We investigate the convergence of the proposed method in both the convex and nonconvex setting.} }
Endnote
%0 Conference Paper %T Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness %A Konstantinos Oikonomidis %A Jan Quan %A Emanuel Laude %A Panagiotis Patrinos %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-oikonomidis25a %I PMLR %P 47132--47154 %U https://proceedings.mlr.press/v267/oikonomidis25a.html %V 267 %X We analyze nonlinearly preconditioned gradient methods for solving smooth minimization problems. We introduce a generalized smoothness property, based on the notion of abstract convexity, that is broader than Lipschitz smoothness and provide sufficient first- and second-order conditions. Notably, our framework encapsulates algorithms associated with the gradient clipping method and brings out novel insights for the class of $(L_0,L_1)$-smooth functions that has received widespread interest recently, thus allowing us to extend beyond already established methods. We investigate the convergence of the proposed method in both the convex and nonconvex setting.
APA
Oikonomidis, K., Quan, J., Laude, E. & Patrinos, P.. (2025). Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:47132-47154 Available from https://proceedings.mlr.press/v267/oikonomidis25a.html.

Related Material