Geometric Convergence of Gauss–Newton for Neural Networks: Riemannian Geometry and Adaptive Damping

Semih Cayci
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:12189-12214, 2026.

Abstract

Ill-conditioned kernel matrices can make first-order methods for training neural networks converge slowly. We establish non-asymptotic convergence bounds for the Gauss–Newton method in both under- and overparameterized regimes, showing it avoids these conditioning bottlenecks. In the underparameterized setting, Gauss–Newton gradient flow in parameter space induces a Riemannian gradient flow on a low-dimensional submanifold of function space. Using tools from Riemannian optimization, we show that, under an appropriate output scaling, the loss satisfies geodesic Polyak–Lojasiewicz and Lipschitz-smoothness conditions, implying geometric convergence to the optimal in-class predictor at an explicit rate independent of Gram-matrix conditioning. In the overparameterized setting, we identify adaptive, curvature-aware regularization schedules and prove fast geometric convergence to a global optimum for both Gauss–Newton gradient flow and discrete-time Gauss–Newton iterates, with rates independent of the minimum eigenvalue of the neural tangent kernel. Overall, Gauss–Newton can be provably faster in ill-conditioned regimes where first-order methods slow down.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-cayci26a, title = {Geometric Convergence of {G}auss–{N}ewton for Neural Networks: {R}iemannian Geometry and Adaptive Damping}, author = {Cayci, Semih}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {12189--12214}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/cayci26a/cayci26a.pdf}, url = {https://proceedings.mlr.press/v306/cayci26a.html}, abstract = {Ill-conditioned kernel matrices can make first-order methods for training neural networks converge slowly. We establish non-asymptotic convergence bounds for the Gauss–Newton method in both under- and overparameterized regimes, showing it avoids these conditioning bottlenecks. In the underparameterized setting, Gauss–Newton gradient flow in parameter space induces a Riemannian gradient flow on a low-dimensional submanifold of function space. Using tools from Riemannian optimization, we show that, under an appropriate output scaling, the loss satisfies geodesic Polyak–Lojasiewicz and Lipschitz-smoothness conditions, implying geometric convergence to the optimal in-class predictor at an explicit rate independent of Gram-matrix conditioning. In the overparameterized setting, we identify adaptive, curvature-aware regularization schedules and prove fast geometric convergence to a global optimum for both Gauss–Newton gradient flow and discrete-time Gauss–Newton iterates, with rates independent of the minimum eigenvalue of the neural tangent kernel. Overall, Gauss–Newton can be provably faster in ill-conditioned regimes where first-order methods slow down.} }
Endnote
%0 Conference Paper %T Geometric Convergence of Gauss–Newton for Neural Networks: Riemannian Geometry and Adaptive Damping %A Semih Cayci %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-cayci26a %I PMLR %P 12189--12214 %U https://proceedings.mlr.press/v306/cayci26a.html %V 306 %X Ill-conditioned kernel matrices can make first-order methods for training neural networks converge slowly. We establish non-asymptotic convergence bounds for the Gauss–Newton method in both under- and overparameterized regimes, showing it avoids these conditioning bottlenecks. In the underparameterized setting, Gauss–Newton gradient flow in parameter space induces a Riemannian gradient flow on a low-dimensional submanifold of function space. Using tools from Riemannian optimization, we show that, under an appropriate output scaling, the loss satisfies geodesic Polyak–Lojasiewicz and Lipschitz-smoothness conditions, implying geometric convergence to the optimal in-class predictor at an explicit rate independent of Gram-matrix conditioning. In the overparameterized setting, we identify adaptive, curvature-aware regularization schedules and prove fast geometric convergence to a global optimum for both Gauss–Newton gradient flow and discrete-time Gauss–Newton iterates, with rates independent of the minimum eigenvalue of the neural tangent kernel. Overall, Gauss–Newton can be provably faster in ill-conditioned regimes where first-order methods slow down.
APA
Cayci, S.. (2026). Geometric Convergence of Gauss–Newton for Neural Networks: Riemannian Geometry and Adaptive Damping. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:12189-12214 Available from https://proceedings.mlr.press/v306/cayci26a.html.

Related Material