Generalisation under gradient descent via deterministic PAC-Bayes

Eugenio Clerico, Tyler Farghly, George Deligiannidis, Benjamin Guedj, Arnaud Doucet
Proceedings of The 36th International Conference on Algorithmic Learning Theory, PMLR 272:349-389, 2025.

Abstract

We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows. Contrary to standard practice in the PAC-Bayesian setting, our result applies to optimisation algorithms that are deterministic, without requiring any de-randomisation step. Our bounds are fully computable, depending on the density of the initial distribution and the Hessian of the training objective over the trajectory. We show that our framework can be applied to a variety of iterative optimisation algorithms, including stochastic gradient descent (SGD), momentum-based schemes, and damped Hamiltonian dynamics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v272-clerico25a, title = {Generalisation under gradient descent via deterministic PAC-Bayes}, author = {Clerico, Eugenio and Farghly, Tyler and Deligiannidis, George and Guedj, Benjamin and Doucet, Arnaud}, booktitle = {Proceedings of The 36th International Conference on Algorithmic Learning Theory}, pages = {349--389}, year = {2025}, editor = {Kamath, Gautam and Loh, Po-Ling}, volume = {272}, series = {Proceedings of Machine Learning Research}, month = {24--27 Feb}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v272/main/assets/clerico25a/clerico25a.pdf}, url = {https://proceedings.mlr.press/v272/clerico25a.html}, abstract = {We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows. Contrary to standard practice in the PAC-Bayesian setting, our result applies to optimisation algorithms that are deterministic, without requiring any de-randomisation step. Our bounds are fully computable, depending on the density of the initial distribution and the Hessian of the training objective over the trajectory. We show that our framework can be applied to a variety of iterative optimisation algorithms, including stochastic gradient descent (SGD), momentum-based schemes, and damped Hamiltonian dynamics.} }
Endnote
%0 Conference Paper %T Generalisation under gradient descent via deterministic PAC-Bayes %A Eugenio Clerico %A Tyler Farghly %A George Deligiannidis %A Benjamin Guedj %A Arnaud Doucet %B Proceedings of The 36th International Conference on Algorithmic Learning Theory %C Proceedings of Machine Learning Research %D 2025 %E Gautam Kamath %E Po-Ling Loh %F pmlr-v272-clerico25a %I PMLR %P 349--389 %U https://proceedings.mlr.press/v272/clerico25a.html %V 272 %X We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows. Contrary to standard practice in the PAC-Bayesian setting, our result applies to optimisation algorithms that are deterministic, without requiring any de-randomisation step. Our bounds are fully computable, depending on the density of the initial distribution and the Hessian of the training objective over the trajectory. We show that our framework can be applied to a variety of iterative optimisation algorithms, including stochastic gradient descent (SGD), momentum-based schemes, and damped Hamiltonian dynamics.
APA
Clerico, E., Farghly, T., Deligiannidis, G., Guedj, B. & Doucet, A.. (2025). Generalisation under gradient descent via deterministic PAC-Bayes. Proceedings of The 36th International Conference on Algorithmic Learning Theory, in Proceedings of Machine Learning Research 272:349-389 Available from https://proceedings.mlr.press/v272/clerico25a.html.

Related Material