A PAC-Bayesian Link Between Generalisation and Flat Minima

Maxime Haddouche, Paul Viallard, Umut Simsekli, Benjamin Guedj
Proceedings of The 36th International Conference on Algorithmic Learning Theory, PMLR 272:481-511, 2025.

Abstract

Modern machine learning usually involves predictors in the overparameterised setting (number of trained parameters greater than dataset size), and their training yields not only good performance on training data, but also good generalisation capacity. This phenomenon challenges many theoretical results, and remains an open problem. To reach a better understanding, we provide novel generalisation bounds involving gradient terms. To do so, we combine the PAC-Bayes toolbox with Poincaré and Log-Sobolev inequalities, avoiding an explicit dependency on the dimension of the predictor space. Our results highlight the positive influence of flat minima (being minima with a neighbourhood nearly minimising the learning problem as well) on generalisation performance, involving directly the benefits of the optimisation phase.

Cite this Paper

BibTeX
@InProceedings{pmlr-v272-haddouche25a, title = {A PAC-Bayesian Link Between Generalisation and Flat Minima}, author = {Haddouche, Maxime and Viallard, Paul and Simsekli, Umut and Guedj, Benjamin}, booktitle = {Proceedings of The 36th International Conference on Algorithmic Learning Theory}, pages = {481--511}, year = {2025}, editor = {Kamath, Gautam and Loh, Po-Ling}, volume = {272}, series = {Proceedings of Machine Learning Research}, month = {24--27 Feb}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v272/main/assets/haddouche25a/haddouche25a.pdf}, url = {https://proceedings.mlr.press/v272/haddouche25a.html}, abstract = {Modern machine learning usually involves predictors in the overparameterised setting (number of trained parameters greater than dataset size), and their training yields not only good performance on training data, but also good generalisation capacity. This phenomenon challenges many theoretical results, and remains an open problem. To reach a better understanding, we provide novel generalisation bounds involving gradient terms. To do so, we combine the PAC-Bayes toolbox with Poincaré and Log-Sobolev inequalities, avoiding an explicit dependency on the dimension of the predictor space. Our results highlight the positive influence of flat minima (being minima with a neighbourhood nearly minimising the learning problem as well) on generalisation performance, involving directly the benefits of the optimisation phase.} }
Endnote
%0 Conference Paper %T A PAC-Bayesian Link Between Generalisation and Flat Minima %A Maxime Haddouche %A Paul Viallard %A Umut Simsekli %A Benjamin Guedj %B Proceedings of The 36th International Conference on Algorithmic Learning Theory %C Proceedings of Machine Learning Research %D 2025 %E Gautam Kamath %E Po-Ling Loh %F pmlr-v272-haddouche25a %I PMLR %P 481--511 %U https://proceedings.mlr.press/v272/haddouche25a.html %V 272 %X Modern machine learning usually involves predictors in the overparameterised setting (number of trained parameters greater than dataset size), and their training yields not only good performance on training data, but also good generalisation capacity. This phenomenon challenges many theoretical results, and remains an open problem. To reach a better understanding, we provide novel generalisation bounds involving gradient terms. To do so, we combine the PAC-Bayes toolbox with Poincaré and Log-Sobolev inequalities, avoiding an explicit dependency on the dimension of the predictor space. Our results highlight the positive influence of flat minima (being minima with a neighbourhood nearly minimising the learning problem as well) on generalisation performance, involving directly the benefits of the optimisation phase.
APA
Haddouche, M., Viallard, P., Simsekli, U. & Guedj, B.. (2025). A PAC-Bayesian Link Between Generalisation and Flat Minima. Proceedings of The 36th International Conference on Algorithmic Learning Theory, in Proceedings of Machine Learning Research 272:481-511 Available from https://proceedings.mlr.press/v272/haddouche25a.html.

Related Material