Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty

Felix Biggs, Benjamin Guedj
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:8165-8182, 2023.

Abstract

We introduce a modified version of the excess risk, which can be used to obtain empirically tighter, faster-rate PAC-Bayesian generalisation bounds. This modified excess risk leverages information about the relative hardness of data examples to reduce the variance of its empirical counterpart, tightening the bound. We combine this with a new bound for [$-$1, 1]-valued (and potentially non-independent) signed losses, which is more favourable when they empirically have low variance around 0. The primary new technical tool is a novel result for sequences of interdependent random vectors which may be of independent interest. We empirically evaluate these new bounds on a number of real-world datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-biggs23a, title = {Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty}, author = {Biggs, Felix and Guedj, Benjamin}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {8165--8182}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/biggs23a/biggs23a.pdf}, url = {https://proceedings.mlr.press/v206/biggs23a.html}, abstract = {We introduce a modified version of the excess risk, which can be used to obtain empirically tighter, faster-rate PAC-Bayesian generalisation bounds. This modified excess risk leverages information about the relative hardness of data examples to reduce the variance of its empirical counterpart, tightening the bound. We combine this with a new bound for [$-$1, 1]-valued (and potentially non-independent) signed losses, which is more favourable when they empirically have low variance around 0. The primary new technical tool is a novel result for sequences of interdependent random vectors which may be of independent interest. We empirically evaluate these new bounds on a number of real-world datasets.} }
Endnote
%0 Conference Paper %T Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty %A Felix Biggs %A Benjamin Guedj %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-biggs23a %I PMLR %P 8165--8182 %U https://proceedings.mlr.press/v206/biggs23a.html %V 206 %X We introduce a modified version of the excess risk, which can be used to obtain empirically tighter, faster-rate PAC-Bayesian generalisation bounds. This modified excess risk leverages information about the relative hardness of data examples to reduce the variance of its empirical counterpart, tightening the bound. We combine this with a new bound for [$-$1, 1]-valued (and potentially non-independent) signed losses, which is more favourable when they empirically have low variance around 0. The primary new technical tool is a novel result for sequences of interdependent random vectors which may be of independent interest. We empirically evaluate these new bounds on a number of real-world datasets.
APA
Biggs, F. & Guedj, B.. (2023). Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:8165-8182 Available from https://proceedings.mlr.press/v206/biggs23a.html.

Related Material