PAC-Bayesian Contrastive Unsupervised Representation Learning

Kento Nozawa, Pascal Germain, Benjamin Guedj
Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), PMLR 124:21-30, 2020.

Abstract

Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to derive a new representation learning algorithm. Numerical experiments on real-life datasets illustrate that our algorithm achieves competitive accuracy, and yields non-vacuous generalisation bounds.

Cite this Paper

BibTeX
@InProceedings{pmlr-v124-nozawa20a, title = {PAC-Bayesian Contrastive Unsupervised Representation Learning}, author = {Nozawa, Kento and Germain, Pascal and Guedj, Benjamin}, booktitle = {Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)}, pages = {21--30}, year = {2020}, editor = {Peters, Jonas and Sontag, David}, volume = {124}, series = {Proceedings of Machine Learning Research}, month = {03--06 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v124/nozawa20a/nozawa20a.pdf}, url = {https://proceedings.mlr.press/v124/nozawa20a.html}, abstract = {Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to derive a new representation learning algorithm. Numerical experiments on real-life datasets illustrate that our algorithm achieves competitive accuracy, and yields non-vacuous generalisation bounds.} }
Endnote
%0 Conference Paper %T PAC-Bayesian Contrastive Unsupervised Representation Learning %A Kento Nozawa %A Pascal Germain %A Benjamin Guedj %B Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI) %C Proceedings of Machine Learning Research %D 2020 %E Jonas Peters %E David Sontag %F pmlr-v124-nozawa20a %I PMLR %P 21--30 %U https://proceedings.mlr.press/v124/nozawa20a.html %V 124 %X Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to derive a new representation learning algorithm. Numerical experiments on real-life datasets illustrate that our algorithm achieves competitive accuracy, and yields non-vacuous generalisation bounds.
APA
Nozawa, K., Germain, P. & Guedj, B.. (2020). PAC-Bayesian Contrastive Unsupervised Representation Learning. Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), in Proceedings of Machine Learning Research 124:21-30 Available from https://proceedings.mlr.press/v124/nozawa20a.html.

Related Material