The SSL Interplay: Augmentations, Inductive Bias, and Generalization

Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann Lecun, Alberto Bietti
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:3252-3298, 2023.

Abstract

Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision. Yet in practice, engineers face issues such as instability in tuning optimizers and collapse of representations during training. Such challenges motivate the need for a theory to shed light on the complex interplay between the choice of data augmentation, network architecture, and training algorithm. % on the resulting performance in downstream tasks. We study such an interplay with a precise analysis of generalization performance on both pretraining and downstream tasks in kernel regimes, and highlight several insights for SSL practitioners that arise from our theory.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-cabannes23a, title = {The {SSL} Interplay: Augmentations, Inductive Bias, and Generalization}, author = {Cabannes, Vivien and Kiani, Bobak and Balestriero, Randall and Lecun, Yann and Bietti, Alberto}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {3252--3298}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/cabannes23a/cabannes23a.pdf}, url = {https://proceedings.mlr.press/v202/cabannes23a.html}, abstract = {Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision. Yet in practice, engineers face issues such as instability in tuning optimizers and collapse of representations during training. Such challenges motivate the need for a theory to shed light on the complex interplay between the choice of data augmentation, network architecture, and training algorithm. % on the resulting performance in downstream tasks. We study such an interplay with a precise analysis of generalization performance on both pretraining and downstream tasks in kernel regimes, and highlight several insights for SSL practitioners that arise from our theory.} }
Endnote
%0 Conference Paper %T The SSL Interplay: Augmentations, Inductive Bias, and Generalization %A Vivien Cabannes %A Bobak Kiani %A Randall Balestriero %A Yann Lecun %A Alberto Bietti %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-cabannes23a %I PMLR %P 3252--3298 %U https://proceedings.mlr.press/v202/cabannes23a.html %V 202 %X Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision. Yet in practice, engineers face issues such as instability in tuning optimizers and collapse of representations during training. Such challenges motivate the need for a theory to shed light on the complex interplay between the choice of data augmentation, network architecture, and training algorithm. % on the resulting performance in downstream tasks. We study such an interplay with a precise analysis of generalization performance on both pretraining and downstream tasks in kernel regimes, and highlight several insights for SSL practitioners that arise from our theory.
APA
Cabannes, V., Kiani, B., Balestriero, R., Lecun, Y. & Bietti, A.. (2023). The SSL Interplay: Augmentations, Inductive Bias, and Generalization. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:3252-3298 Available from https://proceedings.mlr.press/v202/cabannes23a.html.

Related Material