Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning

Zixin Wen, Yuanzhi Li
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:11112-11122, 2021.

Abstract

We formally study how contrastive learning learns the feature representations for neural networks by investigating its feature learning process. We consider the case where our data are comprised of two types of features: the sparse features which we want to learn from, and the dense features we want to get rid of. Theoretically, we prove that contrastive learning using ReLU networks provably learns the desired features if proper augmentations are adopted. We present an underlying principle called feature decoupling to explain the effects of augmentations, where we theoretically characterize how augmentations can reduce the correlations of dense features between positive samples while keeping the correlations of sparse features intact, thereby forcing the neural networks to learn from the self-supervision of sparse features. Empirically, we verified that the feature decoupling principle matches the underlying mechanism of contrastive learning in practice.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-wen21c, title = {Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning}, author = {Wen, Zixin and Li, Yuanzhi}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {11112--11122}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/wen21c/wen21c.pdf}, url = {https://proceedings.mlr.press/v139/wen21c.html}, abstract = {We formally study how contrastive learning learns the feature representations for neural networks by investigating its feature learning process. We consider the case where our data are comprised of two types of features: the sparse features which we want to learn from, and the dense features we want to get rid of. Theoretically, we prove that contrastive learning using ReLU networks provably learns the desired features if proper augmentations are adopted. We present an underlying principle called feature decoupling to explain the effects of augmentations, where we theoretically characterize how augmentations can reduce the correlations of dense features between positive samples while keeping the correlations of sparse features intact, thereby forcing the neural networks to learn from the self-supervision of sparse features. Empirically, we verified that the feature decoupling principle matches the underlying mechanism of contrastive learning in practice.} }
Endnote
%0 Conference Paper %T Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning %A Zixin Wen %A Yuanzhi Li %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-wen21c %I PMLR %P 11112--11122 %U https://proceedings.mlr.press/v139/wen21c.html %V 139 %X We formally study how contrastive learning learns the feature representations for neural networks by investigating its feature learning process. We consider the case where our data are comprised of two types of features: the sparse features which we want to learn from, and the dense features we want to get rid of. Theoretically, we prove that contrastive learning using ReLU networks provably learns the desired features if proper augmentations are adopted. We present an underlying principle called feature decoupling to explain the effects of augmentations, where we theoretically characterize how augmentations can reduce the correlations of dense features between positive samples while keeping the correlations of sparse features intact, thereby forcing the neural networks to learn from the self-supervision of sparse features. Empirically, we verified that the feature decoupling principle matches the underlying mechanism of contrastive learning in practice.
APA
Wen, Z. & Li, Y.. (2021). Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:11112-11122 Available from https://proceedings.mlr.press/v139/wen21c.html.

Related Material