Understanding Contrastive Learning Requires Incorporating Inductive Biases

Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:19250-19286, 2022.

Abstract

Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of augmentations and the value of contrastive loss of representations. We demonstrate that such analyses, that ignore inductive biases of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even provably leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem – different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-saunshi22a, title = {Understanding Contrastive Learning Requires Incorporating Inductive Biases}, author = {Saunshi, Nikunj and Ash, Jordan and Goel, Surbhi and Misra, Dipendra and Zhang, Cyril and Arora, Sanjeev and Kakade, Sham and Krishnamurthy, Akshay}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {19250--19286}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/saunshi22a/saunshi22a.pdf}, url = {https://proceedings.mlr.press/v162/saunshi22a.html}, abstract = {Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of augmentations and the value of contrastive loss of representations. We demonstrate that such analyses, that ignore inductive biases of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even provably leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem – different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.} }
Endnote
%0 Conference Paper %T Understanding Contrastive Learning Requires Incorporating Inductive Biases %A Nikunj Saunshi %A Jordan Ash %A Surbhi Goel %A Dipendra Misra %A Cyril Zhang %A Sanjeev Arora %A Sham Kakade %A Akshay Krishnamurthy %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-saunshi22a %I PMLR %P 19250--19286 %U https://proceedings.mlr.press/v162/saunshi22a.html %V 162 %X Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of augmentations and the value of contrastive loss of representations. We demonstrate that such analyses, that ignore inductive biases of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even provably leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem – different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.
APA
Saunshi, N., Ash, J., Goel, S., Misra, D., Zhang, C., Arora, S., Kakade, S. & Krishnamurthy, A.. (2022). Understanding Contrastive Learning Requires Incorporating Inductive Biases. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:19250-19286 Available from https://proceedings.mlr.press/v162/saunshi22a.html.

Related Material