Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation

Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. Haochen, Tengyu Ma, Percy Liang
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:19847-19878, 2022.

Abstract

We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e.g., photos) and unlabeled data from a target domain (e.g., sketches) are used to learn a classifier for the target domain. Conventional UDA methods (e.g., domain adversarial training) learn domain-invariant features to generalize from the source domain to the target domain. In this paper, we show that contrastive pre-training, which learns features on unlabeled source and target data and then fine-tunes on labeled source data, is competitive with strong UDA methods. However, we find that contrastive pre-training does not learn domain-invariant features, diverging from conventional UDA intuitions. We show theoretically that contrastive pre-training can learn features that vary subtantially across domains but still generalize to the target domain, by disentangling domain and class information. We empirically validate our theory on benchmark vision datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-shen22d, title = {Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation}, author = {Shen, Kendrick and Jones, Robbie M and Kumar, Ananya and Xie, Sang Michael and Haochen, Jeff Z. and Ma, Tengyu and Liang, Percy}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {19847--19878}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/shen22d/shen22d.pdf}, url = {https://proceedings.mlr.press/v162/shen22d.html}, abstract = {We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e.g., photos) and unlabeled data from a target domain (e.g., sketches) are used to learn a classifier for the target domain. Conventional UDA methods (e.g., domain adversarial training) learn domain-invariant features to generalize from the source domain to the target domain. In this paper, we show that contrastive pre-training, which learns features on unlabeled source and target data and then fine-tunes on labeled source data, is competitive with strong UDA methods. However, we find that contrastive pre-training does not learn domain-invariant features, diverging from conventional UDA intuitions. We show theoretically that contrastive pre-training can learn features that vary subtantially across domains but still generalize to the target domain, by disentangling domain and class information. We empirically validate our theory on benchmark vision datasets.} }
Endnote
%0 Conference Paper %T Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation %A Kendrick Shen %A Robbie M Jones %A Ananya Kumar %A Sang Michael Xie %A Jeff Z. Haochen %A Tengyu Ma %A Percy Liang %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-shen22d %I PMLR %P 19847--19878 %U https://proceedings.mlr.press/v162/shen22d.html %V 162 %X We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e.g., photos) and unlabeled data from a target domain (e.g., sketches) are used to learn a classifier for the target domain. Conventional UDA methods (e.g., domain adversarial training) learn domain-invariant features to generalize from the source domain to the target domain. In this paper, we show that contrastive pre-training, which learns features on unlabeled source and target data and then fine-tunes on labeled source data, is competitive with strong UDA methods. However, we find that contrastive pre-training does not learn domain-invariant features, diverging from conventional UDA intuitions. We show theoretically that contrastive pre-training can learn features that vary subtantially across domains but still generalize to the target domain, by disentangling domain and class information. We empirically validate our theory on benchmark vision datasets.
APA
Shen, K., Jones, R.M., Kumar, A., Xie, S.M., Haochen, J.Z., Ma, T. & Liang, P.. (2022). Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:19847-19878 Available from https://proceedings.mlr.press/v162/shen22d.html.

Related Material