Towards Domain-Agnostic Contrastive Learning

Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, Quoc Le
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:10530-10541, 2021.

Abstract

Despite recent successes, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a domain-agnostic approach to contrastive learning, named DACL, that is applicable to problems where domain-specific data augmentations are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input or hidden-state levels. We theoretically analyze our method and show advantages over the Gaussian-noise based contrastive learning approach. To demonstrate the effectiveness of DACL, we conduct experiments across various domains such as tabular data, images, and graphs. Our results show that DACL not only outperforms other domain-agnostic noising methods, such as Gaussian-noise, but also combines well with domain-specific methods, such as SimCLR, to improve self-supervised visual representation learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-verma21a, title = {Towards Domain-Agnostic Contrastive Learning}, author = {Verma, Vikas and Luong, Thang and Kawaguchi, Kenji and Pham, Hieu and Le, Quoc}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {10530--10541}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/verma21a/verma21a.pdf}, url = {https://proceedings.mlr.press/v139/verma21a.html}, abstract = {Despite recent successes, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a domain-agnostic approach to contrastive learning, named DACL, that is applicable to problems where domain-specific data augmentations are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input or hidden-state levels. We theoretically analyze our method and show advantages over the Gaussian-noise based contrastive learning approach. To demonstrate the effectiveness of DACL, we conduct experiments across various domains such as tabular data, images, and graphs. Our results show that DACL not only outperforms other domain-agnostic noising methods, such as Gaussian-noise, but also combines well with domain-specific methods, such as SimCLR, to improve self-supervised visual representation learning.} }
Endnote
%0 Conference Paper %T Towards Domain-Agnostic Contrastive Learning %A Vikas Verma %A Thang Luong %A Kenji Kawaguchi %A Hieu Pham %A Quoc Le %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-verma21a %I PMLR %P 10530--10541 %U https://proceedings.mlr.press/v139/verma21a.html %V 139 %X Despite recent successes, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a domain-agnostic approach to contrastive learning, named DACL, that is applicable to problems where domain-specific data augmentations are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input or hidden-state levels. We theoretically analyze our method and show advantages over the Gaussian-noise based contrastive learning approach. To demonstrate the effectiveness of DACL, we conduct experiments across various domains such as tabular data, images, and graphs. Our results show that DACL not only outperforms other domain-agnostic noising methods, such as Gaussian-noise, but also combines well with domain-specific methods, such as SimCLR, to improve self-supervised visual representation learning.
APA
Verma, V., Luong, T., Kawaguchi, K., Pham, H. & Le, Q.. (2021). Towards Domain-Agnostic Contrastive Learning. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:10530-10541 Available from https://proceedings.mlr.press/v139/verma21a.html.

Related Material