Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective

Liangliang Shi, Gu Zhang, Haoyu Zhen, Jintao Fan, Junchi Yan
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:31408-31421, 2023.

Abstract

Previous research on contrastive learning (CL) has primarily focused on pairwise views to learn representations by attracting positive samples and repelling negative ones. In this work, we aim to understand and generalize CL from a point set matching perspective, instead of the comparison between two points. Specifically, we formulate CL as a form of inverse optimal transport (IOT), which involves a bilevel optimization procedure for learning where the outter minimization aims to learn the representations and the inner is to learn the coupling (i.e. the probability of matching matrix) between the point sets. Specifically, by adjusting the relaxation degree of constraints in the inner minimization, we obtain three contrastive losses and show that the dominant contrastive loss in literature InfoNCE falls into one of these losses. This reveals a new and more general algorithmic framework for CL. Additionally, the soft matching scheme in IOT induces a uniformity penalty to enhance representation learning which is akin to the CL’s uniformity. Results on vision benchmarks show the effectiveness of our derived loss family and the new uniformity term.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-shi23j, title = {Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective}, author = {Shi, Liangliang and Zhang, Gu and Zhen, Haoyu and Fan, Jintao and Yan, Junchi}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {31408--31421}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/shi23j/shi23j.pdf}, url = {https://proceedings.mlr.press/v202/shi23j.html}, abstract = {Previous research on contrastive learning (CL) has primarily focused on pairwise views to learn representations by attracting positive samples and repelling negative ones. In this work, we aim to understand and generalize CL from a point set matching perspective, instead of the comparison between two points. Specifically, we formulate CL as a form of inverse optimal transport (IOT), which involves a bilevel optimization procedure for learning where the outter minimization aims to learn the representations and the inner is to learn the coupling (i.e. the probability of matching matrix) between the point sets. Specifically, by adjusting the relaxation degree of constraints in the inner minimization, we obtain three contrastive losses and show that the dominant contrastive loss in literature InfoNCE falls into one of these losses. This reveals a new and more general algorithmic framework for CL. Additionally, the soft matching scheme in IOT induces a uniformity penalty to enhance representation learning which is akin to the CL’s uniformity. Results on vision benchmarks show the effectiveness of our derived loss family and the new uniformity term.} }
Endnote
%0 Conference Paper %T Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective %A Liangliang Shi %A Gu Zhang %A Haoyu Zhen %A Jintao Fan %A Junchi Yan %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-shi23j %I PMLR %P 31408--31421 %U https://proceedings.mlr.press/v202/shi23j.html %V 202 %X Previous research on contrastive learning (CL) has primarily focused on pairwise views to learn representations by attracting positive samples and repelling negative ones. In this work, we aim to understand and generalize CL from a point set matching perspective, instead of the comparison between two points. Specifically, we formulate CL as a form of inverse optimal transport (IOT), which involves a bilevel optimization procedure for learning where the outter minimization aims to learn the representations and the inner is to learn the coupling (i.e. the probability of matching matrix) between the point sets. Specifically, by adjusting the relaxation degree of constraints in the inner minimization, we obtain three contrastive losses and show that the dominant contrastive loss in literature InfoNCE falls into one of these losses. This reveals a new and more general algorithmic framework for CL. Additionally, the soft matching scheme in IOT induces a uniformity penalty to enhance representation learning which is akin to the CL’s uniformity. Results on vision benchmarks show the effectiveness of our derived loss family and the new uniformity term.
APA
Shi, L., Zhang, G., Zhen, H., Fan, J. & Yan, J.. (2023). Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:31408-31421 Available from https://proceedings.mlr.press/v202/shi23j.html.

Related Material