InfoNCE Loss Provably Learns Cluster-Preserving Representations

Advait Parulekar, Liam Collins, Karthikeyan Shanmugam, Aryan Mokhtari, Sanjay Shakkottai
Proceedings of Thirty Sixth Conference on Learning Theory, PMLR 195:1914-1961, 2023.

Abstract

The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g. the “dogness” of a dog, close to each other in the space generated by the representation. A common and successful approach for tackling this unsupervised learning problem is minimizing the InfoNCE loss associated with the training samples, where each sample is associated with their augmentations (positive samples such as rotation, crop) and a batch of negative samples (unrelated samples). To the best of our knowledge, it was unanswered if the representation learned by minimizing the InfoNCE loss preserves the underlying data clusters, as it only promotes learning a representation that is faithful to augmentations, i.e., an image and its augmentations have the same representation. Our main result is to show that the representation learned by InfoNCE with a finite number of negative samples is also consistent with respect to {\em clusters} in the data, under the condition that the augmentation sets within clusters may be non-overlapping but are close and intertwined, relative to the complexity of the learning function class.

Cite this Paper

BibTeX
@InProceedings{pmlr-v195-parulekar23a, title = {InfoNCE Loss Provably Learns Cluster-Preserving Representations}, author = {Parulekar, Advait and Collins, Liam and Shanmugam, Karthikeyan and Mokhtari, Aryan and Shakkottai, Sanjay}, booktitle = {Proceedings of Thirty Sixth Conference on Learning Theory}, pages = {1914--1961}, year = {2023}, editor = {Neu, Gergely and Rosasco, Lorenzo}, volume = {195}, series = {Proceedings of Machine Learning Research}, month = {12--15 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v195/parulekar23a/parulekar23a.pdf}, url = {https://proceedings.mlr.press/v195/parulekar23a.html}, abstract = {The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g. the “dogness” of a dog, close to each other in the space generated by the representation. A common and successful approach for tackling this unsupervised learning problem is minimizing the InfoNCE loss associated with the training samples, where each sample is associated with their augmentations (positive samples such as rotation, crop) and a batch of negative samples (unrelated samples). To the best of our knowledge, it was unanswered if the representation learned by minimizing the InfoNCE loss preserves the underlying data clusters, as it only promotes learning a representation that is faithful to augmentations, i.e., an image and its augmentations have the same representation. Our main result is to show that the representation learned by InfoNCE with a finite number of negative samples is also consistent with respect to {\em clusters} in the data, under the condition that the augmentation sets within clusters may be non-overlapping but are close and intertwined, relative to the complexity of the learning function class.} }
Endnote
%0 Conference Paper %T InfoNCE Loss Provably Learns Cluster-Preserving Representations %A Advait Parulekar %A Liam Collins %A Karthikeyan Shanmugam %A Aryan Mokhtari %A Sanjay Shakkottai %B Proceedings of Thirty Sixth Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2023 %E Gergely Neu %E Lorenzo Rosasco %F pmlr-v195-parulekar23a %I PMLR %P 1914--1961 %U https://proceedings.mlr.press/v195/parulekar23a.html %V 195 %X The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g. the “dogness” of a dog, close to each other in the space generated by the representation. A common and successful approach for tackling this unsupervised learning problem is minimizing the InfoNCE loss associated with the training samples, where each sample is associated with their augmentations (positive samples such as rotation, crop) and a batch of negative samples (unrelated samples). To the best of our knowledge, it was unanswered if the representation learned by minimizing the InfoNCE loss preserves the underlying data clusters, as it only promotes learning a representation that is faithful to augmentations, i.e., an image and its augmentations have the same representation. Our main result is to show that the representation learned by InfoNCE with a finite number of negative samples is also consistent with respect to {\em clusters} in the data, under the condition that the augmentation sets within clusters may be non-overlapping but are close and intertwined, relative to the complexity of the learning function class.
APA
Parulekar, A., Collins, L., Shanmugam, K., Mokhtari, A. & Shakkottai, S.. (2023). InfoNCE Loss Provably Learns Cluster-Preserving Representations. Proceedings of Thirty Sixth Conference on Learning Theory, in Proceedings of Machine Learning Research 195:1914-1961 Available from https://proceedings.mlr.press/v195/parulekar23a.html.

Related Material