Contrastive Learning with Latent Tension Regularization for Tight Orbits

Ritwik Ghosal
Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations, PMLR 282:174-199, 2026.

Abstract

In self-supervised contrastive learning, multiple augmentations of the same input naturally form a set of latent representations, or an orbit. Ideally, these representations should remain compact and directionally consistent under transformations. Standard methods such as SimCLR prioritize separating different samples but do not explicitly enforce intra-orbit coherence, allowing augmented views of the same input to drift in latent space. We propose Orbit Regularization Loss (ORL), a lightweight extension to the Normalized Temperature-scaled Cross-Entropy (NT-Xent) loss that reweights negative pairs based on a tension score - a measure of alignment between the positive-pair direction and the candidate negative’s displacement. This encourages augmented views to align along stable latent directions, reducing orbit spread without architectural changes or additional supervision. For now, ORL is aimed at improving the geometric structure of embeddings, rather than directly targeting downstream classification accuracy. Experiments on MNIST and CIFAR-10 show that ORL lowers intra-orbit variance, improves directional consistency, and yields a more coherent latent space geometry compared to the NT-Xent baseline.

Cite this Paper

BibTeX
@InProceedings{pmlr-v282-ghosal26a, title = {Contrastive Learning with Latent Tension Regularization for Tight Orbits}, author = {Ghosal, Ritwik}, booktitle = {Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations}, pages = {174--199}, year = {2026}, editor = {Acosta, Francisco and Azeglio, Simone and Tolooshams, Bahareh and van de Geijn, Chase and Shewmake, Christian and Sanborn, Sophia and Miolane, Nina}, volume = {282}, series = {Proceedings of Machine Learning Research}, month = {14 Dec 2024--07 Dec 2025}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v282/main/assets/ghosal26a/ghosal26a.pdf}, url = {https://proceedings.mlr.press/v282/ghosal26a.html}, abstract = {In self-supervised contrastive learning, multiple augmentations of the same input naturally form a set of latent representations, or an orbit. Ideally, these representations should remain compact and directionally consistent under transformations. Standard methods such as SimCLR prioritize separating different samples but do not explicitly enforce intra-orbit coherence, allowing augmented views of the same input to drift in latent space. We propose Orbit Regularization Loss (ORL), a lightweight extension to the Normalized Temperature-scaled Cross-Entropy (NT-Xent) loss that reweights negative pairs based on a tension score - a measure of alignment between the positive-pair direction and the candidate negative’s displacement. This encourages augmented views to align along stable latent directions, reducing orbit spread without architectural changes or additional supervision. For now, ORL is aimed at improving the geometric structure of embeddings, rather than directly targeting downstream classification accuracy. Experiments on MNIST and CIFAR-10 show that ORL lowers intra-orbit variance, improves directional consistency, and yields a more coherent latent space geometry compared to the NT-Xent baseline.} }
Endnote
%0 Conference Paper %T Contrastive Learning with Latent Tension Regularization for Tight Orbits %A Ritwik Ghosal %B Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations %C Proceedings of Machine Learning Research %D 2026 %E Francisco Acosta %E Simone Azeglio %E Bahareh Tolooshams %E Chase van de Geijn %E Christian Shewmake %E Sophia Sanborn %E Nina Miolane %F pmlr-v282-ghosal26a %I PMLR %P 174--199 %U https://proceedings.mlr.press/v282/ghosal26a.html %V 282 %X In self-supervised contrastive learning, multiple augmentations of the same input naturally form a set of latent representations, or an orbit. Ideally, these representations should remain compact and directionally consistent under transformations. Standard methods such as SimCLR prioritize separating different samples but do not explicitly enforce intra-orbit coherence, allowing augmented views of the same input to drift in latent space. We propose Orbit Regularization Loss (ORL), a lightweight extension to the Normalized Temperature-scaled Cross-Entropy (NT-Xent) loss that reweights negative pairs based on a tension score - a measure of alignment between the positive-pair direction and the candidate negative’s displacement. This encourages augmented views to align along stable latent directions, reducing orbit spread without architectural changes or additional supervision. For now, ORL is aimed at improving the geometric structure of embeddings, rather than directly targeting downstream classification accuracy. Experiments on MNIST and CIFAR-10 show that ORL lowers intra-orbit variance, improves directional consistency, and yields a more coherent latent space geometry compared to the NT-Xent baseline.
APA
Ghosal, R.. (2026). Contrastive Learning with Latent Tension Regularization for Tight Orbits. Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations, in Proceedings of Machine Learning Research 282:174-199 Available from https://proceedings.mlr.press/v282/ghosal26a.html.

Related Material