A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learning

Chungpa Lee, Jeongheon Oh, Kibok Lee, Jy-yong Sohn
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:487-495, 2025.

Abstract

Supervised contrastive learning (SupCL) has emerged as a prominent approach in representation learning, leveraging both supervised and self-supervised losses. However, achieving an optimal balance between these losses is challenging; failing to do so can lead to class collapse, reducing discrimination among individual embeddings in the same class. In this paper, we present theoretically grounded guidelines for SupCL to prevent class collapse in learned representations. Specifically, we introduce the Simplex-to-Simplex Embedding Model (SSEM), a theoretical framework that models various embedding structures, including all embeddings that minimize the supervised contrastive loss. Through SSEM, we analyze how hyperparameters affect learned representations, offering practical guidelines for hyperparameter selection to mitigate the risk of class collapse. Our theoretical findings are supported by empirical results across synthetic and real-world datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-lee25a, title = {A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learning}, author = {Lee, Chungpa and Oh, Jeongheon and Lee, Kibok and Sohn, Jy-yong}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {487--495}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/lee25a/lee25a.pdf}, url = {https://proceedings.mlr.press/v258/lee25a.html}, abstract = {Supervised contrastive learning (SupCL) has emerged as a prominent approach in representation learning, leveraging both supervised and self-supervised losses. However, achieving an optimal balance between these losses is challenging; failing to do so can lead to class collapse, reducing discrimination among individual embeddings in the same class. In this paper, we present theoretically grounded guidelines for SupCL to prevent class collapse in learned representations. Specifically, we introduce the Simplex-to-Simplex Embedding Model (SSEM), a theoretical framework that models various embedding structures, including all embeddings that minimize the supervised contrastive loss. Through SSEM, we analyze how hyperparameters affect learned representations, offering practical guidelines for hyperparameter selection to mitigate the risk of class collapse. Our theoretical findings are supported by empirical results across synthetic and real-world datasets.} }
Endnote
%0 Conference Paper %T A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learning %A Chungpa Lee %A Jeongheon Oh %A Kibok Lee %A Jy-yong Sohn %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-lee25a %I PMLR %P 487--495 %U https://proceedings.mlr.press/v258/lee25a.html %V 258 %X Supervised contrastive learning (SupCL) has emerged as a prominent approach in representation learning, leveraging both supervised and self-supervised losses. However, achieving an optimal balance between these losses is challenging; failing to do so can lead to class collapse, reducing discrimination among individual embeddings in the same class. In this paper, we present theoretically grounded guidelines for SupCL to prevent class collapse in learned representations. Specifically, we introduce the Simplex-to-Simplex Embedding Model (SSEM), a theoretical framework that models various embedding structures, including all embeddings that minimize the supervised contrastive loss. Through SSEM, we analyze how hyperparameters affect learned representations, offering practical guidelines for hyperparameter selection to mitigate the risk of class collapse. Our theoretical findings are supported by empirical results across synthetic and real-world datasets.
APA
Lee, C., Oh, J., Lee, K. & Sohn, J.. (2025). A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learning. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:487-495 Available from https://proceedings.mlr.press/v258/lee25a.html.

Related Material