When Softmax Fails at the Top: Extreme-Value Corrections for InfoNCE

Melihcan Erol, Suat Evren, Oktay Ozel, Alexander Morgan, Jongha Jon Ryu, Lizhong Zheng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:28187-28211, 2026.

Abstract

InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected. Using extreme value theory, we show that this assumption is often misaligned with the normalized embedding setting used in modern contrastive learning. Motivated by this mismatch, we propose WEINCE, a simple modification of InfoNCE that uses anchor-wise online batch statistics to blend the usual softmax logits with an endpoint shortfall correction, adding no trainable parameters. Across five vision benchmarks, WEINCE yields consistent improvements in frozen-feature evaluation. These results show that a more faithful statistical treatment of hard negatives can improve contrastive objectives.[Code: https://github.com/hsme98/weince.]

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-erol26a, title = {When Softmax Fails at the Top: {E}xtreme-{V}alue Corrections for {I}nfo{NCE}}, author = {Erol, Melihcan and Evren, Suat and Ozel, Oktay and Morgan, Alexander and Ryu, Jongha Jon and Zheng, Lizhong}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {28187--28211}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/erol26a/erol26a.pdf}, url = {https://proceedings.mlr.press/v306/erol26a.html}, abstract = {InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected. Using extreme value theory, we show that this assumption is often misaligned with the normalized embedding setting used in modern contrastive learning. Motivated by this mismatch, we propose WEINCE, a simple modification of InfoNCE that uses anchor-wise online batch statistics to blend the usual softmax logits with an endpoint shortfall correction, adding no trainable parameters. Across five vision benchmarks, WEINCE yields consistent improvements in frozen-feature evaluation. These results show that a more faithful statistical treatment of hard negatives can improve contrastive objectives.[Code: https://github.com/hsme98/weince.]} }
Endnote
%0 Conference Paper %T When Softmax Fails at the Top: Extreme-Value Corrections for InfoNCE %A Melihcan Erol %A Suat Evren %A Oktay Ozel %A Alexander Morgan %A Jongha Jon Ryu %A Lizhong Zheng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-erol26a %I PMLR %P 28187--28211 %U https://proceedings.mlr.press/v306/erol26a.html %V 306 %X InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected. Using extreme value theory, we show that this assumption is often misaligned with the normalized embedding setting used in modern contrastive learning. Motivated by this mismatch, we propose WEINCE, a simple modification of InfoNCE that uses anchor-wise online batch statistics to blend the usual softmax logits with an endpoint shortfall correction, adding no trainable parameters. Across five vision benchmarks, WEINCE yields consistent improvements in frozen-feature evaluation. These results show that a more faithful statistical treatment of hard negatives can improve contrastive objectives.[Code: https://github.com/hsme98/weince.]
APA
Erol, M., Evren, S., Ozel, O., Morgan, A., Ryu, J.J. & Zheng, L.. (2026). When Softmax Fails at the Top: Extreme-Value Corrections for InfoNCE. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:28187-28211 Available from https://proceedings.mlr.press/v306/erol26a.html.

Related Material