Examining the Bias of In-Batch Sampling in Similarity Learning with Two-Tower Models

Yaxu Liu, Li-Chung Lin, Chih-Jen Lin
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:1036-1044, 2026.

Abstract

Two-tower models are widely used for applications involving learning similarities between pairs of entities, such as user-item pairs in recommender systems. These models are commonly trained using stochastic gradient methods. However, uniformly sampling data often leads to problematic batches that lack positive pairs, especially when positives are a minority of the dataset—a situation particularly common in similarity learning. Instead, a strategy known as in-batch sampling is widely adopted to ensure the presence of positive pairs and training efficiency. Nevertheless, in-batch sampling introduces its own issues, such as mistaking positives for negatives and oversampling popular pairs, resulting in significant performance degradation. In this work, we provide the first systematic analysis of these issues, showing that they all arise from the inconsistency between the expected objective under in-batch sampling and the full-data objective. We refer to this inconsistency as the bias of in-batch sampling. To validate our analysis, we design an unbiased batch loss and conduct rigorous experiments directly comparing unbiased and biased losses. The results provide strong empirical confirmation of our theoretical findings.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-liu26a, title = { Examining the Bias of In-Batch Sampling in Similarity Learning with Two-Tower Models }, author = {Liu, Yaxu and Lin, Li-Chung and Lin, Chih-Jen}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {1036--1044}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/liu26a/liu26a.pdf}, url = {https://proceedings.mlr.press/v300/liu26a.html}, abstract = { Two-tower models are widely used for applications involving learning similarities between pairs of entities, such as user-item pairs in recommender systems. These models are commonly trained using stochastic gradient methods. However, uniformly sampling data often leads to problematic batches that lack positive pairs, especially when positives are a minority of the dataset—a situation particularly common in similarity learning. Instead, a strategy known as in-batch sampling is widely adopted to ensure the presence of positive pairs and training efficiency. Nevertheless, in-batch sampling introduces its own issues, such as mistaking positives for negatives and oversampling popular pairs, resulting in significant performance degradation. In this work, we provide the first systematic analysis of these issues, showing that they all arise from the inconsistency between the expected objective under in-batch sampling and the full-data objective. We refer to this inconsistency as the bias of in-batch sampling. To validate our analysis, we design an unbiased batch loss and conduct rigorous experiments directly comparing unbiased and biased losses. The results provide strong empirical confirmation of our theoretical findings. } }
Endnote
%0 Conference Paper %T Examining the Bias of In-Batch Sampling in Similarity Learning with Two-Tower Models %A Yaxu Liu %A Li-Chung Lin %A Chih-Jen Lin %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-liu26a %I PMLR %P 1036--1044 %U https://proceedings.mlr.press/v300/liu26a.html %V 300 %X Two-tower models are widely used for applications involving learning similarities between pairs of entities, such as user-item pairs in recommender systems. These models are commonly trained using stochastic gradient methods. However, uniformly sampling data often leads to problematic batches that lack positive pairs, especially when positives are a minority of the dataset—a situation particularly common in similarity learning. Instead, a strategy known as in-batch sampling is widely adopted to ensure the presence of positive pairs and training efficiency. Nevertheless, in-batch sampling introduces its own issues, such as mistaking positives for negatives and oversampling popular pairs, resulting in significant performance degradation. In this work, we provide the first systematic analysis of these issues, showing that they all arise from the inconsistency between the expected objective under in-batch sampling and the full-data objective. We refer to this inconsistency as the bias of in-batch sampling. To validate our analysis, we design an unbiased batch loss and conduct rigorous experiments directly comparing unbiased and biased losses. The results provide strong empirical confirmation of our theoretical findings.
APA
Liu, Y., Lin, L. & Lin, C.. (2026). Examining the Bias of In-Batch Sampling in Similarity Learning with Two-Tower Models . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:1036-1044 Available from https://proceedings.mlr.press/v300/liu26a.html.

Related Material