Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation

Michal Lukasik, Lin Chen, Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, Felix X. Yu, Sashank J. Reddi, Gang Fu, Mohammadhossein Bateni, Sanjiv Kumar
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:41074-41102, 2025.

Abstract

Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal area under the ROC curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem—loss aggregation and label aggregation—by characterizing their Bayes-optimal solutions. We show that while both approaches can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-lukasik25a, title = {Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation}, author = {Lukasik, Michal and Chen, Lin and Narasimhan, Harikrishna and Menon, Aditya Krishna and Jitkrittum, Wittawat and Yu, Felix X. and J. Reddi, Sashank and Fu, Gang and Bateni, Mohammadhossein and Kumar, Sanjiv}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {41074--41102}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/lukasik25a/lukasik25a.pdf}, url = {https://proceedings.mlr.press/v267/lukasik25a.html}, abstract = {Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal area under the ROC curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem—loss aggregation and label aggregation—by characterizing their Bayes-optimal solutions. We show that while both approaches can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.} }
Endnote
%0 Conference Paper %T Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation %A Michal Lukasik %A Lin Chen %A Harikrishna Narasimhan %A Aditya Krishna Menon %A Wittawat Jitkrittum %A Felix X. Yu %A Sashank J. Reddi %A Gang Fu %A Mohammadhossein Bateni %A Sanjiv Kumar %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-lukasik25a %I PMLR %P 41074--41102 %U https://proceedings.mlr.press/v267/lukasik25a.html %V 267 %X Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal area under the ROC curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem—loss aggregation and label aggregation—by characterizing their Bayes-optimal solutions. We show that while both approaches can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.
APA
Lukasik, M., Chen, L., Narasimhan, H., Menon, A.K., Jitkrittum, W., Yu, F.X., J. Reddi, S., Fu, G., Bateni, M. & Kumar, S.. (2025). Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:41074-41102 Available from https://proceedings.mlr.press/v267/lukasik25a.html.

Related Material