What Does Preference Learning Recover from Pairwise Comparison Data?

Rattana Pukdee, Maria Florina Balcan, Pradeep Ravikumar
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:100021-100050, 2026.

Abstract

Pairwise preference learning is central to machine learning, with recent applications in aligning language models with human preferences. A typical dataset consists of triplets $(x, y^+, y^-)$, where response $y^+$ is preferred over response $y^-$ for context $x$. The Bradley–Terry (BT) model is the predominant approach, modeling preference probabilities as a function of latent score differences. Standard practice assumes data follows this model and learns the latent scores accordingly. However, real data may violate this assumption, and it remains unclear what BT learning recovers in such cases. Starting from triplet comparison data, we formalize the preference information it encodes through the conditional preference distribution (CPRD). We give precise conditions for when BT is appropriate for modeling the CPRD, and identify factors governing sample efficiency—namely, margin and connectivity. Together, these results offer a data-centric foundation for understanding what preference learning actually recovers.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-pukdee26a, title = {What Does Preference Learning Recover from Pairwise Comparison Data?}, author = {Pukdee, Rattana and Balcan, Maria Florina and Ravikumar, Pradeep}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {100021--100050}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/pukdee26a/pukdee26a.pdf}, url = {https://proceedings.mlr.press/v306/pukdee26a.html}, abstract = {Pairwise preference learning is central to machine learning, with recent applications in aligning language models with human preferences. A typical dataset consists of triplets $(x, y^+, y^-)$, where response $y^+$ is preferred over response $y^-$ for context $x$. The Bradley–Terry (BT) model is the predominant approach, modeling preference probabilities as a function of latent score differences. Standard practice assumes data follows this model and learns the latent scores accordingly. However, real data may violate this assumption, and it remains unclear what BT learning recovers in such cases. Starting from triplet comparison data, we formalize the preference information it encodes through the conditional preference distribution (CPRD). We give precise conditions for when BT is appropriate for modeling the CPRD, and identify factors governing sample efficiency—namely, margin and connectivity. Together, these results offer a data-centric foundation for understanding what preference learning actually recovers.} }
Endnote
%0 Conference Paper %T What Does Preference Learning Recover from Pairwise Comparison Data? %A Rattana Pukdee %A Maria Florina Balcan %A Pradeep Ravikumar %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-pukdee26a %I PMLR %P 100021--100050 %U https://proceedings.mlr.press/v306/pukdee26a.html %V 306 %X Pairwise preference learning is central to machine learning, with recent applications in aligning language models with human preferences. A typical dataset consists of triplets $(x, y^+, y^-)$, where response $y^+$ is preferred over response $y^-$ for context $x$. The Bradley–Terry (BT) model is the predominant approach, modeling preference probabilities as a function of latent score differences. Standard practice assumes data follows this model and learns the latent scores accordingly. However, real data may violate this assumption, and it remains unclear what BT learning recovers in such cases. Starting from triplet comparison data, we formalize the preference information it encodes through the conditional preference distribution (CPRD). We give precise conditions for when BT is appropriate for modeling the CPRD, and identify factors governing sample efficiency—namely, margin and connectivity. Together, these results offer a data-centric foundation for understanding what preference learning actually recovers.
APA
Pukdee, R., Balcan, M.F. & Ravikumar, P.. (2026). What Does Preference Learning Recover from Pairwise Comparison Data?. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:100021-100050 Available from https://proceedings.mlr.press/v306/pukdee26a.html.

Related Material