Venn-Abers Predictors

Vladimir Vovk Royal Holloway, Ivan Petej
Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, PMLR R12:539-548, 2014.

Abstract

ListMLE is a state-of-the-art listwise learning-to- rank algorithm, which has been shown to work very well in application. It defines the probabil- ity distribution based on Plackett-Luce Model in a top-down style to take into account the position information. However, both empirical contradic- tion and theoretical results indicate that ListM- LE cannot well capture the position importance, which is a key factor in ranking. To amend the problem, this paper proposes a new listwise rank- ing method, called position-aware ListMLE (p- ListMLE for short). It views the ranking prob- lem as a sequential learning process, with each step learning a subset of parameters which maxi- mize the corresponding stepwise probability dis- tribution. To solve this sequential multi-objective optimization problem, we propose to use lin- ear scalarization strategy to transform it into a single-objective optimization problem, which is efficient for computation. Our theoretical s- tudy shows that p-ListMLE is better than ListM- LE in statistical consistency with respect to typi- cal ranking evaluation measure NDCG. Further- more, our experiments on benchmark datasets demonstrate that the proposed method can sig- nificantly improve the performance of ListMLE and outperform state-of-the-art listwise learning- to-rank algorithms as well.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR12-holloway14a, title = {Venn-Abers Predictors}, author = {Holloway, Vladimir Vovk Royal and Petej, Ivan}, booktitle = {Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence}, pages = {539--548}, year = {2014}, editor = {Zhang, Nevin L. and Tian, Jin}, volume = {R12}, series = {Proceedings of Machine Learning Research}, month = {23--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r12/main/assets/holloway14a/holloway14a.pdf}, url = {https://proceedings.mlr.press/r12/holloway14a.html}, abstract = {ListMLE is a state-of-the-art listwise learning-to- rank algorithm, which has been shown to work very well in application. It defines the probabil- ity distribution based on Plackett-Luce Model in a top-down style to take into account the position information. However, both empirical contradic- tion and theoretical results indicate that ListM- LE cannot well capture the position importance, which is a key factor in ranking. To amend the problem, this paper proposes a new listwise rank- ing method, called position-aware ListMLE (p- ListMLE for short). It views the ranking prob- lem as a sequential learning process, with each step learning a subset of parameters which maxi- mize the corresponding stepwise probability dis- tribution. To solve this sequential multi-objective optimization problem, we propose to use lin- ear scalarization strategy to transform it into a single-objective optimization problem, which is efficient for computation. Our theoretical s- tudy shows that p-ListMLE is better than ListM- LE in statistical consistency with respect to typi- cal ranking evaluation measure NDCG. Further- more, our experiments on benchmark datasets demonstrate that the proposed method can sig- nificantly improve the performance of ListMLE and outperform state-of-the-art listwise learning- to-rank algorithms as well.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Venn-Abers Predictors %A Vladimir Vovk Royal Holloway %A Ivan Petej %B Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2014 %E Nevin L. Zhang %E Jin Tian %F pmlr-vR12-holloway14a %I PMLR %P 539--548 %U https://proceedings.mlr.press/r12/holloway14a.html %V R12 %X ListMLE is a state-of-the-art listwise learning-to- rank algorithm, which has been shown to work very well in application. It defines the probabil- ity distribution based on Plackett-Luce Model in a top-down style to take into account the position information. However, both empirical contradic- tion and theoretical results indicate that ListM- LE cannot well capture the position importance, which is a key factor in ranking. To amend the problem, this paper proposes a new listwise rank- ing method, called position-aware ListMLE (p- ListMLE for short). It views the ranking prob- lem as a sequential learning process, with each step learning a subset of parameters which maxi- mize the corresponding stepwise probability dis- tribution. To solve this sequential multi-objective optimization problem, we propose to use lin- ear scalarization strategy to transform it into a single-objective optimization problem, which is efficient for computation. Our theoretical s- tudy shows that p-ListMLE is better than ListM- LE in statistical consistency with respect to typi- cal ranking evaluation measure NDCG. Further- more, our experiments on benchmark datasets demonstrate that the proposed method can sig- nificantly improve the performance of ListMLE and outperform state-of-the-art listwise learning- to-rank algorithms as well. %Z Reissued by PMLR on 04 October 2026.
APA
Holloway, V.V.R. & Petej, I.. (2014). Venn-Abers Predictors. Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R12:539-548 Available from https://proceedings.mlr.press/r12/holloway14a.html. Reissued by PMLR on 04 October 2026.

Related Material