Venn-Abers Predictors

Vladimir Vovk Royal Holloway, Ivan Petej
Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, PMLR R12:539-548, 2014.

Abstract

ListMLE is a state-of-the-art listwise learning-to rank algorithm, which has been shown to work very well in application. It defines the probability distribution based on Plackett-Luce Model in a top-down style to take into account the position information. However, both empirical contradiction and theoretical results indicate that ListM- LE cannot well capture the position importance, which is a key factor in ranking. To amend the problem, this paper proposes a new listwise ranking method, called position-aware ListMLE (p-ListMLE for short). It views the ranking problem as a sequential learning process, with each step learning a subset of parameters which maximize the corresponding stepwise probability distribution. To solve this sequential multi-objective optimization problem, we propose to use linear scalarization strategy to transform it into a single-objective optimization problem, which is efficient for computation. Our theoretical study shows that p-ListMLE is better than ListM- LE in statistical consistency with respect to typical ranking evaluation measure NDCG. Furthermore, our experiments on benchmark datasets demonstrate that the proposed method can significantly improve the performance of ListMLE and outperform state-of-the-art listwise learning-to-rank algorithms as well.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR12-holloway14a, title = {Venn-Abers Predictors}, author = {Holloway, Vladimir Vovk Royal and Petej, Ivan}, booktitle = {Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence}, pages = {539--548}, year = {2014}, editor = {Zhang, Nevin L. and Tian, Jin}, volume = {R12}, series = {Proceedings of Machine Learning Research}, month = {23--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r12/main/assets/holloway14a/holloway14a.pdf}, url = {https://proceedings.mlr.press/r12/holloway14a.html}, abstract = {ListMLE is a state-of-the-art listwise learning-to rank algorithm, which has been shown to work very well in application. It defines the probability distribution based on Plackett-Luce Model in a top-down style to take into account the position information. However, both empirical contradiction and theoretical results indicate that ListM- LE cannot well capture the position importance, which is a key factor in ranking. To amend the problem, this paper proposes a new listwise ranking method, called position-aware ListMLE (p-ListMLE for short). It views the ranking problem as a sequential learning process, with each step learning a subset of parameters which maximize the corresponding stepwise probability distribution. To solve this sequential multi-objective optimization problem, we propose to use linear scalarization strategy to transform it into a single-objective optimization problem, which is efficient for computation. Our theoretical study shows that p-ListMLE is better than ListM- LE in statistical consistency with respect to typical ranking evaluation measure NDCG. Furthermore, our experiments on benchmark datasets demonstrate that the proposed method can significantly improve the performance of ListMLE and outperform state-of-the-art listwise learning-to-rank algorithms as well.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Venn-Abers Predictors %A Vladimir Vovk Royal Holloway %A Ivan Petej %B Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2014 %E Nevin L. Zhang %E Jin Tian %F pmlr-vR12-holloway14a %I PMLR %P 539--548 %U https://proceedings.mlr.press/r12/holloway14a.html %V R12 %X ListMLE is a state-of-the-art listwise learning-to rank algorithm, which has been shown to work very well in application. It defines the probability distribution based on Plackett-Luce Model in a top-down style to take into account the position information. However, both empirical contradiction and theoretical results indicate that ListM- LE cannot well capture the position importance, which is a key factor in ranking. To amend the problem, this paper proposes a new listwise ranking method, called position-aware ListMLE (p-ListMLE for short). It views the ranking problem as a sequential learning process, with each step learning a subset of parameters which maximize the corresponding stepwise probability distribution. To solve this sequential multi-objective optimization problem, we propose to use linear scalarization strategy to transform it into a single-objective optimization problem, which is efficient for computation. Our theoretical study shows that p-ListMLE is better than ListM- LE in statistical consistency with respect to typical ranking evaluation measure NDCG. Furthermore, our experiments on benchmark datasets demonstrate that the proposed method can significantly improve the performance of ListMLE and outperform state-of-the-art listwise learning-to-rank algorithms as well. %Z Reissued by PMLR on 04 October 2026.
APA
Holloway, V.V.R. & Petej, I.. (2014). Venn-Abers Predictors. Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R12:539-548 Available from https://proceedings.mlr.press/r12/holloway14a.html. Reissued by PMLR on 04 October 2026.

Related Material