Venn predictors for well-calibrated probability estimation trees

Ulf Johansson, Tuwe Löfström, Håkan Sundell, Henrik Linusson, Anders Gidenstam, Henrik Boström
Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, PMLR 91:3-14, 2018.

Abstract

Successful use of probabilistic classification requires well-calibrated probability estimates, i.e., the predicted class probabilities must correspond to the true probabilities. The standard solution is to employ an additional step, transforming the outputs from a classifier into probability estimates. In this paper, Venn predictors are compared to Platt scaling and isotonic regression, for the purpose of producing well-calibrated probabilistic predictions from decision trees. The empirical investigation, using 22 publicly available data sets, showed that the probability estimates from the Venn predictor were extremely well-calibrated. In fact, in a direct comparison using the accepted reliability metric, the Venn predictor estimates were the most exact on every data set.

Cite this Paper

BibTeX
@InProceedings{pmlr-v91-johansson18a, title = {Venn predictors for well-calibrated probability estimation trees}, author = {Johansson, Ulf and Löfström, Tuwe and Sundell, Håkan and Linusson, Henrik and Gidenstam, Anders and Boström, Henrik}, booktitle = {Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications}, pages = {3--14}, year = {2018}, editor = {Gammerman, Alex and Vovk, Vladimir and Luo, Zhiyuan and Smirnov, Evgueni and Peeters, Ralf}, volume = {91}, series = {Proceedings of Machine Learning Research}, month = {11--13 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v91/johansson18a/johansson18a.pdf}, url = {https://proceedings.mlr.press/v91/johansson18a.html}, abstract = {Successful use of probabilistic classification requires well-calibrated probability estimates, i.e., the predicted class probabilities must correspond to the true probabilities. The standard solution is to employ an additional step, transforming the outputs from a classifier into probability estimates. In this paper, Venn predictors are compared to Platt scaling and isotonic regression, for the purpose of producing well-calibrated probabilistic predictions from decision trees. The empirical investigation, using 22 publicly available data sets, showed that the probability estimates from the Venn predictor were extremely well-calibrated. In fact, in a direct comparison using the accepted reliability metric, the Venn predictor estimates were the most exact on every data set.} }
Endnote
%0 Conference Paper %T Venn predictors for well-calibrated probability estimation trees %A Ulf Johansson %A Tuwe Löfström %A Håkan Sundell %A Henrik Linusson %A Anders Gidenstam %A Henrik Boström %B Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications %C Proceedings of Machine Learning Research %D 2018 %E Alex Gammerman %E Vladimir Vovk %E Zhiyuan Luo %E Evgueni Smirnov %E Ralf Peeters %F pmlr-v91-johansson18a %I PMLR %P 3--14 %U https://proceedings.mlr.press/v91/johansson18a.html %V 91 %X Successful use of probabilistic classification requires well-calibrated probability estimates, i.e., the predicted class probabilities must correspond to the true probabilities. The standard solution is to employ an additional step, transforming the outputs from a classifier into probability estimates. In this paper, Venn predictors are compared to Platt scaling and isotonic regression, for the purpose of producing well-calibrated probabilistic predictions from decision trees. The empirical investigation, using 22 publicly available data sets, showed that the probability estimates from the Venn predictor were extremely well-calibrated. In fact, in a direct comparison using the accepted reliability metric, the Venn predictor estimates were the most exact on every data set.
APA
Johansson, U., Löfström, T., Sundell, H., Linusson, H., Gidenstam, A. & Boström, H.. (2018). Venn predictors for well-calibrated probability estimation trees. Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, in Proceedings of Machine Learning Research 91:3-14 Available from https://proceedings.mlr.press/v91/johansson18a.html.

Related Material