Venn predictors for well-calibrated probability estimation trees

[edit]

Ulf Johansson, Tuwe Löfström, Håkan Sundell, Henrik Linusson, Anders Gidenstam, Henrik Boström ;
Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, PMLR 91:3-14, 2018.

Abstract

Successful use of probabilistic classification requires well-calibrated probability estimates, i.e., the predicted class probabilities must correspond to the true probabilities. The standard solution is to employ an additional step, transforming the outputs from a classifier into probability estimates. In this paper, Venn predictors are compared to Platt scaling and isotonic regression, for the purpose of producing well-calibrated probabilistic predictions from decision trees. The empirical investigation, using 22 publicly available data sets, showed that the probability estimates from the Venn predictor were extremely well-calibrated. In fact, in a direct comparison using the accepted reliability metric, the Venn predictor estimates were the most exact on every data set.

Related Material