Probability Calibration Trees

Tim Leathart, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer
Proceedings of the Ninth Asian Conference on Machine Learning, PMLR 77:145-160, 2017.

Abstract

Obtaining accurate and well calibrated probability estimates from classifiers is useful in many applications, for example, when minimising the expected cost of classifications. Existing methods of calibrating probability estimates are applied globally, ignoring the potential for improvements by applying a more fine-grained model. We propose probability calibration trees, a modification of logistic model trees that identifies regions of the input space in which different probability calibration models are learned to improve performance. We compare probability calibration trees to two widely used calibration methods—isotonic regression and Platt scaling—and show that our method results in lower root mean squared error on average than both methods, for estimates produced by a variety of base learners.

Cite this Paper

BibTeX
@InProceedings{pmlr-v77-leathart17a, title = {Probability Calibration Trees}, author = {Leathart, Tim and Frank, Eibe and Holmes, Geoffrey and Pfahringer, Bernhard}, booktitle = {Proceedings of the Ninth Asian Conference on Machine Learning}, pages = {145--160}, year = {2017}, editor = {Zhang, Min-Ling and Noh, Yung-Kyun}, volume = {77}, series = {Proceedings of Machine Learning Research}, address = {Yonsei University, Seoul, Republic of Korea}, month = {15--17 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v77/leathart17a/leathart17a.pdf}, url = {https://proceedings.mlr.press/v77/leathart17a.html}, abstract = {Obtaining accurate and well calibrated probability estimates from classifiers is useful in many applications, for example, when minimising the expected cost of classifications. Existing methods of calibrating probability estimates are applied globally, ignoring the potential for improvements by applying a more fine-grained model. We propose probability calibration trees, a modification of logistic model trees that identifies regions of the input space in which different probability calibration models are learned to improve performance. We compare probability calibration trees to two widely used calibration methods—isotonic regression and Platt scaling—and show that our method results in lower root mean squared error on average than both methods, for estimates produced by a variety of base learners.} }
Endnote
%0 Conference Paper %T Probability Calibration Trees %A Tim Leathart %A Eibe Frank %A Geoffrey Holmes %A Bernhard Pfahringer %B Proceedings of the Ninth Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Min-Ling Zhang %E Yung-Kyun Noh %F pmlr-v77-leathart17a %I PMLR %P 145--160 %U https://proceedings.mlr.press/v77/leathart17a.html %V 77 %X Obtaining accurate and well calibrated probability estimates from classifiers is useful in many applications, for example, when minimising the expected cost of classifications. Existing methods of calibrating probability estimates are applied globally, ignoring the potential for improvements by applying a more fine-grained model. We propose probability calibration trees, a modification of logistic model trees that identifies regions of the input space in which different probability calibration models are learned to improve performance. We compare probability calibration trees to two widely used calibration methods—isotonic regression and Platt scaling—and show that our method results in lower root mean squared error on average than both methods, for estimates produced by a variety of base learners.
APA
Leathart, T., Frank, E., Holmes, G. & Pfahringer, B.. (2017). Probability Calibration Trees. Proceedings of the Ninth Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 77:145-160 Available from https://proceedings.mlr.press/v77/leathart17a.html.

Related Material