Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain

Takahiro Mimori, Keiko Sasada, Hirotaka Matsui, Issei Sato
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:3664-3672, 2021.

Abstract

We propose an evaluation framework for class probability estimates (CPEs) in the presence of label uncertainty, which is commonly observed as diagnosis disagreement between experts in the medical domain. We also formalize evaluation metrics for higher-order statistics, including inter-rater disagreement, to assess predictions on label uncertainty. Moreover, we propose a novel post-hoc method called alpha-calibration, that equips neural network classifiers with calibrated distributions over CPEs. Using synthetic experiments and a large-scale medical imaging application, we show that our approach significantly enhances the reliability of uncertainty estimates: disagreement probabilities and posterior CPEs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-mimori21a, title = { Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain }, author = {Mimori, Takahiro and Sasada, Keiko and Matsui, Hirotaka and Sato, Issei}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {3664--3672}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/mimori21a/mimori21a.pdf}, url = {https://proceedings.mlr.press/v130/mimori21a.html}, abstract = { We propose an evaluation framework for class probability estimates (CPEs) in the presence of label uncertainty, which is commonly observed as diagnosis disagreement between experts in the medical domain. We also formalize evaluation metrics for higher-order statistics, including inter-rater disagreement, to assess predictions on label uncertainty. Moreover, we propose a novel post-hoc method called alpha-calibration, that equips neural network classifiers with calibrated distributions over CPEs. Using synthetic experiments and a large-scale medical imaging application, we show that our approach significantly enhances the reliability of uncertainty estimates: disagreement probabilities and posterior CPEs. } }
Endnote
%0 Conference Paper %T Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain %A Takahiro Mimori %A Keiko Sasada %A Hirotaka Matsui %A Issei Sato %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-mimori21a %I PMLR %P 3664--3672 %U https://proceedings.mlr.press/v130/mimori21a.html %V 130 %X We propose an evaluation framework for class probability estimates (CPEs) in the presence of label uncertainty, which is commonly observed as diagnosis disagreement between experts in the medical domain. We also formalize evaluation metrics for higher-order statistics, including inter-rater disagreement, to assess predictions on label uncertainty. Moreover, we propose a novel post-hoc method called alpha-calibration, that equips neural network classifiers with calibrated distributions over CPEs. Using synthetic experiments and a large-scale medical imaging application, we show that our approach significantly enhances the reliability of uncertainty estimates: disagreement probabilities and posterior CPEs.
APA
Mimori, T., Sasada, K., Matsui, H. & Sato, I.. (2021). Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:3664-3672 Available from https://proceedings.mlr.press/v130/mimori21a.html.

Related Material