A geometric method for improved uncertainty estimation in real-time

Gabriella Chouraqui, Liron Cohen, Gil Einziger, Liel Leman
Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, PMLR 180:422-432, 2022.

Abstract

Machine learning classifiers are probabilistic in nature, and thus inevitably involve uncertainty. Predicting the probability of a specific input to be correct is called uncertainty (or confidence) estimation and is crucial for risk management.Post-hoc model calibrations can improve models’ uncertainty estimations without the need for retraining, and without changing the model.Our work puts forward a geometric-based approach for uncertainty estimation. Roughly speaking, we use the geometric distance of the current input from the existing training inputs as a signal for estimating uncertainty and then calibrate that signal (instead of the model’s estimation) using standard post-hoc calibration techniques. We show that our method yields better uncertainty estimations than recently proposed approaches by extensively evaluating multiple datasets and models. In addition, we also demonstrate the possibility of performing our approach in near real-time applications. Our code is available at our Github: https://github.com/NoSleepDeveloper/Geometric-Calibrator

Cite this Paper

BibTeX
@InProceedings{pmlr-v180-chouraqui22a, title = {A geometric method for improved uncertainty estimation in real-time}, author = {Chouraqui, Gabriella and Cohen, Liron and Einziger, Gil and Leman, Liel}, booktitle = {Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence}, pages = {422--432}, year = {2022}, editor = {Cussens, James and Zhang, Kun}, volume = {180}, series = {Proceedings of Machine Learning Research}, month = {01--05 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v180/chouraqui22a/chouraqui22a.pdf}, url = {https://proceedings.mlr.press/v180/chouraqui22a.html}, abstract = {Machine learning classifiers are probabilistic in nature, and thus inevitably involve uncertainty. Predicting the probability of a specific input to be correct is called uncertainty (or confidence) estimation and is crucial for risk management.Post-hoc model calibrations can improve models’ uncertainty estimations without the need for retraining, and without changing the model.Our work puts forward a geometric-based approach for uncertainty estimation. Roughly speaking, we use the geometric distance of the current input from the existing training inputs as a signal for estimating uncertainty and then calibrate that signal (instead of the model’s estimation) using standard post-hoc calibration techniques. We show that our method yields better uncertainty estimations than recently proposed approaches by extensively evaluating multiple datasets and models. In addition, we also demonstrate the possibility of performing our approach in near real-time applications. Our code is available at our Github: https://github.com/NoSleepDeveloper/Geometric-Calibrator } }
Endnote
%0 Conference Paper %T A geometric method for improved uncertainty estimation in real-time %A Gabriella Chouraqui %A Liron Cohen %A Gil Einziger %A Liel Leman %B Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2022 %E James Cussens %E Kun Zhang %F pmlr-v180-chouraqui22a %I PMLR %P 422--432 %U https://proceedings.mlr.press/v180/chouraqui22a.html %V 180 %X Machine learning classifiers are probabilistic in nature, and thus inevitably involve uncertainty. Predicting the probability of a specific input to be correct is called uncertainty (or confidence) estimation and is crucial for risk management.Post-hoc model calibrations can improve models’ uncertainty estimations without the need for retraining, and without changing the model.Our work puts forward a geometric-based approach for uncertainty estimation. Roughly speaking, we use the geometric distance of the current input from the existing training inputs as a signal for estimating uncertainty and then calibrate that signal (instead of the model’s estimation) using standard post-hoc calibration techniques. We show that our method yields better uncertainty estimations than recently proposed approaches by extensively evaluating multiple datasets and models. In addition, we also demonstrate the possibility of performing our approach in near real-time applications. Our code is available at our Github: https://github.com/NoSleepDeveloper/Geometric-Calibrator
APA
Chouraqui, G., Cohen, L., Einziger, G. & Leman, L.. (2022). A geometric method for improved uncertainty estimation in real-time. Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 180:422-432 Available from https://proceedings.mlr.press/v180/chouraqui22a.html.

Related Material