Label-wise Aleatoric and Epistemic Uncertainty Quantification

Yusuf Sale, Paul Hofman, Timo Löhr, Lisa Wimmer, Thomas Nagler, Eyke Hüllermeier
Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, PMLR 244:3159-3179, 2024.

Abstract

We present a novel approach to uncertainty quantification in classification tasks based on label-wise decomposition of uncertainty measures. This label-wise perspective allows uncertainty to be quantified at the individual class level, thereby improving cost-sensitive decision-making and helping understand the sources of uncertainty. Furthermore, it allows to define total, aleatoric, and epistemic uncertainty on the basis of non-categorical measures such as variance, going beyond common entropy-based measures. In particular, variance-based measures address some of the limitations associated with established methods that have recently been discussed in the literature. We show that our proposed measures adhere to a number of desirable properties. Through empirical evaluation on a variety of benchmark data sets – including applications in the medical domain where accurate uncertainty quantification is crucial – we establish the effectiveness of label-wise uncertainty quantification.

Cite this Paper

BibTeX
@InProceedings{pmlr-v244-sale24a, title = {Label-wise Aleatoric and Epistemic Uncertainty Quantification}, author = {Sale, Yusuf and Hofman, Paul and L\"ohr, Timo and Wimmer, Lisa and Nagler, Thomas and H\"ullermeier, Eyke}, booktitle = {Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence}, pages = {3159--3179}, year = {2024}, editor = {Kiyavash, Negar and Mooij, Joris M.}, volume = {244}, series = {Proceedings of Machine Learning Research}, month = {15--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v244/main/assets/sale24a/sale24a.pdf}, url = {https://proceedings.mlr.press/v244/sale24a.html}, abstract = {We present a novel approach to uncertainty quantification in classification tasks based on label-wise decomposition of uncertainty measures. This label-wise perspective allows uncertainty to be quantified at the individual class level, thereby improving cost-sensitive decision-making and helping understand the sources of uncertainty. Furthermore, it allows to define total, aleatoric, and epistemic uncertainty on the basis of non-categorical measures such as variance, going beyond common entropy-based measures. In particular, variance-based measures address some of the limitations associated with established methods that have recently been discussed in the literature. We show that our proposed measures adhere to a number of desirable properties. Through empirical evaluation on a variety of benchmark data sets – including applications in the medical domain where accurate uncertainty quantification is crucial – we establish the effectiveness of label-wise uncertainty quantification.} }
Endnote
%0 Conference Paper %T Label-wise Aleatoric and Epistemic Uncertainty Quantification %A Yusuf Sale %A Paul Hofman %A Timo Löhr %A Lisa Wimmer %A Thomas Nagler %A Eyke Hüllermeier %B Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2024 %E Negar Kiyavash %E Joris M. Mooij %F pmlr-v244-sale24a %I PMLR %P 3159--3179 %U https://proceedings.mlr.press/v244/sale24a.html %V 244 %X We present a novel approach to uncertainty quantification in classification tasks based on label-wise decomposition of uncertainty measures. This label-wise perspective allows uncertainty to be quantified at the individual class level, thereby improving cost-sensitive decision-making and helping understand the sources of uncertainty. Furthermore, it allows to define total, aleatoric, and epistemic uncertainty on the basis of non-categorical measures such as variance, going beyond common entropy-based measures. In particular, variance-based measures address some of the limitations associated with established methods that have recently been discussed in the literature. We show that our proposed measures adhere to a number of desirable properties. Through empirical evaluation on a variety of benchmark data sets – including applications in the medical domain where accurate uncertainty quantification is crucial – we establish the effectiveness of label-wise uncertainty quantification.
APA
Sale, Y., Hofman, P., Löhr, T., Wimmer, L., Nagler, T. & Hüllermeier, E.. (2024). Label-wise Aleatoric and Epistemic Uncertainty Quantification. Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 244:3159-3179 Available from https://proceedings.mlr.press/v244/sale24a.html.

Related Material