A Variational Information Theoretic Approach to Out-of-Distribution Detection

Sudeepta Mondal, Zhuolin Jiang, Ganesh Sundaramoorthi
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:44627-44651, 2025.

Abstract

We present a theory for the construction of out-of-distribution (OOD) detection features for neural networks. We introduce random features for OOD through a novel information-theoretic loss functional consisting of two terms, the first based on the KL divergence separates resulting in-distribution (ID) and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information. We formulate a variational procedure to optimize the loss and obtain OOD features. Based on assumptions on OOD distributions, one can recover properties of existing OOD features, i.e., shaping functions. Furthermore, we show that our theory can predict a new shaping function that out-performs existing ones on OOD benchmarks. Our theory provides a general framework for constructing a variety of new features with clear explainability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-mondal25a, title = {A Variational Information Theoretic Approach to Out-of-Distribution Detection}, author = {Mondal, Sudeepta and Jiang, Zhuolin and Sundaramoorthi, Ganesh}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {44627--44651}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/mondal25a/mondal25a.pdf}, url = {https://proceedings.mlr.press/v267/mondal25a.html}, abstract = {We present a theory for the construction of out-of-distribution (OOD) detection features for neural networks. We introduce random features for OOD through a novel information-theoretic loss functional consisting of two terms, the first based on the KL divergence separates resulting in-distribution (ID) and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information. We formulate a variational procedure to optimize the loss and obtain OOD features. Based on assumptions on OOD distributions, one can recover properties of existing OOD features, i.e., shaping functions. Furthermore, we show that our theory can predict a new shaping function that out-performs existing ones on OOD benchmarks. Our theory provides a general framework for constructing a variety of new features with clear explainability.} }
Endnote
%0 Conference Paper %T A Variational Information Theoretic Approach to Out-of-Distribution Detection %A Sudeepta Mondal %A Zhuolin Jiang %A Ganesh Sundaramoorthi %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-mondal25a %I PMLR %P 44627--44651 %U https://proceedings.mlr.press/v267/mondal25a.html %V 267 %X We present a theory for the construction of out-of-distribution (OOD) detection features for neural networks. We introduce random features for OOD through a novel information-theoretic loss functional consisting of two terms, the first based on the KL divergence separates resulting in-distribution (ID) and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information. We formulate a variational procedure to optimize the loss and obtain OOD features. Based on assumptions on OOD distributions, one can recover properties of existing OOD features, i.e., shaping functions. Furthermore, we show that our theory can predict a new shaping function that out-performs existing ones on OOD benchmarks. Our theory provides a general framework for constructing a variety of new features with clear explainability.
APA
Mondal, S., Jiang, Z. & Sundaramoorthi, G.. (2025). A Variational Information Theoretic Approach to Out-of-Distribution Detection. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:44627-44651 Available from https://proceedings.mlr.press/v267/mondal25a.html.

Related Material