Density of States Estimation for Out of Distribution Detection

Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, Joshua Dillon
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:3232-3240, 2021.

Abstract

Perhaps surprisingly, recent studies have shown probabilistic model likelihoods have poor specificity for out-of-distribution (OOD) detection and often assign higher likelihoods to OOD data than in-distribution data. To ameliorate this issue we propose DoSE, the density of states estimator. Drawing on the statistical physics notion of “density of states,” the DoSE decision rule avoids direct comparison of model probabilities, and instead utilizes the “probability of the model probability,” or indeed the frequency of any reasonable statistic. The frequency is calculated using nonparametric density estimators (e.g., KDE and one-class SVM) which measure the typicality of various model statistics given the training data and from which we can flag test points with low typicality as anomalous. Unlike many other methods, DoSE requires neither labeled data nor OOD examples. DoSE is modular and can be trivially applied to any existing, trained model. We demonstrate DoSE’s state-of-the-art performance against other unsupervised OOD detectors on previously established “hard” benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-morningstar21a, title = { Density of States Estimation for Out of Distribution Detection }, author = {Morningstar, Warren and Ham, Cusuh and Gallagher, Andrew and Lakshminarayanan, Balaji and Alemi, Alex and Dillon, Joshua}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {3232--3240}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/morningstar21a/morningstar21a.pdf}, url = {https://proceedings.mlr.press/v130/morningstar21a.html}, abstract = { Perhaps surprisingly, recent studies have shown probabilistic model likelihoods have poor specificity for out-of-distribution (OOD) detection and often assign higher likelihoods to OOD data than in-distribution data. To ameliorate this issue we propose DoSE, the density of states estimator. Drawing on the statistical physics notion of “density of states,” the DoSE decision rule avoids direct comparison of model probabilities, and instead utilizes the “probability of the model probability,” or indeed the frequency of any reasonable statistic. The frequency is calculated using nonparametric density estimators (e.g., KDE and one-class SVM) which measure the typicality of various model statistics given the training data and from which we can flag test points with low typicality as anomalous. Unlike many other methods, DoSE requires neither labeled data nor OOD examples. DoSE is modular and can be trivially applied to any existing, trained model. We demonstrate DoSE’s state-of-the-art performance against other unsupervised OOD detectors on previously established “hard” benchmarks. } }
Endnote
%0 Conference Paper %T Density of States Estimation for Out of Distribution Detection %A Warren Morningstar %A Cusuh Ham %A Andrew Gallagher %A Balaji Lakshminarayanan %A Alex Alemi %A Joshua Dillon %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-morningstar21a %I PMLR %P 3232--3240 %U https://proceedings.mlr.press/v130/morningstar21a.html %V 130 %X Perhaps surprisingly, recent studies have shown probabilistic model likelihoods have poor specificity for out-of-distribution (OOD) detection and often assign higher likelihoods to OOD data than in-distribution data. To ameliorate this issue we propose DoSE, the density of states estimator. Drawing on the statistical physics notion of “density of states,” the DoSE decision rule avoids direct comparison of model probabilities, and instead utilizes the “probability of the model probability,” or indeed the frequency of any reasonable statistic. The frequency is calculated using nonparametric density estimators (e.g., KDE and one-class SVM) which measure the typicality of various model statistics given the training data and from which we can flag test points with low typicality as anomalous. Unlike many other methods, DoSE requires neither labeled data nor OOD examples. DoSE is modular and can be trivially applied to any existing, trained model. We demonstrate DoSE’s state-of-the-art performance against other unsupervised OOD detectors on previously established “hard” benchmarks.
APA
Morningstar, W., Ham, C., Gallagher, A., Lakshminarayanan, B., Alemi, A. & Dillon, J.. (2021). Density of States Estimation for Out of Distribution Detection . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:3232-3240 Available from https://proceedings.mlr.press/v130/morningstar21a.html.

Related Material