Sobolev Space Regularised Pre Density Models

Mark Kozdoba, Binyamin Perets, Shie Mannor
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:25494-25533, 2024.

Abstract

We propose a new approach to non-parametric density estimation that is based on regularizing a Sobolev norm of the density. This method is statistically consistent, and makes the inductive bias of the model clear and interpretable. While there is no closed analytic form for the associated kernel, we show that one can approximate it using sampling. The optimization problem needed to determine the density is non-convex, and standard gradient methods do not perform well. However, we show that with an appropriate initialization and using natural gradients, one can obtain well performing solutions. Finally, while the approach provides pre-densities (i.e. not necessarily integrating to 1), which prevents the use of log-likelihood for cross validation, we show that one can instead adapt Fisher divergence based score matching methods for this task. We evaluate the resulting method on the comprehensive recent anomaly detection benchmark suite, ADBench, and find that it ranks second best, among more than 15 algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-kozdoba24a, title = {Sobolev Space Regularised Pre Density Models}, author = {Kozdoba, Mark and Perets, Binyamin and Mannor, Shie}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {25494--25533}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/kozdoba24a/kozdoba24a.pdf}, url = {https://proceedings.mlr.press/v235/kozdoba24a.html}, abstract = {We propose a new approach to non-parametric density estimation that is based on regularizing a Sobolev norm of the density. This method is statistically consistent, and makes the inductive bias of the model clear and interpretable. While there is no closed analytic form for the associated kernel, we show that one can approximate it using sampling. The optimization problem needed to determine the density is non-convex, and standard gradient methods do not perform well. However, we show that with an appropriate initialization and using natural gradients, one can obtain well performing solutions. Finally, while the approach provides pre-densities (i.e. not necessarily integrating to 1), which prevents the use of log-likelihood for cross validation, we show that one can instead adapt Fisher divergence based score matching methods for this task. We evaluate the resulting method on the comprehensive recent anomaly detection benchmark suite, ADBench, and find that it ranks second best, among more than 15 algorithms.} }
Endnote
%0 Conference Paper %T Sobolev Space Regularised Pre Density Models %A Mark Kozdoba %A Binyamin Perets %A Shie Mannor %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-kozdoba24a %I PMLR %P 25494--25533 %U https://proceedings.mlr.press/v235/kozdoba24a.html %V 235 %X We propose a new approach to non-parametric density estimation that is based on regularizing a Sobolev norm of the density. This method is statistically consistent, and makes the inductive bias of the model clear and interpretable. While there is no closed analytic form for the associated kernel, we show that one can approximate it using sampling. The optimization problem needed to determine the density is non-convex, and standard gradient methods do not perform well. However, we show that with an appropriate initialization and using natural gradients, one can obtain well performing solutions. Finally, while the approach provides pre-densities (i.e. not necessarily integrating to 1), which prevents the use of log-likelihood for cross validation, we show that one can instead adapt Fisher divergence based score matching methods for this task. We evaluate the resulting method on the comprehensive recent anomaly detection benchmark suite, ADBench, and find that it ranks second best, among more than 15 algorithms.
APA
Kozdoba, M., Perets, B. & Mannor, S.. (2024). Sobolev Space Regularised Pre Density Models. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:25494-25533 Available from https://proceedings.mlr.press/v235/kozdoba24a.html.

Related Material