Fat–Tailed Variational Inference with Anisotropic Tail Adaptive Flows

Feynman Liang, Michael Mahoney, Liam Hodgkinson
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:13257-13270, 2022.

Abstract

While fat-tailed densities commonly arise as posterior and marginal distributions in robust models and scale mixtures, they present a problematic scenario when Gaussian-based variational inference fails to accurately capture tail decay. We first improve previous theory on tails of Lipschitz flows by quantifying how they affect the rate of tail decay and expanding the theory to non-Lipschitz polynomial flows. Next, we develop an alternative theory for multivariate tail parameters which is sensitive to tail-anisotropy. In doing so, we unveil a fundamental problem which plagues many existing flow-based methods: they can only model tail-isotropic distributions (i.e., distributions having the same tail parameter in every direction). To mitigate this and enable modeling of tail-anisotropic targets, we propose anisotropic tail-adaptive flows (ATAF). Experimental results confirm ATAF on both synthetic and real-world targets is competitive with prior work while also exhibiting appropriate tail-anisotropy.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-liang22a, title = {{F}at{–}{T}ailed Variational Inference with Anisotropic Tail Adaptive Flows}, author = {Liang, Feynman and Mahoney, Michael and Hodgkinson, Liam}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {13257--13270}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/liang22a/liang22a.pdf}, url = {https://proceedings.mlr.press/v162/liang22a.html}, abstract = {While fat-tailed densities commonly arise as posterior and marginal distributions in robust models and scale mixtures, they present a problematic scenario when Gaussian-based variational inference fails to accurately capture tail decay. We first improve previous theory on tails of Lipschitz flows by quantifying how they affect the rate of tail decay and expanding the theory to non-Lipschitz polynomial flows. Next, we develop an alternative theory for multivariate tail parameters which is sensitive to tail-anisotropy. In doing so, we unveil a fundamental problem which plagues many existing flow-based methods: they can only model tail-isotropic distributions (i.e., distributions having the same tail parameter in every direction). To mitigate this and enable modeling of tail-anisotropic targets, we propose anisotropic tail-adaptive flows (ATAF). Experimental results confirm ATAF on both synthetic and real-world targets is competitive with prior work while also exhibiting appropriate tail-anisotropy.} }
Endnote
%0 Conference Paper %T Fat–Tailed Variational Inference with Anisotropic Tail Adaptive Flows %A Feynman Liang %A Michael Mahoney %A Liam Hodgkinson %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-liang22a %I PMLR %P 13257--13270 %U https://proceedings.mlr.press/v162/liang22a.html %V 162 %X While fat-tailed densities commonly arise as posterior and marginal distributions in robust models and scale mixtures, they present a problematic scenario when Gaussian-based variational inference fails to accurately capture tail decay. We first improve previous theory on tails of Lipschitz flows by quantifying how they affect the rate of tail decay and expanding the theory to non-Lipschitz polynomial flows. Next, we develop an alternative theory for multivariate tail parameters which is sensitive to tail-anisotropy. In doing so, we unveil a fundamental problem which plagues many existing flow-based methods: they can only model tail-isotropic distributions (i.e., distributions having the same tail parameter in every direction). To mitigate this and enable modeling of tail-anisotropic targets, we propose anisotropic tail-adaptive flows (ATAF). Experimental results confirm ATAF on both synthetic and real-world targets is competitive with prior work while also exhibiting appropriate tail-anisotropy.
APA
Liang, F., Mahoney, M. & Hodgkinson, L.. (2022). Fat–Tailed Variational Inference with Anisotropic Tail Adaptive Flows. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:13257-13270 Available from https://proceedings.mlr.press/v162/liang22a.html.

Related Material