On the difficulty of unbiased alpha divergence minimization

Tomas Geffner, Justin Domke
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:3650-3659, 2021.

Abstract

Several approximate inference algorithms have been proposed to minimize an alpha-divergence between an approximating distribution and a target distribution. Many of these algorithms introduce bias, the magnitude of which becomes problematic in high dimensions. Other algorithms are unbiased. These often seem to suffer from high variance, but little is rigorously known. In this work we study unbiased methods for alpha-divergence minimization through the Signal-to-Noise Ratio (SNR) of the gradient estimator. We study several representative scenarios where strong analytical results are possible, such as fully-factorized or Gaussian distributions. We find that when alpha is not zero, the SNR worsens exponentially in the dimensionality of the problem. This casts doubt on the practicality of these methods. We empirically confirm these theoretical results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-geffner21a, title = {On the difficulty of unbiased alpha divergence minimization}, author = {Geffner, Tomas and Domke, Justin}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {3650--3659}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/geffner21a/geffner21a.pdf}, url = {https://proceedings.mlr.press/v139/geffner21a.html}, abstract = {Several approximate inference algorithms have been proposed to minimize an alpha-divergence between an approximating distribution and a target distribution. Many of these algorithms introduce bias, the magnitude of which becomes problematic in high dimensions. Other algorithms are unbiased. These often seem to suffer from high variance, but little is rigorously known. In this work we study unbiased methods for alpha-divergence minimization through the Signal-to-Noise Ratio (SNR) of the gradient estimator. We study several representative scenarios where strong analytical results are possible, such as fully-factorized or Gaussian distributions. We find that when alpha is not zero, the SNR worsens exponentially in the dimensionality of the problem. This casts doubt on the practicality of these methods. We empirically confirm these theoretical results.} }
Endnote
%0 Conference Paper %T On the difficulty of unbiased alpha divergence minimization %A Tomas Geffner %A Justin Domke %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-geffner21a %I PMLR %P 3650--3659 %U https://proceedings.mlr.press/v139/geffner21a.html %V 139 %X Several approximate inference algorithms have been proposed to minimize an alpha-divergence between an approximating distribution and a target distribution. Many of these algorithms introduce bias, the magnitude of which becomes problematic in high dimensions. Other algorithms are unbiased. These often seem to suffer from high variance, but little is rigorously known. In this work we study unbiased methods for alpha-divergence minimization through the Signal-to-Noise Ratio (SNR) of the gradient estimator. We study several representative scenarios where strong analytical results are possible, such as fully-factorized or Gaussian distributions. We find that when alpha is not zero, the SNR worsens exponentially in the dimensionality of the problem. This casts doubt on the practicality of these methods. We empirically confirm these theoretical results.
APA
Geffner, T. & Domke, J.. (2021). On the difficulty of unbiased alpha divergence minimization. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:3650-3659 Available from https://proceedings.mlr.press/v139/geffner21a.html.

Related Material