Stein Variational Inference for Discrete Distributions

Jun Han, Fan Ding, Xianglong Liu, Lorenzo Torresani, Jian Peng, Qiang Liu
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:4563-4572, 2020.

Abstract

Gradient-based approximate inference methods, such as Stein variational gradient descent (SVGD) \cite{liu2016stein}, provide simple and general-purpose inference engines for differentiable continuous distributions. However, existing forms of SVGD can not be directly applied to discrete distributions. In this work, we fill this gap by proposing a simple general-purpose framework that transforms discrete distributions to equivalent piecewise continuous distribution, on which we apply gradient-free Stein variational gradient descent to perform efficient approximate inference. Our empirical results show that our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo on various challenging benchmarks of discrete graphical models. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN), outperforming other widely used ensemble methods on learning binarized AlexNet on CIFAR-10. In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions. Our proposed method outperforms existing GOF test methods for intractable discrete distributions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-han20c, title = {Stein Variational Inference for Discrete Distributions}, author = {Han, Jun and Ding, Fan and Liu, Xianglong and Torresani, Lorenzo and Peng, Jian and Liu, Qiang}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {4563--4572}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/han20c/han20c.pdf}, url = {https://proceedings.mlr.press/v108/han20c.html}, abstract = {Gradient-based approximate inference methods, such as Stein variational gradient descent (SVGD) \cite{liu2016stein}, provide simple and general-purpose inference engines for differentiable continuous distributions. However, existing forms of SVGD can not be directly applied to discrete distributions. In this work, we fill this gap by proposing a simple general-purpose framework that transforms discrete distributions to equivalent piecewise continuous distribution, on which we apply gradient-free Stein variational gradient descent to perform efficient approximate inference. Our empirical results show that our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo on various challenging benchmarks of discrete graphical models. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN), outperforming other widely used ensemble methods on learning binarized AlexNet on CIFAR-10. In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions. Our proposed method outperforms existing GOF test methods for intractable discrete distributions.} }
Endnote
%0 Conference Paper %T Stein Variational Inference for Discrete Distributions %A Jun Han %A Fan Ding %A Xianglong Liu %A Lorenzo Torresani %A Jian Peng %A Qiang Liu %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-han20c %I PMLR %P 4563--4572 %U https://proceedings.mlr.press/v108/han20c.html %V 108 %X Gradient-based approximate inference methods, such as Stein variational gradient descent (SVGD) \cite{liu2016stein}, provide simple and general-purpose inference engines for differentiable continuous distributions. However, existing forms of SVGD can not be directly applied to discrete distributions. In this work, we fill this gap by proposing a simple general-purpose framework that transforms discrete distributions to equivalent piecewise continuous distribution, on which we apply gradient-free Stein variational gradient descent to perform efficient approximate inference. Our empirical results show that our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo on various challenging benchmarks of discrete graphical models. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN), outperforming other widely used ensemble methods on learning binarized AlexNet on CIFAR-10. In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions. Our proposed method outperforms existing GOF test methods for intractable discrete distributions.
APA
Han, J., Ding, F., Liu, X., Torresani, L., Peng, J. & Liu, Q.. (2020). Stein Variational Inference for Discrete Distributions. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:4563-4572 Available from https://proceedings.mlr.press/v108/han20c.html.

Related Material