A trust-region method for stochastic variational inference with applications to streaming data

Lucas Theis, Matt Hoffman
Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:2503-2511, 2015.

Abstract

Stochastic variational inference allows for fast posterior inference in complex Bayesian models. However, the algorithm is prone to local optima which can make the quality of the posterior approximation sensitive to the choice of hyperparameters and initialization. We address this problem by replacing the natural gradient step of stochastic varitional inference with a trust-region update. We show that this leads to generally better results and reduced sensitivity to hyperparameters. We also describe a new strategy for variational inference on streaming data and show that here our trust-region method is crucial for getting good performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v37-theis15, title = {A trust-region method for stochastic variational inference with applications to streaming data}, author = {Theis, Lucas and Hoffman, Matt}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning}, pages = {2503--2511}, year = {2015}, editor = {Bach, Francis and Blei, David}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/theis15.pdf}, url = {https://proceedings.mlr.press/v37/theis15.html}, abstract = {Stochastic variational inference allows for fast posterior inference in complex Bayesian models. However, the algorithm is prone to local optima which can make the quality of the posterior approximation sensitive to the choice of hyperparameters and initialization. We address this problem by replacing the natural gradient step of stochastic varitional inference with a trust-region update. We show that this leads to generally better results and reduced sensitivity to hyperparameters. We also describe a new strategy for variational inference on streaming data and show that here our trust-region method is crucial for getting good performance.} }
Endnote
%0 Conference Paper %T A trust-region method for stochastic variational inference with applications to streaming data %A Lucas Theis %A Matt Hoffman %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-theis15 %I PMLR %P 2503--2511 %U https://proceedings.mlr.press/v37/theis15.html %V 37 %X Stochastic variational inference allows for fast posterior inference in complex Bayesian models. However, the algorithm is prone to local optima which can make the quality of the posterior approximation sensitive to the choice of hyperparameters and initialization. We address this problem by replacing the natural gradient step of stochastic varitional inference with a trust-region update. We show that this leads to generally better results and reduced sensitivity to hyperparameters. We also describe a new strategy for variational inference on streaming data and show that here our trust-region method is crucial for getting good performance.
RIS
TY - CPAPER TI - A trust-region method for stochastic variational inference with applications to streaming data AU - Lucas Theis AU - Matt Hoffman BT - Proceedings of the 32nd International Conference on Machine Learning DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-theis15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 37 SP - 2503 EP - 2511 L1 - http://proceedings.mlr.press/v37/theis15.pdf UR - https://proceedings.mlr.press/v37/theis15.html AB - Stochastic variational inference allows for fast posterior inference in complex Bayesian models. However, the algorithm is prone to local optima which can make the quality of the posterior approximation sensitive to the choice of hyperparameters and initialization. We address this problem by replacing the natural gradient step of stochastic varitional inference with a trust-region update. We show that this leads to generally better results and reduced sensitivity to hyperparameters. We also describe a new strategy for variational inference on streaming data and show that here our trust-region method is crucial for getting good performance. ER -
APA
Theis, L. & Hoffman, M.. (2015). A trust-region method for stochastic variational inference with applications to streaming data. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:2503-2511 Available from https://proceedings.mlr.press/v37/theis15.html.

Related Material