Variational Tempering

Stephan Mandt, James McInerney, Farhan Abrol, Rajesh Ranganath, David Blei
; Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, PMLR 51:704-712, 2016.

Abstract

Variational inference (VI) combined with data subsampling enables approximate posterior inference with large data sets for otherwise intractable models, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This algorithm uses a temperature parameter that deterministically deforms the objective and reduces this parameter over the course of the optimization. A well-known drawback in annealing is the choice of the annealing schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.

Cite this Paper


BibTeX
@InProceedings{pmlr-v51-mandt16, title = {Variational Tempering}, author = {Stephan Mandt and James McInerney and Farhan Abrol and Rajesh Ranganath and David Blei}, booktitle = {Proceedings of the 19th International Conference on Artificial Intelligence and Statistics}, pages = {704--712}, year = {2016}, editor = {Arthur Gretton and Christian C. Robert}, volume = {51}, series = {Proceedings of Machine Learning Research}, address = {Cadiz, Spain}, month = {09--11 May}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v51/mandt16.pdf}, url = {http://proceedings.mlr.press/v51/mandt16.html}, abstract = {Variational inference (VI) combined with data subsampling enables approximate posterior inference with large data sets for otherwise intractable models, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This algorithm uses a temperature parameter that deterministically deforms the objective and reduces this parameter over the course of the optimization. A well-known drawback in annealing is the choice of the annealing schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.} }
Endnote
%0 Conference Paper %T Variational Tempering %A Stephan Mandt %A James McInerney %A Farhan Abrol %A Rajesh Ranganath %A David Blei %B Proceedings of the 19th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2016 %E Arthur Gretton %E Christian C. Robert %F pmlr-v51-mandt16 %I PMLR %J Proceedings of Machine Learning Research %P 704--712 %U http://proceedings.mlr.press %V 51 %W PMLR %X Variational inference (VI) combined with data subsampling enables approximate posterior inference with large data sets for otherwise intractable models, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This algorithm uses a temperature parameter that deterministically deforms the objective and reduces this parameter over the course of the optimization. A well-known drawback in annealing is the choice of the annealing schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.
RIS
TY - CPAPER TI - Variational Tempering AU - Stephan Mandt AU - James McInerney AU - Farhan Abrol AU - Rajesh Ranganath AU - David Blei BT - Proceedings of the 19th International Conference on Artificial Intelligence and Statistics PY - 2016/05/02 DA - 2016/05/02 ED - Arthur Gretton ED - Christian C. Robert ID - pmlr-v51-mandt16 PB - PMLR SP - 704 DP - PMLR EP - 712 L1 - http://proceedings.mlr.press/v51/mandt16.pdf UR - http://proceedings.mlr.press/v51/mandt16.html AB - Variational inference (VI) combined with data subsampling enables approximate posterior inference with large data sets for otherwise intractable models, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This algorithm uses a temperature parameter that deterministically deforms the objective and reduces this parameter over the course of the optimization. A well-known drawback in annealing is the choice of the annealing schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data. ER -
APA
Mandt, S., McInerney, J., Abrol, F., Ranganath, R. & Blei, D.. (2016). Variational Tempering. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, in PMLR 51:704-712

Related Material