An Instability in Variational Inference for Topic Models

Behrooz Ghorbani, Hamid Javadi, Andrea Montanari
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:2221-2231, 2019.

Abstract

Naive mean field variational methods are the state of-the-art approach to inference in topic modeling. We show that these methods suffer from an instability that can produce misleading conclusions. Namely, for certain regimes of the model parameters, variational inference outputs a non-trivial decomposition into topics. However -for the same parameter values- the data contain no actual information about the true topic decomposition, and the output of the algorithm is uncorrelated with it. In particular, the estimated posterior mean is wrong, and estimated credible regions do not achieve the nominal coverage. We discuss how this instability is remedied by more accurate mean field approximations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-ghorbani19a, title = {An Instability in Variational Inference for Topic Models}, author = {Ghorbani, Behrooz and Javadi, Hamid and Montanari, Andrea}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {2221--2231}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/ghorbani19a/ghorbani19a.pdf}, url = {https://proceedings.mlr.press/v97/ghorbani19a.html}, abstract = {Naive mean field variational methods are the state of-the-art approach to inference in topic modeling. We show that these methods suffer from an instability that can produce misleading conclusions. Namely, for certain regimes of the model parameters, variational inference outputs a non-trivial decomposition into topics. However -for the same parameter values- the data contain no actual information about the true topic decomposition, and the output of the algorithm is uncorrelated with it. In particular, the estimated posterior mean is wrong, and estimated credible regions do not achieve the nominal coverage. We discuss how this instability is remedied by more accurate mean field approximations.} }
Endnote
%0 Conference Paper %T An Instability in Variational Inference for Topic Models %A Behrooz Ghorbani %A Hamid Javadi %A Andrea Montanari %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-ghorbani19a %I PMLR %P 2221--2231 %U https://proceedings.mlr.press/v97/ghorbani19a.html %V 97 %X Naive mean field variational methods are the state of-the-art approach to inference in topic modeling. We show that these methods suffer from an instability that can produce misleading conclusions. Namely, for certain regimes of the model parameters, variational inference outputs a non-trivial decomposition into topics. However -for the same parameter values- the data contain no actual information about the true topic decomposition, and the output of the algorithm is uncorrelated with it. In particular, the estimated posterior mean is wrong, and estimated credible regions do not achieve the nominal coverage. We discuss how this instability is remedied by more accurate mean field approximations.
APA
Ghorbani, B., Javadi, H. & Montanari, A.. (2019). An Instability in Variational Inference for Topic Models. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:2221-2231 Available from https://proceedings.mlr.press/v97/ghorbani19a.html.

Related Material