Discovering Discrete Latent Topics with Neural Variational Inference

Yishu Miao, Edward Grefenstette, Phil Blunsom
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:2410-2419, 2017.

Abstract

Topic models have been widely explored as probabilistic generative models of documents. Traditional inference methods have sought closed-form derivations for updating the models, however as the expressiveness of these models grows, so does the difficulty of performing fast and accurate inference over their parameters. This paper presents alternative neural approaches to topic modelling by providing parameterisable distributions over topics which permit training by backpropagation in the framework of neural variational inference. In addition, with the help of a stick-breaking construction, we propose a recurrent network that is able to discover a notionally unbounded number of topics, analogous to Bayesian non-parametric topic models. Experimental results on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the effectiveness and efficiency of these neural topic models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-miao17a, title = {Discovering Discrete Latent Topics with Neural Variational Inference}, author = {Yishu Miao and Edward Grefenstette and Phil Blunsom}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {2410--2419}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/miao17a/miao17a.pdf}, url = {https://proceedings.mlr.press/v70/miao17a.html}, abstract = {Topic models have been widely explored as probabilistic generative models of documents. Traditional inference methods have sought closed-form derivations for updating the models, however as the expressiveness of these models grows, so does the difficulty of performing fast and accurate inference over their parameters. This paper presents alternative neural approaches to topic modelling by providing parameterisable distributions over topics which permit training by backpropagation in the framework of neural variational inference. In addition, with the help of a stick-breaking construction, we propose a recurrent network that is able to discover a notionally unbounded number of topics, analogous to Bayesian non-parametric topic models. Experimental results on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the effectiveness and efficiency of these neural topic models.} }
Endnote
%0 Conference Paper %T Discovering Discrete Latent Topics with Neural Variational Inference %A Yishu Miao %A Edward Grefenstette %A Phil Blunsom %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-miao17a %I PMLR %P 2410--2419 %U https://proceedings.mlr.press/v70/miao17a.html %V 70 %X Topic models have been widely explored as probabilistic generative models of documents. Traditional inference methods have sought closed-form derivations for updating the models, however as the expressiveness of these models grows, so does the difficulty of performing fast and accurate inference over their parameters. This paper presents alternative neural approaches to topic modelling by providing parameterisable distributions over topics which permit training by backpropagation in the framework of neural variational inference. In addition, with the help of a stick-breaking construction, we propose a recurrent network that is able to discover a notionally unbounded number of topics, analogous to Bayesian non-parametric topic models. Experimental results on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the effectiveness and efficiency of these neural topic models.
APA
Miao, Y., Grefenstette, E. & Blunsom, P.. (2017). Discovering Discrete Latent Topics with Neural Variational Inference. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:2410-2419 Available from https://proceedings.mlr.press/v70/miao17a.html.

Related Material