Neural Variational Inference for Text Processing

Yishu Miao, Lei Yu, Phil Blunsom
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:1727-1736, 2016.

Abstract

Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering. Our neural variational document model combines a continuous stochastic document representation with a bag-of-words generative model and achieves the lowest reported perplexities on two standard test corpora. The neural answer selection model employs a stochastic representation layer within an attention mechanism to extract the semantics between a question and answer pair. On two question answering benchmarks this model exceeds all previous published benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-miao16, title = {Neural Variational Inference for Text Processing}, author = {Miao, Yishu and Yu, Lei and Blunsom, Phil}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {1727--1736}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/miao16.pdf}, url = {https://proceedings.mlr.press/v48/miao16.html}, abstract = {Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering. Our neural variational document model combines a continuous stochastic document representation with a bag-of-words generative model and achieves the lowest reported perplexities on two standard test corpora. The neural answer selection model employs a stochastic representation layer within an attention mechanism to extract the semantics between a question and answer pair. On two question answering benchmarks this model exceeds all previous published benchmarks.} }
Endnote
%0 Conference Paper %T Neural Variational Inference for Text Processing %A Yishu Miao %A Lei Yu %A Phil Blunsom %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-miao16 %I PMLR %P 1727--1736 %U https://proceedings.mlr.press/v48/miao16.html %V 48 %X Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering. Our neural variational document model combines a continuous stochastic document representation with a bag-of-words generative model and achieves the lowest reported perplexities on two standard test corpora. The neural answer selection model employs a stochastic representation layer within an attention mechanism to extract the semantics between a question and answer pair. On two question answering benchmarks this model exceeds all previous published benchmarks.
RIS
TY - CPAPER TI - Neural Variational Inference for Text Processing AU - Yishu Miao AU - Lei Yu AU - Phil Blunsom BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-miao16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 1727 EP - 1736 L1 - http://proceedings.mlr.press/v48/miao16.pdf UR - https://proceedings.mlr.press/v48/miao16.html AB - Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering. Our neural variational document model combines a continuous stochastic document representation with a bag-of-words generative model and achieves the lowest reported perplexities on two standard test corpora. The neural answer selection model employs a stochastic representation layer within an attention mechanism to extract the semantics between a question and answer pair. On two question answering benchmarks this model exceeds all previous published benchmarks. ER -
APA
Miao, Y., Yu, L. & Blunsom, P.. (2016). Neural Variational Inference for Text Processing. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:1727-1736 Available from https://proceedings.mlr.press/v48/miao16.html.

Related Material