Grammar Variational Autoencoder

Matt J. Kusner, Brooks Paige, José Miguel Hernández-Lobato
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1945-1954, 2017.

Abstract

Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as natural images, artwork, and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often produce outputs that are not valid. We make the key observation that frequently, discrete data can be represented as a parse tree from a context-free grammar. We propose a variational autoencoder which directly encodes from and decodes to these parse trees, ensuring the generated outputs are always syntactically valid. Surprisingly, we show that not only does our model more often generate valid outputs, it also learns a more coherent latent space in which nearby points decode to similar discrete outputs. We demonstrate the effectiveness of our learned models by showing their improved performance in Bayesian optimization for symbolic regression and molecule generation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-kusner17a, title = {Grammar Variational Autoencoder}, author = {Matt J. Kusner and Brooks Paige and Jos{\'e} Miguel Hern{\'a}ndez-Lobato}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {1945--1954}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/kusner17a/kusner17a.pdf}, url = {https://proceedings.mlr.press/v70/kusner17a.html}, abstract = {Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as natural images, artwork, and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often produce outputs that are not valid. We make the key observation that frequently, discrete data can be represented as a parse tree from a context-free grammar. We propose a variational autoencoder which directly encodes from and decodes to these parse trees, ensuring the generated outputs are always syntactically valid. Surprisingly, we show that not only does our model more often generate valid outputs, it also learns a more coherent latent space in which nearby points decode to similar discrete outputs. We demonstrate the effectiveness of our learned models by showing their improved performance in Bayesian optimization for symbolic regression and molecule generation.} }
Endnote
%0 Conference Paper %T Grammar Variational Autoencoder %A Matt J. Kusner %A Brooks Paige %A José Miguel Hernández-Lobato %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-kusner17a %I PMLR %P 1945--1954 %U https://proceedings.mlr.press/v70/kusner17a.html %V 70 %X Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as natural images, artwork, and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often produce outputs that are not valid. We make the key observation that frequently, discrete data can be represented as a parse tree from a context-free grammar. We propose a variational autoencoder which directly encodes from and decodes to these parse trees, ensuring the generated outputs are always syntactically valid. Surprisingly, we show that not only does our model more often generate valid outputs, it also learns a more coherent latent space in which nearby points decode to similar discrete outputs. We demonstrate the effectiveness of our learned models by showing their improved performance in Bayesian optimization for symbolic regression and molecule generation.
APA
Kusner, M.J., Paige, B. & Hernández-Lobato, J.M.. (2017). Grammar Variational Autoencoder. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:1945-1954 Available from https://proceedings.mlr.press/v70/kusner17a.html.

Related Material