Autoregressive Energy Machines

Charlie Nash, Conor Durkan
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:1735-1744, 2019.

Abstract

Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which can subsequently be normalized to obtain a valid distribution. The challenge with this approach lies in accurately estimating the normalizing constant of the high-dimensional energy function. We propose the Autoregressive Energy Machine, an energy-based model which simultaneously learns an unnormalized density and computes an importance-sampling estimate of the normalizing constant for each conditional in an autoregressive decomposition. The Autoregressive Energy Machine achieves state-of-the-art performance on a suite of density-estimation tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-durkan19a, title = {Autoregressive Energy Machines}, author = {Nash, Charlie and Durkan, Conor}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {1735--1744}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/durkan19a/durkan19a.pdf}, url = {https://proceedings.mlr.press/v97/durkan19a.html}, abstract = {Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which can subsequently be normalized to obtain a valid distribution. The challenge with this approach lies in accurately estimating the normalizing constant of the high-dimensional energy function. We propose the Autoregressive Energy Machine, an energy-based model which simultaneously learns an unnormalized density and computes an importance-sampling estimate of the normalizing constant for each conditional in an autoregressive decomposition. The Autoregressive Energy Machine achieves state-of-the-art performance on a suite of density-estimation tasks.} }
Endnote
%0 Conference Paper %T Autoregressive Energy Machines %A Charlie Nash %A Conor Durkan %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-durkan19a %I PMLR %P 1735--1744 %U https://proceedings.mlr.press/v97/durkan19a.html %V 97 %X Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which can subsequently be normalized to obtain a valid distribution. The challenge with this approach lies in accurately estimating the normalizing constant of the high-dimensional energy function. We propose the Autoregressive Energy Machine, an energy-based model which simultaneously learns an unnormalized density and computes an importance-sampling estimate of the normalizing constant for each conditional in an autoregressive decomposition. The Autoregressive Energy Machine achieves state-of-the-art performance on a suite of density-estimation tasks.
APA
Nash, C. & Durkan, C.. (2019). Autoregressive Energy Machines. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:1735-1744 Available from https://proceedings.mlr.press/v97/durkan19a.html.

Related Material