A Deep and Tractable Density Estimator

Benigno Uria, Iain Murray, Hugo Larochelle
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(1):467-475, 2014.

Abstract

The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-uria14, title = {A Deep and Tractable Density Estimator}, author = {Uria, Benigno and Murray, Iain and Larochelle, Hugo}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {467--475}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {1}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/uria14.pdf}, url = {https://proceedings.mlr.press/v32/uria14.html}, abstract = {The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance.} }
Endnote
%0 Conference Paper %T A Deep and Tractable Density Estimator %A Benigno Uria %A Iain Murray %A Hugo Larochelle %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-uria14 %I PMLR %P 467--475 %U https://proceedings.mlr.press/v32/uria14.html %V 32 %N 1 %X The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance.
RIS
TY - CPAPER TI - A Deep and Tractable Density Estimator AU - Benigno Uria AU - Iain Murray AU - Hugo Larochelle BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/01/27 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-uria14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 1 SP - 467 EP - 475 L1 - http://proceedings.mlr.press/v32/uria14.pdf UR - https://proceedings.mlr.press/v32/uria14.html AB - The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance. ER -
APA
Uria, B., Murray, I. & Larochelle, H.. (2014). A Deep and Tractable Density Estimator. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(1):467-475 Available from https://proceedings.mlr.press/v32/uria14.html.

Related Material