PixelSNAIL: An Improved Autoregressive Generative Model

XI Chen, Nikhil Mishra, Mostafa Rohaninejad, Pieter Abbeel
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:864-872, 2018.

Abstract

Autoregressive generative models achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this paper, we describe the resulting model and present state-of-the-art log-likelihood results on heavily benchmarked datasets: CIFAR-10, $32 \times 32$ ImageNet and $64 \times 64$ ImageNet. Our implementation will be made available at \url{https://github.com/neocxi/pixelsnail-public}.

Cite this Paper


BibTeX
@InProceedings{pmlr-v80-chen18h, title = {{P}ixel{SNAIL}: An Improved Autoregressive Generative Model}, author = {Chen, XI and Mishra, Nikhil and Rohaninejad, Mostafa and Abbeel, Pieter}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {864--872}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/chen18h/chen18h.pdf}, url = {https://proceedings.mlr.press/v80/chen18h.html}, abstract = {Autoregressive generative models achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this paper, we describe the resulting model and present state-of-the-art log-likelihood results on heavily benchmarked datasets: CIFAR-10, $32 \times 32$ ImageNet and $64 \times 64$ ImageNet. Our implementation will be made available at \url{https://github.com/neocxi/pixelsnail-public}.} }
Endnote
%0 Conference Paper %T PixelSNAIL: An Improved Autoregressive Generative Model %A XI Chen %A Nikhil Mishra %A Mostafa Rohaninejad %A Pieter Abbeel %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-chen18h %I PMLR %P 864--872 %U https://proceedings.mlr.press/v80/chen18h.html %V 80 %X Autoregressive generative models achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this paper, we describe the resulting model and present state-of-the-art log-likelihood results on heavily benchmarked datasets: CIFAR-10, $32 \times 32$ ImageNet and $64 \times 64$ ImageNet. Our implementation will be made available at \url{https://github.com/neocxi/pixelsnail-public}.
APA
Chen, X., Mishra, N., Rohaninejad, M. & Abbeel, P.. (2018). PixelSNAIL: An Improved Autoregressive Generative Model. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:864-872 Available from https://proceedings.mlr.press/v80/chen18h.html.

Related Material