Predictive Sampling with Forecasting Autoregressive Models

Auke Wiggers, Emiel Hoogeboom
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:10260-10269, 2020.

Abstract

Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-wiggers20a, title = {Predictive Sampling with Forecasting Autoregressive Models}, author = {Wiggers, Auke and Hoogeboom, Emiel}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {10260--10269}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/wiggers20a/wiggers20a.pdf}, url = {https://proceedings.mlr.press/v119/wiggers20a.html}, abstract = {Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.} }
Endnote
%0 Conference Paper %T Predictive Sampling with Forecasting Autoregressive Models %A Auke Wiggers %A Emiel Hoogeboom %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-wiggers20a %I PMLR %P 10260--10269 %U https://proceedings.mlr.press/v119/wiggers20a.html %V 119 %X Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.
APA
Wiggers, A. & Hoogeboom, E.. (2020). Predictive Sampling with Forecasting Autoregressive Models. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:10260-10269 Available from https://proceedings.mlr.press/v119/wiggers20a.html.

Related Material