Simple and Effective VAE Training with Calibrated Decoders

Oleh Rybkin, Kostas Daniilidis, Sergey Levine
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9179-9189, 2021.

Abstract

Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-rybkin21a, title = {Simple and Effective VAE Training with Calibrated Decoders}, author = {Rybkin, Oleh and Daniilidis, Kostas and Levine, Sergey}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9179--9189}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/rybkin21a/rybkin21a.pdf}, url = {https://proceedings.mlr.press/v139/rybkin21a.html}, abstract = {Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method.} }
Endnote
%0 Conference Paper %T Simple and Effective VAE Training with Calibrated Decoders %A Oleh Rybkin %A Kostas Daniilidis %A Sergey Levine %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-rybkin21a %I PMLR %P 9179--9189 %U https://proceedings.mlr.press/v139/rybkin21a.html %V 139 %X Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method.
APA
Rybkin, O., Daniilidis, K. & Levine, S.. (2021). Simple and Effective VAE Training with Calibrated Decoders. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:9179-9189 Available from https://proceedings.mlr.press/v139/rybkin21a.html.

Related Material