Tighter Variational Bounds are Not Necessarily Better

Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, Yee Whye Teh
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:4277-4285, 2018.

Abstract

We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted autoencoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Furthermore, our results suggest that PIWAE may be able to deliver simultaneous improvements in the training of both the inference and generative networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-rainforth18b, title = {Tighter Variational Bounds are Not Necessarily Better}, author = {Rainforth, Tom and Kosiorek, Adam and Le, Tuan Anh and Maddison, Chris and Igl, Maximilian and Wood, Frank and Teh, Yee Whye}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {4277--4285}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/rainforth18b/rainforth18b.pdf}, url = {https://proceedings.mlr.press/v80/rainforth18b.html}, abstract = {We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted autoencoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Furthermore, our results suggest that PIWAE may be able to deliver simultaneous improvements in the training of both the inference and generative networks.} }
Endnote
%0 Conference Paper %T Tighter Variational Bounds are Not Necessarily Better %A Tom Rainforth %A Adam Kosiorek %A Tuan Anh Le %A Chris Maddison %A Maximilian Igl %A Frank Wood %A Yee Whye Teh %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-rainforth18b %I PMLR %P 4277--4285 %U https://proceedings.mlr.press/v80/rainforth18b.html %V 80 %X We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted autoencoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Furthermore, our results suggest that PIWAE may be able to deliver simultaneous improvements in the training of both the inference and generative networks.
APA
Rainforth, T., Kosiorek, A., Le, T.A., Maddison, C., Igl, M., Wood, F. & Teh, Y.W.. (2018). Tighter Variational Bounds are Not Necessarily Better. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:4277-4285 Available from https://proceedings.mlr.press/v80/rainforth18b.html.

Related Material