Evaluating Lossy Compression Rates of Deep Generative Models

Sicong Huang, Alireza Makhzani, Yanshuai Cao, Roger Grosse
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4444-4454, 2020.

Abstract

The field of deep generative modeling has succeeded in producing astonishingly realistic-seeming images and audio, but quantitative evaluation remains a challenge. Log-likelihood is an appealing metric due to its grounding in statistics and information theory, but it can be challenging to estimate for implicit generative models, and scalar-valued metrics give an incomplete picture of a model’s quality. In this work, we propose to use rate distortion (RD) curves to evaluate and compare deep generative models. While estimating RD curves is seemingly even more computationally demanding than log-likelihood estimation, we show that we can approximate the entire RD curve using nearly the same computations as were previously used to achieve a single log-likelihood estimate. We evaluate lossy compression rates of VAEs, GANs, and adversarial autoencoders (AAEs) on the MNIST and CIFAR10 datasets. Measuring the entire RD curve gives a more complete picture than scalar-valued metrics, and we arrive at a number of insights not obtainable from log-likelihoods alone.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-huang20c, title = {Evaluating Lossy Compression Rates of Deep Generative Models}, author = {Huang, Sicong and Makhzani, Alireza and Cao, Yanshuai and Grosse, Roger}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {4444--4454}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/huang20c/huang20c.pdf}, url = {https://proceedings.mlr.press/v119/huang20c.html}, abstract = {The field of deep generative modeling has succeeded in producing astonishingly realistic-seeming images and audio, but quantitative evaluation remains a challenge. Log-likelihood is an appealing metric due to its grounding in statistics and information theory, but it can be challenging to estimate for implicit generative models, and scalar-valued metrics give an incomplete picture of a model’s quality. In this work, we propose to use rate distortion (RD) curves to evaluate and compare deep generative models. While estimating RD curves is seemingly even more computationally demanding than log-likelihood estimation, we show that we can approximate the entire RD curve using nearly the same computations as were previously used to achieve a single log-likelihood estimate. We evaluate lossy compression rates of VAEs, GANs, and adversarial autoencoders (AAEs) on the MNIST and CIFAR10 datasets. Measuring the entire RD curve gives a more complete picture than scalar-valued metrics, and we arrive at a number of insights not obtainable from log-likelihoods alone.} }
Endnote
%0 Conference Paper %T Evaluating Lossy Compression Rates of Deep Generative Models %A Sicong Huang %A Alireza Makhzani %A Yanshuai Cao %A Roger Grosse %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-huang20c %I PMLR %P 4444--4454 %U https://proceedings.mlr.press/v119/huang20c.html %V 119 %X The field of deep generative modeling has succeeded in producing astonishingly realistic-seeming images and audio, but quantitative evaluation remains a challenge. Log-likelihood is an appealing metric due to its grounding in statistics and information theory, but it can be challenging to estimate for implicit generative models, and scalar-valued metrics give an incomplete picture of a model’s quality. In this work, we propose to use rate distortion (RD) curves to evaluate and compare deep generative models. While estimating RD curves is seemingly even more computationally demanding than log-likelihood estimation, we show that we can approximate the entire RD curve using nearly the same computations as were previously used to achieve a single log-likelihood estimate. We evaluate lossy compression rates of VAEs, GANs, and adversarial autoencoders (AAEs) on the MNIST and CIFAR10 datasets. Measuring the entire RD curve gives a more complete picture than scalar-valued metrics, and we arrive at a number of insights not obtainable from log-likelihoods alone.
APA
Huang, S., Makhzani, A., Cao, Y. & Grosse, R.. (2020). Evaluating Lossy Compression Rates of Deep Generative Models. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:4444-4454 Available from https://proceedings.mlr.press/v119/huang20c.html.

Related Material