Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding

Akira Nakagawa, Keizo Kato, Taiji Suzuki
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:7916-7926, 2021.

Abstract

Variational autoencoder (VAE) estimates the posterior parameters (mean and variance) of latent variables corresponding to each input data. While it is used for many tasks, the transparency of the model is still an underlying issue. This paper provides a quantitative understanding of VAE property through the differential geometric and information-theoretic interpretations of VAE. According to the Rate-distortion theory, the optimal transform coding is achieved by using an orthonormal transform with PCA basis where the transform space is isometric to the input. Considering the analogy of transform coding to VAE, we clarify theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. As a result, we can estimate the data probabilities in the input space from the prior, loss metrics, and corresponding posterior parameters, and further, the quantitative importance of each latent variable can be evaluated like the eigenvalue of PCA.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-nakagawa21a, title = {Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding}, author = {Nakagawa, Akira and Kato, Keizo and Suzuki, Taiji}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {7916--7926}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/nakagawa21a/nakagawa21a.pdf}, url = {https://proceedings.mlr.press/v139/nakagawa21a.html}, abstract = {Variational autoencoder (VAE) estimates the posterior parameters (mean and variance) of latent variables corresponding to each input data. While it is used for many tasks, the transparency of the model is still an underlying issue. This paper provides a quantitative understanding of VAE property through the differential geometric and information-theoretic interpretations of VAE. According to the Rate-distortion theory, the optimal transform coding is achieved by using an orthonormal transform with PCA basis where the transform space is isometric to the input. Considering the analogy of transform coding to VAE, we clarify theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. As a result, we can estimate the data probabilities in the input space from the prior, loss metrics, and corresponding posterior parameters, and further, the quantitative importance of each latent variable can be evaluated like the eigenvalue of PCA.} }
Endnote
%0 Conference Paper %T Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding %A Akira Nakagawa %A Keizo Kato %A Taiji Suzuki %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-nakagawa21a %I PMLR %P 7916--7926 %U https://proceedings.mlr.press/v139/nakagawa21a.html %V 139 %X Variational autoencoder (VAE) estimates the posterior parameters (mean and variance) of latent variables corresponding to each input data. While it is used for many tasks, the transparency of the model is still an underlying issue. This paper provides a quantitative understanding of VAE property through the differential geometric and information-theoretic interpretations of VAE. According to the Rate-distortion theory, the optimal transform coding is achieved by using an orthonormal transform with PCA basis where the transform space is isometric to the input. Considering the analogy of transform coding to VAE, we clarify theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. As a result, we can estimate the data probabilities in the input space from the prior, loss metrics, and corresponding posterior parameters, and further, the quantitative importance of each latent variable can be evaluated like the eigenvalue of PCA.
APA
Nakagawa, A., Kato, K. & Suzuki, T.. (2021). Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:7916-7926 Available from https://proceedings.mlr.press/v139/nakagawa21a.html.

Related Material