Spectral Smoothing Unveils Phase Transitions in Hierarchical Variational Autoencoders

Adeel Pervez, Efstratios Gavves
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8536-8545, 2021.

Abstract

Variational autoencoders with deep hierarchies of stochastic layers have been known to suffer from the problem of posterior collapse, where the top layers fall back to the prior and become independent of input. We suggest that the hierarchical VAE objective explicitly includes the variance of the function parameterizing the mean and variance of the latent Gaussian distribution which itself is often a high variance function. Building on this we generalize VAE neural networks by incorporating a smoothing parameter motivated by Gaussian analysis to reduce higher frequency components and consequently the variance in parameterizing functions and show that this can help to solve the problem of posterior collapse. We further show that under such smoothing the VAE loss exhibits a phase transition, where the top layer KL divergence sharply drops to zero at a critical value of the smoothing parameter that is similar for the same model across datasets. We validate the phenomenon across model configurations and datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-pervez21a, title = {Spectral Smoothing Unveils Phase Transitions in Hierarchical Variational Autoencoders}, author = {Pervez, Adeel and Gavves, Efstratios}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {8536--8545}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/pervez21a/pervez21a.pdf}, url = {https://proceedings.mlr.press/v139/pervez21a.html}, abstract = {Variational autoencoders with deep hierarchies of stochastic layers have been known to suffer from the problem of posterior collapse, where the top layers fall back to the prior and become independent of input. We suggest that the hierarchical VAE objective explicitly includes the variance of the function parameterizing the mean and variance of the latent Gaussian distribution which itself is often a high variance function. Building on this we generalize VAE neural networks by incorporating a smoothing parameter motivated by Gaussian analysis to reduce higher frequency components and consequently the variance in parameterizing functions and show that this can help to solve the problem of posterior collapse. We further show that under such smoothing the VAE loss exhibits a phase transition, where the top layer KL divergence sharply drops to zero at a critical value of the smoothing parameter that is similar for the same model across datasets. We validate the phenomenon across model configurations and datasets.} }
Endnote
%0 Conference Paper %T Spectral Smoothing Unveils Phase Transitions in Hierarchical Variational Autoencoders %A Adeel Pervez %A Efstratios Gavves %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-pervez21a %I PMLR %P 8536--8545 %U https://proceedings.mlr.press/v139/pervez21a.html %V 139 %X Variational autoencoders with deep hierarchies of stochastic layers have been known to suffer from the problem of posterior collapse, where the top layers fall back to the prior and become independent of input. We suggest that the hierarchical VAE objective explicitly includes the variance of the function parameterizing the mean and variance of the latent Gaussian distribution which itself is often a high variance function. Building on this we generalize VAE neural networks by incorporating a smoothing parameter motivated by Gaussian analysis to reduce higher frequency components and consequently the variance in parameterizing functions and show that this can help to solve the problem of posterior collapse. We further show that under such smoothing the VAE loss exhibits a phase transition, where the top layer KL divergence sharply drops to zero at a critical value of the smoothing parameter that is similar for the same model across datasets. We validate the phenomenon across model configurations and datasets.
APA
Pervez, A. & Gavves, E.. (2021). Spectral Smoothing Unveils Phase Transitions in Hierarchical Variational Autoencoders. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:8536-8545 Available from https://proceedings.mlr.press/v139/pervez21a.html.

Related Material