Information-Theoretic Generalization Bounds for VAEs: A Role of Encoder and Latent Variable

Futoshi Futami, Masahiro Fujisawa
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:32384-32440, 2026.

Abstract

Despite their remarkable success, a rigorous theoretical understanding of how latent variables (LVs) govern the generalization performance of Variational Autoencoders (VAEs) remains largely elusive. Existing theoretical analyses are confined to supervised learning or models with discrete latent spaces, leaving their role in standard VAEs with continuous LVs poorly understood. This paper establishes the first information-theoretic analysis for VAEs by adapting a theoretical framework from supervised learning—the leave-one-out conditional mutual information framework—to the unsupervised, continuous latent space of these models. Our analysis reveals that their generalization error is bounded solely by the information complexity of the encoder and LVs, independent of the decoder. The versatility of our framework is demonstrated through its extension to both hierarchical VAEs, for which we provide layer-wise bounds, and data generation, where we link our information-theoretic principles to a novel bound on the 2-Wasserstein distance between true and generated distributions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-futami26a, title = {Information-Theoretic Generalization Bounds for {VAE}s: A Role of Encoder and Latent Variable}, author = {Futami, Futoshi and Fujisawa, Masahiro}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {32384--32440}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/futami26a/futami26a.pdf}, url = {https://proceedings.mlr.press/v306/futami26a.html}, abstract = {Despite their remarkable success, a rigorous theoretical understanding of how latent variables (LVs) govern the generalization performance of Variational Autoencoders (VAEs) remains largely elusive. Existing theoretical analyses are confined to supervised learning or models with discrete latent spaces, leaving their role in standard VAEs with continuous LVs poorly understood. This paper establishes the first information-theoretic analysis for VAEs by adapting a theoretical framework from supervised learning—the leave-one-out conditional mutual information framework—to the unsupervised, continuous latent space of these models. Our analysis reveals that their generalization error is bounded solely by the information complexity of the encoder and LVs, independent of the decoder. The versatility of our framework is demonstrated through its extension to both hierarchical VAEs, for which we provide layer-wise bounds, and data generation, where we link our information-theoretic principles to a novel bound on the 2-Wasserstein distance between true and generated distributions.} }
Endnote
%0 Conference Paper %T Information-Theoretic Generalization Bounds for VAEs: A Role of Encoder and Latent Variable %A Futoshi Futami %A Masahiro Fujisawa %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-futami26a %I PMLR %P 32384--32440 %U https://proceedings.mlr.press/v306/futami26a.html %V 306 %X Despite their remarkable success, a rigorous theoretical understanding of how latent variables (LVs) govern the generalization performance of Variational Autoencoders (VAEs) remains largely elusive. Existing theoretical analyses are confined to supervised learning or models with discrete latent spaces, leaving their role in standard VAEs with continuous LVs poorly understood. This paper establishes the first information-theoretic analysis for VAEs by adapting a theoretical framework from supervised learning—the leave-one-out conditional mutual information framework—to the unsupervised, continuous latent space of these models. Our analysis reveals that their generalization error is bounded solely by the information complexity of the encoder and LVs, independent of the decoder. The versatility of our framework is demonstrated through its extension to both hierarchical VAEs, for which we provide layer-wise bounds, and data generation, where we link our information-theoretic principles to a novel bound on the 2-Wasserstein distance between true and generated distributions.
APA
Futami, F. & Fujisawa, M.. (2026). Information-Theoretic Generalization Bounds for VAEs: A Role of Encoder and Latent Variable. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:32384-32440 Available from https://proceedings.mlr.press/v306/futami26a.html.

Related Material