Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspective

Ruixiang Zhang, Masanori Koyama, Katsuhiko Ishiguro
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:11141-11152, 2020.

Abstract

Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. We represent the inductive bias in the form of mask variables to model the dependency structure in the graphical model and extend the theory of multivariate information bottleneck (Friedman et al., 2001) to enforce it. Our model provides a principled approach to learn a set of semantically meaningful latent factors that reflect various types of desired structures like capturing correlation or encoding invariance, while also offering the flexibility to automatically estimate the dependency structure from data. We show that our framework unifies many existing generative models and can be applied to a variety of tasks, including multimodal data modeling, algorithmic fairness, and out-of-distribution generalization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-zhang20m, title = {Learning Structured Latent Factors from Dependent {D}ata:{A} Generative Model Framework from Information-Theoretic Perspective}, author = {Zhang, Ruixiang and Koyama, Masanori and Ishiguro, Katsuhiko}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {11141--11152}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/zhang20m/zhang20m.pdf}, url = {https://proceedings.mlr.press/v119/zhang20m.html}, abstract = {Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. We represent the inductive bias in the form of mask variables to model the dependency structure in the graphical model and extend the theory of multivariate information bottleneck (Friedman et al., 2001) to enforce it. Our model provides a principled approach to learn a set of semantically meaningful latent factors that reflect various types of desired structures like capturing correlation or encoding invariance, while also offering the flexibility to automatically estimate the dependency structure from data. We show that our framework unifies many existing generative models and can be applied to a variety of tasks, including multimodal data modeling, algorithmic fairness, and out-of-distribution generalization.} }
Endnote
%0 Conference Paper %T Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspective %A Ruixiang Zhang %A Masanori Koyama %A Katsuhiko Ishiguro %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-zhang20m %I PMLR %P 11141--11152 %U https://proceedings.mlr.press/v119/zhang20m.html %V 119 %X Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. We represent the inductive bias in the form of mask variables to model the dependency structure in the graphical model and extend the theory of multivariate information bottleneck (Friedman et al., 2001) to enforce it. Our model provides a principled approach to learn a set of semantically meaningful latent factors that reflect various types of desired structures like capturing correlation or encoding invariance, while also offering the flexibility to automatically estimate the dependency structure from data. We show that our framework unifies many existing generative models and can be applied to a variety of tasks, including multimodal data modeling, algorithmic fairness, and out-of-distribution generalization.
APA
Zhang, R., Koyama, M. & Ishiguro, K.. (2020). Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspective. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:11141-11152 Available from https://proceedings.mlr.press/v119/zhang20m.html.

Related Material