Learning Hierarchical Features from Deep Generative Models

Shengjia Zhao, Jiaming Song, Stefano Ermon
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:4091-4099, 2017.

Abstract

Deep neural networks have been shown to be very successful at learning feature hierarchies in supervised learning tasks. Generative models, on the other hand, have benefited less from hierarchical models with multiple layers of latent variables. In this paper, we prove that hierarchical latent variable models do not take advantage of the hierarchical structure when trained with existing variational methods, and provide some limitations on the kind of features existing models can learn. Finally we propose an alternative architecture that do not suffer from these limitations. Our model is able to learn highly interpretable and disentangled hierarchical features on several natural image datasets with no task specific regularization or prior knowledge.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-zhao17c, title = {Learning Hierarchical Features from Deep Generative Models}, author = {Shengjia Zhao and Jiaming Song and Stefano Ermon}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {4091--4099}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/zhao17c/zhao17c.pdf}, url = {https://proceedings.mlr.press/v70/zhao17c.html}, abstract = {Deep neural networks have been shown to be very successful at learning feature hierarchies in supervised learning tasks. Generative models, on the other hand, have benefited less from hierarchical models with multiple layers of latent variables. In this paper, we prove that hierarchical latent variable models do not take advantage of the hierarchical structure when trained with existing variational methods, and provide some limitations on the kind of features existing models can learn. Finally we propose an alternative architecture that do not suffer from these limitations. Our model is able to learn highly interpretable and disentangled hierarchical features on several natural image datasets with no task specific regularization or prior knowledge.} }
Endnote
%0 Conference Paper %T Learning Hierarchical Features from Deep Generative Models %A Shengjia Zhao %A Jiaming Song %A Stefano Ermon %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-zhao17c %I PMLR %P 4091--4099 %U https://proceedings.mlr.press/v70/zhao17c.html %V 70 %X Deep neural networks have been shown to be very successful at learning feature hierarchies in supervised learning tasks. Generative models, on the other hand, have benefited less from hierarchical models with multiple layers of latent variables. In this paper, we prove that hierarchical latent variable models do not take advantage of the hierarchical structure when trained with existing variational methods, and provide some limitations on the kind of features existing models can learn. Finally we propose an alternative architecture that do not suffer from these limitations. Our model is able to learn highly interpretable and disentangled hierarchical features on several natural image datasets with no task specific regularization or prior knowledge.
APA
Zhao, S., Song, J. & Ermon, S.. (2017). Learning Hierarchical Features from Deep Generative Models. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:4091-4099 Available from https://proceedings.mlr.press/v70/zhao17c.html.

Related Material