Disentangled Representations for Sequence Data using Information Bottleneck Principle

Masanori Yamada, Heecheol Kim, Kosuke Miyoshi, Tomoharu Iwata, Hiroshi Yamakawa
Proceedings of The 12th Asian Conference on Machine Learning, PMLR 129:305-320, 2020.

Abstract

We propose the factorizing variational autoencoder (FAVAE), a generative model for learning dis- entangled representations from sequential data via the information bottleneck principle without supervision. Real-world data are often generated by a few explanatory factors of variation, and disentangled representation learning obtains these factors from the data. We focus on the disen- tangled representation of sequential data which can be useful in a wide range of applications, such as video, speech, and stock markets. Factors in sequential data are categorized into dynamic and static ones: dynamic factors are time dependent, and static factors are time independent. Previous models disentangle between static and dynamic factors and between dynamic factors with different time dependencies by explicitly modeling the priors of latent variables. However, these models cannot disentangle representations between dynamic factors with the same time dependency, such as disentangling “picking up” and “throwing” in robotic tasks. On the other hand, FAVAE can disentangle multiple dynamic factors via the information bottleneck principle where it does not require modeling priors. We conducted experiments to show that FAVAE can extract disentangled dynamic factors on synthetic, video, and speech datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v129-yamada20a, title = {Disentangled Representations for Sequence Data using Information Bottleneck Principle}, author = {Yamada, Masanori and Kim, Heecheol and Miyoshi, Kosuke and Iwata, Tomoharu and Yamakawa, Hiroshi}, booktitle = {Proceedings of The 12th Asian Conference on Machine Learning}, pages = {305--320}, year = {2020}, editor = {Pan, Sinno Jialin and Sugiyama, Masashi}, volume = {129}, series = {Proceedings of Machine Learning Research}, month = {18--20 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v129/yamada20a/yamada20a.pdf}, url = {https://proceedings.mlr.press/v129/yamada20a.html}, abstract = {We propose the factorizing variational autoencoder (FAVAE), a generative model for learning dis- entangled representations from sequential data via the information bottleneck principle without supervision. Real-world data are often generated by a few explanatory factors of variation, and disentangled representation learning obtains these factors from the data. We focus on the disen- tangled representation of sequential data which can be useful in a wide range of applications, such as video, speech, and stock markets. Factors in sequential data are categorized into dynamic and static ones: dynamic factors are time dependent, and static factors are time independent. Previous models disentangle between static and dynamic factors and between dynamic factors with different time dependencies by explicitly modeling the priors of latent variables. However, these models cannot disentangle representations between dynamic factors with the same time dependency, such as disentangling “picking up” and “throwing” in robotic tasks. On the other hand, FAVAE can disentangle multiple dynamic factors via the information bottleneck principle where it does not require modeling priors. We conducted experiments to show that FAVAE can extract disentangled dynamic factors on synthetic, video, and speech datasets.} }
Endnote
%0 Conference Paper %T Disentangled Representations for Sequence Data using Information Bottleneck Principle %A Masanori Yamada %A Heecheol Kim %A Kosuke Miyoshi %A Tomoharu Iwata %A Hiroshi Yamakawa %B Proceedings of The 12th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Sinno Jialin Pan %E Masashi Sugiyama %F pmlr-v129-yamada20a %I PMLR %P 305--320 %U https://proceedings.mlr.press/v129/yamada20a.html %V 129 %X We propose the factorizing variational autoencoder (FAVAE), a generative model for learning dis- entangled representations from sequential data via the information bottleneck principle without supervision. Real-world data are often generated by a few explanatory factors of variation, and disentangled representation learning obtains these factors from the data. We focus on the disen- tangled representation of sequential data which can be useful in a wide range of applications, such as video, speech, and stock markets. Factors in sequential data are categorized into dynamic and static ones: dynamic factors are time dependent, and static factors are time independent. Previous models disentangle between static and dynamic factors and between dynamic factors with different time dependencies by explicitly modeling the priors of latent variables. However, these models cannot disentangle representations between dynamic factors with the same time dependency, such as disentangling “picking up” and “throwing” in robotic tasks. On the other hand, FAVAE can disentangle multiple dynamic factors via the information bottleneck principle where it does not require modeling priors. We conducted experiments to show that FAVAE can extract disentangled dynamic factors on synthetic, video, and speech datasets.
APA
Yamada, M., Kim, H., Miyoshi, K., Iwata, T. & Yamakawa, H.. (2020). Disentangled Representations for Sequence Data using Information Bottleneck Principle. Proceedings of The 12th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 129:305-320 Available from https://proceedings.mlr.press/v129/yamada20a.html.

Related Material