Better Mixing via Deep Representations

Yoshua Bengio, Gregoire Mesnil, Yann Dauphin, Salah Rifai
Proceedings of the 30th International Conference on Machine Learning, PMLR 28(1):552-560, 2013.

Abstract

It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples.

Cite this Paper

BibTeX
@InProceedings{pmlr-v28-bengio13, title = {Better Mixing via Deep Representations}, author = {Bengio, Yoshua and Mesnil, Gregoire and Dauphin, Yann and Rifai, Salah}, booktitle = {Proceedings of the 30th International Conference on Machine Learning}, pages = {552--560}, year = {2013}, editor = {Dasgupta, Sanjoy and McAllester, David}, volume = {28}, number = {1}, series = {Proceedings of Machine Learning Research}, address = {Atlanta, Georgia, USA}, month = {17--19 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v28/bengio13.pdf}, url = {https://proceedings.mlr.press/v28/bengio13.html}, abstract = {It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples.} }
Endnote
%0 Conference Paper %T Better Mixing via Deep Representations %A Yoshua Bengio %A Gregoire Mesnil %A Yann Dauphin %A Salah Rifai %B Proceedings of the 30th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2013 %E Sanjoy Dasgupta %E David McAllester %F pmlr-v28-bengio13 %I PMLR %P 552--560 %U https://proceedings.mlr.press/v28/bengio13.html %V 28 %N 1 %X It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples.
RIS
TY - CPAPER TI - Better Mixing via Deep Representations AU - Yoshua Bengio AU - Gregoire Mesnil AU - Yann Dauphin AU - Salah Rifai BT - Proceedings of the 30th International Conference on Machine Learning DA - 2013/02/13 ED - Sanjoy Dasgupta ED - David McAllester ID - pmlr-v28-bengio13 PB - PMLR DP - Proceedings of Machine Learning Research VL - 28 IS - 1 SP - 552 EP - 560 L1 - http://proceedings.mlr.press/v28/bengio13.pdf UR - https://proceedings.mlr.press/v28/bengio13.html AB - It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples. ER -
APA
Bengio, Y., Mesnil, G., Dauphin, Y. & Rifai, S.. (2013). Better Mixing via Deep Representations. Proceedings of the 30th International Conference on Machine Learning, in Proceedings of Machine Learning Research 28(1):552-560 Available from https://proceedings.mlr.press/v28/bengio13.html.

Related Material