Learning Multilevel Distributed Representations for High-Dimensional Sequences

Ilya Sutskever, Geoffrey Hinton
Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, PMLR 2:548-555, 2007.

Abstract

We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.

Cite this Paper

BibTeX
@InProceedings{pmlr-v2-sutskever07a, title = {Learning Multilevel Distributed Representations for High-Dimensional Sequences}, author = {Sutskever, Ilya and Hinton, Geoffrey}, booktitle = {Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics}, pages = {548--555}, year = {2007}, editor = {Meila, Marina and Shen, Xiaotong}, volume = {2}, series = {Proceedings of Machine Learning Research}, address = {San Juan, Puerto Rico}, month = {21--24 Mar}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v2/sutskever07a/sutskever07a.pdf}, url = {https://proceedings.mlr.press/v2/sutskever07a.html}, abstract = {We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.} }
Endnote
%0 Conference Paper %T Learning Multilevel Distributed Representations for High-Dimensional Sequences %A Ilya Sutskever %A Geoffrey Hinton %B Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2007 %E Marina Meila %E Xiaotong Shen %F pmlr-v2-sutskever07a %I PMLR %P 548--555 %U https://proceedings.mlr.press/v2/sutskever07a.html %V 2 %X We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.
RIS
TY - CPAPER TI - Learning Multilevel Distributed Representations for High-Dimensional Sequences AU - Ilya Sutskever AU - Geoffrey Hinton BT - Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics DA - 2007/03/11 ED - Marina Meila ED - Xiaotong Shen ID - pmlr-v2-sutskever07a PB - PMLR DP - Proceedings of Machine Learning Research VL - 2 SP - 548 EP - 555 L1 - http://proceedings.mlr.press/v2/sutskever07a/sutskever07a.pdf UR - https://proceedings.mlr.press/v2/sutskever07a.html AB - We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box. ER -
APA
Sutskever, I. & Hinton, G.. (2007). Learning Multilevel Distributed Representations for High-Dimensional Sequences. Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 2:548-555 Available from https://proceedings.mlr.press/v2/sutskever07a.html.

Related Material