Learning Multilevel Distributed Representations for High-Dimensional Sequences

Ilya Sutskever, Geoffrey Hinton
; Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, PMLR 2:548-555, 2007.

Abstract

We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.

Cite this Paper


BibTeX
@InProceedings{pmlr-v2-sutskever07a, title = {Learning Multilevel Distributed Representations for High-Dimensional Sequences}, author = {Ilya Sutskever and Geoffrey Hinton}, booktitle = {Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics}, pages = {548--555}, year = {2007}, editor = {Marina Meila and Xiaotong Shen}, volume = {2}, series = {Proceedings of Machine Learning Research}, address = {San Juan, Puerto Rico}, month = {21--24 Mar}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v2/sutskever07a/sutskever07a.pdf}, url = {http://proceedings.mlr.press/v2/sutskever07a.html}, abstract = {We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.} }
Endnote
%0 Conference Paper %T Learning Multilevel Distributed Representations for High-Dimensional Sequences %A Ilya Sutskever %A Geoffrey Hinton %B Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2007 %E Marina Meila %E Xiaotong Shen %F pmlr-v2-sutskever07a %I PMLR %J Proceedings of Machine Learning Research %P 548--555 %U http://proceedings.mlr.press %V 2 %W PMLR %X We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.
RIS
TY - CPAPER TI - Learning Multilevel Distributed Representations for High-Dimensional Sequences AU - Ilya Sutskever AU - Geoffrey Hinton BT - Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics PY - 2007/03/11 DA - 2007/03/11 ED - Marina Meila ED - Xiaotong Shen ID - pmlr-v2-sutskever07a PB - PMLR SP - 548 DP - PMLR EP - 555 L1 - http://proceedings.mlr.press/v2/sutskever07a/sutskever07a.pdf UR - http://proceedings.mlr.press/v2/sutskever07a.html AB - We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box. ER -
APA
Sutskever, I. & Hinton, G.. (2007). Learning Multilevel Distributed Representations for High-Dimensional Sequences. Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, in PMLR 2:548-555

Related Material