Extracting Latent State Representations with Linear Dynamics from Rich Observations

Abraham Frandsen, Rong Ge, Holden Lee
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:6705-6725, 2022.

Abstract

Recently, many reinforcement learning techniques have been shown to have provable guarantees in the simple case of linear dynamics, especially in problems like linear quadratic regulators. However, in practice many tasks require learning a policy from rich, high-dimensional features such as images, which are unlikely to be linear. We consider a setting where there is a hidden linear subspace of the high-dimensional feature space in which the dynamics are linear. We design natural objectives based on forward and inverse dynamics models. We prove that these objectives can be efficiently optimized and their local optimizers extract the hidden linear subspace. We empirically verify our theoretical results with synthetic data and explore the effectiveness of our approach (generalized to nonlinear settings) in simple control tasks with rich observations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-frandsen22a, title = {Extracting Latent State Representations with Linear Dynamics from Rich Observations}, author = {Frandsen, Abraham and Ge, Rong and Lee, Holden}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {6705--6725}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/frandsen22a/frandsen22a.pdf}, url = {https://proceedings.mlr.press/v162/frandsen22a.html}, abstract = {Recently, many reinforcement learning techniques have been shown to have provable guarantees in the simple case of linear dynamics, especially in problems like linear quadratic regulators. However, in practice many tasks require learning a policy from rich, high-dimensional features such as images, which are unlikely to be linear. We consider a setting where there is a hidden linear subspace of the high-dimensional feature space in which the dynamics are linear. We design natural objectives based on forward and inverse dynamics models. We prove that these objectives can be efficiently optimized and their local optimizers extract the hidden linear subspace. We empirically verify our theoretical results with synthetic data and explore the effectiveness of our approach (generalized to nonlinear settings) in simple control tasks with rich observations.} }
Endnote
%0 Conference Paper %T Extracting Latent State Representations with Linear Dynamics from Rich Observations %A Abraham Frandsen %A Rong Ge %A Holden Lee %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-frandsen22a %I PMLR %P 6705--6725 %U https://proceedings.mlr.press/v162/frandsen22a.html %V 162 %X Recently, many reinforcement learning techniques have been shown to have provable guarantees in the simple case of linear dynamics, especially in problems like linear quadratic regulators. However, in practice many tasks require learning a policy from rich, high-dimensional features such as images, which are unlikely to be linear. We consider a setting where there is a hidden linear subspace of the high-dimensional feature space in which the dynamics are linear. We design natural objectives based on forward and inverse dynamics models. We prove that these objectives can be efficiently optimized and their local optimizers extract the hidden linear subspace. We empirically verify our theoretical results with synthetic data and explore the effectiveness of our approach (generalized to nonlinear settings) in simple control tasks with rich observations.
APA
Frandsen, A., Ge, R. & Lee, H.. (2022). Extracting Latent State Representations with Linear Dynamics from Rich Observations. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:6705-6725 Available from https://proceedings.mlr.press/v162/frandsen22a.html.

Related Material