Can Direct Latent Model Learning Solve Linear Quadratic Gaussian Control?

Yi Tian, Kaiqing Zhang, Russ Tedrake, Suvrit Sra
Proceedings of The 5th Annual Learning for Dynamics and Control Conference, PMLR 211:51-63, 2023.

Abstract

We study the task of learning state representations from potentially high-dimensional observations, with the goal of controlling an unknown partially observable system. We pursue a direct latent model learning approach, where a dynamic model in some latent state space is learned by predicting quantities directly related to planning (e.g., costs) without reconstructing the observations. In particular, we focus on an intuitive cost-driven state representation learning method for solving Linear Quadratic Gaussian (LQG) control, one of the most fundamental partially observable control problems. As our main results, we establish finite-sample guarantees of finding a near-optimal state representation function and a near-optimal controller using the directly learned latent model. To the best of our knowledge, despite various empirical successes, prior to this work it was unclear if such a cost-driven latent model learner enjoys finite-sample guarantees. Our work underscores the value of predicting multi-step costs, an idea that is key to our theory, and notably also an idea that is known to be empirically valuable for learning state representations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v211-tian23a, title = {Can Direct Latent Model Learning Solve Linear Quadratic Gaussian Control?}, author = {Tian, Yi and Zhang, Kaiqing and Tedrake, Russ and Sra, Suvrit}, booktitle = {Proceedings of The 5th Annual Learning for Dynamics and Control Conference}, pages = {51--63}, year = {2023}, editor = {Matni, Nikolai and Morari, Manfred and Pappas, George J.}, volume = {211}, series = {Proceedings of Machine Learning Research}, month = {15--16 Jun}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v211/tian23a/tian23a.pdf}, url = {https://proceedings.mlr.press/v211/tian23a.html}, abstract = {We study the task of learning state representations from potentially high-dimensional observations, with the goal of controlling an unknown partially observable system. We pursue a direct latent model learning approach, where a dynamic model in some latent state space is learned by predicting quantities directly related to planning (e.g., costs) without reconstructing the observations. In particular, we focus on an intuitive cost-driven state representation learning method for solving Linear Quadratic Gaussian (LQG) control, one of the most fundamental partially observable control problems. As our main results, we establish finite-sample guarantees of finding a near-optimal state representation function and a near-optimal controller using the directly learned latent model. To the best of our knowledge, despite various empirical successes, prior to this work it was unclear if such a cost-driven latent model learner enjoys finite-sample guarantees. Our work underscores the value of predicting multi-step costs, an idea that is key to our theory, and notably also an idea that is known to be empirically valuable for learning state representations.} }
Endnote
%0 Conference Paper %T Can Direct Latent Model Learning Solve Linear Quadratic Gaussian Control? %A Yi Tian %A Kaiqing Zhang %A Russ Tedrake %A Suvrit Sra %B Proceedings of The 5th Annual Learning for Dynamics and Control Conference %C Proceedings of Machine Learning Research %D 2023 %E Nikolai Matni %E Manfred Morari %E George J. Pappas %F pmlr-v211-tian23a %I PMLR %P 51--63 %U https://proceedings.mlr.press/v211/tian23a.html %V 211 %X We study the task of learning state representations from potentially high-dimensional observations, with the goal of controlling an unknown partially observable system. We pursue a direct latent model learning approach, where a dynamic model in some latent state space is learned by predicting quantities directly related to planning (e.g., costs) without reconstructing the observations. In particular, we focus on an intuitive cost-driven state representation learning method for solving Linear Quadratic Gaussian (LQG) control, one of the most fundamental partially observable control problems. As our main results, we establish finite-sample guarantees of finding a near-optimal state representation function and a near-optimal controller using the directly learned latent model. To the best of our knowledge, despite various empirical successes, prior to this work it was unclear if such a cost-driven latent model learner enjoys finite-sample guarantees. Our work underscores the value of predicting multi-step costs, an idea that is key to our theory, and notably also an idea that is known to be empirically valuable for learning state representations.
APA
Tian, Y., Zhang, K., Tedrake, R. & Sra, S.. (2023). Can Direct Latent Model Learning Solve Linear Quadratic Gaussian Control?. Proceedings of The 5th Annual Learning for Dynamics and Control Conference, in Proceedings of Machine Learning Research 211:51-63 Available from https://proceedings.mlr.press/v211/tian23a.html.

Related Material