The Power of Learned Locally Linear Models for Nonlinear Policy Optimization

Daniel Pfrommer, Max Simchowitz, Tyler Westenbroek, Nikolai Matni, Stephen Tu
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:27737-27821, 2023.

Abstract

A common pipeline in learning-based control is to iteratively estimate a model of system dynamics, and apply a trajectory optimization algorithm - e.g. $\mathtt{iLQR}$ - on the learned model to minimize a target cost. This paper conducts a rigorous analysis of a simplified variant of this strategy for general nonlinear systems. We analyze an algorithm which iterates between estimating local linear models of nonlinear system dynamics and performing $\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains sample complexity polynomial in relevant problem parameters, and, by synthesizing locally stabilizing gains, overcomes exponential dependence in problem horizon. Experimental results validate the performance of our algorithm, and compare to natural deep-learning baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-pfrommer23a, title = {The Power of Learned Locally Linear Models for Nonlinear Policy Optimization}, author = {Pfrommer, Daniel and Simchowitz, Max and Westenbroek, Tyler and Matni, Nikolai and Tu, Stephen}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {27737--27821}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/pfrommer23a/pfrommer23a.pdf}, url = {https://proceedings.mlr.press/v202/pfrommer23a.html}, abstract = {A common pipeline in learning-based control is to iteratively estimate a model of system dynamics, and apply a trajectory optimization algorithm - e.g. $\mathtt{iLQR}$ - on the learned model to minimize a target cost. This paper conducts a rigorous analysis of a simplified variant of this strategy for general nonlinear systems. We analyze an algorithm which iterates between estimating local linear models of nonlinear system dynamics and performing $\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains sample complexity polynomial in relevant problem parameters, and, by synthesizing locally stabilizing gains, overcomes exponential dependence in problem horizon. Experimental results validate the performance of our algorithm, and compare to natural deep-learning baselines.} }
Endnote
%0 Conference Paper %T The Power of Learned Locally Linear Models for Nonlinear Policy Optimization %A Daniel Pfrommer %A Max Simchowitz %A Tyler Westenbroek %A Nikolai Matni %A Stephen Tu %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-pfrommer23a %I PMLR %P 27737--27821 %U https://proceedings.mlr.press/v202/pfrommer23a.html %V 202 %X A common pipeline in learning-based control is to iteratively estimate a model of system dynamics, and apply a trajectory optimization algorithm - e.g. $\mathtt{iLQR}$ - on the learned model to minimize a target cost. This paper conducts a rigorous analysis of a simplified variant of this strategy for general nonlinear systems. We analyze an algorithm which iterates between estimating local linear models of nonlinear system dynamics and performing $\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains sample complexity polynomial in relevant problem parameters, and, by synthesizing locally stabilizing gains, overcomes exponential dependence in problem horizon. Experimental results validate the performance of our algorithm, and compare to natural deep-learning baselines.
APA
Pfrommer, D., Simchowitz, M., Westenbroek, T., Matni, N. & Tu, S.. (2023). The Power of Learned Locally Linear Models for Nonlinear Policy Optimization. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:27737-27821 Available from https://proceedings.mlr.press/v202/pfrommer23a.html.

Related Material