MPC-inspired reinforcement learning for verifiable model-free control

Yiwen Lu, Zishuo Li, Yihan Zhou, Na Li, Yilin Mo
Proceedings of the 6th Annual Learning for Dynamics & Control Conference, PMLR 242:399-413, 2024.

Abstract

In this paper, we introduce a new class of parameterized controllers, drawing inspiration from Model Predictive Control (MPC). These controllers adopt an unrolled Quadratic Programming (QP) solver, structured similarly to a deep neural network, with parameters from the QP problem that is similar to linear MPC. The parameters are learned rather than derived from models. This approach addresses the limitations of commonly learned controllers with Multi-Layer Perceptron (MLP) or other general neural network architecture in deep reinforcement learning, in terms of explainability and performance guarantees. The learned controllers not only possess verifiable properties like persistent feasibility and asymptotic stability akin to MPC, but they also empirically match MPC and MLP controllers in control performance. Moreover, they are more computationally efficient in implementation compared to MPC and require significantly fewer learnable policy parameters than MLP controllers. Practical application is demonstrated through a vehicle drift maneuvering task, showcasing the potential of these controllers in real-world scenarios.

Cite this Paper

BibTeX
@InProceedings{pmlr-v242-lu24a, title = {{MPC}-inspired reinforcement learning for verifiable model-free control}, author = {Lu, Yiwen and Li, Zishuo and Zhou, Yihan and Li, Na and Mo, Yilin}, booktitle = {Proceedings of the 6th Annual Learning for Dynamics & Control Conference}, pages = {399--413}, year = {2024}, editor = {Abate, Alessandro and Cannon, Mark and Margellos, Kostas and Papachristodoulou, Antonis}, volume = {242}, series = {Proceedings of Machine Learning Research}, month = {15--17 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v242/lu24a/lu24a.pdf}, url = {https://proceedings.mlr.press/v242/lu24a.html}, abstract = {In this paper, we introduce a new class of parameterized controllers, drawing inspiration from Model Predictive Control (MPC). These controllers adopt an unrolled Quadratic Programming (QP) solver, structured similarly to a deep neural network, with parameters from the QP problem that is similar to linear MPC. The parameters are learned rather than derived from models. This approach addresses the limitations of commonly learned controllers with Multi-Layer Perceptron (MLP) or other general neural network architecture in deep reinforcement learning, in terms of explainability and performance guarantees. The learned controllers not only possess verifiable properties like persistent feasibility and asymptotic stability akin to MPC, but they also empirically match MPC and MLP controllers in control performance. Moreover, they are more computationally efficient in implementation compared to MPC and require significantly fewer learnable policy parameters than MLP controllers. Practical application is demonstrated through a vehicle drift maneuvering task, showcasing the potential of these controllers in real-world scenarios.} }
Endnote
%0 Conference Paper %T MPC-inspired reinforcement learning for verifiable model-free control %A Yiwen Lu %A Zishuo Li %A Yihan Zhou %A Na Li %A Yilin Mo %B Proceedings of the 6th Annual Learning for Dynamics & Control Conference %C Proceedings of Machine Learning Research %D 2024 %E Alessandro Abate %E Mark Cannon %E Kostas Margellos %E Antonis Papachristodoulou %F pmlr-v242-lu24a %I PMLR %P 399--413 %U https://proceedings.mlr.press/v242/lu24a.html %V 242 %X In this paper, we introduce a new class of parameterized controllers, drawing inspiration from Model Predictive Control (MPC). These controllers adopt an unrolled Quadratic Programming (QP) solver, structured similarly to a deep neural network, with parameters from the QP problem that is similar to linear MPC. The parameters are learned rather than derived from models. This approach addresses the limitations of commonly learned controllers with Multi-Layer Perceptron (MLP) or other general neural network architecture in deep reinforcement learning, in terms of explainability and performance guarantees. The learned controllers not only possess verifiable properties like persistent feasibility and asymptotic stability akin to MPC, but they also empirically match MPC and MLP controllers in control performance. Moreover, they are more computationally efficient in implementation compared to MPC and require significantly fewer learnable policy parameters than MLP controllers. Practical application is demonstrated through a vehicle drift maneuvering task, showcasing the potential of these controllers in real-world scenarios.
APA
Lu, Y., Li, Z., Zhou, Y., Li, N. & Mo, Y.. (2024). MPC-inspired reinforcement learning for verifiable model-free control. Proceedings of the 6th Annual Learning for Dynamics & Control Conference, in Proceedings of Machine Learning Research 242:399-413 Available from https://proceedings.mlr.press/v242/lu24a.html.

Related Material