Combining model-based controller and ML advice via convex reparameterization

Junxuan Shen, Adam Wierman, Guannan Qu
Proceedings of the 6th Annual Learning for Dynamics & Control Conference, PMLR 242:679-693, 2024.

Abstract

Machine Learning (ML) based control, particularly Reinforcement Learning (RL), has achieved impressive advancements but is often black-box and lacks worst-case guarantees in safety-critical systems. In contrast, classical model-based control offers stability guarantees but usually underperforms the machine-learned black-box controller. This motivates us to combine machine-learned black-box and model-based controllers. Due to the nonconvexity of the space of stable controllers, a simple convex combination of the two controllers can lead to instability. We propose using Disturbance Response Control (DRC) to reparameterize the two controllers, ensuring the convexity of the stable controller space. We then propose lambdaCLEAC, which adaptively combines the machine-learned black-box controller and the model-based controller in the DRC parameterization. We prove that our approach achieves the best of both worlds: stability as in model-based control and similar regret bounds as the machine-learned controller.

Cite this Paper

BibTeX
@InProceedings{pmlr-v242-shen24a, title = {Combining model-based controller and {ML} advice via convex reparameterization}, author = {Shen, Junxuan and Wierman, Adam and Qu, Guannan}, booktitle = {Proceedings of the 6th Annual Learning for Dynamics & Control Conference}, pages = {679--693}, year = {2024}, editor = {Abate, Alessandro and Cannon, Mark and Margellos, Kostas and Papachristodoulou, Antonis}, volume = {242}, series = {Proceedings of Machine Learning Research}, month = {15--17 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v242/shen24a/shen24a.pdf}, url = {https://proceedings.mlr.press/v242/shen24a.html}, abstract = {Machine Learning (ML) based control, particularly Reinforcement Learning (RL), has achieved impressive advancements but is often black-box and lacks worst-case guarantees in safety-critical systems. In contrast, classical model-based control offers stability guarantees but usually underperforms the machine-learned black-box controller. This motivates us to combine machine-learned black-box and model-based controllers. Due to the nonconvexity of the space of stable controllers, a simple convex combination of the two controllers can lead to instability. We propose using Disturbance Response Control (DRC) to reparameterize the two controllers, ensuring the convexity of the stable controller space. We then propose lambdaCLEAC, which adaptively combines the machine-learned black-box controller and the model-based controller in the DRC parameterization. We prove that our approach achieves the best of both worlds: stability as in model-based control and similar regret bounds as the machine-learned controller.} }
Endnote
%0 Conference Paper %T Combining model-based controller and ML advice via convex reparameterization %A Junxuan Shen %A Adam Wierman %A Guannan Qu %B Proceedings of the 6th Annual Learning for Dynamics & Control Conference %C Proceedings of Machine Learning Research %D 2024 %E Alessandro Abate %E Mark Cannon %E Kostas Margellos %E Antonis Papachristodoulou %F pmlr-v242-shen24a %I PMLR %P 679--693 %U https://proceedings.mlr.press/v242/shen24a.html %V 242 %X Machine Learning (ML) based control, particularly Reinforcement Learning (RL), has achieved impressive advancements but is often black-box and lacks worst-case guarantees in safety-critical systems. In contrast, classical model-based control offers stability guarantees but usually underperforms the machine-learned black-box controller. This motivates us to combine machine-learned black-box and model-based controllers. Due to the nonconvexity of the space of stable controllers, a simple convex combination of the two controllers can lead to instability. We propose using Disturbance Response Control (DRC) to reparameterize the two controllers, ensuring the convexity of the stable controller space. We then propose lambdaCLEAC, which adaptively combines the machine-learned black-box controller and the model-based controller in the DRC parameterization. We prove that our approach achieves the best of both worlds: stability as in model-based control and similar regret bounds as the machine-learned controller.
APA
Shen, J., Wierman, A. & Qu, G.. (2024). Combining model-based controller and ML advice via convex reparameterization. Proceedings of the 6th Annual Learning for Dynamics & Control Conference, in Proceedings of Machine Learning Research 242:679-693 Available from https://proceedings.mlr.press/v242/shen24a.html.

Related Material