Real-world fluid directed rigid body control via deep reinforcement learning

Mohak Bhardwaj, Thomas Lampe, Michael Neunert, Francesco Romano, Abbas Abdolmaleki, Arunkumar Byravan, Markus Wulfmeier, Martin Riedmiller, Jonas Buchli
Proceedings of the 6th Annual Learning for Dynamics & Control Conference, PMLR 242:414-427, 2024.

Abstract

Recent advances in real-world applications of reinforcement learning (RL) have relied on the ability to accurately simulate systems at scale. However, domains such as fluid dynamical systems exhibit complex dynamic phenomena that are hard to simulate at high integration rates, limiting the direct application of modern deep RL algorithms to often expensive or safety critical hardware. In this work, we introduce “Box o’ Flows”, a novel benchtop experimental control system for systematically evaluating RL algorithms in dynamic real-world scenarios. We describe the key components of the Box o’ Flows, and through a series of experiments demonstrate how state-of-the-art model-free RL algorithms can synthesize a variety of complex behaviors via simple reward specifications. Furthermore, we explore the role of offline RL in data-efficient hypothesis testing by reusing past experiences. We believe that the insights gained from this preliminary study and the availability of systems like the Box o’ Flows support the way forward for developing systematic RL algorithms that can be generally applied to complex, dynamical systems. Supplementary material and videos of experiments are available at https://sites.google.com/view/box-o-flows/home.

Cite this Paper

BibTeX
@InProceedings{pmlr-v242-bhardwaj24a, title = {Real-world fluid directed rigid body control via deep reinforcement learning}, author = {Bhardwaj, Mohak and Lampe, Thomas and Neunert, Michael and Romano, Francesco and Abdolmaleki, Abbas and Byravan, Arunkumar and Wulfmeier, Markus and Riedmiller, Martin and Buchli, Jonas}, booktitle = {Proceedings of the 6th Annual Learning for Dynamics & Control Conference}, pages = {414--427}, year = {2024}, editor = {Abate, Alessandro and Cannon, Mark and Margellos, Kostas and Papachristodoulou, Antonis}, volume = {242}, series = {Proceedings of Machine Learning Research}, month = {15--17 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v242/bhardwaj24a/bhardwaj24a.pdf}, url = {https://proceedings.mlr.press/v242/bhardwaj24a.html}, abstract = {Recent advances in real-world applications of reinforcement learning (RL) have relied on the ability to accurately simulate systems at scale. However, domains such as fluid dynamical systems exhibit complex dynamic phenomena that are hard to simulate at high integration rates, limiting the direct application of modern deep RL algorithms to often expensive or safety critical hardware. In this work, we introduce “Box o’ Flows”, a novel benchtop experimental control system for systematically evaluating RL algorithms in dynamic real-world scenarios. We describe the key components of the Box o’ Flows, and through a series of experiments demonstrate how state-of-the-art model-free RL algorithms can synthesize a variety of complex behaviors via simple reward specifications. Furthermore, we explore the role of offline RL in data-efficient hypothesis testing by reusing past experiences. We believe that the insights gained from this preliminary study and the availability of systems like the Box o’ Flows support the way forward for developing systematic RL algorithms that can be generally applied to complex, dynamical systems. Supplementary material and videos of experiments are available at https://sites.google.com/view/box-o-flows/home.} }
Endnote
%0 Conference Paper %T Real-world fluid directed rigid body control via deep reinforcement learning %A Mohak Bhardwaj %A Thomas Lampe %A Michael Neunert %A Francesco Romano %A Abbas Abdolmaleki %A Arunkumar Byravan %A Markus Wulfmeier %A Martin Riedmiller %A Jonas Buchli %B Proceedings of the 6th Annual Learning for Dynamics & Control Conference %C Proceedings of Machine Learning Research %D 2024 %E Alessandro Abate %E Mark Cannon %E Kostas Margellos %E Antonis Papachristodoulou %F pmlr-v242-bhardwaj24a %I PMLR %P 414--427 %U https://proceedings.mlr.press/v242/bhardwaj24a.html %V 242 %X Recent advances in real-world applications of reinforcement learning (RL) have relied on the ability to accurately simulate systems at scale. However, domains such as fluid dynamical systems exhibit complex dynamic phenomena that are hard to simulate at high integration rates, limiting the direct application of modern deep RL algorithms to often expensive or safety critical hardware. In this work, we introduce “Box o’ Flows”, a novel benchtop experimental control system for systematically evaluating RL algorithms in dynamic real-world scenarios. We describe the key components of the Box o’ Flows, and through a series of experiments demonstrate how state-of-the-art model-free RL algorithms can synthesize a variety of complex behaviors via simple reward specifications. Furthermore, we explore the role of offline RL in data-efficient hypothesis testing by reusing past experiences. We believe that the insights gained from this preliminary study and the availability of systems like the Box o’ Flows support the way forward for developing systematic RL algorithms that can be generally applied to complex, dynamical systems. Supplementary material and videos of experiments are available at https://sites.google.com/view/box-o-flows/home.
APA
Bhardwaj, M., Lampe, T., Neunert, M., Romano, F., Abdolmaleki, A., Byravan, A., Wulfmeier, M., Riedmiller, M. & Buchli, J.. (2024). Real-world fluid directed rigid body control via deep reinforcement learning. Proceedings of the 6th Annual Learning for Dynamics & Control Conference, in Proceedings of Machine Learning Research 242:414-427 Available from https://proceedings.mlr.press/v242/bhardwaj24a.html.

Related Material