Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments

Jun Yamada, Youngwoon Lee, Gautam Salhotra, Karl Pertsch, Max Pflueger, Gaurav Sukhatme, Joseph Lim, Peter Englert
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:589-603, 2021.

Abstract

Deep reinforcement learning (RL) agents are able to learn contact-rich manipulation tasks by maximizing a reward signal, but require large amounts of experience, especially in environments with many obstacles that complicate exploration. In contrast, motion planners use explicit models of the agent and environment to plan collision-free paths to faraway goals, but suffer from inaccurate models in tasks that require contacts with the environment. To combine the benefits of both approaches, we propose motion planner augmented RL (MoPA-RL) which augments the action space of an RL agent with the long-horizon planning capabilities of motion planners. Based on the magnitude of the action, our approach smoothly transitions between directly executing the action and invoking a motion planner. We evaluate our approach on various simulated manipulation tasks and compare it to alternative action spaces in terms of learning efficiency and safety. The experiments demonstrate that MoPA-RL increases learning efficiency, leads to a faster exploration, and results in safer policies that avoid collisions with the environment. Videos and code are available at https://clvrai.com/mopa-rl.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-yamada21a, title = {Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments}, author = {Yamada, Jun and Lee, Youngwoon and Salhotra, Gautam and Pertsch, Karl and Pflueger, Max and Sukhatme, Gaurav and Lim, Joseph and Englert, Peter}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {589--603}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/yamada21a/yamada21a.pdf}, url = {https://proceedings.mlr.press/v155/yamada21a.html}, abstract = {Deep reinforcement learning (RL) agents are able to learn contact-rich manipulation tasks by maximizing a reward signal, but require large amounts of experience, especially in environments with many obstacles that complicate exploration. In contrast, motion planners use explicit models of the agent and environment to plan collision-free paths to faraway goals, but suffer from inaccurate models in tasks that require contacts with the environment. To combine the benefits of both approaches, we propose motion planner augmented RL (MoPA-RL) which augments the action space of an RL agent with the long-horizon planning capabilities of motion planners. Based on the magnitude of the action, our approach smoothly transitions between directly executing the action and invoking a motion planner. We evaluate our approach on various simulated manipulation tasks and compare it to alternative action spaces in terms of learning efficiency and safety. The experiments demonstrate that MoPA-RL increases learning efficiency, leads to a faster exploration, and results in safer policies that avoid collisions with the environment. Videos and code are available at https://clvrai.com/mopa-rl.} }
Endnote
%0 Conference Paper %T Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments %A Jun Yamada %A Youngwoon Lee %A Gautam Salhotra %A Karl Pertsch %A Max Pflueger %A Gaurav Sukhatme %A Joseph Lim %A Peter Englert %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-yamada21a %I PMLR %P 589--603 %U https://proceedings.mlr.press/v155/yamada21a.html %V 155 %X Deep reinforcement learning (RL) agents are able to learn contact-rich manipulation tasks by maximizing a reward signal, but require large amounts of experience, especially in environments with many obstacles that complicate exploration. In contrast, motion planners use explicit models of the agent and environment to plan collision-free paths to faraway goals, but suffer from inaccurate models in tasks that require contacts with the environment. To combine the benefits of both approaches, we propose motion planner augmented RL (MoPA-RL) which augments the action space of an RL agent with the long-horizon planning capabilities of motion planners. Based on the magnitude of the action, our approach smoothly transitions between directly executing the action and invoking a motion planner. We evaluate our approach on various simulated manipulation tasks and compare it to alternative action spaces in terms of learning efficiency and safety. The experiments demonstrate that MoPA-RL increases learning efficiency, leads to a faster exploration, and results in safer policies that avoid collisions with the environment. Videos and code are available at https://clvrai.com/mopa-rl.
APA
Yamada, J., Lee, Y., Salhotra, G., Pertsch, K., Pflueger, M., Sukhatme, G., Lim, J. & Englert, P.. (2021). Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:589-603 Available from https://proceedings.mlr.press/v155/yamada21a.html.

Related Material