Policy Stitching: Learning Transferable Robot Policies

Pingcheng Jian, Easop Lee, Zachary Bell, Michael M. Zavlanos, Boyuan Chen
Proceedings of The 7th Conference on Robot Learning, PMLR 229:3789-3808, 2023.

Abstract

Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensitive to changes in task environments and robot kinematics. Transfer RL aims to leverage previous knowledge to accelerate learning of new tasks or new body configurations. However, existing methods struggle to generalize to novel robot-task combinations and scale to realistic tasks due to complex architecture design or strong regularization that limits the capacity of the learned policy. We propose Policy Stitching, a novel framework that facilitates robot transfer learning for novel combinations of robots and tasks. Our key idea is to apply modular policy design and align the latent representations between the modular interfaces. Our method allows direct stitching of the robot and task modules trained separately to form a new policy for fast adaptation. Our simulated and real-world experiments on various 3D manipulation tasks demonstrate the superior zero-shot and few-shot transfer learning performances of our method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-jian23a, title = {Policy Stitching: Learning Transferable Robot Policies}, author = {Jian, Pingcheng and Lee, Easop and Bell, Zachary and Zavlanos, Michael M. and Chen, Boyuan}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {3789--3808}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/jian23a/jian23a.pdf}, url = {https://proceedings.mlr.press/v229/jian23a.html}, abstract = {Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensitive to changes in task environments and robot kinematics. Transfer RL aims to leverage previous knowledge to accelerate learning of new tasks or new body configurations. However, existing methods struggle to generalize to novel robot-task combinations and scale to realistic tasks due to complex architecture design or strong regularization that limits the capacity of the learned policy. We propose Policy Stitching, a novel framework that facilitates robot transfer learning for novel combinations of robots and tasks. Our key idea is to apply modular policy design and align the latent representations between the modular interfaces. Our method allows direct stitching of the robot and task modules trained separately to form a new policy for fast adaptation. Our simulated and real-world experiments on various 3D manipulation tasks demonstrate the superior zero-shot and few-shot transfer learning performances of our method.} }
Endnote
%0 Conference Paper %T Policy Stitching: Learning Transferable Robot Policies %A Pingcheng Jian %A Easop Lee %A Zachary Bell %A Michael M. Zavlanos %A Boyuan Chen %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-jian23a %I PMLR %P 3789--3808 %U https://proceedings.mlr.press/v229/jian23a.html %V 229 %X Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensitive to changes in task environments and robot kinematics. Transfer RL aims to leverage previous knowledge to accelerate learning of new tasks or new body configurations. However, existing methods struggle to generalize to novel robot-task combinations and scale to realistic tasks due to complex architecture design or strong regularization that limits the capacity of the learned policy. We propose Policy Stitching, a novel framework that facilitates robot transfer learning for novel combinations of robots and tasks. Our key idea is to apply modular policy design and align the latent representations between the modular interfaces. Our method allows direct stitching of the robot and task modules trained separately to form a new policy for fast adaptation. Our simulated and real-world experiments on various 3D manipulation tasks demonstrate the superior zero-shot and few-shot transfer learning performances of our method.
APA
Jian, P., Lee, E., Bell, Z., Zavlanos, M.M. & Chen, B.. (2023). Policy Stitching: Learning Transferable Robot Policies. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:3789-3808 Available from https://proceedings.mlr.press/v229/jian23a.html.

Related Material