Dynamic Handover: Throw and Catch with Bimanual Hands

Binghao Huang, Yuanpei Chen, Tianyu Wang, Yuzhe Qin, Yaodong Yang, Nikolay Atanasov, Xiaolong Wang
Proceedings of The 7th Conference on Robot Learning, PMLR 229:1887-1902, 2023.

Abstract

Humans throw and catch objects all the time. However, such a seemingly common skill introduces a lot of challenges for robots to achieve: The robots need to operate such dynamic actions at high-speed, collaborate precisely, and interact with diverse objects. In this paper, we design a system with two multi-finger hands attached to robot arms to solve this problem. We train our system using Multi-Agent Reinforcement Learning in simulation and perform Sim2Real transfer to deploy on the real robots. To overcome the Sim2Real gap, we provide multiple novel algorithm designs including learning a trajectory prediction model for the object. Such a model can help the robot catcher has a real-time estimation of where the object will be heading, and then react accordingly. We conduct our experiments with multiple objects in the real-world system, and show significant improvements over multiple baselines. Our project page is available at https://binghao-huang.github.io/dynamic_handover/

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-huang23d, title = {Dynamic Handover: Throw and Catch with Bimanual Hands}, author = {Huang, Binghao and Chen, Yuanpei and Wang, Tianyu and Qin, Yuzhe and Yang, Yaodong and Atanasov, Nikolay and Wang, Xiaolong}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {1887--1902}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/huang23d/huang23d.pdf}, url = {https://proceedings.mlr.press/v229/huang23d.html}, abstract = {Humans throw and catch objects all the time. However, such a seemingly common skill introduces a lot of challenges for robots to achieve: The robots need to operate such dynamic actions at high-speed, collaborate precisely, and interact with diverse objects. In this paper, we design a system with two multi-finger hands attached to robot arms to solve this problem. We train our system using Multi-Agent Reinforcement Learning in simulation and perform Sim2Real transfer to deploy on the real robots. To overcome the Sim2Real gap, we provide multiple novel algorithm designs including learning a trajectory prediction model for the object. Such a model can help the robot catcher has a real-time estimation of where the object will be heading, and then react accordingly. We conduct our experiments with multiple objects in the real-world system, and show significant improvements over multiple baselines. Our project page is available at https://binghao-huang.github.io/dynamic_handover/} }
Endnote
%0 Conference Paper %T Dynamic Handover: Throw and Catch with Bimanual Hands %A Binghao Huang %A Yuanpei Chen %A Tianyu Wang %A Yuzhe Qin %A Yaodong Yang %A Nikolay Atanasov %A Xiaolong Wang %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-huang23d %I PMLR %P 1887--1902 %U https://proceedings.mlr.press/v229/huang23d.html %V 229 %X Humans throw and catch objects all the time. However, such a seemingly common skill introduces a lot of challenges for robots to achieve: The robots need to operate such dynamic actions at high-speed, collaborate precisely, and interact with diverse objects. In this paper, we design a system with two multi-finger hands attached to robot arms to solve this problem. We train our system using Multi-Agent Reinforcement Learning in simulation and perform Sim2Real transfer to deploy on the real robots. To overcome the Sim2Real gap, we provide multiple novel algorithm designs including learning a trajectory prediction model for the object. Such a model can help the robot catcher has a real-time estimation of where the object will be heading, and then react accordingly. We conduct our experiments with multiple objects in the real-world system, and show significant improvements over multiple baselines. Our project page is available at https://binghao-huang.github.io/dynamic_handover/
APA
Huang, B., Chen, Y., Wang, T., Qin, Y., Yang, Y., Atanasov, N. & Wang, X.. (2023). Dynamic Handover: Throw and Catch with Bimanual Hands. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:1887-1902 Available from https://proceedings.mlr.press/v229/huang23d.html.

Related Material