ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Daniel Seita, Yufei Wang, Sarthak J Shetty, Edward Yao Li, Zackory Erickson, David Held
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1038-1049, 2023.

Abstract

Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense per-point flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform physical scooping experiments with ToolFlowNet and find that we can attain 82% scooping success. See https://sites.google.com/view/point-cloud-policy/home for supplementary material.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-seita23a, title = {ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds}, author = {Seita, Daniel and Wang, Yufei and Shetty, Sarthak J and Li, Edward Yao and Erickson, Zackory and Held, David}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1038--1049}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/seita23a/seita23a.pdf}, url = {https://proceedings.mlr.press/v205/seita23a.html}, abstract = {Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense per-point flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform physical scooping experiments with ToolFlowNet and find that we can attain 82% scooping success. See https://sites.google.com/view/point-cloud-policy/home for supplementary material.} }
Endnote
%0 Conference Paper %T ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds %A Daniel Seita %A Yufei Wang %A Sarthak J Shetty %A Edward Yao Li %A Zackory Erickson %A David Held %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-seita23a %I PMLR %P 1038--1049 %U https://proceedings.mlr.press/v205/seita23a.html %V 205 %X Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense per-point flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform physical scooping experiments with ToolFlowNet and find that we can attain 82% scooping success. See https://sites.google.com/view/point-cloud-policy/home for supplementary material.
APA
Seita, D., Wang, Y., Shetty, S.J., Li, E.Y., Erickson, Z. & Held, D.. (2023). ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1038-1049 Available from https://proceedings.mlr.press/v205/seita23a.html.

Related Material