General In-hand Object Rotation with Vision and Touch

Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, Jitendra Malik
Proceedings of The 7th Conference on Robot Learning, PMLR 229:2549-2564, 2023.

Abstract

We introduce Rotateit, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and highlight the importance of visual and tactile sensing.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-qi23a, title = {General In-hand Object Rotation with Vision and Touch}, author = {Qi, Haozhi and Yi, Brent and Suresh, Sudharshan and Lambeta, Mike and Ma, Yi and Calandra, Roberto and Malik, Jitendra}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {2549--2564}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/qi23a/qi23a.pdf}, url = {https://proceedings.mlr.press/v229/qi23a.html}, abstract = {We introduce Rotateit, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and highlight the importance of visual and tactile sensing.} }
Endnote
%0 Conference Paper %T General In-hand Object Rotation with Vision and Touch %A Haozhi Qi %A Brent Yi %A Sudharshan Suresh %A Mike Lambeta %A Yi Ma %A Roberto Calandra %A Jitendra Malik %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-qi23a %I PMLR %P 2549--2564 %U https://proceedings.mlr.press/v229/qi23a.html %V 229 %X We introduce Rotateit, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and highlight the importance of visual and tactile sensing.
APA
Qi, H., Yi, B., Suresh, S., Lambeta, M., Ma, Y., Calandra, R. & Malik, J.. (2023). General In-hand Object Rotation with Vision and Touch. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:2549-2564 Available from https://proceedings.mlr.press/v229/qi23a.html.

Related Material