VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects

Youngsun Wi, Andy Zeng, Pete Florence, Nima Fazeli
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1806-1816, 2023.

Abstract

Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation (VIRDO) via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information. In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-wi23a, title = {VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects}, author = {Wi, Youngsun and Zeng, Andy and Florence, Pete and Fazeli, Nima}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1806--1816}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/wi23a/wi23a.pdf}, url = {https://proceedings.mlr.press/v205/wi23a.html}, abstract = {Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation (VIRDO) via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information. In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations. } }
Endnote
%0 Conference Paper %T VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects %A Youngsun Wi %A Andy Zeng %A Pete Florence %A Nima Fazeli %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-wi23a %I PMLR %P 1806--1816 %U https://proceedings.mlr.press/v205/wi23a.html %V 205 %X Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation (VIRDO) via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information. In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.
APA
Wi, Y., Zeng, A., Florence, P. & Fazeli, N.. (2023). VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1806-1816 Available from https://proceedings.mlr.press/v205/wi23a.html.

Related Material