Im2Contact: Vision-Based Contact Localization Without Touch or Force Sensing

Leon Kim, Yunshuang Li, Michael Posa, Dinesh Jayaraman
Proceedings of The 7th Conference on Robot Learning, PMLR 229:1533-1546, 2023.

Abstract

Contacts play a critical role in most manipulation tasks. Robots today mainly use proximal touch/force sensors to sense contacts, but the information they provide must be calibrated and is inherently local, with practical applications relying either on extensive surface coverage or restrictive assumptions to resolve ambiguities. We propose a vision-based extrinsic contact localization task: with only a single RGB-D camera view of a robot workspace, identify when and where an object held by the robot contacts the rest of the environment. We show that careful task-attuned design is critical for a neural network trained in simulation to discover solutions that transfer well to a real robot. Our final approach im2contact demonstrates the promise of versatile general-purpose contact perception from vision alone, performing well for localizing various contact types (point, line, or planar; sticking, sliding, or rolling; single or multiple), and even under occlusions in its camera view. Video results can be found at: https://sites.google.com/view/im2contact/home

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-kim23b, title = {Im2Contact: Vision-Based Contact Localization Without Touch or Force Sensing}, author = {Kim, Leon and Li, Yunshuang and Posa, Michael and Jayaraman, Dinesh}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {1533--1546}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/kim23b/kim23b.pdf}, url = {https://proceedings.mlr.press/v229/kim23b.html}, abstract = {Contacts play a critical role in most manipulation tasks. Robots today mainly use proximal touch/force sensors to sense contacts, but the information they provide must be calibrated and is inherently local, with practical applications relying either on extensive surface coverage or restrictive assumptions to resolve ambiguities. We propose a vision-based extrinsic contact localization task: with only a single RGB-D camera view of a robot workspace, identify when and where an object held by the robot contacts the rest of the environment. We show that careful task-attuned design is critical for a neural network trained in simulation to discover solutions that transfer well to a real robot. Our final approach im2contact demonstrates the promise of versatile general-purpose contact perception from vision alone, performing well for localizing various contact types (point, line, or planar; sticking, sliding, or rolling; single or multiple), and even under occlusions in its camera view. Video results can be found at: https://sites.google.com/view/im2contact/home} }
Endnote
%0 Conference Paper %T Im2Contact: Vision-Based Contact Localization Without Touch or Force Sensing %A Leon Kim %A Yunshuang Li %A Michael Posa %A Dinesh Jayaraman %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-kim23b %I PMLR %P 1533--1546 %U https://proceedings.mlr.press/v229/kim23b.html %V 229 %X Contacts play a critical role in most manipulation tasks. Robots today mainly use proximal touch/force sensors to sense contacts, but the information they provide must be calibrated and is inherently local, with practical applications relying either on extensive surface coverage or restrictive assumptions to resolve ambiguities. We propose a vision-based extrinsic contact localization task: with only a single RGB-D camera view of a robot workspace, identify when and where an object held by the robot contacts the rest of the environment. We show that careful task-attuned design is critical for a neural network trained in simulation to discover solutions that transfer well to a real robot. Our final approach im2contact demonstrates the promise of versatile general-purpose contact perception from vision alone, performing well for localizing various contact types (point, line, or planar; sticking, sliding, or rolling; single or multiple), and even under occlusions in its camera view. Video results can be found at: https://sites.google.com/view/im2contact/home
APA
Kim, L., Li, Y., Posa, M. & Jayaraman, D.. (2023). Im2Contact: Vision-Based Contact Localization Without Touch or Force Sensing. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:1533-1546 Available from https://proceedings.mlr.press/v229/kim23b.html.

Related Material