Seeing Glass: Joint Point-Cloud and Depth Completion for Transparent Objects

Haoping Xu, Yi Ru Wang, Sagi Eppel, Alan Aspuru-Guzik, Florian Shkurti, Animesh Garg
Proceedings of the 5th Conference on Robot Learning, PMLR 164:827-838, 2022.

Abstract

The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literature, we also propose an automated dataset creation workflow that consists of robot-controlled image collection and vision-based automatic annotation. Through this automated workflow, we created Transparent Object Depth Dataset (TODD), which consists of nearly 15000 RGB-D images. Our experimental evaluation demonstrates that TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets, including ClearGrasp, and that it also handles cluttered scenes when trained on TODD. Code and dataset will be released at https://www.pair.toronto.edu/TranspareNet/

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-xu22b, title = {Seeing Glass: Joint Point-Cloud and Depth Completion for Transparent Objects}, author = {Xu, Haoping and Wang, Yi Ru and Eppel, Sagi and Aspuru-Guzik, Alan and Shkurti, Florian and Garg, Animesh}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {827--838}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/xu22b/xu22b.pdf}, url = {https://proceedings.mlr.press/v164/xu22b.html}, abstract = {The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literature, we also propose an automated dataset creation workflow that consists of robot-controlled image collection and vision-based automatic annotation. Through this automated workflow, we created Transparent Object Depth Dataset (TODD), which consists of nearly 15000 RGB-D images. Our experimental evaluation demonstrates that TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets, including ClearGrasp, and that it also handles cluttered scenes when trained on TODD. Code and dataset will be released at https://www.pair.toronto.edu/TranspareNet/} }
Endnote
%0 Conference Paper %T Seeing Glass: Joint Point-Cloud and Depth Completion for Transparent Objects %A Haoping Xu %A Yi Ru Wang %A Sagi Eppel %A Alan Aspuru-Guzik %A Florian Shkurti %A Animesh Garg %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-xu22b %I PMLR %P 827--838 %U https://proceedings.mlr.press/v164/xu22b.html %V 164 %X The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literature, we also propose an automated dataset creation workflow that consists of robot-controlled image collection and vision-based automatic annotation. Through this automated workflow, we created Transparent Object Depth Dataset (TODD), which consists of nearly 15000 RGB-D images. Our experimental evaluation demonstrates that TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets, including ClearGrasp, and that it also handles cluttered scenes when trained on TODD. Code and dataset will be released at https://www.pair.toronto.edu/TranspareNet/
APA
Xu, H., Wang, Y.R., Eppel, S., Aspuru-Guzik, A., Shkurti, F. & Garg, A.. (2022). Seeing Glass: Joint Point-Cloud and Depth Completion for Transparent Objects. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:827-838 Available from https://proceedings.mlr.press/v164/xu22b.html.

Related Material