HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation

Vainavi Viswanath, Kaushik Shivakumar, Mallika Parulekar, Jainil Ajmera, Justin Kerr, Jeffrey Ichnowski, Richard Cheng, Thomas Kollar, Ken Goldberg
Proceedings of The 7th Conference on Robot Learning, PMLR 229:341-357, 2023.

Abstract

Tracing – estimating the spatial state of – long deformable linear objects such as cables, threads, hoses, or ropes, is useful for a broad range of tasks in homes, retail, factories, construction, transportation, and healthcare. For long deformable linear objects (DLOs or simply cables) with many (over 25) crossings, we present HANDLOOM (Heterogeneous Autoregressive Learned Deformable Linear Object Observation and Manipulation) a learning-based algorithm that fits a trace to a greyscale image of cables. We evaluate HANDLOOM on semi-planar DLO configurations where each crossing involves at most 2 segments. HANDLOOM makes use of neural networks trained with 30,000 simulated examples and 568 real examples to autoregressively estimate traces of cables and classify crossings. Experiments find that in settings with multiple identical cables, HANDLOOM can trace each cable with $80%$ accuracy. In single-cable images, HANDLOOM can trace and identify knots with $77%$ accuracy. When HANDLOOM is incorporated into a bimanual robot system, it enables state-based imitation of knot tying with $80%$ accuracy, and it successfully untangles $64%$ of cable configurations across 3 levels of difficulty. Additionally, HANDLOOM demonstrates generalization to knot types and materials (rubber, cloth rope) not present in the training dataset with $85%$ accuracy. Supplementary material, including all code and an annotated dataset of RGB-D images of cables along with ground-truth traces, is at https://sites.google.com/view/cable-tracing.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-viswanath23a, title = {HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation}, author = {Viswanath, Vainavi and Shivakumar, Kaushik and Parulekar, Mallika and Ajmera, Jainil and Kerr, Justin and Ichnowski, Jeffrey and Cheng, Richard and Kollar, Thomas and Goldberg, Ken}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {341--357}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/viswanath23a/viswanath23a.pdf}, url = {https://proceedings.mlr.press/v229/viswanath23a.html}, abstract = {Tracing – estimating the spatial state of – long deformable linear objects such as cables, threads, hoses, or ropes, is useful for a broad range of tasks in homes, retail, factories, construction, transportation, and healthcare. For long deformable linear objects (DLOs or simply cables) with many (over 25) crossings, we present HANDLOOM (Heterogeneous Autoregressive Learned Deformable Linear Object Observation and Manipulation) a learning-based algorithm that fits a trace to a greyscale image of cables. We evaluate HANDLOOM on semi-planar DLO configurations where each crossing involves at most 2 segments. HANDLOOM makes use of neural networks trained with 30,000 simulated examples and 568 real examples to autoregressively estimate traces of cables and classify crossings. Experiments find that in settings with multiple identical cables, HANDLOOM can trace each cable with $80%$ accuracy. In single-cable images, HANDLOOM can trace and identify knots with $77%$ accuracy. When HANDLOOM is incorporated into a bimanual robot system, it enables state-based imitation of knot tying with $80%$ accuracy, and it successfully untangles $64%$ of cable configurations across 3 levels of difficulty. Additionally, HANDLOOM demonstrates generalization to knot types and materials (rubber, cloth rope) not present in the training dataset with $85%$ accuracy. Supplementary material, including all code and an annotated dataset of RGB-D images of cables along with ground-truth traces, is at https://sites.google.com/view/cable-tracing.} }
Endnote
%0 Conference Paper %T HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation %A Vainavi Viswanath %A Kaushik Shivakumar %A Mallika Parulekar %A Jainil Ajmera %A Justin Kerr %A Jeffrey Ichnowski %A Richard Cheng %A Thomas Kollar %A Ken Goldberg %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-viswanath23a %I PMLR %P 341--357 %U https://proceedings.mlr.press/v229/viswanath23a.html %V 229 %X Tracing – estimating the spatial state of – long deformable linear objects such as cables, threads, hoses, or ropes, is useful for a broad range of tasks in homes, retail, factories, construction, transportation, and healthcare. For long deformable linear objects (DLOs or simply cables) with many (over 25) crossings, we present HANDLOOM (Heterogeneous Autoregressive Learned Deformable Linear Object Observation and Manipulation) a learning-based algorithm that fits a trace to a greyscale image of cables. We evaluate HANDLOOM on semi-planar DLO configurations where each crossing involves at most 2 segments. HANDLOOM makes use of neural networks trained with 30,000 simulated examples and 568 real examples to autoregressively estimate traces of cables and classify crossings. Experiments find that in settings with multiple identical cables, HANDLOOM can trace each cable with $80%$ accuracy. In single-cable images, HANDLOOM can trace and identify knots with $77%$ accuracy. When HANDLOOM is incorporated into a bimanual robot system, it enables state-based imitation of knot tying with $80%$ accuracy, and it successfully untangles $64%$ of cable configurations across 3 levels of difficulty. Additionally, HANDLOOM demonstrates generalization to knot types and materials (rubber, cloth rope) not present in the training dataset with $85%$ accuracy. Supplementary material, including all code and an annotated dataset of RGB-D images of cables along with ground-truth traces, is at https://sites.google.com/view/cable-tracing.
APA
Viswanath, V., Shivakumar, K., Parulekar, M., Ajmera, J., Kerr, J., Ichnowski, J., Cheng, R., Kollar, T. & Goldberg, K.. (2023). HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:341-357 Available from https://proceedings.mlr.press/v229/viswanath23a.html.

Related Material