Leveraging Language for Accelerated Learning of Tool Manipulation

Allen Z. Ren, Bharat Govil, Tsung-Yen Yang, Karthik R Narasimhan, Anirudha Majumdar
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1531-1541, 2023.

Abstract

Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-ren23a, title = {Leveraging Language for Accelerated Learning of Tool Manipulation}, author = {Ren, Allen Z. and Govil, Bharat and Yang, Tsung-Yen and Narasimhan, Karthik R and Majumdar, Anirudha}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1531--1541}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/ren23a/ren23a.pdf}, url = {https://proceedings.mlr.press/v205/ren23a.html}, abstract = {Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.} }
Endnote
%0 Conference Paper %T Leveraging Language for Accelerated Learning of Tool Manipulation %A Allen Z. Ren %A Bharat Govil %A Tsung-Yen Yang %A Karthik R Narasimhan %A Anirudha Majumdar %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-ren23a %I PMLR %P 1531--1541 %U https://proceedings.mlr.press/v205/ren23a.html %V 205 %X Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.
APA
Ren, A.Z., Govil, B., Yang, T., Narasimhan, K.R. & Majumdar, A.. (2023). Leveraging Language for Accelerated Learning of Tool Manipulation. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1531-1541 Available from https://proceedings.mlr.press/v205/ren23a.html.

Related Material