SayTap: Language to Quadrupedal Locomotion

Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada
Proceedings of The 7th Conference on Robot Learning, PMLR 229:3556-3570, 2023.

Abstract

Large language models (LLMs) have demonstrated the potential to perform high-level planning. Yet, it remains a challenge for LLMs to comprehend low-level commands, such as joint angle targets or motor torques. This paper proposes an approach to use foot contact patterns as an interface that bridges human commands in natural language and a locomotion controller that outputs these low-level commands. This results in an interactive system for quadrupedal robots that allows the users to craft diverse locomotion behaviors flexibly. We contribute an LLM prompt design, a reward function, and a method to expose the controller to the feasible distribution of contact patterns. The results are a controller capable of achieving diverse locomotion patterns that can be transferred to real robot hardware. Compared with other design choices, the proposed approach enjoys more than $50%$ success rate in predicting the correct contact patterns and can solve 10 more tasks out of a total of 30 tasks. (https://saytap.github.io)

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-tang23a, title = {SayTap: Language to Quadrupedal Locomotion}, author = {Tang, Yujin and Yu, Wenhao and Tan, Jie and Zen, Heiga and Faust, Aleksandra and Harada, Tatsuya}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {3556--3570}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/tang23a/tang23a.pdf}, url = {https://proceedings.mlr.press/v229/tang23a.html}, abstract = {Large language models (LLMs) have demonstrated the potential to perform high-level planning. Yet, it remains a challenge for LLMs to comprehend low-level commands, such as joint angle targets or motor torques. This paper proposes an approach to use foot contact patterns as an interface that bridges human commands in natural language and a locomotion controller that outputs these low-level commands. This results in an interactive system for quadrupedal robots that allows the users to craft diverse locomotion behaviors flexibly. We contribute an LLM prompt design, a reward function, and a method to expose the controller to the feasible distribution of contact patterns. The results are a controller capable of achieving diverse locomotion patterns that can be transferred to real robot hardware. Compared with other design choices, the proposed approach enjoys more than $50%$ success rate in predicting the correct contact patterns and can solve 10 more tasks out of a total of 30 tasks. (https://saytap.github.io)} }
Endnote
%0 Conference Paper %T SayTap: Language to Quadrupedal Locomotion %A Yujin Tang %A Wenhao Yu %A Jie Tan %A Heiga Zen %A Aleksandra Faust %A Tatsuya Harada %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-tang23a %I PMLR %P 3556--3570 %U https://proceedings.mlr.press/v229/tang23a.html %V 229 %X Large language models (LLMs) have demonstrated the potential to perform high-level planning. Yet, it remains a challenge for LLMs to comprehend low-level commands, such as joint angle targets or motor torques. This paper proposes an approach to use foot contact patterns as an interface that bridges human commands in natural language and a locomotion controller that outputs these low-level commands. This results in an interactive system for quadrupedal robots that allows the users to craft diverse locomotion behaviors flexibly. We contribute an LLM prompt design, a reward function, and a method to expose the controller to the feasible distribution of contact patterns. The results are a controller capable of achieving diverse locomotion patterns that can be transferred to real robot hardware. Compared with other design choices, the proposed approach enjoys more than $50%$ success rate in predicting the correct contact patterns and can solve 10 more tasks out of a total of 30 tasks. (https://saytap.github.io)
APA
Tang, Y., Yu, W., Tan, J., Zen, H., Faust, A. & Harada, T.. (2023). SayTap: Language to Quadrupedal Locomotion. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:3556-3570 Available from https://proceedings.mlr.press/v229/tang23a.html.

Related Material