Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning

Dhruv Shah, Michael Robert Equi, Błażej Osiński, Fei Xia, Brian Ichter, Sergey Levine
Proceedings of The 7th Conference on Robot Learning, PMLR 229:2683-2699, 2023.

Abstract

Navigation in unfamiliar environments presents a major challenge for robots: while mapping and planning techniques can be used to build up a representation of the world, quickly discovering a path to a desired goal in unfamiliar settings with such methods often requires lengthy mapping and exploration. Humans can rapidly navigate new environments, particularly indoor environments that are laid out logically, by leveraging semantics — e.g., a kitchen often adjoins a living room, an exit sign indicates the way out, and so forth. Language models can provide robots with such knowledge, but directly using language models to instruct a robot how to reach some destination can also be impractical: while language models might produce a narrative about how to reach some goal, because they are not grounded in real-world observations, this narrative might be arbitrarily wrong. Therefore, in this paper we study how the “semantic guesswork” produced by language models can be utilized as a guiding heuristic for planning algorithms. Our method, Language Frontier Guide (LFG), uses the language model to bias exploration of novel real-world environments by incorporating the semantic knowledge stored in language models as a search heuristic for planning with either topological or metric maps. We evaluate LFG in challenging real-world environments and simulated benchmarks, outperforming uninformed exploration and other ways of using language models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-shah23c, title = {Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning}, author = {Shah, Dhruv and Equi, Michael Robert and Osi\'{n}ski, B\l{}a\.{z}ej and Xia, Fei and Ichter, Brian and Levine, Sergey}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {2683--2699}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/shah23c/shah23c.pdf}, url = {https://proceedings.mlr.press/v229/shah23c.html}, abstract = {Navigation in unfamiliar environments presents a major challenge for robots: while mapping and planning techniques can be used to build up a representation of the world, quickly discovering a path to a desired goal in unfamiliar settings with such methods often requires lengthy mapping and exploration. Humans can rapidly navigate new environments, particularly indoor environments that are laid out logically, by leveraging semantics — e.g., a kitchen often adjoins a living room, an exit sign indicates the way out, and so forth. Language models can provide robots with such knowledge, but directly using language models to instruct a robot how to reach some destination can also be impractical: while language models might produce a narrative about how to reach some goal, because they are not grounded in real-world observations, this narrative might be arbitrarily wrong. Therefore, in this paper we study how the “semantic guesswork” produced by language models can be utilized as a guiding heuristic for planning algorithms. Our method, Language Frontier Guide (LFG), uses the language model to bias exploration of novel real-world environments by incorporating the semantic knowledge stored in language models as a search heuristic for planning with either topological or metric maps. We evaluate LFG in challenging real-world environments and simulated benchmarks, outperforming uninformed exploration and other ways of using language models.} }
Endnote
%0 Conference Paper %T Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning %A Dhruv Shah %A Michael Robert Equi %A Błażej Osiński %A Fei Xia %A Brian Ichter %A Sergey Levine %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-shah23c %I PMLR %P 2683--2699 %U https://proceedings.mlr.press/v229/shah23c.html %V 229 %X Navigation in unfamiliar environments presents a major challenge for robots: while mapping and planning techniques can be used to build up a representation of the world, quickly discovering a path to a desired goal in unfamiliar settings with such methods often requires lengthy mapping and exploration. Humans can rapidly navigate new environments, particularly indoor environments that are laid out logically, by leveraging semantics — e.g., a kitchen often adjoins a living room, an exit sign indicates the way out, and so forth. Language models can provide robots with such knowledge, but directly using language models to instruct a robot how to reach some destination can also be impractical: while language models might produce a narrative about how to reach some goal, because they are not grounded in real-world observations, this narrative might be arbitrarily wrong. Therefore, in this paper we study how the “semantic guesswork” produced by language models can be utilized as a guiding heuristic for planning algorithms. Our method, Language Frontier Guide (LFG), uses the language model to bias exploration of novel real-world environments by incorporating the semantic knowledge stored in language models as a search heuristic for planning with either topological or metric maps. We evaluate LFG in challenging real-world environments and simulated benchmarks, outperforming uninformed exploration and other ways of using language models.
APA
Shah, D., Equi, M.R., Osiński, B., Xia, F., Ichter, B. & Levine, S.. (2023). Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:2683-2699 Available from https://proceedings.mlr.press/v229/shah23c.html.

Related Material