Meta-Optimization and Program Search using Language Models for Task and Motion Planning

Denis Shcherba, Eckart Cobo-Briesewitz, Cornelius V. Braun, Marc Toussaint
Proceedings of The 9th Conference on Robot Learning, PMLR 305:5339-5361, 2025.

Abstract

Intelligent interaction with the real world requires robotic agents to jointly reason over high-level plans and low-level controls. This requirement is formalized in the task and motion planning (TAMP) problem, in which symbolic planning and continuous trajectory generation must be solved in a coordinated manner. Recently, foundation model-based approaches to TAMP have presented impressive results, including fast planning times and the execution of natural language instructions. Yet, the optimal interface between high-level plan and low-level motion generation remains to be found: prior approaches are limited by either too much abstraction (e.g., chaining simplified skill primitives) or a lack thereof (e.g., direct joint angle prediction). Our method introduces a novel technique employing a form of meta-optimization to address these shortcomings by: (i) using program search over trajectory optimization problems as an interface between foundation model and robot controllers, and (ii) leveraging a zero-order method to optimize numerical values in the foundation model output. Results on challenging object manipulation and drawing tasks confirm that our proposed method improves over prior TAMP approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-shcherba25a, title = {Meta-Optimization and Program Search using Language Models for Task and Motion Planning}, author = {Shcherba, Denis and Cobo-Briesewitz, Eckart and Braun, Cornelius V. and Toussaint, Marc}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {5339--5361}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/shcherba25a/shcherba25a.pdf}, url = {https://proceedings.mlr.press/v305/shcherba25a.html}, abstract = {Intelligent interaction with the real world requires robotic agents to jointly reason over high-level plans and low-level controls. This requirement is formalized in the task and motion planning (TAMP) problem, in which symbolic planning and continuous trajectory generation must be solved in a coordinated manner. Recently, foundation model-based approaches to TAMP have presented impressive results, including fast planning times and the execution of natural language instructions. Yet, the optimal interface between high-level plan and low-level motion generation remains to be found: prior approaches are limited by either too much abstraction (e.g., chaining simplified skill primitives) or a lack thereof (e.g., direct joint angle prediction). Our method introduces a novel technique employing a form of meta-optimization to address these shortcomings by: (i) using program search over trajectory optimization problems as an interface between foundation model and robot controllers, and (ii) leveraging a zero-order method to optimize numerical values in the foundation model output. Results on challenging object manipulation and drawing tasks confirm that our proposed method improves over prior TAMP approaches.} }
Endnote
%0 Conference Paper %T Meta-Optimization and Program Search using Language Models for Task and Motion Planning %A Denis Shcherba %A Eckart Cobo-Briesewitz %A Cornelius V. Braun %A Marc Toussaint %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-shcherba25a %I PMLR %P 5339--5361 %U https://proceedings.mlr.press/v305/shcherba25a.html %V 305 %X Intelligent interaction with the real world requires robotic agents to jointly reason over high-level plans and low-level controls. This requirement is formalized in the task and motion planning (TAMP) problem, in which symbolic planning and continuous trajectory generation must be solved in a coordinated manner. Recently, foundation model-based approaches to TAMP have presented impressive results, including fast planning times and the execution of natural language instructions. Yet, the optimal interface between high-level plan and low-level motion generation remains to be found: prior approaches are limited by either too much abstraction (e.g., chaining simplified skill primitives) or a lack thereof (e.g., direct joint angle prediction). Our method introduces a novel technique employing a form of meta-optimization to address these shortcomings by: (i) using program search over trajectory optimization problems as an interface between foundation model and robot controllers, and (ii) leveraging a zero-order method to optimize numerical values in the foundation model output. Results on challenging object manipulation and drawing tasks confirm that our proposed method improves over prior TAMP approaches.
APA
Shcherba, D., Cobo-Briesewitz, E., Braun, C.V. & Toussaint, M.. (2025). Meta-Optimization and Program Search using Language Models for Task and Motion Planning. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:5339-5361 Available from https://proceedings.mlr.press/v305/shcherba25a.html.

Related Material