Language-guided Manipulator Motion Planning with Bounded Task Space

Thies Oelerich, Christian Hartl-Nesic, Andreas Kugi
Proceedings of The 8th Conference on Robot Learning, PMLR 270:749-779, 2025.

Abstract

Language-based robot control is a powerful and versatile method to control a robot manipulator where large language models (LLMs) are used to reason about the environment. However, the generated robot motions by these controllers often lack safety and performance, resulting in jerky movements. In this work, a novel modular framework for zero-shot motion planning for manipulation tasks is developed. The modular components do not require any motion-planning-specific training. An LLM is combined with a vision model to create Python code that interacts with a novel path planner, which creates a piecewise linear reference path with bounds around the path that ensure safety. An optimization-based planner, the BoundMPC framework, is utilized to execute optimal, safe, and collision-free trajectories along the reference path. The effectiveness of the approach is shown on various everyday manipulation tasks in simulation and experiment, shown in the video at www.acin.tuwien.ac.at/42d2.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-oelerich25a, title = {Language-guided Manipulator Motion Planning with Bounded Task Space}, author = {Oelerich, Thies and Hartl-Nesic, Christian and Kugi, Andreas}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {749--779}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/oelerich25a/oelerich25a.pdf}, url = {https://proceedings.mlr.press/v270/oelerich25a.html}, abstract = {Language-based robot control is a powerful and versatile method to control a robot manipulator where large language models (LLMs) are used to reason about the environment. However, the generated robot motions by these controllers often lack safety and performance, resulting in jerky movements. In this work, a novel modular framework for zero-shot motion planning for manipulation tasks is developed. The modular components do not require any motion-planning-specific training. An LLM is combined with a vision model to create Python code that interacts with a novel path planner, which creates a piecewise linear reference path with bounds around the path that ensure safety. An optimization-based planner, the BoundMPC framework, is utilized to execute optimal, safe, and collision-free trajectories along the reference path. The effectiveness of the approach is shown on various everyday manipulation tasks in simulation and experiment, shown in the video at www.acin.tuwien.ac.at/42d2.} }
Endnote
%0 Conference Paper %T Language-guided Manipulator Motion Planning with Bounded Task Space %A Thies Oelerich %A Christian Hartl-Nesic %A Andreas Kugi %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-oelerich25a %I PMLR %P 749--779 %U https://proceedings.mlr.press/v270/oelerich25a.html %V 270 %X Language-based robot control is a powerful and versatile method to control a robot manipulator where large language models (LLMs) are used to reason about the environment. However, the generated robot motions by these controllers often lack safety and performance, resulting in jerky movements. In this work, a novel modular framework for zero-shot motion planning for manipulation tasks is developed. The modular components do not require any motion-planning-specific training. An LLM is combined with a vision model to create Python code that interacts with a novel path planner, which creates a piecewise linear reference path with bounds around the path that ensure safety. An optimization-based planner, the BoundMPC framework, is utilized to execute optimal, safe, and collision-free trajectories along the reference path. The effectiveness of the approach is shown on various everyday manipulation tasks in simulation and experiment, shown in the video at www.acin.tuwien.ac.at/42d2.
APA
Oelerich, T., Hartl-Nesic, C. & Kugi, A.. (2025). Language-guided Manipulator Motion Planning with Bounded Task Space. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:749-779 Available from https://proceedings.mlr.press/v270/oelerich25a.html.

Related Material