LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations

Weikang Wan, Jiawei Fu, Xiaodi Yuan, Yifeng Zhu, Hao Su
Proceedings of The 9th Conference on Robot Learning, PMLR 305:4994-5021, 2025.

Abstract

Developing robotic systems capable of robustly executing long-horizon manipulation tasks with human-level dexterity is challenging, as such tasks require both physical dexterity and seamless sequencing of manipulation skills while robustly handling environment variations. While imitation learning offers a promising approach, acquiring comprehensive datasets is resource-intensive. In this work, we propose a learning framework and system LodeStar that automatically decomposes task demonstrations into semantically meaningful skills using off-the-shelf foundation models, and generates diverse synthetic demonstration datasets from a few human demos through reinforcement learning. These sim-augmented datasets enable robust skill training, with a Skill Routing Transformer (SRT) policy effectively chaining the learned skills together to execute complex long-horizon manipulation tasks. Experimental evaluations on three challenging real-world long-horizon dexterous manipulation tasks demonstrate that our approach significantly improves task performance and robustness compared to previous baselines. Videos are available at lodestar-robot.github.io.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-wan25a, title = {LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations}, author = {Wan, Weikang and Fu, Jiawei and Yuan, Xiaodi and Zhu, Yifeng and Su, Hao}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {4994--5021}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/wan25a/wan25a.pdf}, url = {https://proceedings.mlr.press/v305/wan25a.html}, abstract = {Developing robotic systems capable of robustly executing long-horizon manipulation tasks with human-level dexterity is challenging, as such tasks require both physical dexterity and seamless sequencing of manipulation skills while robustly handling environment variations. While imitation learning offers a promising approach, acquiring comprehensive datasets is resource-intensive. In this work, we propose a learning framework and system LodeStar that automatically decomposes task demonstrations into semantically meaningful skills using off-the-shelf foundation models, and generates diverse synthetic demonstration datasets from a few human demos through reinforcement learning. These sim-augmented datasets enable robust skill training, with a Skill Routing Transformer (SRT) policy effectively chaining the learned skills together to execute complex long-horizon manipulation tasks. Experimental evaluations on three challenging real-world long-horizon dexterous manipulation tasks demonstrate that our approach significantly improves task performance and robustness compared to previous baselines. Videos are available at lodestar-robot.github.io.} }
Endnote
%0 Conference Paper %T LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations %A Weikang Wan %A Jiawei Fu %A Xiaodi Yuan %A Yifeng Zhu %A Hao Su %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-wan25a %I PMLR %P 4994--5021 %U https://proceedings.mlr.press/v305/wan25a.html %V 305 %X Developing robotic systems capable of robustly executing long-horizon manipulation tasks with human-level dexterity is challenging, as such tasks require both physical dexterity and seamless sequencing of manipulation skills while robustly handling environment variations. While imitation learning offers a promising approach, acquiring comprehensive datasets is resource-intensive. In this work, we propose a learning framework and system LodeStar that automatically decomposes task demonstrations into semantically meaningful skills using off-the-shelf foundation models, and generates diverse synthetic demonstration datasets from a few human demos through reinforcement learning. These sim-augmented datasets enable robust skill training, with a Skill Routing Transformer (SRT) policy effectively chaining the learned skills together to execute complex long-horizon manipulation tasks. Experimental evaluations on three challenging real-world long-horizon dexterous manipulation tasks demonstrate that our approach significantly improves task performance and robustness compared to previous baselines. Videos are available at lodestar-robot.github.io.
APA
Wan, W., Fu, J., Yuan, X., Zhu, Y. & Su, H.. (2025). LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:4994-5021 Available from https://proceedings.mlr.press/v305/wan25a.html.

Related Material