TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach

Weikun Peng, Jun Lv, Yuwei Zeng, Haonan Chen, Siheng Zhao, Jichen Sun, Cewu Lu, Lin Shao
Proceedings of The 8th Conference on Robot Learning, PMLR 270:318-339, 2025.

Abstract

The tie-knotting task is highly challenging due to the tie’s high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie’s meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found on https://tiebots.github.io/.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-peng25a, title = {TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach}, author = {Peng, Weikun and Lv, Jun and Zeng, Yuwei and Chen, Haonan and Zhao, Siheng and Sun, Jichen and Lu, Cewu and Shao, Lin}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {318--339}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/peng25a/peng25a.pdf}, url = {https://proceedings.mlr.press/v270/peng25a.html}, abstract = {The tie-knotting task is highly challenging due to the tie’s high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie’s meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found on https://tiebots.github.io/.} }
Endnote
%0 Conference Paper %T TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach %A Weikun Peng %A Jun Lv %A Yuwei Zeng %A Haonan Chen %A Siheng Zhao %A Jichen Sun %A Cewu Lu %A Lin Shao %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-peng25a %I PMLR %P 318--339 %U https://proceedings.mlr.press/v270/peng25a.html %V 270 %X The tie-knotting task is highly challenging due to the tie’s high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie’s meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found on https://tiebots.github.io/.
APA
Peng, W., Lv, J., Zeng, Y., Chen, H., Zhao, S., Sun, J., Lu, C. & Shao, L.. (2025). TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:318-339 Available from https://proceedings.mlr.press/v270/peng25a.html.

Related Material