TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types

Yuhao Lin, Yi-Lin Wei, Haoran Liao, Mu Lin, Chengyi Xing, Hao Li, Dandan Zhang, Mark Cutkosky, Wei-Shi Zheng
Proceedings of The 9th Conference on Robot Learning, PMLR 305:4975-4993, 2025.

Abstract

Dexterous teleoperation plays a crucial role in robotic manipulation for real-world data collection and remote robot control. Previous dexterous teleoperation mostly relies on hand retargeting to closely mimic human hand postures. However, these approaches may fail to fully leverage the inherent dexterity of dexterous hands, which can execute unique actions through their structural advantages compared to human hands. To address this limitation, we propose TypeTele, a type-guided dexterous teleoperation system, which enables dexterous hands to perform actions that are not constrained by human motion patterns. This is achieved by introducing dexterous manipulation types into the teleoperation system, allowing operators to employ appropriate types to complete specific tasks. To support this system, we build an extensible dexterous manipulation type library to cover comprehensive dexterous postures used in manipulation tasks. During teleoperation, we employ a MLLM (Multi-modality Large Language Model)-assisted type retrieval module to identify the most suitable manipulation type based on the specific task and operator commands. Extensive experiments of real-world teleoperation and imitation learning demonstrate that the incorporation of manipulation types significantly takes full advantage of the dexterous robot’s ability to perform diverse and complex tasks with higher success rates.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-lin25d, title = {TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types}, author = {Lin, Yuhao and Wei, Yi-Lin and Liao, Haoran and Lin, Mu and Xing, Chengyi and Li, Hao and Zhang, Dandan and Cutkosky, Mark and Zheng, Wei-Shi}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {4975--4993}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/lin25d/lin25d.pdf}, url = {https://proceedings.mlr.press/v305/lin25d.html}, abstract = {Dexterous teleoperation plays a crucial role in robotic manipulation for real-world data collection and remote robot control. Previous dexterous teleoperation mostly relies on hand retargeting to closely mimic human hand postures. However, these approaches may fail to fully leverage the inherent dexterity of dexterous hands, which can execute unique actions through their structural advantages compared to human hands. To address this limitation, we propose TypeTele, a type-guided dexterous teleoperation system, which enables dexterous hands to perform actions that are not constrained by human motion patterns. This is achieved by introducing dexterous manipulation types into the teleoperation system, allowing operators to employ appropriate types to complete specific tasks. To support this system, we build an extensible dexterous manipulation type library to cover comprehensive dexterous postures used in manipulation tasks. During teleoperation, we employ a MLLM (Multi-modality Large Language Model)-assisted type retrieval module to identify the most suitable manipulation type based on the specific task and operator commands. Extensive experiments of real-world teleoperation and imitation learning demonstrate that the incorporation of manipulation types significantly takes full advantage of the dexterous robot’s ability to perform diverse and complex tasks with higher success rates.} }
Endnote
%0 Conference Paper %T TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types %A Yuhao Lin %A Yi-Lin Wei %A Haoran Liao %A Mu Lin %A Chengyi Xing %A Hao Li %A Dandan Zhang %A Mark Cutkosky %A Wei-Shi Zheng %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-lin25d %I PMLR %P 4975--4993 %U https://proceedings.mlr.press/v305/lin25d.html %V 305 %X Dexterous teleoperation plays a crucial role in robotic manipulation for real-world data collection and remote robot control. Previous dexterous teleoperation mostly relies on hand retargeting to closely mimic human hand postures. However, these approaches may fail to fully leverage the inherent dexterity of dexterous hands, which can execute unique actions through their structural advantages compared to human hands. To address this limitation, we propose TypeTele, a type-guided dexterous teleoperation system, which enables dexterous hands to perform actions that are not constrained by human motion patterns. This is achieved by introducing dexterous manipulation types into the teleoperation system, allowing operators to employ appropriate types to complete specific tasks. To support this system, we build an extensible dexterous manipulation type library to cover comprehensive dexterous postures used in manipulation tasks. During teleoperation, we employ a MLLM (Multi-modality Large Language Model)-assisted type retrieval module to identify the most suitable manipulation type based on the specific task and operator commands. Extensive experiments of real-world teleoperation and imitation learning demonstrate that the incorporation of manipulation types significantly takes full advantage of the dexterous robot’s ability to perform diverse and complex tasks with higher success rates.
APA
Lin, Y., Wei, Y., Liao, H., Lin, M., Xing, C., Li, H., Zhang, D., Cutkosky, M. & Zheng, W.. (2025). TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:4975-4993 Available from https://proceedings.mlr.press/v305/lin25d.html.

Related Material