MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation

Kelin Yu, Yunhai Han, Qixian Wang, Vaibhav Saxena, Danfei Xu, Ye Zhao
Proceedings of The 8th Conference on Robot Learning, PMLR 270:4844-4865, 2025.

Abstract

Tactile sensing is critical to fine-grained, contact-rich manipulation tasks, such as insertion and assembly. Prior research has shown the possibility of learning tactile-guided policy from teleoperated demonstration data. However, to provide the demonstration, human users often rely on visual feedback to control the robot. This creates a gap between the sensing modality used for controlling the robot (visual) and the modality of interest (tactile). To bridge this gap, we introduce “MimicTouch”, a novel framework for learning policies directly from demonstrations provided by human users with their hands. The key innovations are i) a human tactile data collection system which collects multi-modal tactile dataset for learning human’s tactile-guided control strategy, ii) an imitation learning-based framework for learning human’s tactile-guided control strategy through such data, and iii) an online residual RL framework to bridge the embodiment gap between the human hand and the robot gripper. Through comprehensive experiments, we highlight the efficacy of utilizing human’s tactile-guided control strategy to resolve contact-rich manipulation tasks. The project website is at https://sites.google.com/view/MimicTouch.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-yu25c, title = {MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation}, author = {Yu, Kelin and Han, Yunhai and Wang, Qixian and Saxena, Vaibhav and Xu, Danfei and Zhao, Ye}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {4844--4865}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/yu25c/yu25c.pdf}, url = {https://proceedings.mlr.press/v270/yu25c.html}, abstract = {Tactile sensing is critical to fine-grained, contact-rich manipulation tasks, such as insertion and assembly. Prior research has shown the possibility of learning tactile-guided policy from teleoperated demonstration data. However, to provide the demonstration, human users often rely on visual feedback to control the robot. This creates a gap between the sensing modality used for controlling the robot (visual) and the modality of interest (tactile). To bridge this gap, we introduce “MimicTouch”, a novel framework for learning policies directly from demonstrations provided by human users with their hands. The key innovations are i) a human tactile data collection system which collects multi-modal tactile dataset for learning human’s tactile-guided control strategy, ii) an imitation learning-based framework for learning human’s tactile-guided control strategy through such data, and iii) an online residual RL framework to bridge the embodiment gap between the human hand and the robot gripper. Through comprehensive experiments, we highlight the efficacy of utilizing human’s tactile-guided control strategy to resolve contact-rich manipulation tasks. The project website is at https://sites.google.com/view/MimicTouch.} }
Endnote
%0 Conference Paper %T MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation %A Kelin Yu %A Yunhai Han %A Qixian Wang %A Vaibhav Saxena %A Danfei Xu %A Ye Zhao %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-yu25c %I PMLR %P 4844--4865 %U https://proceedings.mlr.press/v270/yu25c.html %V 270 %X Tactile sensing is critical to fine-grained, contact-rich manipulation tasks, such as insertion and assembly. Prior research has shown the possibility of learning tactile-guided policy from teleoperated demonstration data. However, to provide the demonstration, human users often rely on visual feedback to control the robot. This creates a gap between the sensing modality used for controlling the robot (visual) and the modality of interest (tactile). To bridge this gap, we introduce “MimicTouch”, a novel framework for learning policies directly from demonstrations provided by human users with their hands. The key innovations are i) a human tactile data collection system which collects multi-modal tactile dataset for learning human’s tactile-guided control strategy, ii) an imitation learning-based framework for learning human’s tactile-guided control strategy through such data, and iii) an online residual RL framework to bridge the embodiment gap between the human hand and the robot gripper. Through comprehensive experiments, we highlight the efficacy of utilizing human’s tactile-guided control strategy to resolve contact-rich manipulation tasks. The project website is at https://sites.google.com/view/MimicTouch.
APA
Yu, K., Han, Y., Wang, Q., Saxena, V., Xu, D. & Zhao, Y.. (2025). MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:4844-4865 Available from https://proceedings.mlr.press/v270/yu25c.html.

Related Material