Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions

Harrison Field, Max Yang, Yijiong Lin, Efi Psomopoulou, David A.W. Barton, Nathan F. Lepora
Proceedings of The 9th Conference on Robot Learning, PMLR 305:2847-2887, 2025.

Abstract

Large language models (LLMs) are beginning to automate reward design for dexterous manipulation. However, no prior work has considered tactile sensing, which is known to be critical for human-like dexterity. We present Text2Touch, bringing LLM-crafted rewards to the challenging task of multi-axis in-hand object rotation with real-world vision based tactile sensing in palm-up and palm-down configurations. Our prompt engineering strategy scales to over 70 environment variables, and sim-to-real distillation enables successful policy transfer to a tactile-enabled fully actuated four-fingered dexterous robot hand. Text2Touch significantly outperforms a carefully tuned human-engineered baseline, demonstrating superior rotation speed and stability while relying on reward functions that are an order of magnitude shorter and simpler. These results illustrate how LLM-designed rewards can significantly reduce the time from concept to deployable dexterous tactile skills, supporting more rapid and scalable multimodal robot learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-field25a, title = {Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions}, author = {Field, Harrison and Yang, Max and Lin, Yijiong and Psomopoulou, Efi and Barton, David A.W. and Lepora, Nathan F.}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {2847--2887}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/field25a/field25a.pdf}, url = {https://proceedings.mlr.press/v305/field25a.html}, abstract = {Large language models (LLMs) are beginning to automate reward design for dexterous manipulation. However, no prior work has considered tactile sensing, which is known to be critical for human-like dexterity. We present Text2Touch, bringing LLM-crafted rewards to the challenging task of multi-axis in-hand object rotation with real-world vision based tactile sensing in palm-up and palm-down configurations. Our prompt engineering strategy scales to over 70 environment variables, and sim-to-real distillation enables successful policy transfer to a tactile-enabled fully actuated four-fingered dexterous robot hand. Text2Touch significantly outperforms a carefully tuned human-engineered baseline, demonstrating superior rotation speed and stability while relying on reward functions that are an order of magnitude shorter and simpler. These results illustrate how LLM-designed rewards can significantly reduce the time from concept to deployable dexterous tactile skills, supporting more rapid and scalable multimodal robot learning.} }
Endnote
%0 Conference Paper %T Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions %A Harrison Field %A Max Yang %A Yijiong Lin %A Efi Psomopoulou %A David A.W. Barton %A Nathan F. Lepora %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-field25a %I PMLR %P 2847--2887 %U https://proceedings.mlr.press/v305/field25a.html %V 305 %X Large language models (LLMs) are beginning to automate reward design for dexterous manipulation. However, no prior work has considered tactile sensing, which is known to be critical for human-like dexterity. We present Text2Touch, bringing LLM-crafted rewards to the challenging task of multi-axis in-hand object rotation with real-world vision based tactile sensing in palm-up and palm-down configurations. Our prompt engineering strategy scales to over 70 environment variables, and sim-to-real distillation enables successful policy transfer to a tactile-enabled fully actuated four-fingered dexterous robot hand. Text2Touch significantly outperforms a carefully tuned human-engineered baseline, demonstrating superior rotation speed and stability while relying on reward functions that are an order of magnitude shorter and simpler. These results illustrate how LLM-designed rewards can significantly reduce the time from concept to deployable dexterous tactile skills, supporting more rapid and scalable multimodal robot learning.
APA
Field, H., Yang, M., Lin, Y., Psomopoulou, E., Barton, D.A. & Lepora, N.F.. (2025). Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:2847-2887 Available from https://proceedings.mlr.press/v305/field25a.html.

Related Material