SimShear: Sim-to-Real Shear-based Tactile Servoing

Kipp Freud, Yijiong Lin, Nathan F. Lepora
Proceedings of The 9th Conference on Robot Learning, PMLR 305:3401-3412, 2025.

Abstract

We present SimShear: a sim-to-real pipeline for tactile control that allows use of shear information without explicitly modeling shear dynamics in simulation. Shear, which arises from lateral movements across contact surfaces, are critical for tasks involving dynamic object interactions but are challenging to simulate. We introduce shPix2pix: a shear-conditioned U-Net GAN that transforms simulated tactile images absent of shear plus a vector encoding shear information into realistic equivalents that include shear deformations, and show this outperforms baseline pix2pix methods for simulating tactile images and pose/shear prediction. This is applied to two control tasks using a pair of low-cost desktop robotic arms equipped with a vision-based tactile sensor: first, a tactile tracking task, where a follower arm tracks a surface moved by the leader arm; second, a collaborative co-lift task, where both arms jointly hold an object while the leader arm moves along a prescribed trajectory. Our method maintain contact errors within 1-2 mm across varied trajectories where shear sensing is essential for task performance. This work validates the use of sim-to-real shear modeling with rigid-body simulators, opening new possibilities for simulation in tactile robotics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-freud25a, title = {SimShear: Sim-to-Real Shear-based Tactile Servoing}, author = {Freud, Kipp and Lin, Yijiong and Lepora, Nathan F.}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {3401--3412}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/freud25a/freud25a.pdf}, url = {https://proceedings.mlr.press/v305/freud25a.html}, abstract = {We present SimShear: a sim-to-real pipeline for tactile control that allows use of shear information without explicitly modeling shear dynamics in simulation. Shear, which arises from lateral movements across contact surfaces, are critical for tasks involving dynamic object interactions but are challenging to simulate. We introduce shPix2pix: a shear-conditioned U-Net GAN that transforms simulated tactile images absent of shear plus a vector encoding shear information into realistic equivalents that include shear deformations, and show this outperforms baseline pix2pix methods for simulating tactile images and pose/shear prediction. This is applied to two control tasks using a pair of low-cost desktop robotic arms equipped with a vision-based tactile sensor: first, a tactile tracking task, where a follower arm tracks a surface moved by the leader arm; second, a collaborative co-lift task, where both arms jointly hold an object while the leader arm moves along a prescribed trajectory. Our method maintain contact errors within 1-2 mm across varied trajectories where shear sensing is essential for task performance. This work validates the use of sim-to-real shear modeling with rigid-body simulators, opening new possibilities for simulation in tactile robotics.} }
Endnote
%0 Conference Paper %T SimShear: Sim-to-Real Shear-based Tactile Servoing %A Kipp Freud %A Yijiong Lin %A Nathan F. Lepora %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-freud25a %I PMLR %P 3401--3412 %U https://proceedings.mlr.press/v305/freud25a.html %V 305 %X We present SimShear: a sim-to-real pipeline for tactile control that allows use of shear information without explicitly modeling shear dynamics in simulation. Shear, which arises from lateral movements across contact surfaces, are critical for tasks involving dynamic object interactions but are challenging to simulate. We introduce shPix2pix: a shear-conditioned U-Net GAN that transforms simulated tactile images absent of shear plus a vector encoding shear information into realistic equivalents that include shear deformations, and show this outperforms baseline pix2pix methods for simulating tactile images and pose/shear prediction. This is applied to two control tasks using a pair of low-cost desktop robotic arms equipped with a vision-based tactile sensor: first, a tactile tracking task, where a follower arm tracks a surface moved by the leader arm; second, a collaborative co-lift task, where both arms jointly hold an object while the leader arm moves along a prescribed trajectory. Our method maintain contact errors within 1-2 mm across varied trajectories where shear sensing is essential for task performance. This work validates the use of sim-to-real shear modeling with rigid-body simulators, opening new possibilities for simulation in tactile robotics.
APA
Freud, K., Lin, Y. & Lepora, N.F.. (2025). SimShear: Sim-to-Real Shear-based Tactile Servoing. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:3401-3412 Available from https://proceedings.mlr.press/v305/freud25a.html.

Related Material