Ensuring Force Safety in Vision-Guided Robotic Manipulation via Implicit Tactile Calibration

Lai Wei, Jiahua Ma, Yibo Hu, Ruimao Zhang
Proceedings of The 9th Conference on Robot Learning, PMLR 305:1049-1062, 2025.

Abstract

In unstructured environments, robotic manipulation tasks involving objects with constrained motion trajectories—such as door opening—often experience discrepancies between the robot’s vision-guided end-effector trajectory and the object’s constrained motion path. Such discrepancies generate unintended harmful forces, which, if exacerbated, may lead to task failure and potential damage to the manipulated objects or the robot itself. To address this issue, this paper introduces a novel diffusion framework, termed SafeDiff. Unlike conventional methods that sequentially fuse visual and tactile data to predict future robot states, our approach generates a prospective state sequence based on the current robot state and visual context observations, using real-time force feedback as a calibration signal. This implicitly adjusts the robot’s state within the state space, enhancing operational success rates and significantly reducing harmful forces during manipulation, thus ensuring manipulation force safety. Additionally, we develop a large-scale simulation dataset named SafeDoorManip50k, offering extensive multimodal data to train and evaluate the proposed method. Extensive experiments show that our visual-tactile model substantially mitigates the risk of harmful forces in the door opening task, across both simulated and real-world settings.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-wei25a, title = {Ensuring Force Safety in Vision-Guided Robotic Manipulation via Implicit Tactile Calibration}, author = {Wei, Lai and Ma, Jiahua and Hu, Yibo and Zhang, Ruimao}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {1049--1062}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/wei25a/wei25a.pdf}, url = {https://proceedings.mlr.press/v305/wei25a.html}, abstract = {In unstructured environments, robotic manipulation tasks involving objects with constrained motion trajectories—such as door opening—often experience discrepancies between the robot’s vision-guided end-effector trajectory and the object’s constrained motion path. Such discrepancies generate unintended harmful forces, which, if exacerbated, may lead to task failure and potential damage to the manipulated objects or the robot itself. To address this issue, this paper introduces a novel diffusion framework, termed SafeDiff. Unlike conventional methods that sequentially fuse visual and tactile data to predict future robot states, our approach generates a prospective state sequence based on the current robot state and visual context observations, using real-time force feedback as a calibration signal. This implicitly adjusts the robot’s state within the state space, enhancing operational success rates and significantly reducing harmful forces during manipulation, thus ensuring manipulation force safety. Additionally, we develop a large-scale simulation dataset named SafeDoorManip50k, offering extensive multimodal data to train and evaluate the proposed method. Extensive experiments show that our visual-tactile model substantially mitigates the risk of harmful forces in the door opening task, across both simulated and real-world settings.} }
Endnote
%0 Conference Paper %T Ensuring Force Safety in Vision-Guided Robotic Manipulation via Implicit Tactile Calibration %A Lai Wei %A Jiahua Ma %A Yibo Hu %A Ruimao Zhang %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-wei25a %I PMLR %P 1049--1062 %U https://proceedings.mlr.press/v305/wei25a.html %V 305 %X In unstructured environments, robotic manipulation tasks involving objects with constrained motion trajectories—such as door opening—often experience discrepancies between the robot’s vision-guided end-effector trajectory and the object’s constrained motion path. Such discrepancies generate unintended harmful forces, which, if exacerbated, may lead to task failure and potential damage to the manipulated objects or the robot itself. To address this issue, this paper introduces a novel diffusion framework, termed SafeDiff. Unlike conventional methods that sequentially fuse visual and tactile data to predict future robot states, our approach generates a prospective state sequence based on the current robot state and visual context observations, using real-time force feedback as a calibration signal. This implicitly adjusts the robot’s state within the state space, enhancing operational success rates and significantly reducing harmful forces during manipulation, thus ensuring manipulation force safety. Additionally, we develop a large-scale simulation dataset named SafeDoorManip50k, offering extensive multimodal data to train and evaluate the proposed method. Extensive experiments show that our visual-tactile model substantially mitigates the risk of harmful forces in the door opening task, across both simulated and real-world settings.
APA
Wei, L., Ma, J., Hu, Y. & Zhang, R.. (2025). Ensuring Force Safety in Vision-Guided Robotic Manipulation via Implicit Tactile Calibration. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:1049-1062 Available from https://proceedings.mlr.press/v305/wei25a.html.

Related Material