Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions

Juan Del Aguila Ferrandis, Joao Moura, Sethu Vijayakumar
Proceedings of The 8th Conference on Robot Learning, PMLR 270:1501-1515, 2025.

Abstract

Manipulation without grasping, known as non-prehensile manipulation, is essential for dexterous robots in contact-rich environments, but presents many challenges relating with underactuation, hybrid-dynamics, and frictional uncertainty. Additionally, object occlusions in a scenario of contact uncertainty and where the motion of the object evolves independently from the robot becomes a critical problem, which previous literature fails to address. We present a method for learning visuotactile state estimators and uncertainty-aware control policies for non-prehensile manipulation under occlusions, by leveraging diverse interaction data from privileged policies trained in simulation. We formulate the estimator within a Bayesian deep learning framework, to model its uncertainty, and then train uncertainty-aware control policies by incorporating the pre-learned estimator into the reinforcement learning (RL) loop, both of which lead to significantly improved estimator and policy performance. Therefore, unlike prior non-prehensile research that relies on complex external perception set-ups, our method successfully handles occlusions after sim-to-real transfer to robotic hardware with a simple onboard camera.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-ferrandis25a, title = {Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions}, author = {Ferrandis, Juan Del Aguila and Moura, Joao and Vijayakumar, Sethu}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {1501--1515}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/ferrandis25a/ferrandis25a.pdf}, url = {https://proceedings.mlr.press/v270/ferrandis25a.html}, abstract = {Manipulation without grasping, known as non-prehensile manipulation, is essential for dexterous robots in contact-rich environments, but presents many challenges relating with underactuation, hybrid-dynamics, and frictional uncertainty. Additionally, object occlusions in a scenario of contact uncertainty and where the motion of the object evolves independently from the robot becomes a critical problem, which previous literature fails to address. We present a method for learning visuotactile state estimators and uncertainty-aware control policies for non-prehensile manipulation under occlusions, by leveraging diverse interaction data from privileged policies trained in simulation. We formulate the estimator within a Bayesian deep learning framework, to model its uncertainty, and then train uncertainty-aware control policies by incorporating the pre-learned estimator into the reinforcement learning (RL) loop, both of which lead to significantly improved estimator and policy performance. Therefore, unlike prior non-prehensile research that relies on complex external perception set-ups, our method successfully handles occlusions after sim-to-real transfer to robotic hardware with a simple onboard camera.} }
Endnote
%0 Conference Paper %T Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions %A Juan Del Aguila Ferrandis %A Joao Moura %A Sethu Vijayakumar %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-ferrandis25a %I PMLR %P 1501--1515 %U https://proceedings.mlr.press/v270/ferrandis25a.html %V 270 %X Manipulation without grasping, known as non-prehensile manipulation, is essential for dexterous robots in contact-rich environments, but presents many challenges relating with underactuation, hybrid-dynamics, and frictional uncertainty. Additionally, object occlusions in a scenario of contact uncertainty and where the motion of the object evolves independently from the robot becomes a critical problem, which previous literature fails to address. We present a method for learning visuotactile state estimators and uncertainty-aware control policies for non-prehensile manipulation under occlusions, by leveraging diverse interaction data from privileged policies trained in simulation. We formulate the estimator within a Bayesian deep learning framework, to model its uncertainty, and then train uncertainty-aware control policies by incorporating the pre-learned estimator into the reinforcement learning (RL) loop, both of which lead to significantly improved estimator and policy performance. Therefore, unlike prior non-prehensile research that relies on complex external perception set-ups, our method successfully handles occlusions after sim-to-real transfer to robotic hardware with a simple onboard camera.
APA
Ferrandis, J.D.A., Moura, J. & Vijayakumar, S.. (2025). Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:1501-1515 Available from https://proceedings.mlr.press/v270/ferrandis25a.html.

Related Material