Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding

Priya Sundaresan, Suneel Belkhale, Dorsa Sadigh
Proceedings of The 6th Conference on Robot Learning, PMLR 205:332-341, 2023.

Abstract

Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within a single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71% success over 69 skewering attempts total. Supplementary material, code, and videos can be found on our website: https://sites.google.com/view/hapticvisualnet-corl22/home.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-sundaresan23a, title = {Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding}, author = {Sundaresan, Priya and Belkhale, Suneel and Sadigh, Dorsa}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {332--341}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/sundaresan23a/sundaresan23a.pdf}, url = {https://proceedings.mlr.press/v205/sundaresan23a.html}, abstract = {Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within a single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71% success over 69 skewering attempts total. Supplementary material, code, and videos can be found on our website: https://sites.google.com/view/hapticvisualnet-corl22/home.} }
Endnote
%0 Conference Paper %T Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding %A Priya Sundaresan %A Suneel Belkhale %A Dorsa Sadigh %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-sundaresan23a %I PMLR %P 332--341 %U https://proceedings.mlr.press/v205/sundaresan23a.html %V 205 %X Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within a single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71% success over 69 skewering attempts total. Supplementary material, code, and videos can be found on our website: https://sites.google.com/view/hapticvisualnet-corl22/home.
APA
Sundaresan, P., Belkhale, S. & Sadigh, D.. (2023). Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:332-341 Available from https://proceedings.mlr.press/v205/sundaresan23a.html.

Related Material