Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization

Albert Wu, Michelle Guo, Karen Liu
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1938-1948, 2023.

Abstract

To fully utilize the versatility of a multi-fingered dexterous robotic hand for executing diverse object grasps, one must consider the rich physical constraints introduced by hand-object interaction and object geometry. We propose an integrative approach of combining a generative model and a bilevel optimization (BO) to plan diverse grasp configurations on novel objects. First, a conditional variational autoencoder trained on merely six YCB objects predicts the finger placement directly from the object point cloud. The prediction is then used to seed a nonconvex BO that solves for a grasp configuration under collision, reachability, wrench closure, and friction constraints. Our method achieved an 86.7% success over 120 real world grasping trials on 20 household objects, including unseen and challenging geometries. Through quantitative empirical evaluations, we confirm that grasp configurations produced by our pipeline are indeed guaranteed to satisfy kinematic and dynamic constraints. A video summary of our results is available at youtu.be/9DTrImbN99I.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-wu23b, title = {Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization}, author = {Wu, Albert and Guo, Michelle and Liu, Karen}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1938--1948}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/wu23b/wu23b.pdf}, url = {https://proceedings.mlr.press/v205/wu23b.html}, abstract = {To fully utilize the versatility of a multi-fingered dexterous robotic hand for executing diverse object grasps, one must consider the rich physical constraints introduced by hand-object interaction and object geometry. We propose an integrative approach of combining a generative model and a bilevel optimization (BO) to plan diverse grasp configurations on novel objects. First, a conditional variational autoencoder trained on merely six YCB objects predicts the finger placement directly from the object point cloud. The prediction is then used to seed a nonconvex BO that solves for a grasp configuration under collision, reachability, wrench closure, and friction constraints. Our method achieved an 86.7% success over 120 real world grasping trials on 20 household objects, including unseen and challenging geometries. Through quantitative empirical evaluations, we confirm that grasp configurations produced by our pipeline are indeed guaranteed to satisfy kinematic and dynamic constraints. A video summary of our results is available at youtu.be/9DTrImbN99I.} }
Endnote
%0 Conference Paper %T Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization %A Albert Wu %A Michelle Guo %A Karen Liu %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-wu23b %I PMLR %P 1938--1948 %U https://proceedings.mlr.press/v205/wu23b.html %V 205 %X To fully utilize the versatility of a multi-fingered dexterous robotic hand for executing diverse object grasps, one must consider the rich physical constraints introduced by hand-object interaction and object geometry. We propose an integrative approach of combining a generative model and a bilevel optimization (BO) to plan diverse grasp configurations on novel objects. First, a conditional variational autoencoder trained on merely six YCB objects predicts the finger placement directly from the object point cloud. The prediction is then used to seed a nonconvex BO that solves for a grasp configuration under collision, reachability, wrench closure, and friction constraints. Our method achieved an 86.7% success over 120 real world grasping trials on 20 household objects, including unseen and challenging geometries. Through quantitative empirical evaluations, we confirm that grasp configurations produced by our pipeline are indeed guaranteed to satisfy kinematic and dynamic constraints. A video summary of our results is available at youtu.be/9DTrImbN99I.
APA
Wu, A., Guo, M. & Liu, K.. (2023). Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1938-1948 Available from https://proceedings.mlr.press/v205/wu23b.html.

Related Material