Self-Supervised Learning of Scene-Graph Representations for Robotic Sequential Manipulation Planning

Son Nguyen, Ozgur Oguz, Valentin Hartmann, Marc Toussaint
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:2104-2119, 2021.

Abstract

We present a self-supervised representation learning approach for visual reasoning and integrate it into a nonlinear program formulation for motion optimization to tackle sequential manipulation tasks. Such problems have usually been addressed by combined task and motion planning approaches, for which spatial relations and logical rules that rely on symbolic representations have to be predefined by the user. We propose to learn relational structures by leveraging visual perception to alleviate the resulting knowledge acquisition bottleneck. In particular, we learn constructing {\em scene-graphs}, that represent objects (“red box"), and their spatial relationships (‘yellow cylinder on red box"). This representation allows us to plan high-level discrete decisions effectively using graph search algorithms. We integrate the visual reasoning module with a nonlinear optimization method for robot motion planning and verify its feasibility on the classic blocks-world domain. Our proposed framework successfully finds the sequence of actions and enables the robot to execute feasible motion plans to realize the given tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-nguyen21b, title = {Self-Supervised Learning of Scene-Graph Representations for Robotic Sequential Manipulation Planning}, author = {Nguyen, Son and Oguz, Ozgur and Hartmann, Valentin and Toussaint, Marc}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {2104--2119}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/nguyen21b/nguyen21b.pdf}, url = {https://proceedings.mlr.press/v155/nguyen21b.html}, abstract = {We present a self-supervised representation learning approach for visual reasoning and integrate it into a nonlinear program formulation for motion optimization to tackle sequential manipulation tasks. Such problems have usually been addressed by combined task and motion planning approaches, for which spatial relations and logical rules that rely on symbolic representations have to be predefined by the user. We propose to learn relational structures by leveraging visual perception to alleviate the resulting knowledge acquisition bottleneck. In particular, we learn constructing {\em scene-graphs}, that represent objects (“red box"), and their spatial relationships (‘yellow cylinder on red box"). This representation allows us to plan high-level discrete decisions effectively using graph search algorithms. We integrate the visual reasoning module with a nonlinear optimization method for robot motion planning and verify its feasibility on the classic blocks-world domain. Our proposed framework successfully finds the sequence of actions and enables the robot to execute feasible motion plans to realize the given tasks.} }
Endnote
%0 Conference Paper %T Self-Supervised Learning of Scene-Graph Representations for Robotic Sequential Manipulation Planning %A Son Nguyen %A Ozgur Oguz %A Valentin Hartmann %A Marc Toussaint %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-nguyen21b %I PMLR %P 2104--2119 %U https://proceedings.mlr.press/v155/nguyen21b.html %V 155 %X We present a self-supervised representation learning approach for visual reasoning and integrate it into a nonlinear program formulation for motion optimization to tackle sequential manipulation tasks. Such problems have usually been addressed by combined task and motion planning approaches, for which spatial relations and logical rules that rely on symbolic representations have to be predefined by the user. We propose to learn relational structures by leveraging visual perception to alleviate the resulting knowledge acquisition bottleneck. In particular, we learn constructing {\em scene-graphs}, that represent objects (“red box"), and their spatial relationships (‘yellow cylinder on red box"). This representation allows us to plan high-level discrete decisions effectively using graph search algorithms. We integrate the visual reasoning module with a nonlinear optimization method for robot motion planning and verify its feasibility on the classic blocks-world domain. Our proposed framework successfully finds the sequence of actions and enables the robot to execute feasible motion plans to realize the given tasks.
APA
Nguyen, S., Oguz, O., Hartmann, V. & Toussaint, M.. (2021). Self-Supervised Learning of Scene-Graph Representations for Robotic Sequential Manipulation Planning. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:2104-2119 Available from https://proceedings.mlr.press/v155/nguyen21b.html.

Related Material