Taskography: Evaluating robot task planning over large 3D scene graphs

Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, Florian Shkurti
Proceedings of the 5th Conference on Robot Learning, PMLR 164:46-58, 2022.

Abstract

3D scene graphs (3DSGs) are an emerging description; unifying symbolic, topological, and metric scene representations. However, typical 3DSGs contain hundreds of objects and symbols even for small environments; rendering task planning on the \emph{full} graph impractical. We construct \textbf{Taskography}, the first large-scale robotic task planning benchmark over 3DSGs. While most benchmarking efforts in this area focus on \emph{vision-based planning}, we systematically study \emph{symbolic} planning, to decouple planning performance from visual representation learning. We observe that, among existing methods, neither classical nor learning-based planners are capable of real-time planning over \emph{full} 3DSGs. Enabling real-time planning demands progress on \emph{both} (a) sparsifying 3DSGs for tractable planning and (b) designing planners that better exploit 3DSG hierarchies. Towards the former goal, we propose \textbf{Scrub}, a task-conditioned 3DSG sparsification method; enabling classical planners to match (and surpass) state-of-the-art learning-based planners. Towards the latter goal, we propose \textbf{Seek}, a procedure enabling learning-based planners to exploit 3DSG structure, reducing the number of replanning queries required by current best approaches by an order of magnitude. We will open-source all code and baselines to spur further research along the intersections of robot task planning, learning and 3DSGs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-agia22a, title = {Taskography: Evaluating robot task planning over large 3D scene graphs}, author = {Agia, Christopher and Jatavallabhula, Krishna Murthy and Khodeir, Mohamed and Miksik, Ondrej and Vineet, Vibhav and Mukadam, Mustafa and Paull, Liam and Shkurti, Florian}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {46--58}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/agia22a/agia22a.pdf}, url = {https://proceedings.mlr.press/v164/agia22a.html}, abstract = {3D scene graphs (3DSGs) are an emerging description; unifying symbolic, topological, and metric scene representations. However, typical 3DSGs contain hundreds of objects and symbols even for small environments; rendering task planning on the \emph{full} graph impractical. We construct \textbf{Taskography}, the first large-scale robotic task planning benchmark over 3DSGs. While most benchmarking efforts in this area focus on \emph{vision-based planning}, we systematically study \emph{symbolic} planning, to decouple planning performance from visual representation learning. We observe that, among existing methods, neither classical nor learning-based planners are capable of real-time planning over \emph{full} 3DSGs. Enabling real-time planning demands progress on \emph{both} (a) sparsifying 3DSGs for tractable planning and (b) designing planners that better exploit 3DSG hierarchies. Towards the former goal, we propose \textbf{Scrub}, a task-conditioned 3DSG sparsification method; enabling classical planners to match (and surpass) state-of-the-art learning-based planners. Towards the latter goal, we propose \textbf{Seek}, a procedure enabling learning-based planners to exploit 3DSG structure, reducing the number of replanning queries required by current best approaches by an order of magnitude. We will open-source all code and baselines to spur further research along the intersections of robot task planning, learning and 3DSGs.} }
Endnote
%0 Conference Paper %T Taskography: Evaluating robot task planning over large 3D scene graphs %A Christopher Agia %A Krishna Murthy Jatavallabhula %A Mohamed Khodeir %A Ondrej Miksik %A Vibhav Vineet %A Mustafa Mukadam %A Liam Paull %A Florian Shkurti %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-agia22a %I PMLR %P 46--58 %U https://proceedings.mlr.press/v164/agia22a.html %V 164 %X 3D scene graphs (3DSGs) are an emerging description; unifying symbolic, topological, and metric scene representations. However, typical 3DSGs contain hundreds of objects and symbols even for small environments; rendering task planning on the \emph{full} graph impractical. We construct \textbf{Taskography}, the first large-scale robotic task planning benchmark over 3DSGs. While most benchmarking efforts in this area focus on \emph{vision-based planning}, we systematically study \emph{symbolic} planning, to decouple planning performance from visual representation learning. We observe that, among existing methods, neither classical nor learning-based planners are capable of real-time planning over \emph{full} 3DSGs. Enabling real-time planning demands progress on \emph{both} (a) sparsifying 3DSGs for tractable planning and (b) designing planners that better exploit 3DSG hierarchies. Towards the former goal, we propose \textbf{Scrub}, a task-conditioned 3DSG sparsification method; enabling classical planners to match (and surpass) state-of-the-art learning-based planners. Towards the latter goal, we propose \textbf{Seek}, a procedure enabling learning-based planners to exploit 3DSG structure, reducing the number of replanning queries required by current best approaches by an order of magnitude. We will open-source all code and baselines to spur further research along the intersections of robot task planning, learning and 3DSGs.
APA
Agia, C., Jatavallabhula, K.M., Khodeir, M., Miksik, O., Vineet, V., Mukadam, M., Paull, L. & Shkurti, F.. (2022). Taskography: Evaluating robot task planning over large 3D scene graphs. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:46-58 Available from https://proceedings.mlr.press/v164/agia22a.html.

Related Material