Tracking and Planning with Spatial World Models

Baris Kayalibay, Atanas Mirchev, Patrick van der Smagt, Justin Bayer
Proceedings of The 4th Annual Learning for Dynamics and Control Conference, PMLR 168:124-137, 2022.

Abstract

We introduce a method for real-time navigation and tracking with differentiably rendered world models. Learning models for control has led to impressive results in robotics and computer games, but this success has yet to be extended to vision-based navigation. To address this, we transfer advances in the emergent field of differentiable rendering to model-based control. We do this by planning in a learned 3D spatial world model, combined with a pose estimation algorithm previously used in the context of TSDF fusion, but now tailored to our setting and improved to incorporate agent dynamics. We evaluate over six simulated environments based on complex human-designed floor plans and provide quantitative results. We achieve up to 92% navigation success rate at a frequency of 15 Hz using only image and depth observations under stochastic, continuous dynamics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v168-kayalibay22a, title = {Tracking and Planning with Spatial World Models}, author = {Kayalibay, Baris and Mirchev, Atanas and van der Smagt, Patrick and Bayer, Justin}, booktitle = {Proceedings of The 4th Annual Learning for Dynamics and Control Conference}, pages = {124--137}, year = {2022}, editor = {Firoozi, Roya and Mehr, Negar and Yel, Esen and Antonova, Rika and Bohg, Jeannette and Schwager, Mac and Kochenderfer, Mykel}, volume = {168}, series = {Proceedings of Machine Learning Research}, month = {23--24 Jun}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v168/kayalibay22a/kayalibay22a.pdf}, url = {https://proceedings.mlr.press/v168/kayalibay22a.html}, abstract = {We introduce a method for real-time navigation and tracking with differentiably rendered world models. Learning models for control has led to impressive results in robotics and computer games, but this success has yet to be extended to vision-based navigation. To address this, we transfer advances in the emergent field of differentiable rendering to model-based control. We do this by planning in a learned 3D spatial world model, combined with a pose estimation algorithm previously used in the context of TSDF fusion, but now tailored to our setting and improved to incorporate agent dynamics. We evaluate over six simulated environments based on complex human-designed floor plans and provide quantitative results. We achieve up to 92% navigation success rate at a frequency of 15 Hz using only image and depth observations under stochastic, continuous dynamics.} }
Endnote
%0 Conference Paper %T Tracking and Planning with Spatial World Models %A Baris Kayalibay %A Atanas Mirchev %A Patrick van der Smagt %A Justin Bayer %B Proceedings of The 4th Annual Learning for Dynamics and Control Conference %C Proceedings of Machine Learning Research %D 2022 %E Roya Firoozi %E Negar Mehr %E Esen Yel %E Rika Antonova %E Jeannette Bohg %E Mac Schwager %E Mykel Kochenderfer %F pmlr-v168-kayalibay22a %I PMLR %P 124--137 %U https://proceedings.mlr.press/v168/kayalibay22a.html %V 168 %X We introduce a method for real-time navigation and tracking with differentiably rendered world models. Learning models for control has led to impressive results in robotics and computer games, but this success has yet to be extended to vision-based navigation. To address this, we transfer advances in the emergent field of differentiable rendering to model-based control. We do this by planning in a learned 3D spatial world model, combined with a pose estimation algorithm previously used in the context of TSDF fusion, but now tailored to our setting and improved to incorporate agent dynamics. We evaluate over six simulated environments based on complex human-designed floor plans and provide quantitative results. We achieve up to 92% navigation success rate at a frequency of 15 Hz using only image and depth observations under stochastic, continuous dynamics.
APA
Kayalibay, B., Mirchev, A., van der Smagt, P. & Bayer, J.. (2022). Tracking and Planning with Spatial World Models. Proceedings of The 4th Annual Learning for Dynamics and Control Conference, in Proceedings of Machine Learning Research 168:124-137 Available from https://proceedings.mlr.press/v168/kayalibay22a.html.

Related Material