LENS: Localization enhanced by NeRF synthesis

Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, Arnaud de La Fortelle
Proceedings of the 5th Conference on Robot Learning, PMLR 164:1347-1356, 2022.

Abstract

Neural Radiance Fields (NeRF) have recently demonstrated photorealistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrelevant places we selected virtual camera locations from NeRF internal representation of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training. At the time of publication, our approach improved state of the art with a 60% lower error on Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy becomes comparable to structure-based methods, without any architecture modification or domain adaptation constraints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribution of data used for training on public benchmarks. We concluded that pose regression accuracy is mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-moreau22a, title = {LENS: Localization enhanced by NeRF synthesis}, author = {Moreau, Arthur and Piasco, Nathan and Tsishkou, Dzmitry and Stanciulescu, Bogdan and Fortelle, Arnaud de La}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {1347--1356}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/moreau22a/moreau22a.pdf}, url = {https://proceedings.mlr.press/v164/moreau22a.html}, abstract = {Neural Radiance Fields (NeRF) have recently demonstrated photorealistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrelevant places we selected virtual camera locations from NeRF internal representation of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training. At the time of publication, our approach improved state of the art with a 60% lower error on Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy becomes comparable to structure-based methods, without any architecture modification or domain adaptation constraints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribution of data used for training on public benchmarks. We concluded that pose regression accuracy is mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.} }
Endnote
%0 Conference Paper %T LENS: Localization enhanced by NeRF synthesis %A Arthur Moreau %A Nathan Piasco %A Dzmitry Tsishkou %A Bogdan Stanciulescu %A Arnaud de La Fortelle %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-moreau22a %I PMLR %P 1347--1356 %U https://proceedings.mlr.press/v164/moreau22a.html %V 164 %X Neural Radiance Fields (NeRF) have recently demonstrated photorealistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrelevant places we selected virtual camera locations from NeRF internal representation of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training. At the time of publication, our approach improved state of the art with a 60% lower error on Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy becomes comparable to structure-based methods, without any architecture modification or domain adaptation constraints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribution of data used for training on public benchmarks. We concluded that pose regression accuracy is mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.
APA
Moreau, A., Piasco, N., Tsishkou, D., Stanciulescu, B. & Fortelle, A.d.L.. (2022). LENS: Localization enhanced by NeRF synthesis. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:1347-1356 Available from https://proceedings.mlr.press/v164/moreau22a.html.

Related Material