Robust Camera Pose Refinement for Multi-Resolution Hash Encoding

Hwan Heo, Taekyung Kim, Jiyoung Lee, Jaewon Lee, Soohyun Kim, Hyunwoo J. Kim, Jin-Hwa Kim
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:13000-13016, 2023.

Abstract

Multi-resolution hash encoding has recently been proposed to reduce the computational cost of neural renderings, such as NeRF. This method requires accurate camera poses for the neural renderings of given scenes. However, contrary to previous methods jointly optimizing camera poses and 3D scenes, the naive gradient-based camera pose refinement method using multi-resolution hash encoding severely deteriorates performance. We propose a joint optimization algorithm to calibrate the camera pose and learn a geometric representation using efficient multi-resolution hash encoding. Showing that the oscillating gradient flows of hash encoding interfere with the registration of camera poses, our method addresses the issue by utilizing smooth interpolation weighting to stabilize the gradient oscillation for the ray samplings across hash grids. Moreover, the curriculum training procedure helps to learn the level-wise hash encoding, further increasing the pose refinement. Experiments on the novel-view synthesis datasets validate that our learning frameworks achieve state-of-the-art performance and rapid convergence of neural rendering.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-heo23a, title = {Robust Camera Pose Refinement for Multi-Resolution Hash Encoding}, author = {Heo, Hwan and Kim, Taekyung and Lee, Jiyoung and Lee, Jaewon and Kim, Soohyun and Kim, Hyunwoo J. and Kim, Jin-Hwa}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {13000--13016}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/heo23a/heo23a.pdf}, url = {https://proceedings.mlr.press/v202/heo23a.html}, abstract = {Multi-resolution hash encoding has recently been proposed to reduce the computational cost of neural renderings, such as NeRF. This method requires accurate camera poses for the neural renderings of given scenes. However, contrary to previous methods jointly optimizing camera poses and 3D scenes, the naive gradient-based camera pose refinement method using multi-resolution hash encoding severely deteriorates performance. We propose a joint optimization algorithm to calibrate the camera pose and learn a geometric representation using efficient multi-resolution hash encoding. Showing that the oscillating gradient flows of hash encoding interfere with the registration of camera poses, our method addresses the issue by utilizing smooth interpolation weighting to stabilize the gradient oscillation for the ray samplings across hash grids. Moreover, the curriculum training procedure helps to learn the level-wise hash encoding, further increasing the pose refinement. Experiments on the novel-view synthesis datasets validate that our learning frameworks achieve state-of-the-art performance and rapid convergence of neural rendering.} }
Endnote
%0 Conference Paper %T Robust Camera Pose Refinement for Multi-Resolution Hash Encoding %A Hwan Heo %A Taekyung Kim %A Jiyoung Lee %A Jaewon Lee %A Soohyun Kim %A Hyunwoo J. Kim %A Jin-Hwa Kim %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-heo23a %I PMLR %P 13000--13016 %U https://proceedings.mlr.press/v202/heo23a.html %V 202 %X Multi-resolution hash encoding has recently been proposed to reduce the computational cost of neural renderings, such as NeRF. This method requires accurate camera poses for the neural renderings of given scenes. However, contrary to previous methods jointly optimizing camera poses and 3D scenes, the naive gradient-based camera pose refinement method using multi-resolution hash encoding severely deteriorates performance. We propose a joint optimization algorithm to calibrate the camera pose and learn a geometric representation using efficient multi-resolution hash encoding. Showing that the oscillating gradient flows of hash encoding interfere with the registration of camera poses, our method addresses the issue by utilizing smooth interpolation weighting to stabilize the gradient oscillation for the ray samplings across hash grids. Moreover, the curriculum training procedure helps to learn the level-wise hash encoding, further increasing the pose refinement. Experiments on the novel-view synthesis datasets validate that our learning frameworks achieve state-of-the-art performance and rapid convergence of neural rendering.
APA
Heo, H., Kim, T., Lee, J., Lee, J., Kim, S., Kim, H.J. & Kim, J.. (2023). Robust Camera Pose Refinement for Multi-Resolution Hash Encoding. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:13000-13016 Available from https://proceedings.mlr.press/v202/heo23a.html.

Related Material