GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency

Min-Seop Kwak, Jiuhn Song, Seungryong Kim
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:18023-18036, 2023.

Abstract

We present a novel framework to regularize Neural Radiance Field (NeRF) in a few-shot setting with a geometry-aware consistency regularization. The proposed approach leverages a rendered depth map at unobserved viewpoint to warp sparse input images to the unobserved viewpoint and impose them as pseudo ground truths to facilitate learning of NeRF. By encouraging such geometry-aware consistency at a feature-level instead of using pixel-level reconstruction loss, we regularize the NeRF at semantic and structural levels while allowing for modeling view dependent radiance to account for color variations across viewpoints. We also propose an effective method to filter out erroneous warped solutions, along with training strategies to stabilize training during optimization. We show that our model achieves competitive results compared to state-of-the-art few-shot NeRF models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-kwak23a, title = {{G}e{C}o{N}e{RF}: Few-shot Neural Radiance Fields via Geometric Consistency}, author = {Kwak, Min-Seop and Song, Jiuhn and Kim, Seungryong}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {18023--18036}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/kwak23a/kwak23a.pdf}, url = {https://proceedings.mlr.press/v202/kwak23a.html}, abstract = {We present a novel framework to regularize Neural Radiance Field (NeRF) in a few-shot setting with a geometry-aware consistency regularization. The proposed approach leverages a rendered depth map at unobserved viewpoint to warp sparse input images to the unobserved viewpoint and impose them as pseudo ground truths to facilitate learning of NeRF. By encouraging such geometry-aware consistency at a feature-level instead of using pixel-level reconstruction loss, we regularize the NeRF at semantic and structural levels while allowing for modeling view dependent radiance to account for color variations across viewpoints. We also propose an effective method to filter out erroneous warped solutions, along with training strategies to stabilize training during optimization. We show that our model achieves competitive results compared to state-of-the-art few-shot NeRF models.} }
Endnote
%0 Conference Paper %T GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency %A Min-Seop Kwak %A Jiuhn Song %A Seungryong Kim %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-kwak23a %I PMLR %P 18023--18036 %U https://proceedings.mlr.press/v202/kwak23a.html %V 202 %X We present a novel framework to regularize Neural Radiance Field (NeRF) in a few-shot setting with a geometry-aware consistency regularization. The proposed approach leverages a rendered depth map at unobserved viewpoint to warp sparse input images to the unobserved viewpoint and impose them as pseudo ground truths to facilitate learning of NeRF. By encouraging such geometry-aware consistency at a feature-level instead of using pixel-level reconstruction loss, we regularize the NeRF at semantic and structural levels while allowing for modeling view dependent radiance to account for color variations across viewpoints. We also propose an effective method to filter out erroneous warped solutions, along with training strategies to stabilize training during optimization. We show that our model achieves competitive results compared to state-of-the-art few-shot NeRF models.
APA
Kwak, M., Song, J. & Kim, S.. (2023). GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:18023-18036 Available from https://proceedings.mlr.press/v202/kwak23a.html.

Related Material