Equivariant Neural Rendering

Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, Qi Shan
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2761-2770, 2020.

Abstract

We propose a framework for learning neural scene representations directly from images, without 3D supervision. Our key insight is that 3D structure can be imposed by ensuring that the learned representation transforms like a real 3D scene. Specifically, we introduce a loss which enforces equivariance of the scene representation with respect to 3D transformations. Our formulation allows us to infer and render scenes in real time while achieving comparable results to models requiring minutes for inference. In addition, we introduce two challenging new datasets for scene representation and neural rendering, including scenes with complex lighting and backgrounds. Through experiments, we show that our model achieves compelling results on these datasets as well as on standard ShapeNet benchmarks.

Cite this Paper


BibTeX
@InProceedings{pmlr-v119-dupont20a, title = {Equivariant Neural Rendering}, author = {Dupont, Emilien and Martin, Miguel Bautista and Colburn, Alex and Sankar, Aditya and Susskind, Josh and Shan, Qi}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {2761--2770}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/dupont20a/dupont20a.pdf}, url = {https://proceedings.mlr.press/v119/dupont20a.html}, abstract = {We propose a framework for learning neural scene representations directly from images, without 3D supervision. Our key insight is that 3D structure can be imposed by ensuring that the learned representation transforms like a real 3D scene. Specifically, we introduce a loss which enforces equivariance of the scene representation with respect to 3D transformations. Our formulation allows us to infer and render scenes in real time while achieving comparable results to models requiring minutes for inference. In addition, we introduce two challenging new datasets for scene representation and neural rendering, including scenes with complex lighting and backgrounds. Through experiments, we show that our model achieves compelling results on these datasets as well as on standard ShapeNet benchmarks.} }
Endnote
%0 Conference Paper %T Equivariant Neural Rendering %A Emilien Dupont %A Miguel Bautista Martin %A Alex Colburn %A Aditya Sankar %A Josh Susskind %A Qi Shan %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-dupont20a %I PMLR %P 2761--2770 %U https://proceedings.mlr.press/v119/dupont20a.html %V 119 %X We propose a framework for learning neural scene representations directly from images, without 3D supervision. Our key insight is that 3D structure can be imposed by ensuring that the learned representation transforms like a real 3D scene. Specifically, we introduce a loss which enforces equivariance of the scene representation with respect to 3D transformations. Our formulation allows us to infer and render scenes in real time while achieving comparable results to models requiring minutes for inference. In addition, we introduce two challenging new datasets for scene representation and neural rendering, including scenes with complex lighting and backgrounds. Through experiments, we show that our model achieves compelling results on these datasets as well as on standard ShapeNet benchmarks.
APA
Dupont, E., Martin, M.B., Colburn, A., Sankar, A., Susskind, J. & Shan, Q.. (2020). Equivariant Neural Rendering. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:2761-2770 Available from https://proceedings.mlr.press/v119/dupont20a.html.

Related Material