PolyGen: An Autoregressive Generative Model of 3D Meshes

Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, Peter Battaglia
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7220-7229, 2020.

Abstract

Polygon meshes are an efficient representation of 3D geometry, and are of central importance in computer graphics, robotics and games development. Existing learning-based approaches for object synthesis have avoided the challenges of working with 3D meshes, instead using alternative object representations that are more compatible with neural architectures and training approaches. We present PolyGen, a generative model of 3D objects which models the mesh directly, predicting vertices and faces sequentially using a Transformer-based architecture. Our model can condition on a range of inputs, including object classes, voxels, and images, and because the model is probabilistic it can produce samples that capture uncertainty in ambiguous scenarios. We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task. We also evaluate the conditional models on surface reconstruction metrics against alternative methods, and demonstrate competitive performance despite not training directly on this task.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-nash20a, title = {{P}oly{G}en: An Autoregressive Generative Model of 3{D} Meshes}, author = {Nash, Charlie and Ganin, Yaroslav and Eslami, S. M. Ali and Battaglia, Peter}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {7220--7229}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/nash20a/nash20a.pdf}, url = {https://proceedings.mlr.press/v119/nash20a.html}, abstract = {Polygon meshes are an efficient representation of 3D geometry, and are of central importance in computer graphics, robotics and games development. Existing learning-based approaches for object synthesis have avoided the challenges of working with 3D meshes, instead using alternative object representations that are more compatible with neural architectures and training approaches. We present PolyGen, a generative model of 3D objects which models the mesh directly, predicting vertices and faces sequentially using a Transformer-based architecture. Our model can condition on a range of inputs, including object classes, voxels, and images, and because the model is probabilistic it can produce samples that capture uncertainty in ambiguous scenarios. We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task. We also evaluate the conditional models on surface reconstruction metrics against alternative methods, and demonstrate competitive performance despite not training directly on this task.} }
Endnote
%0 Conference Paper %T PolyGen: An Autoregressive Generative Model of 3D Meshes %A Charlie Nash %A Yaroslav Ganin %A S. M. Ali Eslami %A Peter Battaglia %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-nash20a %I PMLR %P 7220--7229 %U https://proceedings.mlr.press/v119/nash20a.html %V 119 %X Polygon meshes are an efficient representation of 3D geometry, and are of central importance in computer graphics, robotics and games development. Existing learning-based approaches for object synthesis have avoided the challenges of working with 3D meshes, instead using alternative object representations that are more compatible with neural architectures and training approaches. We present PolyGen, a generative model of 3D objects which models the mesh directly, predicting vertices and faces sequentially using a Transformer-based architecture. Our model can condition on a range of inputs, including object classes, voxels, and images, and because the model is probabilistic it can produce samples that capture uncertainty in ambiguous scenarios. We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task. We also evaluate the conditional models on surface reconstruction metrics against alternative methods, and demonstrate competitive performance despite not training directly on this task.
APA
Nash, C., Ganin, Y., Eslami, S.M.A. & Battaglia, P.. (2020). PolyGen: An Autoregressive Generative Model of 3D Meshes. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:7220-7229 Available from https://proceedings.mlr.press/v119/nash20a.html.

Related Material