Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics

Jad Abou-Chakra, Krishan Rana, Feras Dayoub, Niko Suenderhauf
Proceedings of The 8th Conference on Robot Learning, PMLR 270:513-530, 2025.

Abstract

For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation – modelling geometry, physics, and visual observations – that informs perception, planning, and control algorithms. We propose a novel dual “Gaussian-Particle” representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates “visual forces” that correct the particle positions while respecting known physical constraints. By integrating predictive physical modeling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-abou-chakra25a, title = {Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics}, author = {Abou-Chakra, Jad and Rana, Krishan and Dayoub, Feras and Suenderhauf, Niko}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {513--530}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/abou-chakra25a/abou-chakra25a.pdf}, url = {https://proceedings.mlr.press/v270/abou-chakra25a.html}, abstract = {For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation – modelling geometry, physics, and visual observations – that informs perception, planning, and control algorithms. We propose a novel dual “Gaussian-Particle” representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates “visual forces” that correct the particle positions while respecting known physical constraints. By integrating predictive physical modeling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/} }
Endnote
%0 Conference Paper %T Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics %A Jad Abou-Chakra %A Krishan Rana %A Feras Dayoub %A Niko Suenderhauf %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-abou-chakra25a %I PMLR %P 513--530 %U https://proceedings.mlr.press/v270/abou-chakra25a.html %V 270 %X For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation – modelling geometry, physics, and visual observations – that informs perception, planning, and control algorithms. We propose a novel dual “Gaussian-Particle” representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates “visual forces” that correct the particle positions while respecting known physical constraints. By integrating predictive physical modeling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/
APA
Abou-Chakra, J., Rana, K., Dayoub, F. & Suenderhauf, N.. (2025). Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:513-530 Available from https://proceedings.mlr.press/v270/abou-chakra25a.html.

Related Material