Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision

Alberta Longhini, Marcel Büsching, Bardienus Pieter Duisterhof, Jens Lundell, Jeffrey Ichnowski, Mårten Björkman, Danica Kragic
Proceedings of The 8th Conference on Robot Learning, PMLR 270:2845-2865, 2025.

Abstract

We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth’s state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by ∼85 %.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-longhini25a, title = {Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision}, author = {Longhini, Alberta and B{\"{u}}sching, Marcel and Duisterhof, Bardienus Pieter and Lundell, Jens and Ichnowski, Jeffrey and Bj{\"{o}}rkman, M\aa{}rten and Kragic, Danica}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {2845--2865}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/longhini25a/longhini25a.pdf}, url = {https://proceedings.mlr.press/v270/longhini25a.html}, abstract = {We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth’s state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by $\sim 85$ %.} }
Endnote
%0 Conference Paper %T Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision %A Alberta Longhini %A Marcel Büsching %A Bardienus Pieter Duisterhof %A Jens Lundell %A Jeffrey Ichnowski %A Mårten Björkman %A Danica Kragic %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-longhini25a %I PMLR %P 2845--2865 %U https://proceedings.mlr.press/v270/longhini25a.html %V 270 %X We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth’s state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by $\sim 85$ %.
APA
Longhini, A., Büsching, M., Duisterhof, B.P., Lundell, J., Ichnowski, J., Björkman, M. & Kragic, D.. (2025). Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:2845-2865 Available from https://proceedings.mlr.press/v270/longhini25a.html.

Related Material