Automated Creation of Digital Cousins for Robust Policy Learning

Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, Li Fei-Fei
Proceedings of The 8th Conference on Robot Learning, PMLR 270:4912-4943, 2025.

Abstract

Training robot policies in the real world can be unsafe, costly, and difficult to scale. Simulation serves as an inexpensive and potentially limitless source of training data, but suffers from the semantics and physics disparity between simulated and real-world environments. These discrepancies can be minimized by training in *digital twins*, which serve as virtual replicas of a real scene but are expensive to generate and cannot produce cross-domain generalization. To address these limitations, we propose the concept of ***digital cousins***, a virtual asset or scene that, unlike a *digital twin*, does not explicitly model a real-world counterpart but still exhibits similar geometric and semantic affordances. As a result, *digital cousins* simultaneously reduce the cost of generating an analogous virtual environment while also facilitating better robustness during sim-to-real domain transfer by providing a distribution of similar training scenes. Leveraging digital cousins, we introduce a novel method for their automated creation, and propose a fully automated real-to-sim-to-real pipeline for generating fully interactive scenes and training robot policies that can be deployed zero-shot in the original scene. We find that digital cousin scenes that preserve geometric and semantic affordances can be produced automatically, and can be used to train policies that outperform policies trained on digital twins, achieving 90% vs. 25% success rates under zero-shot sim-to-real transfer. Additional details are available at https://digital-cousins.github.io/.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-dai25a, title = {Automated Creation of Digital Cousins for Robust Policy Learning}, author = {Dai, Tianyuan and Wong, Josiah and Jiang, Yunfan and Wang, Chen and Gokmen, Cem and Zhang, Ruohan and Wu, Jiajun and Fei-Fei, Li}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {4912--4943}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/dai25a/dai25a.pdf}, url = {https://proceedings.mlr.press/v270/dai25a.html}, abstract = {Training robot policies in the real world can be unsafe, costly, and difficult to scale. Simulation serves as an inexpensive and potentially limitless source of training data, but suffers from the semantics and physics disparity between simulated and real-world environments. These discrepancies can be minimized by training in *digital twins*, which serve as virtual replicas of a real scene but are expensive to generate and cannot produce cross-domain generalization. To address these limitations, we propose the concept of ***digital cousins***, a virtual asset or scene that, unlike a *digital twin*, does not explicitly model a real-world counterpart but still exhibits similar geometric and semantic affordances. As a result, *digital cousins* simultaneously reduce the cost of generating an analogous virtual environment while also facilitating better robustness during sim-to-real domain transfer by providing a distribution of similar training scenes. Leveraging digital cousins, we introduce a novel method for their automated creation, and propose a fully automated real-to-sim-to-real pipeline for generating fully interactive scenes and training robot policies that can be deployed zero-shot in the original scene. We find that digital cousin scenes that preserve geometric and semantic affordances can be produced automatically, and can be used to train policies that outperform policies trained on digital twins, achieving 90% vs. 25% success rates under zero-shot sim-to-real transfer. Additional details are available at https://digital-cousins.github.io/.} }
Endnote
%0 Conference Paper %T Automated Creation of Digital Cousins for Robust Policy Learning %A Tianyuan Dai %A Josiah Wong %A Yunfan Jiang %A Chen Wang %A Cem Gokmen %A Ruohan Zhang %A Jiajun Wu %A Li Fei-Fei %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-dai25a %I PMLR %P 4912--4943 %U https://proceedings.mlr.press/v270/dai25a.html %V 270 %X Training robot policies in the real world can be unsafe, costly, and difficult to scale. Simulation serves as an inexpensive and potentially limitless source of training data, but suffers from the semantics and physics disparity between simulated and real-world environments. These discrepancies can be minimized by training in *digital twins*, which serve as virtual replicas of a real scene but are expensive to generate and cannot produce cross-domain generalization. To address these limitations, we propose the concept of ***digital cousins***, a virtual asset or scene that, unlike a *digital twin*, does not explicitly model a real-world counterpart but still exhibits similar geometric and semantic affordances. As a result, *digital cousins* simultaneously reduce the cost of generating an analogous virtual environment while also facilitating better robustness during sim-to-real domain transfer by providing a distribution of similar training scenes. Leveraging digital cousins, we introduce a novel method for their automated creation, and propose a fully automated real-to-sim-to-real pipeline for generating fully interactive scenes and training robot policies that can be deployed zero-shot in the original scene. We find that digital cousin scenes that preserve geometric and semantic affordances can be produced automatically, and can be used to train policies that outperform policies trained on digital twins, achieving 90% vs. 25% success rates under zero-shot sim-to-real transfer. Additional details are available at https://digital-cousins.github.io/.
APA
Dai, T., Wong, J., Jiang, Y., Wang, C., Gokmen, C., Zhang, R., Wu, J. & Fei-Fei, L.. (2025). Automated Creation of Digital Cousins for Robust Policy Learning. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:4912-4943 Available from https://proceedings.mlr.press/v270/dai25a.html.

Related Material