Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation

Yajvan Ravan, Adam Rashid, Alan Yu, Kai McClennen, Gio Huh, Kevin Yang, Zhutian Yang, Qinxi Yu, Xiaolong Wang, Phillip Isola, Ge Yang
Proceedings of The 9th Conference on Robot Learning, PMLR 305:5151-5169, 2025.

Abstract

We introduce Lucid-XR, a generative data engine for creating diverse and realistic-looking data to train real-world robot systems. At the core of Lucid-XR is vuer, a web-based physics simulation environment that runs directly on the XR headset, enabling internet-scale access to immersive, latency-free virtual interactions without requiring specialized equipment. The complete system integrates on-device physics simulation with on-device human-to-robot pose retargeting, that are further amplified by a physics-guided video generation pipeline commandable with natural language specifications. We demonstrate zero-shot sim-to-real transfer of robot visual policies, trained entirely on Lucid-XR’s synthetic data, across bimanual and dexterous manipulation tasks that involve flexible materials, adhesive interaction between particles, and rigid body contact.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-ravan25a, title = {Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation}, author = {Ravan, Yajvan and Rashid, Adam and Yu, Alan and McClennen, Kai and Huh, Gio and Yang, Kevin and Yang, Zhutian and Yu, Qinxi and Wang, Xiaolong and Isola, Phillip and Yang, Ge}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {5151--5169}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/ravan25a/ravan25a.pdf}, url = {https://proceedings.mlr.press/v305/ravan25a.html}, abstract = {We introduce Lucid-XR, a generative data engine for creating diverse and realistic-looking data to train real-world robot systems. At the core of Lucid-XR is vuer, a web-based physics simulation environment that runs directly on the XR headset, enabling internet-scale access to immersive, latency-free virtual interactions without requiring specialized equipment. The complete system integrates on-device physics simulation with on-device human-to-robot pose retargeting, that are further amplified by a physics-guided video generation pipeline commandable with natural language specifications. We demonstrate zero-shot sim-to-real transfer of robot visual policies, trained entirely on Lucid-XR’s synthetic data, across bimanual and dexterous manipulation tasks that involve flexible materials, adhesive interaction between particles, and rigid body contact.} }
Endnote
%0 Conference Paper %T Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation %A Yajvan Ravan %A Adam Rashid %A Alan Yu %A Kai McClennen %A Gio Huh %A Kevin Yang %A Zhutian Yang %A Qinxi Yu %A Xiaolong Wang %A Phillip Isola %A Ge Yang %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-ravan25a %I PMLR %P 5151--5169 %U https://proceedings.mlr.press/v305/ravan25a.html %V 305 %X We introduce Lucid-XR, a generative data engine for creating diverse and realistic-looking data to train real-world robot systems. At the core of Lucid-XR is vuer, a web-based physics simulation environment that runs directly on the XR headset, enabling internet-scale access to immersive, latency-free virtual interactions without requiring specialized equipment. The complete system integrates on-device physics simulation with on-device human-to-robot pose retargeting, that are further amplified by a physics-guided video generation pipeline commandable with natural language specifications. We demonstrate zero-shot sim-to-real transfer of robot visual policies, trained entirely on Lucid-XR’s synthetic data, across bimanual and dexterous manipulation tasks that involve flexible materials, adhesive interaction between particles, and rigid body contact.
APA
Ravan, Y., Rashid, A., Yu, A., McClennen, K., Huh, G., Yang, K., Yang, Z., Yu, Q., Wang, X., Isola, P. & Yang, G.. (2025). Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:5151-5169 Available from https://proceedings.mlr.press/v305/ravan25a.html.

Related Material