Learning Visual Parkour from Generated Images

Alan Yu, Ge Yang, Ran Choi, Yajvan Ravan, John Leonard, Phillip Isola
Proceedings of The 8th Conference on Robot Learning, PMLR 270:2500-2516, 2025.

Abstract

Fast and accurate physics simulation is an essential component of robot learning, where robots can explore failure scenarios that are difficult to produce in the real world and learn from unlimited on-policy data. Yet, it remains challenging to incorporate RGB-color perception into the sim-to-real pipeline that matches the real world in its richness and realism. In this work, we train a robot dog in simulation for visual parkour. We propose a way to use generative models to synthesize diverse and physically accurate image sequences of the scene from the robot’s ego-centric perspective. We present demonstrations of zero-shot transfer to the RGB-only observations of the real world on a robot equipped with a low-cost, off-the-shelf color camera.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-yu25b, title = {Learning Visual Parkour from Generated Images}, author = {Yu, Alan and Yang, Ge and Choi, Ran and Ravan, Yajvan and Leonard, John and Isola, Phillip}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {2500--2516}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/yu25b/yu25b.pdf}, url = {https://proceedings.mlr.press/v270/yu25b.html}, abstract = {Fast and accurate physics simulation is an essential component of robot learning, where robots can explore failure scenarios that are difficult to produce in the real world and learn from unlimited on-policy data. Yet, it remains challenging to incorporate RGB-color perception into the sim-to-real pipeline that matches the real world in its richness and realism. In this work, we train a robot dog in simulation for visual parkour. We propose a way to use generative models to synthesize diverse and physically accurate image sequences of the scene from the robot’s ego-centric perspective. We present demonstrations of zero-shot transfer to the RGB-only observations of the real world on a robot equipped with a low-cost, off-the-shelf color camera.} }
Endnote
%0 Conference Paper %T Learning Visual Parkour from Generated Images %A Alan Yu %A Ge Yang %A Ran Choi %A Yajvan Ravan %A John Leonard %A Phillip Isola %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-yu25b %I PMLR %P 2500--2516 %U https://proceedings.mlr.press/v270/yu25b.html %V 270 %X Fast and accurate physics simulation is an essential component of robot learning, where robots can explore failure scenarios that are difficult to produce in the real world and learn from unlimited on-policy data. Yet, it remains challenging to incorporate RGB-color perception into the sim-to-real pipeline that matches the real world in its richness and realism. In this work, we train a robot dog in simulation for visual parkour. We propose a way to use generative models to synthesize diverse and physically accurate image sequences of the scene from the robot’s ego-centric perspective. We present demonstrations of zero-shot transfer to the RGB-only observations of the real world on a robot equipped with a low-cost, off-the-shelf color camera.
APA
Yu, A., Yang, G., Choi, R., Ravan, Y., Leonard, J. & Isola, P.. (2025). Learning Visual Parkour from Generated Images. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:2500-2516 Available from https://proceedings.mlr.press/v270/yu25b.html.

Related Material