Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data

John So, Amber Xie, Sunggoo Jung, Jeffrey Edlund, Rohan Thakker, Ali-akbar Agha-mohammadi, Pieter Abbeel, Stephen James
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1871-1881, 2023.

Abstract

Autonomous driving is complex, requiring sophisticated 3D scene understanding, localization, mapping, and control. Rather than explicitly modelling and fusing each of these components, we instead consider an end-to-end approach via reinforcement learning (RL). However, collecting exploration driving data in the real world is impractical and dangerous. While training in simulation and deploying visual sim-to-real techniques has worked well for robot manipulation, deploying beyond controlled workspace viewpoints remains a challenge. In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps, subsequently enabling real-world images to also be translated. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world. Our approach, which can be trained in 48 hours on 1 GPU, can perform equally as well as a classical perception and control stack that took thousands of engineering hours over several months to build. We hope this work motivates future end-to-end autonomous driving research.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-so23a, title = {Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data}, author = {So, John and Xie, Amber and Jung, Sunggoo and Edlund, Jeffrey and Thakker, Rohan and Agha-mohammadi, Ali-akbar and Abbeel, Pieter and James, Stephen}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1871--1881}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/so23a/so23a.pdf}, url = {https://proceedings.mlr.press/v205/so23a.html}, abstract = {Autonomous driving is complex, requiring sophisticated 3D scene understanding, localization, mapping, and control. Rather than explicitly modelling and fusing each of these components, we instead consider an end-to-end approach via reinforcement learning (RL). However, collecting exploration driving data in the real world is impractical and dangerous. While training in simulation and deploying visual sim-to-real techniques has worked well for robot manipulation, deploying beyond controlled workspace viewpoints remains a challenge. In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps, subsequently enabling real-world images to also be translated. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world. Our approach, which can be trained in 48 hours on 1 GPU, can perform equally as well as a classical perception and control stack that took thousands of engineering hours over several months to build. We hope this work motivates future end-to-end autonomous driving research.} }
Endnote
%0 Conference Paper %T Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data %A John So %A Amber Xie %A Sunggoo Jung %A Jeffrey Edlund %A Rohan Thakker %A Ali-akbar Agha-mohammadi %A Pieter Abbeel %A Stephen James %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-so23a %I PMLR %P 1871--1881 %U https://proceedings.mlr.press/v205/so23a.html %V 205 %X Autonomous driving is complex, requiring sophisticated 3D scene understanding, localization, mapping, and control. Rather than explicitly modelling and fusing each of these components, we instead consider an end-to-end approach via reinforcement learning (RL). However, collecting exploration driving data in the real world is impractical and dangerous. While training in simulation and deploying visual sim-to-real techniques has worked well for robot manipulation, deploying beyond controlled workspace viewpoints remains a challenge. In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps, subsequently enabling real-world images to also be translated. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world. Our approach, which can be trained in 48 hours on 1 GPU, can perform equally as well as a classical perception and control stack that took thousands of engineering hours over several months to build. We hope this work motivates future end-to-end autonomous driving research.
APA
So, J., Xie, A., Jung, S., Edlund, J., Thakker, R., Agha-mohammadi, A., Abbeel, P. & James, S.. (2023). Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1871-1881 Available from https://proceedings.mlr.press/v205/so23a.html.

Related Material