Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients

Oliver Scheel, Luca Bergamini, Maciej Wolczyk, Błażej Osiński, Peter Ondruska
Proceedings of the 5th Conference on Robot Learning, PMLR 164:718-728, 2022.

Abstract

In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method on 100 hours of expert demonstrations on urban roads and show that it learns complex driving policies that generalize well and can perform a variety of driving maneuvers. We demonstrate this in simulation as well as deploy our model to self-driving vehicles in the real-world. Our method outperforms previously demonstrated state-of-the-art for urban driving scenarios - all this without the need for complex state perturbations or collecting additional on-policy data during training. We make code and data publicly available.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-scheel22a, title = {Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients}, author = {Scheel, Oliver and Bergamini, Luca and Wolczyk, Maciej and Osi\'nski, B\l{a}\.{z}ej and Ondruska, Peter}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {718--728}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/scheel22a/scheel22a.pdf}, url = {https://proceedings.mlr.press/v164/scheel22a.html}, abstract = {In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method on 100 hours of expert demonstrations on urban roads and show that it learns complex driving policies that generalize well and can perform a variety of driving maneuvers. We demonstrate this in simulation as well as deploy our model to self-driving vehicles in the real-world. Our method outperforms previously demonstrated state-of-the-art for urban driving scenarios - all this without the need for complex state perturbations or collecting additional on-policy data during training. We make code and data publicly available.} }
Endnote
%0 Conference Paper %T Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients %A Oliver Scheel %A Luca Bergamini %A Maciej Wolczyk %A Błażej Osiński %A Peter Ondruska %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-scheel22a %I PMLR %P 718--728 %U https://proceedings.mlr.press/v164/scheel22a.html %V 164 %X In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method on 100 hours of expert demonstrations on urban roads and show that it learns complex driving policies that generalize well and can perform a variety of driving maneuvers. We demonstrate this in simulation as well as deploy our model to self-driving vehicles in the real-world. Our method outperforms previously demonstrated state-of-the-art for urban driving scenarios - all this without the need for complex state perturbations or collecting additional on-policy data during training. We make code and data publicly available.
APA
Scheel, O., Bergamini, L., Wolczyk, M., Osiński, B. & Ondruska, P.. (2022). Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:718-728 Available from https://proceedings.mlr.press/v164/scheel22a.html.

Related Material