FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing

Kyle Stachowicz, Dhruv Shah, Arjun Bhorkar, Ilya Kostrikov, Sergey Levine
Proceedings of The 7th Conference on Robot Learning, PMLR 229:3100-3111, 2023.

Abstract

We present a system that enables an autonomous small-scale RC car to drive aggressively from visual observations using reinforcement learning (RL). Our system, FastRLAP, trains autonomously in the real world, without human interventions, and without requiring any simulation or expert demonstrations. Our system integrates a number of important components to make this possible: we initialize the representations for the RL policy and value function from a large prior dataset of other robots navigating in other environments (at low speed), which provides a navigation-relevant representation. From here, a sample-efficient online RL method uses a single low-speed user-provided demonstration to determine the desired driving course, extracts a set of navigational checkpoints, and autonomously practices driving through these checkpoints, resetting automatically on collision or failure. Perhaps surprisingly, we find that with appropriate initialization and choice of algorithm, our system can learn to drive over a variety of racing courses with less than 20 minutes of online training. The resulting policies exhibit emergent aggressive driving skills, such as timing braking and acceleration around turns and avoiding areas which impede the robot’s motion, approaching the performance of a human driver using a similar first-person interface over the course of training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-stachowicz23a, title = {FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing}, author = {Stachowicz, Kyle and Shah, Dhruv and Bhorkar, Arjun and Kostrikov, Ilya and Levine, Sergey}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {3100--3111}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/stachowicz23a/stachowicz23a.pdf}, url = {https://proceedings.mlr.press/v229/stachowicz23a.html}, abstract = {We present a system that enables an autonomous small-scale RC car to drive aggressively from visual observations using reinforcement learning (RL). Our system, FastRLAP, trains autonomously in the real world, without human interventions, and without requiring any simulation or expert demonstrations. Our system integrates a number of important components to make this possible: we initialize the representations for the RL policy and value function from a large prior dataset of other robots navigating in other environments (at low speed), which provides a navigation-relevant representation. From here, a sample-efficient online RL method uses a single low-speed user-provided demonstration to determine the desired driving course, extracts a set of navigational checkpoints, and autonomously practices driving through these checkpoints, resetting automatically on collision or failure. Perhaps surprisingly, we find that with appropriate initialization and choice of algorithm, our system can learn to drive over a variety of racing courses with less than 20 minutes of online training. The resulting policies exhibit emergent aggressive driving skills, such as timing braking and acceleration around turns and avoiding areas which impede the robot’s motion, approaching the performance of a human driver using a similar first-person interface over the course of training.} }
Endnote
%0 Conference Paper %T FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing %A Kyle Stachowicz %A Dhruv Shah %A Arjun Bhorkar %A Ilya Kostrikov %A Sergey Levine %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-stachowicz23a %I PMLR %P 3100--3111 %U https://proceedings.mlr.press/v229/stachowicz23a.html %V 229 %X We present a system that enables an autonomous small-scale RC car to drive aggressively from visual observations using reinforcement learning (RL). Our system, FastRLAP, trains autonomously in the real world, without human interventions, and without requiring any simulation or expert demonstrations. Our system integrates a number of important components to make this possible: we initialize the representations for the RL policy and value function from a large prior dataset of other robots navigating in other environments (at low speed), which provides a navigation-relevant representation. From here, a sample-efficient online RL method uses a single low-speed user-provided demonstration to determine the desired driving course, extracts a set of navigational checkpoints, and autonomously practices driving through these checkpoints, resetting automatically on collision or failure. Perhaps surprisingly, we find that with appropriate initialization and choice of algorithm, our system can learn to drive over a variety of racing courses with less than 20 minutes of online training. The resulting policies exhibit emergent aggressive driving skills, such as timing braking and acceleration around turns and avoiding areas which impede the robot’s motion, approaching the performance of a human driver using a similar first-person interface over the course of training.
APA
Stachowicz, K., Shah, D., Bhorkar, A., Kostrikov, I. & Levine, S.. (2023). FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:3100-3111 Available from https://proceedings.mlr.press/v229/stachowicz23a.html.

Related Material