Action-based Representation Learning for Autonomous Driving

Yi Xiao, Felipe Codevilla, Christopher Pal, Antonio Lopez
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:232-246, 2021.

Abstract

Human drivers produce a vast amount of data which could, in principle, be used to improve autonomous driving systems. Unfortunately, seemingly straightforward approaches for creating end-to-end driving models that map sensor data directly into driving actions are problematic in terms of interpretability, and typically have significant difficulty dealing with spurious correlations. Alternatively, we propose to use this kind of action-based driving data for learning representations. Our experiments show that an affordance-based driving model pre-trained with this approach can leverage a relatively small amount of weakly annotated imagery and outperform pure end-to-end driving models, while being more interpretable. Further, we demonstrate how this strategy outperforms previous methods based on learning inverse dynamics models as well as other methods based on heavy human supervision (ImageNet).

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-xiao21a, title = {Action-based Representation Learning for Autonomous Driving}, author = {Xiao, Yi and Codevilla, Felipe and Pal, Christopher and Lopez, Antonio}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {232--246}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/xiao21a/xiao21a.pdf}, url = {https://proceedings.mlr.press/v155/xiao21a.html}, abstract = {Human drivers produce a vast amount of data which could, in principle, be used to improve autonomous driving systems. Unfortunately, seemingly straightforward approaches for creating end-to-end driving models that map sensor data directly into driving actions are problematic in terms of interpretability, and typically have significant difficulty dealing with spurious correlations. Alternatively, we propose to use this kind of action-based driving data for learning representations. Our experiments show that an affordance-based driving model pre-trained with this approach can leverage a relatively small amount of weakly annotated imagery and outperform pure end-to-end driving models, while being more interpretable. Further, we demonstrate how this strategy outperforms previous methods based on learning inverse dynamics models as well as other methods based on heavy human supervision (ImageNet).} }
Endnote
%0 Conference Paper %T Action-based Representation Learning for Autonomous Driving %A Yi Xiao %A Felipe Codevilla %A Christopher Pal %A Antonio Lopez %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-xiao21a %I PMLR %P 232--246 %U https://proceedings.mlr.press/v155/xiao21a.html %V 155 %X Human drivers produce a vast amount of data which could, in principle, be used to improve autonomous driving systems. Unfortunately, seemingly straightforward approaches for creating end-to-end driving models that map sensor data directly into driving actions are problematic in terms of interpretability, and typically have significant difficulty dealing with spurious correlations. Alternatively, we propose to use this kind of action-based driving data for learning representations. Our experiments show that an affordance-based driving model pre-trained with this approach can leverage a relatively small amount of weakly annotated imagery and outperform pure end-to-end driving models, while being more interpretable. Further, we demonstrate how this strategy outperforms previous methods based on learning inverse dynamics models as well as other methods based on heavy human supervision (ImageNet).
APA
Xiao, Y., Codevilla, F., Pal, C. & Lopez, A.. (2021). Action-based Representation Learning for Autonomous Driving. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:232-246 Available from https://proceedings.mlr.press/v155/xiao21a.html.

Related Material