SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning

Albert Zhao, Tong He, Yitao Liang, Haibin Huang, Guy Van den Broeck, Stefano Soatto
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:156-175, 2021.

Abstract

We describe a policy learning approach to map visual inputs to driving controls conditioned on turning command that leverages side tasks on semantics and object affordances via a learned representation trained for driving. To learn this representation, we train a squeeze network to drive using annotations for the side task as input. This representation encodes the driving-relevant information associated with the side task while ideally throwing out side task-relevant but driving-irrelevant nuisances. We then train a mimic network to drive using only images as input and use the squeeze network’s latent representation to supervise the mimic network via a mimicking loss. Notably, we do not aim to achieve the side task nor to learn features for it; instead, we aim to learn, via the mimicking loss, a representation of the side task annotations directly useful for driving. We test our approach using the CARLA simulator. In addition, we introduce a more challenging but realistic evaluation protocol that considers a run that reaches the destination successful only if it does not violate common traffic rules.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-zhao21a, title = {SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning}, author = {Zhao, Albert and He, Tong and Liang, Yitao and Huang, Haibin and Broeck, Guy Van den and Soatto, Stefano}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {156--175}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/zhao21a/zhao21a.pdf}, url = {https://proceedings.mlr.press/v155/zhao21a.html}, abstract = {We describe a policy learning approach to map visual inputs to driving controls conditioned on turning command that leverages side tasks on semantics and object affordances via a learned representation trained for driving. To learn this representation, we train a squeeze network to drive using annotations for the side task as input. This representation encodes the driving-relevant information associated with the side task while ideally throwing out side task-relevant but driving-irrelevant nuisances. We then train a mimic network to drive using only images as input and use the squeeze network’s latent representation to supervise the mimic network via a mimicking loss. Notably, we do not aim to achieve the side task nor to learn features for it; instead, we aim to learn, via the mimicking loss, a representation of the side task annotations directly useful for driving. We test our approach using the CARLA simulator. In addition, we introduce a more challenging but realistic evaluation protocol that considers a run that reaches the destination successful only if it does not violate common traffic rules.} }
Endnote
%0 Conference Paper %T SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning %A Albert Zhao %A Tong He %A Yitao Liang %A Haibin Huang %A Guy Van den Broeck %A Stefano Soatto %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-zhao21a %I PMLR %P 156--175 %U https://proceedings.mlr.press/v155/zhao21a.html %V 155 %X We describe a policy learning approach to map visual inputs to driving controls conditioned on turning command that leverages side tasks on semantics and object affordances via a learned representation trained for driving. To learn this representation, we train a squeeze network to drive using annotations for the side task as input. This representation encodes the driving-relevant information associated with the side task while ideally throwing out side task-relevant but driving-irrelevant nuisances. We then train a mimic network to drive using only images as input and use the squeeze network’s latent representation to supervise the mimic network via a mimicking loss. Notably, we do not aim to achieve the side task nor to learn features for it; instead, we aim to learn, via the mimicking loss, a representation of the side task annotations directly useful for driving. We test our approach using the CARLA simulator. In addition, we introduce a more challenging but realistic evaluation protocol that considers a run that reaches the destination successful only if it does not violate common traffic rules.
APA
Zhao, A., He, T., Liang, Y., Huang, H., Broeck, G.V.d. & Soatto, S.. (2021). SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:156-175 Available from https://proceedings.mlr.press/v155/zhao21a.html.

Related Material