Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs

Sean Segal, Eric Kee, Wenjie Luo, Abbas Sadat, Ersin Yumer, Raquel Urtasun
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:973-983, 2021.

Abstract

In this paper, we tackle the problem of spatio-temporal tagging of self-driving scenes from raw sensor data. Our approach learns a universal embedding for all tags, enabling efficient tagging of many attributes and faster learning of new attributes with limited data. Importantly, the embedding is spatio-temporally aware, allowing the model to naturally output spatio-temporal tag values. Values can then be pooled over arbitrary regions, in order to, for example, compute the pedestrian density in front of the SDV, or determine if a car is blocking another car at a 4-way intersection. We demonstrate the effectiveness of our approach on a new large scale self-driving dataset, SDVScenes, containing 15 attributes relating to vehicle and pedestrian density, the actions of each actor, the speed of each actor, interactions between actors, and the topology of the road map.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-segal21a, title = {Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs}, author = {Segal, Sean and Kee, Eric and Luo, Wenjie and Sadat, Abbas and Yumer, Ersin and Urtasun, Raquel}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {973--983}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/segal21a/segal21a.pdf}, url = {https://proceedings.mlr.press/v155/segal21a.html}, abstract = {In this paper, we tackle the problem of spatio-temporal tagging of self-driving scenes from raw sensor data. Our approach learns a universal embedding for all tags, enabling efficient tagging of many attributes and faster learning of new attributes with limited data. Importantly, the embedding is spatio-temporally aware, allowing the model to naturally output spatio-temporal tag values. Values can then be pooled over arbitrary regions, in order to, for example, compute the pedestrian density in front of the SDV, or determine if a car is blocking another car at a 4-way intersection. We demonstrate the effectiveness of our approach on a new large scale self-driving dataset, SDVScenes, containing 15 attributes relating to vehicle and pedestrian density, the actions of each actor, the speed of each actor, interactions between actors, and the topology of the road map.} }
Endnote
%0 Conference Paper %T Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs %A Sean Segal %A Eric Kee %A Wenjie Luo %A Abbas Sadat %A Ersin Yumer %A Raquel Urtasun %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-segal21a %I PMLR %P 973--983 %U https://proceedings.mlr.press/v155/segal21a.html %V 155 %X In this paper, we tackle the problem of spatio-temporal tagging of self-driving scenes from raw sensor data. Our approach learns a universal embedding for all tags, enabling efficient tagging of many attributes and faster learning of new attributes with limited data. Importantly, the embedding is spatio-temporally aware, allowing the model to naturally output spatio-temporal tag values. Values can then be pooled over arbitrary regions, in order to, for example, compute the pedestrian density in front of the SDV, or determine if a car is blocking another car at a 4-way intersection. We demonstrate the effectiveness of our approach on a new large scale self-driving dataset, SDVScenes, containing 15 attributes relating to vehicle and pedestrian density, the actions of each actor, the speed of each actor, interactions between actors, and the topology of the road map.
APA
Segal, S., Kee, E., Luo, W., Sadat, A., Yumer, E. & Urtasun, R.. (2021). Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:973-983 Available from https://proceedings.mlr.press/v155/segal21a.html.

Related Material