Recovering and Simulating Pedestrians in the Wild

Ze Yang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wei-Chiu Ma, Raquel Urtasun
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:419-431, 2021.

Abstract

Sensor simulation is a key component for testing the performance of self-driving vehicles and for data augmentation to better train perception systems. Typical approaches rely on artists to create both 3D assets and their animations to generate a new scenario. This, however, does not scale. In contrast, we propose to recover the shape and motion of pedestrians from sensor readings captured in the wild by a self-driving car driving around. Towards this goal, we formulate the problem as energy minimization in a deep structured model that exploits human shape priors, reprojection consistency with 2D poses extracted from images, and a ray-caster that encourages the reconstructed mesh to agree with the LiDAR readings. Importantly, we do not require any ground-truth 3D scans or 3D pose annotations. We then incorporate the reconstructed pedestrian assets bank in a realistic LiDAR simulation system by performing motion retargeting, and show that the simulated LiDAR data can be used to significantly reduce the amount of annotated real-world data required for visual perception tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-yang21a, title = {Recovering and Simulating Pedestrians in the Wild}, author = {Yang, Ze and Manivasagam, Sivabalan and Liang, Ming and Yang, Bin and Ma, Wei-Chiu and Urtasun, Raquel}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {419--431}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/yang21a/yang21a.pdf}, url = {https://proceedings.mlr.press/v155/yang21a.html}, abstract = {Sensor simulation is a key component for testing the performance of self-driving vehicles and for data augmentation to better train perception systems. Typical approaches rely on artists to create both 3D assets and their animations to generate a new scenario. This, however, does not scale. In contrast, we propose to recover the shape and motion of pedestrians from sensor readings captured in the wild by a self-driving car driving around. Towards this goal, we formulate the problem as energy minimization in a deep structured model that exploits human shape priors, reprojection consistency with 2D poses extracted from images, and a ray-caster that encourages the reconstructed mesh to agree with the LiDAR readings. Importantly, we do not require any ground-truth 3D scans or 3D pose annotations. We then incorporate the reconstructed pedestrian assets bank in a realistic LiDAR simulation system by performing motion retargeting, and show that the simulated LiDAR data can be used to significantly reduce the amount of annotated real-world data required for visual perception tasks.} }
Endnote
%0 Conference Paper %T Recovering and Simulating Pedestrians in the Wild %A Ze Yang %A Sivabalan Manivasagam %A Ming Liang %A Bin Yang %A Wei-Chiu Ma %A Raquel Urtasun %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-yang21a %I PMLR %P 419--431 %U https://proceedings.mlr.press/v155/yang21a.html %V 155 %X Sensor simulation is a key component for testing the performance of self-driving vehicles and for data augmentation to better train perception systems. Typical approaches rely on artists to create both 3D assets and their animations to generate a new scenario. This, however, does not scale. In contrast, we propose to recover the shape and motion of pedestrians from sensor readings captured in the wild by a self-driving car driving around. Towards this goal, we formulate the problem as energy minimization in a deep structured model that exploits human shape priors, reprojection consistency with 2D poses extracted from images, and a ray-caster that encourages the reconstructed mesh to agree with the LiDAR readings. Importantly, we do not require any ground-truth 3D scans or 3D pose annotations. We then incorporate the reconstructed pedestrian assets bank in a realistic LiDAR simulation system by performing motion retargeting, and show that the simulated LiDAR data can be used to significantly reduce the amount of annotated real-world data required for visual perception tasks.
APA
Yang, Z., Manivasagam, S., Liang, M., Yang, B., Ma, W. & Urtasun, R.. (2021). Recovering and Simulating Pedestrians in the Wild. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:419-431 Available from https://proceedings.mlr.press/v155/yang21a.html.

Related Material