BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities

Binyu Zhao, Wei ZHANG, Zhaonian Zou
Proceedings of The 7th Conference on Robot Learning, PMLR 229:1022-1035, 2023.

Abstract

Collaborative perception enables agents to share complementary perceptual information with nearby agents. This can significantly benefit the perception performance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most proposed approaches mainly focus on single modality (especially LiDAR), and not fully exploit the superiority of multi-modal perception. We propose an collaborative perception paradigm, BM2CP, which employs LiDAR and camera to achieve efficient multi-modal perception. BM2CP utilizes LiDAR-guided modal fusion, cooperative depth generation and modality-guided intermediate fusion to acquire deep interactions between modalities and agents. Moreover, it is capable to cope with the special case that one of the sensors is unavailable. Extensive experiments validate that it outperforms the state-of-the-art methods with 50X lower communication volumes in real-world autonomous driving scenarios. Our code is available at supplementary materials.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-zhao23a, title = {BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities}, author = {Zhao, Binyu and ZHANG, Wei and Zou, Zhaonian}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {1022--1035}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/zhao23a/zhao23a.pdf}, url = {https://proceedings.mlr.press/v229/zhao23a.html}, abstract = {Collaborative perception enables agents to share complementary perceptual information with nearby agents. This can significantly benefit the perception performance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most proposed approaches mainly focus on single modality (especially LiDAR), and not fully exploit the superiority of multi-modal perception. We propose an collaborative perception paradigm, BM2CP, which employs LiDAR and camera to achieve efficient multi-modal perception. BM2CP utilizes LiDAR-guided modal fusion, cooperative depth generation and modality-guided intermediate fusion to acquire deep interactions between modalities and agents. Moreover, it is capable to cope with the special case that one of the sensors is unavailable. Extensive experiments validate that it outperforms the state-of-the-art methods with 50X lower communication volumes in real-world autonomous driving scenarios. Our code is available at supplementary materials.} }
Endnote
%0 Conference Paper %T BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities %A Binyu Zhao %A Wei ZHANG %A Zhaonian Zou %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-zhao23a %I PMLR %P 1022--1035 %U https://proceedings.mlr.press/v229/zhao23a.html %V 229 %X Collaborative perception enables agents to share complementary perceptual information with nearby agents. This can significantly benefit the perception performance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most proposed approaches mainly focus on single modality (especially LiDAR), and not fully exploit the superiority of multi-modal perception. We propose an collaborative perception paradigm, BM2CP, which employs LiDAR and camera to achieve efficient multi-modal perception. BM2CP utilizes LiDAR-guided modal fusion, cooperative depth generation and modality-guided intermediate fusion to acquire deep interactions between modalities and agents. Moreover, it is capable to cope with the special case that one of the sensors is unavailable. Extensive experiments validate that it outperforms the state-of-the-art methods with 50X lower communication volumes in real-world autonomous driving scenarios. Our code is available at supplementary materials.
APA
Zhao, B., ZHANG, W. & Zou, Z.. (2023). BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:1022-1035 Available from https://proceedings.mlr.press/v229/zhao23a.html.

Related Material