Nighttime Semantic Segmentation with Unsupervised Learning and Cross Attention

Jian Cheng, Yang-Feng Hu, Yu Dai, Xue Qiao, Li Yao, Jun-Yan Yang
Proceedings of The 14th Asian Conference on Machine Learning, PMLR 189, 2023.

Abstract

In recent years, semantic segmentation has shown very good performance in daytime scenes. But in nighttime scenes, semantic segmentation greatly reduces its accuracy. Due to the lack of large-scale nighttime semantic segmentation datasets, it is difficult to directly train segmentation models for nighttime scenes. Therefore, it becomes important to adapt the daytime scene segmentation model to the nighttime scene without directly using the nighttime scene segmentation dataset. In this paper, we propose a framework based on unsupervised learning and cross attention. The proposed method fuses supervised daytime scenes and unsupervised nighttime scenes, the supervision information in the daytime scene and the texture information specific to the nighttime scene are fully utilized, and the model is adapted to both the daytime scene and the nighttime scene. Consistency regulation is used to make segmentation model adapt to the complex and changeable night scene texture and illumination. In view of the coarse correspondence of static objects between day and night image pairs in the Dark Zurich dataset, cross attention is proposed to make the model pay more attention to the parts of the night scene which are similar to the daytime scene. Extensive experiments on Dark Zurich and Nighttime Driving datasets show that our method obtains better performance in nighttime semantic segmentation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v189-cheng23a, title = {Nighttime Semantic Segmentation with Unsupervised Learning and Cross Attention}, author = {Cheng, Jian and Hu, Yang{-}Feng and Dai, Yu and Qiao, Xue and Yao, Li and Yang, Jun{-}Yan}, booktitle = {Proceedings of The 14th Asian Conference on Machine Learning}, year = {2023}, editor = {Khan, Emtiyaz and Gonen, Mehmet}, volume = {189}, series = {Proceedings of Machine Learning Research}, month = {12--14 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v189/cheng23a/cheng23a.pdf}, url = {https://proceedings.mlr.press/v189/cheng23a.html}, abstract = {In recent years, semantic segmentation has shown very good performance in daytime scenes. But in nighttime scenes, semantic segmentation greatly reduces its accuracy. Due to the lack of large-scale nighttime semantic segmentation datasets, it is difficult to directly train segmentation models for nighttime scenes. Therefore, it becomes important to adapt the daytime scene segmentation model to the nighttime scene without directly using the nighttime scene segmentation dataset. In this paper, we propose a framework based on unsupervised learning and cross attention. The proposed method fuses supervised daytime scenes and unsupervised nighttime scenes, the supervision information in the daytime scene and the texture information specific to the nighttime scene are fully utilized, and the model is adapted to both the daytime scene and the nighttime scene. Consistency regulation is used to make segmentation model adapt to the complex and changeable night scene texture and illumination. In view of the coarse correspondence of static objects between day and night image pairs in the Dark Zurich dataset, cross attention is proposed to make the model pay more attention to the parts of the night scene which are similar to the daytime scene. Extensive experiments on Dark Zurich and Nighttime Driving datasets show that our method obtains better performance in nighttime semantic segmentation.} }
Endnote
%0 Conference Paper %T Nighttime Semantic Segmentation with Unsupervised Learning and Cross Attention %A Jian Cheng %A Yang-Feng Hu %A Yu Dai %A Xue Qiao %A Li Yao %A Jun-Yan Yang %B Proceedings of The 14th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Emtiyaz Khan %E Mehmet Gonen %F pmlr-v189-cheng23a %I PMLR %U https://proceedings.mlr.press/v189/cheng23a.html %V 189 %X In recent years, semantic segmentation has shown very good performance in daytime scenes. But in nighttime scenes, semantic segmentation greatly reduces its accuracy. Due to the lack of large-scale nighttime semantic segmentation datasets, it is difficult to directly train segmentation models for nighttime scenes. Therefore, it becomes important to adapt the daytime scene segmentation model to the nighttime scene without directly using the nighttime scene segmentation dataset. In this paper, we propose a framework based on unsupervised learning and cross attention. The proposed method fuses supervised daytime scenes and unsupervised nighttime scenes, the supervision information in the daytime scene and the texture information specific to the nighttime scene are fully utilized, and the model is adapted to both the daytime scene and the nighttime scene. Consistency regulation is used to make segmentation model adapt to the complex and changeable night scene texture and illumination. In view of the coarse correspondence of static objects between day and night image pairs in the Dark Zurich dataset, cross attention is proposed to make the model pay more attention to the parts of the night scene which are similar to the daytime scene. Extensive experiments on Dark Zurich and Nighttime Driving datasets show that our method obtains better performance in nighttime semantic segmentation.
APA
Cheng, J., Hu, Y., Dai, Y., Qiao, X., Yao, L. & Yang, J.. (2023). Nighttime Semantic Segmentation with Unsupervised Learning and Cross Attention. Proceedings of The 14th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 189 Available from https://proceedings.mlr.press/v189/cheng23a.html.

Related Material