Testing DNN-based Autonomous Driving Systems under Critical Environmental Conditions

Zhong Li, Minxue Pan, Tian Zhang, Xuandong Li
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:6471-6482, 2021.

Abstract

Due to the increasing usage of Deep Neural Network (DNN) based autonomous driving systems (ADS) where erroneous or unexpected behaviours can lead to catastrophic accidents, testing such systems is of growing importance. Existing approaches often just focus on finding erroneous behaviours and have not thoroughly studied the impact of environmental conditions. In this paper, we propose to test DNN-based ADS under different environmental conditions to identify the critical ones, that is, the environmental conditions under which the ADS are more prone to errors. To tackle the problem of the space of environmental conditions being extremely large, we present a novel approach named TACTIC that employs the search-based method to identify critical environmental conditions generated by an image-to-image translation model. Large-scale experiments show that TACTIC can effectively identify critical environmental conditions and produce realistic testing images, and meanwhile, reveal more erroneous behaviours compared to existing approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-li21r, title = {Testing DNN-based Autonomous Driving Systems under Critical Environmental Conditions}, author = {Li, Zhong and Pan, Minxue and Zhang, Tian and Li, Xuandong}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {6471--6482}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/li21r/li21r.pdf}, url = {https://proceedings.mlr.press/v139/li21r.html}, abstract = {Due to the increasing usage of Deep Neural Network (DNN) based autonomous driving systems (ADS) where erroneous or unexpected behaviours can lead to catastrophic accidents, testing such systems is of growing importance. Existing approaches often just focus on finding erroneous behaviours and have not thoroughly studied the impact of environmental conditions. In this paper, we propose to test DNN-based ADS under different environmental conditions to identify the critical ones, that is, the environmental conditions under which the ADS are more prone to errors. To tackle the problem of the space of environmental conditions being extremely large, we present a novel approach named TACTIC that employs the search-based method to identify critical environmental conditions generated by an image-to-image translation model. Large-scale experiments show that TACTIC can effectively identify critical environmental conditions and produce realistic testing images, and meanwhile, reveal more erroneous behaviours compared to existing approaches.} }
Endnote
%0 Conference Paper %T Testing DNN-based Autonomous Driving Systems under Critical Environmental Conditions %A Zhong Li %A Minxue Pan %A Tian Zhang %A Xuandong Li %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-li21r %I PMLR %P 6471--6482 %U https://proceedings.mlr.press/v139/li21r.html %V 139 %X Due to the increasing usage of Deep Neural Network (DNN) based autonomous driving systems (ADS) where erroneous or unexpected behaviours can lead to catastrophic accidents, testing such systems is of growing importance. Existing approaches often just focus on finding erroneous behaviours and have not thoroughly studied the impact of environmental conditions. In this paper, we propose to test DNN-based ADS under different environmental conditions to identify the critical ones, that is, the environmental conditions under which the ADS are more prone to errors. To tackle the problem of the space of environmental conditions being extremely large, we present a novel approach named TACTIC that employs the search-based method to identify critical environmental conditions generated by an image-to-image translation model. Large-scale experiments show that TACTIC can effectively identify critical environmental conditions and produce realistic testing images, and meanwhile, reveal more erroneous behaviours compared to existing approaches.
APA
Li, Z., Pan, M., Zhang, T. & Li, X.. (2021). Testing DNN-based Autonomous Driving Systems under Critical Environmental Conditions. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:6471-6482 Available from https://proceedings.mlr.press/v139/li21r.html.

Related Material