Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons

Bohang Zhang, Tianle Cai, Zhou Lu, Di He, Liwei Wang
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:12368-12379, 2021.

Abstract

It is well-known that standard neural networks, even with a high classification accuracy, are vulnerable to small ℓ∞-norm bounded adversarial perturbations. Although many attempts have been made, most previous works either can only provide empirical verification of the defense to a particular attack method, or can only develop a certified guarantee of the model robustness in limited scenarios. In this paper, we seek for a new approach to develop a theoretically principled neural network that inherently resists ℓ∞ perturbations. In particular, we design a novel neuron that uses ℓ∞-distance as its basic operation (which we call ℓ∞-dist neuron), and show that any neural network constructed with ℓ∞-dist neurons (called ℓ∞-dist net) is naturally a 1-Lipschitz function with respect to ℓ∞-norm. This directly provides a rigorous guarantee of the certified robustness based on the margin of prediction outputs. We then prove that such networks have enough expressive power to approximate any 1-Lipschitz function with robust generalization guarantee. We further provide a holistic training strategy that can greatly alleviate optimization difficulties. Experimental results show that using ℓ∞-dist nets as basic building blocks, we consistently achieve state-of-the-art performance on commonly used datasets: 93.09% certified accuracy on MNIST (ϵ=0.3), 35.42% on CIFAR-10 (ϵ=8/255) and 16.31% on TinyImageNet (ϵ=1/255).

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-zhang21b, title = {Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons}, author = {Zhang, Bohang and Cai, Tianle and Lu, Zhou and He, Di and Wang, Liwei}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {12368--12379}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/zhang21b/zhang21b.pdf}, url = {https://proceedings.mlr.press/v139/zhang21b.html}, abstract = {It is well-known that standard neural networks, even with a high classification accuracy, are vulnerable to small $\ell_\infty$-norm bounded adversarial perturbations. Although many attempts have been made, most previous works either can only provide empirical verification of the defense to a particular attack method, or can only develop a certified guarantee of the model robustness in limited scenarios. In this paper, we seek for a new approach to develop a theoretically principled neural network that inherently resists $\ell_\infty$ perturbations. In particular, we design a novel neuron that uses $\ell_\infty$-distance as its basic operation (which we call $\ell_\infty$-dist neuron), and show that any neural network constructed with $\ell_\infty$-dist neurons (called $\ell_{\infty}$-dist net) is naturally a 1-Lipschitz function with respect to $\ell_\infty$-norm. This directly provides a rigorous guarantee of the certified robustness based on the margin of prediction outputs. We then prove that such networks have enough expressive power to approximate any 1-Lipschitz function with robust generalization guarantee. We further provide a holistic training strategy that can greatly alleviate optimization difficulties. Experimental results show that using $\ell_{\infty}$-dist nets as basic building blocks, we consistently achieve state-of-the-art performance on commonly used datasets: 93.09% certified accuracy on MNIST ($\epsilon=0.3$), 35.42% on CIFAR-10 ($\epsilon=8/255$) and 16.31% on TinyImageNet ($\epsilon=1/255$).} }
Endnote
%0 Conference Paper %T Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons %A Bohang Zhang %A Tianle Cai %A Zhou Lu %A Di He %A Liwei Wang %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-zhang21b %I PMLR %P 12368--12379 %U https://proceedings.mlr.press/v139/zhang21b.html %V 139 %X It is well-known that standard neural networks, even with a high classification accuracy, are vulnerable to small $\ell_\infty$-norm bounded adversarial perturbations. Although many attempts have been made, most previous works either can only provide empirical verification of the defense to a particular attack method, or can only develop a certified guarantee of the model robustness in limited scenarios. In this paper, we seek for a new approach to develop a theoretically principled neural network that inherently resists $\ell_\infty$ perturbations. In particular, we design a novel neuron that uses $\ell_\infty$-distance as its basic operation (which we call $\ell_\infty$-dist neuron), and show that any neural network constructed with $\ell_\infty$-dist neurons (called $\ell_{\infty}$-dist net) is naturally a 1-Lipschitz function with respect to $\ell_\infty$-norm. This directly provides a rigorous guarantee of the certified robustness based on the margin of prediction outputs. We then prove that such networks have enough expressive power to approximate any 1-Lipschitz function with robust generalization guarantee. We further provide a holistic training strategy that can greatly alleviate optimization difficulties. Experimental results show that using $\ell_{\infty}$-dist nets as basic building blocks, we consistently achieve state-of-the-art performance on commonly used datasets: 93.09% certified accuracy on MNIST ($\epsilon=0.3$), 35.42% on CIFAR-10 ($\epsilon=8/255$) and 16.31% on TinyImageNet ($\epsilon=1/255$).
APA
Zhang, B., Cai, T., Lu, Z., He, D. & Wang, L.. (2021). Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:12368-12379 Available from https://proceedings.mlr.press/v139/zhang21b.html.

Related Material