Lipschitz constant estimation for 1D convolutional neural networks

Patricia Pauli, Dennis Gramlich, Frank Allgöwer
Proceedings of The 5th Annual Learning for Dynamics and Control Conference, PMLR 211:1321-1332, 2023.

Abstract

In this work, we propose a dissipativity-based method for Lipschitz constant estimation of 1D convolutional neural networks (CNNs). In particular, we analyze the dissipativity properties of convolutional, pooling, and fully connected layers making use of incremental quadratic constraints for nonlinear activation functions and pooling operations. The Lipschitz constant of the concatenation of these mappings is then estimated by solving a semidefinite program which we derive from dissipativity theory. To make our method as efficient as possible, we exploit the structure of convolutional layers by realizing these finite impulse response filters as causal dynamical systems in state space and carrying out the dissipativity analysis for the state space realizations. The examples we provide show that our Lipschitz bounds are advantageous in terms of accuracy and scalability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v211-pauli23a, title = {Lipschitz constant estimation for 1D convolutional neural networks}, author = {Pauli, Patricia and Gramlich, Dennis and Allg\"ower, Frank}, booktitle = {Proceedings of The 5th Annual Learning for Dynamics and Control Conference}, pages = {1321--1332}, year = {2023}, editor = {Matni, Nikolai and Morari, Manfred and Pappas, George J.}, volume = {211}, series = {Proceedings of Machine Learning Research}, month = {15--16 Jun}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v211/pauli23a/pauli23a.pdf}, url = {https://proceedings.mlr.press/v211/pauli23a.html}, abstract = {In this work, we propose a dissipativity-based method for Lipschitz constant estimation of 1D convolutional neural networks (CNNs). In particular, we analyze the dissipativity properties of convolutional, pooling, and fully connected layers making use of incremental quadratic constraints for nonlinear activation functions and pooling operations. The Lipschitz constant of the concatenation of these mappings is then estimated by solving a semidefinite program which we derive from dissipativity theory. To make our method as efficient as possible, we exploit the structure of convolutional layers by realizing these finite impulse response filters as causal dynamical systems in state space and carrying out the dissipativity analysis for the state space realizations. The examples we provide show that our Lipschitz bounds are advantageous in terms of accuracy and scalability.} }
Endnote
%0 Conference Paper %T Lipschitz constant estimation for 1D convolutional neural networks %A Patricia Pauli %A Dennis Gramlich %A Frank Allgöwer %B Proceedings of The 5th Annual Learning for Dynamics and Control Conference %C Proceedings of Machine Learning Research %D 2023 %E Nikolai Matni %E Manfred Morari %E George J. Pappas %F pmlr-v211-pauli23a %I PMLR %P 1321--1332 %U https://proceedings.mlr.press/v211/pauli23a.html %V 211 %X In this work, we propose a dissipativity-based method for Lipschitz constant estimation of 1D convolutional neural networks (CNNs). In particular, we analyze the dissipativity properties of convolutional, pooling, and fully connected layers making use of incremental quadratic constraints for nonlinear activation functions and pooling operations. The Lipschitz constant of the concatenation of these mappings is then estimated by solving a semidefinite program which we derive from dissipativity theory. To make our method as efficient as possible, we exploit the structure of convolutional layers by realizing these finite impulse response filters as causal dynamical systems in state space and carrying out the dissipativity analysis for the state space realizations. The examples we provide show that our Lipschitz bounds are advantageous in terms of accuracy and scalability.
APA
Pauli, P., Gramlich, D. & Allgöwer, F.. (2023). Lipschitz constant estimation for 1D convolutional neural networks. Proceedings of The 5th Annual Learning for Dynamics and Control Conference, in Proceedings of Machine Learning Research 211:1321-1332 Available from https://proceedings.mlr.press/v211/pauli23a.html.

Related Material