Training deep residual networks for uniform approximation guarantees

Matteo Marchi, Bahman Gharesifard, Paulo Tabuada
Proceedings of the 3rd Conference on Learning for Dynamics and Control, PMLR 144:677-688, 2021.

Abstract

It has recently been shown that deep residual networks with sufficiently high depth, but bounded width, are capable of universal approximation in the supremum norm sense. Based on these results, we show how to modify existing training algorithms for deep residual networks so as to provide approximation bounds for the test error, in the supremum norm, based on the training error. Our methods are based on control-theoretic interpretations of these networks both in discrete and continuous time, and establish that it is enough to suitably constrain the set of parameters being learned in a way that is compatible with most currently used training algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v144-marchi21a, title = {Training deep residual networks for uniform approximation guarantees}, author = {Marchi, Matteo and Gharesifard, Bahman and Tabuada, Paulo}, booktitle = {Proceedings of the 3rd Conference on Learning for Dynamics and Control}, pages = {677--688}, year = {2021}, editor = {Jadbabaie, Ali and Lygeros, John and Pappas, George J. and A. Parrilo, Pablo and Recht, Benjamin and Tomlin, Claire J. and Zeilinger, Melanie N.}, volume = {144}, series = {Proceedings of Machine Learning Research}, month = {07 -- 08 June}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v144/marchi21a/marchi21a.pdf}, url = {https://proceedings.mlr.press/v144/marchi21a.html}, abstract = {It has recently been shown that deep residual networks with sufficiently high depth, but bounded width, are capable of universal approximation in the supremum norm sense. Based on these results, we show how to modify existing training algorithms for deep residual networks so as to provide approximation bounds for the test error, in the supremum norm, based on the training error. Our methods are based on control-theoretic interpretations of these networks both in discrete and continuous time, and establish that it is enough to suitably constrain the set of parameters being learned in a way that is compatible with most currently used training algorithms.} }
Endnote
%0 Conference Paper %T Training deep residual networks for uniform approximation guarantees %A Matteo Marchi %A Bahman Gharesifard %A Paulo Tabuada %B Proceedings of the 3rd Conference on Learning for Dynamics and Control %C Proceedings of Machine Learning Research %D 2021 %E Ali Jadbabaie %E John Lygeros %E George J. Pappas %E Pablo A. Parrilo %E Benjamin Recht %E Claire J. Tomlin %E Melanie N. Zeilinger %F pmlr-v144-marchi21a %I PMLR %P 677--688 %U https://proceedings.mlr.press/v144/marchi21a.html %V 144 %X It has recently been shown that deep residual networks with sufficiently high depth, but bounded width, are capable of universal approximation in the supremum norm sense. Based on these results, we show how to modify existing training algorithms for deep residual networks so as to provide approximation bounds for the test error, in the supremum norm, based on the training error. Our methods are based on control-theoretic interpretations of these networks both in discrete and continuous time, and establish that it is enough to suitably constrain the set of parameters being learned in a way that is compatible with most currently used training algorithms.
APA
Marchi, M., Gharesifard, B. & Tabuada, P.. (2021). Training deep residual networks for uniform approximation guarantees. Proceedings of the 3rd Conference on Learning for Dynamics and Control, in Proceedings of Machine Learning Research 144:677-688 Available from https://proceedings.mlr.press/v144/marchi21a.html.

Related Material