Convolutional and Residual Networks Provably Contain Lottery Tickets

Rebekka Burkholz
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:2414-2433, 2022.

Abstract

The Lottery Ticket Hypothesis continues to have a profound practical impact on the quest for small scale deep neural networks that solve modern deep learning tasks at competitive performance. These lottery tickets are identified by pruning large randomly initialized neural networks with architectures that are as diverse as their applications. Yet, theoretical insights that attest their existence have been mostly focused on deed fully-connected feed forward networks with ReLU activation functions. We prove that also modern architectures consisting of convolutional and residual layers that can be equipped with almost arbitrary activation functions can contain lottery tickets with high probability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-burkholz22a, title = {Convolutional and Residual Networks Provably Contain Lottery Tickets}, author = {Burkholz, Rebekka}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {2414--2433}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/burkholz22a/burkholz22a.pdf}, url = {https://proceedings.mlr.press/v162/burkholz22a.html}, abstract = {The Lottery Ticket Hypothesis continues to have a profound practical impact on the quest for small scale deep neural networks that solve modern deep learning tasks at competitive performance. These lottery tickets are identified by pruning large randomly initialized neural networks with architectures that are as diverse as their applications. Yet, theoretical insights that attest their existence have been mostly focused on deed fully-connected feed forward networks with ReLU activation functions. We prove that also modern architectures consisting of convolutional and residual layers that can be equipped with almost arbitrary activation functions can contain lottery tickets with high probability.} }
Endnote
%0 Conference Paper %T Convolutional and Residual Networks Provably Contain Lottery Tickets %A Rebekka Burkholz %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-burkholz22a %I PMLR %P 2414--2433 %U https://proceedings.mlr.press/v162/burkholz22a.html %V 162 %X The Lottery Ticket Hypothesis continues to have a profound practical impact on the quest for small scale deep neural networks that solve modern deep learning tasks at competitive performance. These lottery tickets are identified by pruning large randomly initialized neural networks with architectures that are as diverse as their applications. Yet, theoretical insights that attest their existence have been mostly focused on deed fully-connected feed forward networks with ReLU activation functions. We prove that also modern architectures consisting of convolutional and residual layers that can be equipped with almost arbitrary activation functions can contain lottery tickets with high probability.
APA
Burkholz, R.. (2022). Convolutional and Residual Networks Provably Contain Lottery Tickets. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:2414-2433 Available from https://proceedings.mlr.press/v162/burkholz22a.html.

Related Material