Universality of Winning Tickets: A Renormalization Group Perspective

William T Redman, Tianlong Chen, Zhangyang Wang, Akshunna S. Dogra
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:18483-18498, 2022.

Abstract

Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. This has generated broad interest, but methods to study this universality are lacking. We make use of renormalization group theory, a powerful tool from theoretical physics, to address this need. We find that iterative magnitude pruning, the principal algorithm used for discovering winning tickets, is a renormalization group scheme, and can be viewed as inducing a flow in parameter space. We demonstrate that ResNet-50 models with transferable winning tickets have flows with common properties, as would be expected from the theory. Similar observations are made for BERT models, with evidence that their flows are near fixed points. Additionally, we leverage our framework to study winning tickets transferred across ResNet architectures, observing that smaller models have flows with more uniform properties than larger models, complicating transfer between them.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-redman22a, title = {Universality of Winning Tickets: A Renormalization Group Perspective}, author = {Redman, William T and Chen, Tianlong and Wang, Zhangyang and Dogra, Akshunna S.}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {18483--18498}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/redman22a/redman22a.pdf}, url = {https://proceedings.mlr.press/v162/redman22a.html}, abstract = {Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. This has generated broad interest, but methods to study this universality are lacking. We make use of renormalization group theory, a powerful tool from theoretical physics, to address this need. We find that iterative magnitude pruning, the principal algorithm used for discovering winning tickets, is a renormalization group scheme, and can be viewed as inducing a flow in parameter space. We demonstrate that ResNet-50 models with transferable winning tickets have flows with common properties, as would be expected from the theory. Similar observations are made for BERT models, with evidence that their flows are near fixed points. Additionally, we leverage our framework to study winning tickets transferred across ResNet architectures, observing that smaller models have flows with more uniform properties than larger models, complicating transfer between them.} }
Endnote
%0 Conference Paper %T Universality of Winning Tickets: A Renormalization Group Perspective %A William T Redman %A Tianlong Chen %A Zhangyang Wang %A Akshunna S. Dogra %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-redman22a %I PMLR %P 18483--18498 %U https://proceedings.mlr.press/v162/redman22a.html %V 162 %X Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. This has generated broad interest, but methods to study this universality are lacking. We make use of renormalization group theory, a powerful tool from theoretical physics, to address this need. We find that iterative magnitude pruning, the principal algorithm used for discovering winning tickets, is a renormalization group scheme, and can be viewed as inducing a flow in parameter space. We demonstrate that ResNet-50 models with transferable winning tickets have flows with common properties, as would be expected from the theory. Similar observations are made for BERT models, with evidence that their flows are near fixed points. Additionally, we leverage our framework to study winning tickets transferred across ResNet architectures, observing that smaller models have flows with more uniform properties than larger models, complicating transfer between them.
APA
Redman, W.T., Chen, T., Wang, Z. & Dogra, A.S.. (2022). Universality of Winning Tickets: A Renormalization Group Perspective. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:18483-18498 Available from https://proceedings.mlr.press/v162/redman22a.html.

Related Material