Are All Linear Regions Created Equal?

Matteo Gamba, Adrian Chmielewski-Anders, Josephine Sullivan, Hossein Azizpour, Marten Bjorkman
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:6573-6590, 2022.

Abstract

The number of linear regions has been studied as a proxy of complexity for ReLU networks. However, the empirical success of network compression techniques like pruning and knowledge distillation, suggest that in the overparameterized setting, linear regions density might fail to capture the effective nonlinearity. In this work, we propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-gamba22a, title = { Are All Linear Regions Created Equal? }, author = {Gamba, Matteo and Chmielewski-Anders, Adrian and Sullivan, Josephine and Azizpour, Hossein and Bjorkman, Marten}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {6573--6590}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/gamba22a/gamba22a.pdf}, url = {https://proceedings.mlr.press/v151/gamba22a.html}, abstract = { The number of linear regions has been studied as a proxy of complexity for ReLU networks. However, the empirical success of network compression techniques like pruning and knowledge distillation, suggest that in the overparameterized setting, linear regions density might fail to capture the effective nonlinearity. In this work, we propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions. } }
Endnote
%0 Conference Paper %T Are All Linear Regions Created Equal? %A Matteo Gamba %A Adrian Chmielewski-Anders %A Josephine Sullivan %A Hossein Azizpour %A Marten Bjorkman %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-gamba22a %I PMLR %P 6573--6590 %U https://proceedings.mlr.press/v151/gamba22a.html %V 151 %X The number of linear regions has been studied as a proxy of complexity for ReLU networks. However, the empirical success of network compression techniques like pruning and knowledge distillation, suggest that in the overparameterized setting, linear regions density might fail to capture the effective nonlinearity. In this work, we propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions.
APA
Gamba, M., Chmielewski-Anders, A., Sullivan, J., Azizpour, H. & Bjorkman, M.. (2022). Are All Linear Regions Created Equal? . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:6573-6590 Available from https://proceedings.mlr.press/v151/gamba22a.html.

Related Material