Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs

Jie Ren, Mingjie Li, Meng Zhou, Shih-Han Chan, Quanshi Zhang
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:18537-18558, 2022.

Abstract

This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN’s complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN. The code is released at https://github.com/sjtu-XAI-lab/transformation-complexity.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-ren22b, title = {Towards Theoretical Analysis of Transformation Complexity of {R}e{LU} {DNN}s}, author = {Ren, Jie and Li, Mingjie and Zhou, Meng and Chan, Shih-Han and Zhang, Quanshi}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {18537--18558}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/ren22b/ren22b.pdf}, url = {https://proceedings.mlr.press/v162/ren22b.html}, abstract = {This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN’s complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN. The code is released at https://github.com/sjtu-XAI-lab/transformation-complexity.} }
Endnote
%0 Conference Paper %T Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs %A Jie Ren %A Mingjie Li %A Meng Zhou %A Shih-Han Chan %A Quanshi Zhang %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-ren22b %I PMLR %P 18537--18558 %U https://proceedings.mlr.press/v162/ren22b.html %V 162 %X This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN’s complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN. The code is released at https://github.com/sjtu-XAI-lab/transformation-complexity.
APA
Ren, J., Li, M., Zhou, M., Chan, S. & Zhang, Q.. (2022). Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:18537-18558 Available from https://proceedings.mlr.press/v162/ren22b.html.

Related Material