Role Taxonomy of Units in Deep Neural Networks

Yang Zhao, Hao Zhang, Xiuyuan Hu
Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, PMLR 243:291-301, 2024.

Abstract

Identifying the role of network units in deep neural networks (DNNs) is critical in many aspects including giving understandings on the mechanisms of DNNs and building basic connections between deep learning and neuroscience. However, there remains unclear on which roles the units in DNNs with different generalization ability could present. To this end, we give role taxonomy of units in DNNs, where units are categorized into four types in terms of their functional preference on separately the training set and testing set. We show that ratios of the four categories are highly associated with the generalization ability of DNNs from two distinct perspectives, based on which we give signs of DNNs with well generalization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v243-zhao24b, title = {Role Taxonomy of Units in Deep Neural Networks}, author = {Zhao, Yang and Zhang, Hao and Hu, Xiuyuan}, booktitle = {Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models}, pages = {291--301}, year = {2024}, editor = {Fumero, Marco and Rodolá, Emanuele and Domine, Clementine and Locatello, Francesco and Dziugaite, Karolina and Mathilde, Caron}, volume = {243}, series = {Proceedings of Machine Learning Research}, month = {15 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v243/zhao24b/zhao24b.pdf}, url = {https://proceedings.mlr.press/v243/zhao24b.html}, abstract = {Identifying the role of network units in deep neural networks (DNNs) is critical in many aspects including giving understandings on the mechanisms of DNNs and building basic connections between deep learning and neuroscience. However, there remains unclear on which roles the units in DNNs with different generalization ability could present. To this end, we give role taxonomy of units in DNNs, where units are categorized into four types in terms of their functional preference on separately the training set and testing set. We show that ratios of the four categories are highly associated with the generalization ability of DNNs from two distinct perspectives, based on which we give signs of DNNs with well generalization.} }
Endnote
%0 Conference Paper %T Role Taxonomy of Units in Deep Neural Networks %A Yang Zhao %A Hao Zhang %A Xiuyuan Hu %B Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models %C Proceedings of Machine Learning Research %D 2024 %E Marco Fumero %E Emanuele Rodolá %E Clementine Domine %E Francesco Locatello %E Karolina Dziugaite %E Caron Mathilde %F pmlr-v243-zhao24b %I PMLR %P 291--301 %U https://proceedings.mlr.press/v243/zhao24b.html %V 243 %X Identifying the role of network units in deep neural networks (DNNs) is critical in many aspects including giving understandings on the mechanisms of DNNs and building basic connections between deep learning and neuroscience. However, there remains unclear on which roles the units in DNNs with different generalization ability could present. To this end, we give role taxonomy of units in DNNs, where units are categorized into four types in terms of their functional preference on separately the training set and testing set. We show that ratios of the four categories are highly associated with the generalization ability of DNNs from two distinct perspectives, based on which we give signs of DNNs with well generalization.
APA
Zhao, Y., Zhang, H. & Hu, X.. (2024). Role Taxonomy of Units in Deep Neural Networks. Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, in Proceedings of Machine Learning Research 243:291-301 Available from https://proceedings.mlr.press/v243/zhao24b.html.

Related Material