Fiedler Regularization: Learning Neural Networks with Graph Sparsity

Edric Tam, David Dunson
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:9346-9355, 2020.

Abstract

We introduce a novel regularization approach for deep learning that incorporates and respects the underlying graphical structure of the neural network. Existing regularization methods often focus on penalizing weights in a global/uniform manner that ignores the connectivity structure of the neural network. We propose to use the Fiedler value of the neural network’s underlying graph as a tool for regularization. We provide theoretical support for this approach via spectral graph theory. We show several useful properties of the Fiedler value that make it suitable for regularization. We provide an approximate, variational approach for faster computation during training. We provide an alternative formulation of this framework in the form of a structurally weighted L1 penalty, thus linking our approach to sparsity induction. We performed experiments on datasets that compare Fiedler regularization with traditional regularization methods such as Dropout and weight decay. Results demonstrate the efficacy of Fiedler regularization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-tam20a, title = {Fiedler Regularization: Learning Neural Networks with Graph Sparsity}, author = {Tam, Edric and Dunson, David}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {9346--9355}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/tam20a/tam20a.pdf}, url = {https://proceedings.mlr.press/v119/tam20a.html}, abstract = {We introduce a novel regularization approach for deep learning that incorporates and respects the underlying graphical structure of the neural network. Existing regularization methods often focus on penalizing weights in a global/uniform manner that ignores the connectivity structure of the neural network. We propose to use the Fiedler value of the neural network’s underlying graph as a tool for regularization. We provide theoretical support for this approach via spectral graph theory. We show several useful properties of the Fiedler value that make it suitable for regularization. We provide an approximate, variational approach for faster computation during training. We provide an alternative formulation of this framework in the form of a structurally weighted L1 penalty, thus linking our approach to sparsity induction. We performed experiments on datasets that compare Fiedler regularization with traditional regularization methods such as Dropout and weight decay. Results demonstrate the efficacy of Fiedler regularization.} }
Endnote
%0 Conference Paper %T Fiedler Regularization: Learning Neural Networks with Graph Sparsity %A Edric Tam %A David Dunson %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-tam20a %I PMLR %P 9346--9355 %U https://proceedings.mlr.press/v119/tam20a.html %V 119 %X We introduce a novel regularization approach for deep learning that incorporates and respects the underlying graphical structure of the neural network. Existing regularization methods often focus on penalizing weights in a global/uniform manner that ignores the connectivity structure of the neural network. We propose to use the Fiedler value of the neural network’s underlying graph as a tool for regularization. We provide theoretical support for this approach via spectral graph theory. We show several useful properties of the Fiedler value that make it suitable for regularization. We provide an approximate, variational approach for faster computation during training. We provide an alternative formulation of this framework in the form of a structurally weighted L1 penalty, thus linking our approach to sparsity induction. We performed experiments on datasets that compare Fiedler regularization with traditional regularization methods such as Dropout and weight decay. Results demonstrate the efficacy of Fiedler regularization.
APA
Tam, E. & Dunson, D.. (2020). Fiedler Regularization: Learning Neural Networks with Graph Sparsity. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:9346-9355 Available from https://proceedings.mlr.press/v119/tam20a.html.

Related Material