Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regions

Quynh Nguyen, Mahesh Chandra Mukkamala, Matthias Hein
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:3740-3749, 2018.

Abstract

In the recent literature the important role of depth in deep learning has been emphasized. In this paper we argue that sufficient width of a feedforward network is equally important by answering the simple question under which conditions the decision regions of a neural network are connected. It turns out that for a class of activation functions including leaky ReLU, neural networks having a pyramidal structure, that is no layer has more hidden units than the input dimension, produce necessarily connected decision regions. This implies that a sufficiently wide hidden layer is necessary to guarantee that the network can produce disconnected decision regions. We discuss the implications of this result for the construction of neural networks, in particular the relation to the problem of adversarial manipulation of classifiers.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-nguyen18b, title = {Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regions}, author = {Nguyen, Quynh and Mukkamala, Mahesh Chandra and Hein, Matthias}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {3740--3749}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/nguyen18b/nguyen18b.pdf}, url = {https://proceedings.mlr.press/v80/nguyen18b.html}, abstract = {In the recent literature the important role of depth in deep learning has been emphasized. In this paper we argue that sufficient width of a feedforward network is equally important by answering the simple question under which conditions the decision regions of a neural network are connected. It turns out that for a class of activation functions including leaky ReLU, neural networks having a pyramidal structure, that is no layer has more hidden units than the input dimension, produce necessarily connected decision regions. This implies that a sufficiently wide hidden layer is necessary to guarantee that the network can produce disconnected decision regions. We discuss the implications of this result for the construction of neural networks, in particular the relation to the problem of adversarial manipulation of classifiers.} }
Endnote
%0 Conference Paper %T Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regions %A Quynh Nguyen %A Mahesh Chandra Mukkamala %A Matthias Hein %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-nguyen18b %I PMLR %P 3740--3749 %U https://proceedings.mlr.press/v80/nguyen18b.html %V 80 %X In the recent literature the important role of depth in deep learning has been emphasized. In this paper we argue that sufficient width of a feedforward network is equally important by answering the simple question under which conditions the decision regions of a neural network are connected. It turns out that for a class of activation functions including leaky ReLU, neural networks having a pyramidal structure, that is no layer has more hidden units than the input dimension, produce necessarily connected decision regions. This implies that a sufficiently wide hidden layer is necessary to guarantee that the network can produce disconnected decision regions. We discuss the implications of this result for the construction of neural networks, in particular the relation to the problem of adversarial manipulation of classifiers.
APA
Nguyen, Q., Mukkamala, M.C. & Hein, M.. (2018). Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regions. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:3740-3749 Available from https://proceedings.mlr.press/v80/nguyen18b.html.

Related Material