Understanding Geometry of Encoder-Decoder CNNs

Jong Chul Ye, Woon Kyoung Sung
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:7064-7073, 2019.

Abstract

Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems in computer vision, medical imaging, etc. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance. Inspired by recent theoretical understanding on generalizability, expressivity and optimization landscape of neural networks, as well as the theory of convolutional framelets, here we provide a unified theoretical framework that leads to a better understanding of geometry of encoder-decoder CNNs. Our unified mathematical framework shows that encoder-decoder CNN architecture is closely related to nonlinear basis representation using combinatorial convolution frames, whose expressibility increases exponentially with the network depth. We also demonstrate the importance of skipped connection in terms of expressibility, and optimization landscape.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-ye19a, title = {Understanding Geometry of Encoder-Decoder {CNN}s}, author = {Ye, Jong Chul and Sung, Woon Kyoung}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {7064--7073}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/ye19a/ye19a.pdf}, url = {https://proceedings.mlr.press/v97/ye19a.html}, abstract = {Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems in computer vision, medical imaging, etc. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance. Inspired by recent theoretical understanding on generalizability, expressivity and optimization landscape of neural networks, as well as the theory of convolutional framelets, here we provide a unified theoretical framework that leads to a better understanding of geometry of encoder-decoder CNNs. Our unified mathematical framework shows that encoder-decoder CNN architecture is closely related to nonlinear basis representation using combinatorial convolution frames, whose expressibility increases exponentially with the network depth. We also demonstrate the importance of skipped connection in terms of expressibility, and optimization landscape.} }
Endnote
%0 Conference Paper %T Understanding Geometry of Encoder-Decoder CNNs %A Jong Chul Ye %A Woon Kyoung Sung %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-ye19a %I PMLR %P 7064--7073 %U https://proceedings.mlr.press/v97/ye19a.html %V 97 %X Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems in computer vision, medical imaging, etc. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance. Inspired by recent theoretical understanding on generalizability, expressivity and optimization landscape of neural networks, as well as the theory of convolutional framelets, here we provide a unified theoretical framework that leads to a better understanding of geometry of encoder-decoder CNNs. Our unified mathematical framework shows that encoder-decoder CNN architecture is closely related to nonlinear basis representation using combinatorial convolution frames, whose expressibility increases exponentially with the network depth. We also demonstrate the importance of skipped connection in terms of expressibility, and optimization landscape.
APA
Ye, J.C. & Sung, W.K.. (2019). Understanding Geometry of Encoder-Decoder CNNs. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:7064-7073 Available from https://proceedings.mlr.press/v97/ye19a.html.

Related Material