ResNet and Batch-normalization Improve Data Separability

Yasutaka Furusho, Kazushi Ikeda
Proceedings of The Eleventh Asian Conference on Machine Learning, PMLR 101:94-108, 2019.

Abstract

The skip-connection and the batch-normalization (BN) in ResNet enable an extreme deep neural network to be trained with high performance. However, the reasons for its high performance are still unclear. To clear that, we study the effects of the skip-connection and the BN on the class-related signal propagation through hidden layers because a large ratio of the between-class distance to the within-class distance of feature vectors at the last hidden layer induces high performance. Our result shows that the between-class distance and the within-class distance change differently through layers: the deep multilayer perceptron with randomly initialized weights degrades the ratio of the between-class distance to the within-class distance and the skip-connection and the BN relax this degradation. Moreover, our analysis implies that the skip-connection and the BN encourage training to improve this distance ratio. These results imply that the skip-connection and the BN induce high performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v101-furusho19a, title = {ResNet and Batch-normalization Improve Data Separability}, author = {Furusho, Yasutaka and Ikeda, Kazushi}, booktitle = {Proceedings of The Eleventh Asian Conference on Machine Learning}, pages = {94--108}, year = {2019}, editor = {Lee, Wee Sun and Suzuki, Taiji}, volume = {101}, series = {Proceedings of Machine Learning Research}, month = {17--19 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v101/furusho19a/furusho19a.pdf}, url = {https://proceedings.mlr.press/v101/furusho19a.html}, abstract = {The skip-connection and the batch-normalization (BN) in ResNet enable an extreme deep neural network to be trained with high performance. However, the reasons for its high performance are still unclear. To clear that, we study the effects of the skip-connection and the BN on the class-related signal propagation through hidden layers because a large ratio of the between-class distance to the within-class distance of feature vectors at the last hidden layer induces high performance. Our result shows that the between-class distance and the within-class distance change differently through layers: the deep multilayer perceptron with randomly initialized weights degrades the ratio of the between-class distance to the within-class distance and the skip-connection and the BN relax this degradation. Moreover, our analysis implies that the skip-connection and the BN encourage training to improve this distance ratio. These results imply that the skip-connection and the BN induce high performance.} }
Endnote
%0 Conference Paper %T ResNet and Batch-normalization Improve Data Separability %A Yasutaka Furusho %A Kazushi Ikeda %B Proceedings of The Eleventh Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Wee Sun Lee %E Taiji Suzuki %F pmlr-v101-furusho19a %I PMLR %P 94--108 %U https://proceedings.mlr.press/v101/furusho19a.html %V 101 %X The skip-connection and the batch-normalization (BN) in ResNet enable an extreme deep neural network to be trained with high performance. However, the reasons for its high performance are still unclear. To clear that, we study the effects of the skip-connection and the BN on the class-related signal propagation through hidden layers because a large ratio of the between-class distance to the within-class distance of feature vectors at the last hidden layer induces high performance. Our result shows that the between-class distance and the within-class distance change differently through layers: the deep multilayer perceptron with randomly initialized weights degrades the ratio of the between-class distance to the within-class distance and the skip-connection and the BN relax this degradation. Moreover, our analysis implies that the skip-connection and the BN encourage training to improve this distance ratio. These results imply that the skip-connection and the BN induce high performance.
APA
Furusho, Y. & Ikeda, K.. (2019). ResNet and Batch-normalization Improve Data Separability. Proceedings of The Eleventh Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 101:94-108 Available from https://proceedings.mlr.press/v101/furusho19a.html.

Related Material