Deep Isometric Learning for Visual Recognition

Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, Jitendra Malik
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7824-7835, 2020.

Abstract

Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance. This paper shows that deep vanilla ConvNets without normalization nor skip connections can also be trained to achieve surprisingly good performance on standard image recognition benchmarks. This is achieved by enforcing the convolution kernels to be near isometric during initialization and training, as well as by using a variant of ReLU that is shifted towards being isometric. Further experiments show that if combined with skip connections, such near isometric networks can achieve performances on par with (for ImageNet) and better than (for COCO) the standard ResNet, even without normalization at all. Our code is available at https://github.com/HaozhiQi/ISONet.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-qi20a, title = {Deep Isometric Learning for Visual Recognition}, author = {Qi, Haozhi and You, Chong and Wang, Xiaolong and Ma, Yi and Malik, Jitendra}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {7824--7835}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/qi20a/qi20a.pdf}, url = {https://proceedings.mlr.press/v119/qi20a.html}, abstract = {Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance. This paper shows that deep vanilla ConvNets without normalization nor skip connections can also be trained to achieve surprisingly good performance on standard image recognition benchmarks. This is achieved by enforcing the convolution kernels to be near isometric during initialization and training, as well as by using a variant of ReLU that is shifted towards being isometric. Further experiments show that if combined with skip connections, such near isometric networks can achieve performances on par with (for ImageNet) and better than (for COCO) the standard ResNet, even without normalization at all. Our code is available at https://github.com/HaozhiQi/ISONet.} }
Endnote
%0 Conference Paper %T Deep Isometric Learning for Visual Recognition %A Haozhi Qi %A Chong You %A Xiaolong Wang %A Yi Ma %A Jitendra Malik %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-qi20a %I PMLR %P 7824--7835 %U https://proceedings.mlr.press/v119/qi20a.html %V 119 %X Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance. This paper shows that deep vanilla ConvNets without normalization nor skip connections can also be trained to achieve surprisingly good performance on standard image recognition benchmarks. This is achieved by enforcing the convolution kernels to be near isometric during initialization and training, as well as by using a variant of ReLU that is shifted towards being isometric. Further experiments show that if combined with skip connections, such near isometric networks can achieve performances on par with (for ImageNet) and better than (for COCO) the standard ResNet, even without normalization at all. Our code is available at https://github.com/HaozhiQi/ISONet.
APA
Qi, H., You, C., Wang, X., Ma, Y. & Malik, J.. (2020). Deep Isometric Learning for Visual Recognition. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:7824-7835 Available from https://proceedings.mlr.press/v119/qi20a.html.

Related Material