Demystifying Dropout

Hongchang Gao, Jian Pei, Heng Huang
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:2112-2121, 2019.

Abstract

Dropout is a popular technique to train large-scale deep neural networks to alleviate the overfitting problem. To disclose the underlying reasons for its gain, numerous works have tried to explain it from different perspectives. In this paper, unlike existing works, we explore it from a new perspective to provide new insight into this line of research. In detail, we disentangle the forward and backward pass of dropout. Then, we find that these two passes need different levels of noise to improve the generalization performance of deep neural networks. Based on this observation, we propose the augmented dropout which employs different dropping strategies in the forward and backward pass. Experimental results have verified the effectiveness of our proposed method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-gao19d, title = {Demystifying Dropout}, author = {Gao, Hongchang and Pei, Jian and Huang, Heng}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {2112--2121}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/gao19d/gao19d.pdf}, url = {https://proceedings.mlr.press/v97/gao19d.html}, abstract = {Dropout is a popular technique to train large-scale deep neural networks to alleviate the overfitting problem. To disclose the underlying reasons for its gain, numerous works have tried to explain it from different perspectives. In this paper, unlike existing works, we explore it from a new perspective to provide new insight into this line of research. In detail, we disentangle the forward and backward pass of dropout. Then, we find that these two passes need different levels of noise to improve the generalization performance of deep neural networks. Based on this observation, we propose the augmented dropout which employs different dropping strategies in the forward and backward pass. Experimental results have verified the effectiveness of our proposed method.} }
Endnote
%0 Conference Paper %T Demystifying Dropout %A Hongchang Gao %A Jian Pei %A Heng Huang %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-gao19d %I PMLR %P 2112--2121 %U https://proceedings.mlr.press/v97/gao19d.html %V 97 %X Dropout is a popular technique to train large-scale deep neural networks to alleviate the overfitting problem. To disclose the underlying reasons for its gain, numerous works have tried to explain it from different perspectives. In this paper, unlike existing works, we explore it from a new perspective to provide new insight into this line of research. In detail, we disentangle the forward and backward pass of dropout. Then, we find that these two passes need different levels of noise to improve the generalization performance of deep neural networks. Based on this observation, we propose the augmented dropout which employs different dropping strategies in the forward and backward pass. Experimental results have verified the effectiveness of our proposed method.
APA
Gao, H., Pei, J. & Huang, H.. (2019). Demystifying Dropout. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:2112-2121 Available from https://proceedings.mlr.press/v97/gao19d.html.

Related Material