Dropout: Explicit Forms and Capacity Control

Raman Arora, Peter Bartlett, Poorya Mianjy, Nathan Srebro
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:351-361, 2021.

Abstract

We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a distribution-dependent regularizer that equals the weighted trace-norm of the product of the factors. In deep learning, we show that the distribution-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-arora21a, title = {Dropout: Explicit Forms and Capacity Control}, author = {Arora, Raman and Bartlett, Peter and Mianjy, Poorya and Srebro, Nathan}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {351--361}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/arora21a/arora21a.pdf}, url = {https://proceedings.mlr.press/v139/arora21a.html}, abstract = {We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a distribution-dependent regularizer that equals the weighted trace-norm of the product of the factors. In deep learning, we show that the distribution-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks.} }
Endnote
%0 Conference Paper %T Dropout: Explicit Forms and Capacity Control %A Raman Arora %A Peter Bartlett %A Poorya Mianjy %A Nathan Srebro %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-arora21a %I PMLR %P 351--361 %U https://proceedings.mlr.press/v139/arora21a.html %V 139 %X We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a distribution-dependent regularizer that equals the weighted trace-norm of the product of the factors. In deep learning, we show that the distribution-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks.
APA
Arora, R., Bartlett, P., Mianjy, P. & Srebro, N.. (2021). Dropout: Explicit Forms and Capacity Control. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:351-361 Available from https://proceedings.mlr.press/v139/arora21a.html.

Related Material