Failures of Gradient-Based Deep Learning

Shai Shalev-Shwartz, Ohad Shamir, Shaked Shammah
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3067-3075, 2017.

Abstract

In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art. However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms. We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties. We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-shalev-shwartz17a, title = {Failures of Gradient-Based Deep Learning}, author = {Shai Shalev-Shwartz and Ohad Shamir and Shaked Shammah}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {3067--3075}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/shalev-shwartz17a/shalev-shwartz17a.pdf}, url = {https://proceedings.mlr.press/v70/shalev-shwartz17a.html}, abstract = {In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art. However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms. We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties. We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.} }
Endnote
%0 Conference Paper %T Failures of Gradient-Based Deep Learning %A Shai Shalev-Shwartz %A Ohad Shamir %A Shaked Shammah %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-shalev-shwartz17a %I PMLR %P 3067--3075 %U https://proceedings.mlr.press/v70/shalev-shwartz17a.html %V 70 %X In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art. However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms. We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties. We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.
APA
Shalev-Shwartz, S., Shamir, O. & Shammah, S.. (2017). Failures of Gradient-Based Deep Learning. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3067-3075 Available from https://proceedings.mlr.press/v70/shalev-shwartz17a.html.

Related Material