Learning a Single Neuron with Gradient Methods

Gilad Yehudai, Shamir Ohad
Proceedings of Thirty Third Conference on Learning Theory, PMLR 125:3756-3786, 2020.

Abstract

We consider the fundamental problem of learning a single neuron $\mathbf{x}\mapsto \sigma(\mathbf{w}^\top\mathbf{x})$ in a realizable setting, using standard gradient methods with random initialization, and under general families of input distributions and activations. On the one hand, we show that some assumptions on both the distribution and the activation function are necessary. On the other hand, we prove positive guarantees under mild assumptions, which go significantly beyond those studied in the literature so far. We also point out and study the challenges in further strengthening and generalizing our results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v125-yehudai20a, title = {Learning a Single Neuron with Gradient Methods}, author = {Yehudai, Gilad and Ohad, Shamir}, booktitle = {Proceedings of Thirty Third Conference on Learning Theory}, pages = {3756--3786}, year = {2020}, editor = {Abernethy, Jacob and Agarwal, Shivani}, volume = {125}, series = {Proceedings of Machine Learning Research}, month = {09--12 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v125/yehudai20a/yehudai20a.pdf}, url = {https://proceedings.mlr.press/v125/yehudai20a.html}, abstract = { We consider the fundamental problem of learning a single neuron $\mathbf{x}\mapsto \sigma(\mathbf{w}^\top\mathbf{x})$ in a realizable setting, using standard gradient methods with random initialization, and under general families of input distributions and activations. On the one hand, we show that some assumptions on both the distribution and the activation function are necessary. On the other hand, we prove positive guarantees under mild assumptions, which go significantly beyond those studied in the literature so far. We also point out and study the challenges in further strengthening and generalizing our results.} }
Endnote
%0 Conference Paper %T Learning a Single Neuron with Gradient Methods %A Gilad Yehudai %A Shamir Ohad %B Proceedings of Thirty Third Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2020 %E Jacob Abernethy %E Shivani Agarwal %F pmlr-v125-yehudai20a %I PMLR %P 3756--3786 %U https://proceedings.mlr.press/v125/yehudai20a.html %V 125 %X We consider the fundamental problem of learning a single neuron $\mathbf{x}\mapsto \sigma(\mathbf{w}^\top\mathbf{x})$ in a realizable setting, using standard gradient methods with random initialization, and under general families of input distributions and activations. On the one hand, we show that some assumptions on both the distribution and the activation function are necessary. On the other hand, we prove positive guarantees under mild assumptions, which go significantly beyond those studied in the literature so far. We also point out and study the challenges in further strengthening and generalizing our results.
APA
Yehudai, G. & Ohad, S.. (2020). Learning a Single Neuron with Gradient Methods. Proceedings of Thirty Third Conference on Learning Theory, in Proceedings of Machine Learning Research 125:3756-3786 Available from https://proceedings.mlr.press/v125/yehudai20a.html.

Related Material