Learning a Single Neuron with Gradient Methods

Gilad Yehudai, Shamir Ohad
; Proceedings of Thirty Third Conference on Learning Theory, PMLR 125:3756-3786, 2020.

Abstract

We consider the fundamental problem of learning a single neuron $\mathbf{x}\mapsto \sigma(\mathbf{w}^\top\mathbf{x})$ in a realizable setting, using standard gradient methods with random initialization, and under general families of input distributions and activations. On the one hand, we show that some assumptions on both the distribution and the activation function are necessary. On the other hand, we prove positive guarantees under mild assumptions, which go significantly beyond those studied in the literature so far. We also point out and study the challenges in further strengthening and generalizing our results.

Cite this Paper


BibTeX
@InProceedings{pmlr-v125-yehudai20a, title = {Learning a Single Neuron with Gradient Methods}, author = {Yehudai, Gilad and Ohad, Shamir}, pages = {3756--3786}, year = {2020}, editor = {Jacob Abernethy and Shivani Agarwal}, volume = {125}, series = {Proceedings of Machine Learning Research}, address = {}, month = {09--12 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v125/yehudai20a/yehudai20a.pdf}, url = {http://proceedings.mlr.press/v125/yehudai20a.html}, abstract = { We consider the fundamental problem of learning a single neuron $\mathbf{x}\mapsto \sigma(\mathbf{w}^\top\mathbf{x})$ in a realizable setting, using standard gradient methods with random initialization, and under general families of input distributions and activations. On the one hand, we show that some assumptions on both the distribution and the activation function are necessary. On the other hand, we prove positive guarantees under mild assumptions, which go significantly beyond those studied in the literature so far. We also point out and study the challenges in further strengthening and generalizing our results.} }
Endnote
%0 Conference Paper %T Learning a Single Neuron with Gradient Methods %A Gilad Yehudai %A Shamir Ohad %B Proceedings of Thirty Third Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2020 %E Jacob Abernethy %E Shivani Agarwal %F pmlr-v125-yehudai20a %I PMLR %J Proceedings of Machine Learning Research %P 3756--3786 %U http://proceedings.mlr.press %V 125 %W PMLR %X We consider the fundamental problem of learning a single neuron $\mathbf{x}\mapsto \sigma(\mathbf{w}^\top\mathbf{x})$ in a realizable setting, using standard gradient methods with random initialization, and under general families of input distributions and activations. On the one hand, we show that some assumptions on both the distribution and the activation function are necessary. On the other hand, we prove positive guarantees under mild assumptions, which go significantly beyond those studied in the literature so far. We also point out and study the challenges in further strengthening and generalizing our results.
APA
Yehudai, G. & Ohad, S.. (2020). Learning a Single Neuron with Gradient Methods. Proceedings of Thirty Third Conference on Learning Theory, in PMLR 125:3756-3786

Related Material