Iterate Averaging as Regularization for Stochastic Gradient Descent
[edit]
Proceedings of the 31st Conference On Learning Theory, PMLR 75:32223242, 2018.
Abstract
We propose and analyze a variant of the classic Polyak–Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear leastsquares regression, we show that this averaging scheme has the same regularizing effect, and indeed is asymptotically equivalent, to ridge regression. In particular, we derive finitesample bounds for the proposed approach that match the best known results for regularized stochastic gradient methods.
Related Material


