Online Non-Parametric Regression

Alexander Rakhlin, Karthik Sridharan
Proceedings of The 27th Conference on Learning Theory, PMLR 35:1232-1264, 2014.

Abstract

We establish optimal rates for online regression for arbitrary classes of regression functions in terms of the sequential entropy introduced in (Rakhlin et al., 2010). The optimal rates are shown to exhibit a phase transition analogous to the i.i.d./statistical learning case, studied in (Rakhlin et al., 2014b). In the frequently encountered situation when sequential entropy and i.i.d. empirical entropy match, our results point to the interesting phenomenon that the rates for statistical learning with squared loss and online nonparametric regression are the same. In addition to a non-algorithmic study of minimax regret, we exhibit a generic forecaster that enjoys the established optimal rates. We also provide a recipe for designing online regression algorithms that can be computationally efficient. We illustrate the techniques by deriving existing and new forecasters for the case of finite experts and for online linear regression.

Cite this Paper

BibTeX
@InProceedings{pmlr-v35-rakhlin14, title = {Online Non-Parametric Regression}, author = {Rakhlin, Alexander and Sridharan, Karthik}, booktitle = {Proceedings of The 27th Conference on Learning Theory}, pages = {1232--1264}, year = {2014}, editor = {Balcan, Maria Florina and Feldman, Vitaly and Szepesvári, Csaba}, volume = {35}, series = {Proceedings of Machine Learning Research}, address = {Barcelona, Spain}, month = {13--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v35/rakhlin14.pdf}, url = {https://proceedings.mlr.press/v35/rakhlin14.html}, abstract = {We establish optimal rates for online regression for arbitrary classes of regression functions in terms of the sequential entropy introduced in (Rakhlin et al., 2010). The optimal rates are shown to exhibit a phase transition analogous to the i.i.d./statistical learning case, studied in (Rakhlin et al., 2014b). In the frequently encountered situation when sequential entropy and i.i.d. empirical entropy match, our results point to the interesting phenomenon that the rates for statistical learning with squared loss and online nonparametric regression are the same. In addition to a non-algorithmic study of minimax regret, we exhibit a generic forecaster that enjoys the established optimal rates. We also provide a recipe for designing online regression algorithms that can be computationally efficient. We illustrate the techniques by deriving existing and new forecasters for the case of finite experts and for online linear regression.} }
Endnote
%0 Conference Paper %T Online Non-Parametric Regression %A Alexander Rakhlin %A Karthik Sridharan %B Proceedings of The 27th Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2014 %E Maria Florina Balcan %E Vitaly Feldman %E Csaba Szepesvári %F pmlr-v35-rakhlin14 %I PMLR %P 1232--1264 %U https://proceedings.mlr.press/v35/rakhlin14.html %V 35 %X We establish optimal rates for online regression for arbitrary classes of regression functions in terms of the sequential entropy introduced in (Rakhlin et al., 2010). The optimal rates are shown to exhibit a phase transition analogous to the i.i.d./statistical learning case, studied in (Rakhlin et al., 2014b). In the frequently encountered situation when sequential entropy and i.i.d. empirical entropy match, our results point to the interesting phenomenon that the rates for statistical learning with squared loss and online nonparametric regression are the same. In addition to a non-algorithmic study of minimax regret, we exhibit a generic forecaster that enjoys the established optimal rates. We also provide a recipe for designing online regression algorithms that can be computationally efficient. We illustrate the techniques by deriving existing and new forecasters for the case of finite experts and for online linear regression.
RIS
TY - CPAPER TI - Online Non-Parametric Regression AU - Alexander Rakhlin AU - Karthik Sridharan BT - Proceedings of The 27th Conference on Learning Theory DA - 2014/05/29 ED - Maria Florina Balcan ED - Vitaly Feldman ED - Csaba Szepesvári ID - pmlr-v35-rakhlin14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 35 SP - 1232 EP - 1264 L1 - http://proceedings.mlr.press/v35/rakhlin14.pdf UR - https://proceedings.mlr.press/v35/rakhlin14.html AB - We establish optimal rates for online regression for arbitrary classes of regression functions in terms of the sequential entropy introduced in (Rakhlin et al., 2010). The optimal rates are shown to exhibit a phase transition analogous to the i.i.d./statistical learning case, studied in (Rakhlin et al., 2014b). In the frequently encountered situation when sequential entropy and i.i.d. empirical entropy match, our results point to the interesting phenomenon that the rates for statistical learning with squared loss and online nonparametric regression are the same. In addition to a non-algorithmic study of minimax regret, we exhibit a generic forecaster that enjoys the established optimal rates. We also provide a recipe for designing online regression algorithms that can be computationally efficient. We illustrate the techniques by deriving existing and new forecasters for the case of finite experts and for online linear regression. ER -
APA
Rakhlin, A. & Sridharan, K.. (2014). Online Non-Parametric Regression. Proceedings of The 27th Conference on Learning Theory, in Proceedings of Machine Learning Research 35:1232-1264 Available from https://proceedings.mlr.press/v35/rakhlin14.html.

Related Material