Optimal rates for stochastic convex optimization under Tsybakov noise condition

Aaditya Ramdas, Aarti Singh
Proceedings of the 30th International Conference on Machine Learning, PMLR 28(1):365-373, 2013.

Abstract

We focus on the problem of minimizing a convex function f over a convex set S given T queries to a stochastic first order oracle. We argue that the complexity of convex minimization is only determined by the rate of growth of the function around its minimum x^*_f,S, as quantified by a Tsybakov-like noise condition. Specifically, we prove that if f grows at least as fast as \|x-x^*_f,S\|^κaround its minimum, for some κ> 1, then the optimal rate of learning f(x^*_f,S) is Θ(T^-\fracκ2κ-2). The classic rate Θ(1/\sqrt T) for convex functions and Θ(1/T) for strongly convex functions are special cases of our result for κ→∞and κ=2, and even faster rates are attained for 1 < κ< 2. We also derive tight bounds for the complexity of learning x_f,S^*, where the optimal rate is Θ(T^-\frac12κ-2). Interestingly, these precise rates also characterize the complexity of active learning and our results further strengthen the connections between the fields of active learning and convex optimization, both of which rely on feedback-driven queries.

Cite this Paper

BibTeX
@InProceedings{pmlr-v28-ramdas13, title = {Optimal rates for stochastic convex optimization under Tsybakov noise condition}, author = {Ramdas, Aaditya and Singh, Aarti}, booktitle = {Proceedings of the 30th International Conference on Machine Learning}, pages = {365--373}, year = {2013}, editor = {Dasgupta, Sanjoy and McAllester, David}, volume = {28}, number = {1}, series = {Proceedings of Machine Learning Research}, address = {Atlanta, Georgia, USA}, month = {17--19 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v28/ramdas13.pdf}, url = {https://proceedings.mlr.press/v28/ramdas13.html}, abstract = {We focus on the problem of minimizing a convex function f over a convex set S given T queries to a stochastic first order oracle. We argue that the complexity of convex minimization is only determined by the rate of growth of the function around its minimum x^*_f,S, as quantified by a Tsybakov-like noise condition. Specifically, we prove that if f grows at least as fast as \|x-x^*_f,S\|^κaround its minimum, for some κ> 1, then the optimal rate of learning f(x^*_f,S) is Θ(T^-\fracκ2κ-2). The classic rate Θ(1/\sqrt T) for convex functions and Θ(1/T) for strongly convex functions are special cases of our result for κ→∞and κ=2, and even faster rates are attained for 1 < κ< 2. We also derive tight bounds for the complexity of learning x_f,S^*, where the optimal rate is Θ(T^-\frac12κ-2). Interestingly, these precise rates also characterize the complexity of active learning and our results further strengthen the connections between the fields of active learning and convex optimization, both of which rely on feedback-driven queries.} }
Endnote
%0 Conference Paper %T Optimal rates for stochastic convex optimization under Tsybakov noise condition %A Aaditya Ramdas %A Aarti Singh %B Proceedings of the 30th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2013 %E Sanjoy Dasgupta %E David McAllester %F pmlr-v28-ramdas13 %I PMLR %P 365--373 %U https://proceedings.mlr.press/v28/ramdas13.html %V 28 %N 1 %X We focus on the problem of minimizing a convex function f over a convex set S given T queries to a stochastic first order oracle. We argue that the complexity of convex minimization is only determined by the rate of growth of the function around its minimum x^*_f,S, as quantified by a Tsybakov-like noise condition. Specifically, we prove that if f grows at least as fast as \|x-x^*_f,S\|^κaround its minimum, for some κ> 1, then the optimal rate of learning f(x^*_f,S) is Θ(T^-\fracκ2κ-2). The classic rate Θ(1/\sqrt T) for convex functions and Θ(1/T) for strongly convex functions are special cases of our result for κ→∞and κ=2, and even faster rates are attained for 1 < κ< 2. We also derive tight bounds for the complexity of learning x_f,S^*, where the optimal rate is Θ(T^-\frac12κ-2). Interestingly, these precise rates also characterize the complexity of active learning and our results further strengthen the connections between the fields of active learning and convex optimization, both of which rely on feedback-driven queries.
RIS
TY - CPAPER TI - Optimal rates for stochastic convex optimization under Tsybakov noise condition AU - Aaditya Ramdas AU - Aarti Singh BT - Proceedings of the 30th International Conference on Machine Learning DA - 2013/02/13 ED - Sanjoy Dasgupta ED - David McAllester ID - pmlr-v28-ramdas13 PB - PMLR DP - Proceedings of Machine Learning Research VL - 28 IS - 1 SP - 365 EP - 373 L1 - http://proceedings.mlr.press/v28/ramdas13.pdf UR - https://proceedings.mlr.press/v28/ramdas13.html AB - We focus on the problem of minimizing a convex function f over a convex set S given T queries to a stochastic first order oracle. We argue that the complexity of convex minimization is only determined by the rate of growth of the function around its minimum x^*_f,S, as quantified by a Tsybakov-like noise condition. Specifically, we prove that if f grows at least as fast as \|x-x^*_f,S\|^κaround its minimum, for some κ> 1, then the optimal rate of learning f(x^*_f,S) is Θ(T^-\fracκ2κ-2). The classic rate Θ(1/\sqrt T) for convex functions and Θ(1/T) for strongly convex functions are special cases of our result for κ→∞and κ=2, and even faster rates are attained for 1 < κ< 2. We also derive tight bounds for the complexity of learning x_f,S^*, where the optimal rate is Θ(T^-\frac12κ-2). Interestingly, these precise rates also characterize the complexity of active learning and our results further strengthen the connections between the fields of active learning and convex optimization, both of which rely on feedback-driven queries. ER -
APA
Ramdas, A. & Singh, A.. (2013). Optimal rates for stochastic convex optimization under Tsybakov noise condition. Proceedings of the 30th International Conference on Machine Learning, in Proceedings of Machine Learning Research 28(1):365-373 Available from https://proceedings.mlr.press/v28/ramdas13.html.

Related Material