Maximum Margin Multiclass Nearest Neighbors

Aryeh Kontorovich, Roi Weiss
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):892-900, 2014.

Abstract

We develop a general framework for margin-based multicategory classification in metric spaces. The basic work-horse is a margin-regularized version of the nearest-neighbor classifier. We prove generalization bounds that match the state of the art in sample size n and significantly improve the dependence on the number of classes k. Our point of departure is a nearly Bayes-optimal finite-sample risk bound independent of k. Although k-free, this bound is unregularized and non-adaptive, which motivates our main result: Rademacher and scale-sensitive margin bounds with a logarithmic dependence on k. As the best previous risk estimates in this setting were of order \sqrt k, our bound is exponentially sharper. From the algorithmic standpoint, in doubling metric spaces our classifier may be trained on n examples in O(n^2\log n) time and evaluated on new points in O(\log n) time.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-kontorovichb14, title = {Maximum Margin Multiclass Nearest Neighbors}, author = {Kontorovich, Aryeh and Weiss, Roi}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {892--900}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/kontorovichb14.pdf}, url = {https://proceedings.mlr.press/v32/kontorovichb14.html}, abstract = {We develop a general framework for margin-based multicategory classification in metric spaces. The basic work-horse is a margin-regularized version of the nearest-neighbor classifier. We prove generalization bounds that match the state of the art in sample size n and significantly improve the dependence on the number of classes k. Our point of departure is a nearly Bayes-optimal finite-sample risk bound independent of k. Although k-free, this bound is unregularized and non-adaptive, which motivates our main result: Rademacher and scale-sensitive margin bounds with a logarithmic dependence on k. As the best previous risk estimates in this setting were of order \sqrt k, our bound is exponentially sharper. From the algorithmic standpoint, in doubling metric spaces our classifier may be trained on n examples in O(n^2\log n) time and evaluated on new points in O(\log n) time.} }
Endnote
%0 Conference Paper %T Maximum Margin Multiclass Nearest Neighbors %A Aryeh Kontorovich %A Roi Weiss %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-kontorovichb14 %I PMLR %P 892--900 %U https://proceedings.mlr.press/v32/kontorovichb14.html %V 32 %N 2 %X We develop a general framework for margin-based multicategory classification in metric spaces. The basic work-horse is a margin-regularized version of the nearest-neighbor classifier. We prove generalization bounds that match the state of the art in sample size n and significantly improve the dependence on the number of classes k. Our point of departure is a nearly Bayes-optimal finite-sample risk bound independent of k. Although k-free, this bound is unregularized and non-adaptive, which motivates our main result: Rademacher and scale-sensitive margin bounds with a logarithmic dependence on k. As the best previous risk estimates in this setting were of order \sqrt k, our bound is exponentially sharper. From the algorithmic standpoint, in doubling metric spaces our classifier may be trained on n examples in O(n^2\log n) time and evaluated on new points in O(\log n) time.
RIS
TY - CPAPER TI - Maximum Margin Multiclass Nearest Neighbors AU - Aryeh Kontorovich AU - Roi Weiss BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/06/18 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-kontorovichb14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 2 SP - 892 EP - 900 L1 - http://proceedings.mlr.press/v32/kontorovichb14.pdf UR - https://proceedings.mlr.press/v32/kontorovichb14.html AB - We develop a general framework for margin-based multicategory classification in metric spaces. The basic work-horse is a margin-regularized version of the nearest-neighbor classifier. We prove generalization bounds that match the state of the art in sample size n and significantly improve the dependence on the number of classes k. Our point of departure is a nearly Bayes-optimal finite-sample risk bound independent of k. Although k-free, this bound is unregularized and non-adaptive, which motivates our main result: Rademacher and scale-sensitive margin bounds with a logarithmic dependence on k. As the best previous risk estimates in this setting were of order \sqrt k, our bound is exponentially sharper. From the algorithmic standpoint, in doubling metric spaces our classifier may be trained on n examples in O(n^2\log n) time and evaluated on new points in O(\log n) time. ER -
APA
Kontorovich, A. & Weiss, R.. (2014). Maximum Margin Multiclass Nearest Neighbors. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(2):892-900 Available from https://proceedings.mlr.press/v32/kontorovichb14.html.

Related Material