Maximum Margin Multiclass Nearest Neighbors


Aryeh Kontorovich, Roi Weiss ;
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):892-900, 2014.


We develop a general framework for margin-based multicategory classification in metric spaces. The basic work-horse is a margin-regularized version of the nearest-neighbor classifier. We prove generalization bounds that match the state of the art in sample size n and significantly improve the dependence on the number of classes k. Our point of departure is a nearly Bayes-optimal finite-sample risk bound independent of k. Although k-free, this bound is unregularized and non-adaptive, which motivates our main result: Rademacher and scale-sensitive margin bounds with a logarithmic dependence on k. As the best previous risk estimates in this setting were of order \sqrt k, our bound is exponentially sharper. From the algorithmic standpoint, in doubling metric spaces our classifier may be trained on n examples in O(n^2\log n) time and evaluated on new points in O(\log n) time.

Related Material