Majority-of-Three: The Simplest Optimal Learner?

Ishaq Aden-Ali, Mikael Møller Høandgsgaard, Kasper Green Larsen, Nikita Zhivotovskiy
Proceedings of Thirty Seventh Conference on Learning Theory, PMLR 247:22-45, 2024.

Abstract

Developing an optimal PAC learning algorithm in the realizable setting, where empirical risk minimization (ERM) is suboptimal, was a major open problem in learning theory for decades. The problem was finally resolved by Hanneke a few years ago. Unfortunately, Hanneke’s algorithm is quite complex as it returns the majority vote of many ERM classifiers that are trained on carefully selected subsets of the data. It is thus a natural goal to determine the simplest algorithm that is optimal. In this work we study the arguably simplest algorithm that could be optimal: returning the majority vote of three ERM classifiers. We show that this algorithm achieves the optimal in-expectation bound on its error which is provably unattainable by a single ERM classifier. Furthermore, we prove a near-optimal high-probability bound on this algorithm’s error. We conjecture that a better analysis will prove that this algorithm is in fact optimal in the high-probability regime.

Cite this Paper

BibTeX
@InProceedings{pmlr-v247-aden-ali24a, title = {Majority-of-Three: The Simplest Optimal Learner?}, author = {Aden-Ali, Ishaq and H\o{}andgsgaard, Mikael M\o{}ller and Larsen, Kasper Green and Zhivotovskiy, Nikita}, booktitle = {Proceedings of Thirty Seventh Conference on Learning Theory}, pages = {22--45}, year = {2024}, editor = {Agrawal, Shipra and Roth, Aaron}, volume = {247}, series = {Proceedings of Machine Learning Research}, month = {30 Jun--03 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v247/aden-ali24a/aden-ali24a.pdf}, url = {https://proceedings.mlr.press/v247/aden-ali24a.html}, abstract = {Developing an optimal PAC learning algorithm in the realizable setting, where empirical risk minimization (ERM) is suboptimal, was a major open problem in learning theory for decades. The problem was finally resolved by Hanneke a few years ago. Unfortunately, Hanneke’s algorithm is quite complex as it returns the majority vote of many ERM classifiers that are trained on carefully selected subsets of the data. It is thus a natural goal to determine the simplest algorithm that is optimal. In this work we study the arguably simplest algorithm that could be optimal: returning the majority vote of three ERM classifiers. We show that this algorithm achieves the optimal in-expectation bound on its error which is provably unattainable by a single ERM classifier. Furthermore, we prove a near-optimal high-probability bound on this algorithm’s error. We conjecture that a better analysis will prove that this algorithm is in fact optimal in the high-probability regime.} }
Endnote
%0 Conference Paper %T Majority-of-Three: The Simplest Optimal Learner? %A Ishaq Aden-Ali %A Mikael Møller Høandgsgaard %A Kasper Green Larsen %A Nikita Zhivotovskiy %B Proceedings of Thirty Seventh Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2024 %E Shipra Agrawal %E Aaron Roth %F pmlr-v247-aden-ali24a %I PMLR %P 22--45 %U https://proceedings.mlr.press/v247/aden-ali24a.html %V 247 %X Developing an optimal PAC learning algorithm in the realizable setting, where empirical risk minimization (ERM) is suboptimal, was a major open problem in learning theory for decades. The problem was finally resolved by Hanneke a few years ago. Unfortunately, Hanneke’s algorithm is quite complex as it returns the majority vote of many ERM classifiers that are trained on carefully selected subsets of the data. It is thus a natural goal to determine the simplest algorithm that is optimal. In this work we study the arguably simplest algorithm that could be optimal: returning the majority vote of three ERM classifiers. We show that this algorithm achieves the optimal in-expectation bound on its error which is provably unattainable by a single ERM classifier. Furthermore, we prove a near-optimal high-probability bound on this algorithm’s error. We conjecture that a better analysis will prove that this algorithm is in fact optimal in the high-probability regime.
APA
Aden-Ali, I., Høandgsgaard, M.M., Larsen, K.G. & Zhivotovskiy, N.. (2024). Majority-of-Three: The Simplest Optimal Learner?. Proceedings of Thirty Seventh Conference on Learning Theory, in Proceedings of Machine Learning Research 247:22-45 Available from https://proceedings.mlr.press/v247/aden-ali24a.html.

Related Material