Optimal multiclass overfitting by sequence reconstruction from Hamming queries

Jayadev Acharya, Ananda Theertha Suresh
Proceedings of the 31st International Conference on Algorithmic Learning Theory, PMLR 117:3-21, 2020.

Abstract

A primary concern of excessive reuse of test datasets in machine learning is that it can lead to overfitting. Multiclass classification was recently shown to be more resistant to overfitting than binary classification. In an open problem of COLT 2019, Feldman, Frostig, and Hardt ask to characterize the dependence of the amount of overfitting bias with the number of classes $m$, the number of accuracy queries $k$, and the number of examples in the dataset $n$. We resolve this problem and determine the amount of overfitting possible in multi-class classification. We provide computationally efficient algorithms that achieve overfitting bias of $\tilde{\Theta}(\max\{\sqrt{{k}/{(mn)}}, k/n\})$, matching the known upper bounds.

Cite this Paper

BibTeX
@InProceedings{pmlr-v117-acharya20a, title = {Optimal multiclass overfitting by sequence reconstruction from Hamming queries}, author = {Acharya, Jayadev and Suresh, Ananda Theertha}, booktitle = {Proceedings of the 31st International Conference on Algorithmic Learning Theory}, pages = {3--21}, year = {2020}, editor = {Kontorovich, Aryeh and Neu, Gergely}, volume = {117}, series = {Proceedings of Machine Learning Research}, month = {08 Feb--11 Feb}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v117/acharya20a/acharya20a.pdf}, url = {https://proceedings.mlr.press/v117/acharya20a.html}, abstract = {A primary concern of excessive reuse of test datasets in machine learning is that it can lead to overfitting. Multiclass classification was recently shown to be more resistant to overfitting than binary classification. In an open problem of COLT 2019, Feldman, Frostig, and Hardt ask to characterize the dependence of the amount of overfitting bias with the number of classes $m$, the number of accuracy queries $k$, and the number of examples in the dataset $n$. We resolve this problem and determine the amount of overfitting possible in multi-class classification. We provide computationally efficient algorithms that achieve overfitting bias of $\tilde{\Theta}(\max\{\sqrt{{k}/{(mn)}}, k/n\})$, matching the known upper bounds.} }
Endnote
%0 Conference Paper %T Optimal multiclass overfitting by sequence reconstruction from Hamming queries %A Jayadev Acharya %A Ananda Theertha Suresh %B Proceedings of the 31st International Conference on Algorithmic Learning Theory %C Proceedings of Machine Learning Research %D 2020 %E Aryeh Kontorovich %E Gergely Neu %F pmlr-v117-acharya20a %I PMLR %P 3--21 %U https://proceedings.mlr.press/v117/acharya20a.html %V 117 %X A primary concern of excessive reuse of test datasets in machine learning is that it can lead to overfitting. Multiclass classification was recently shown to be more resistant to overfitting than binary classification. In an open problem of COLT 2019, Feldman, Frostig, and Hardt ask to characterize the dependence of the amount of overfitting bias with the number of classes $m$, the number of accuracy queries $k$, and the number of examples in the dataset $n$. We resolve this problem and determine the amount of overfitting possible in multi-class classification. We provide computationally efficient algorithms that achieve overfitting bias of $\tilde{\Theta}(\max\{\sqrt{{k}/{(mn)}}, k/n\})$, matching the known upper bounds.
APA
Acharya, J. & Suresh, A.T.. (2020). Optimal multiclass overfitting by sequence reconstruction from Hamming queries. Proceedings of the 31st International Conference on Algorithmic Learning Theory, in Proceedings of Machine Learning Research 117:3-21 Available from https://proceedings.mlr.press/v117/acharya20a.html.

Related Material