Can Infinitely Wide Deep Nets Help Small-data Multi-label Learning?

Guoqiang Wu, Jun Zhu
Proceedings of the 15th Asian Conference on Machine Learning, PMLR 222:1494-1509, 2024.

Abstract

In Multi-label Learning (MLL), kernel methods and deep neural networks (DNNs) are two typical families of approaches. Recent theory discovers an interesting connection between infinitely wide DNNs and neural tangent kernel (NTK) based methods. Further, recent work has shown the promising performance of NTK-based methods in \emph{small-data single-labeled tasks}. Then, a natural question arises: can infinitely wide DNNs help small-data multi-label learning? To answer this question, in this paper, we present to utilize infinitely wide DNNs for the MLL task. Specifically, we propose an NTK-based kernel method for MLL, which aims to minimize Hamming and ranking loss simultaneously. Moreover, to efficiently train the model, we use the Nystr{ö}m method, which has rarely been used in MLL. Further, we give rigorous theoretical analyses on learning guarantees of the proposed algorithm w.r.t. these two measures. Finally, empirical results on small-scale datasets illustrate its superior performance along with efficiency over several related baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v222-wu24b, title = {Can Infinitely Wide Deep Nets Help Small-data Multi-label Learning?}, author = {Wu, Guoqiang and Zhu, Jun}, booktitle = {Proceedings of the 15th Asian Conference on Machine Learning}, pages = {1494--1509}, year = {2024}, editor = {Yanıkoğlu, Berrin and Buntine, Wray}, volume = {222}, series = {Proceedings of Machine Learning Research}, month = {11--14 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v222/wu24b/wu24b.pdf}, url = {https://proceedings.mlr.press/v222/wu24b.html}, abstract = {In Multi-label Learning (MLL), kernel methods and deep neural networks (DNNs) are two typical families of approaches. Recent theory discovers an interesting connection between infinitely wide DNNs and neural tangent kernel (NTK) based methods. Further, recent work has shown the promising performance of NTK-based methods in \emph{small-data single-labeled tasks}. Then, a natural question arises: can infinitely wide DNNs help small-data multi-label learning? To answer this question, in this paper, we present to utilize infinitely wide DNNs for the MLL task. Specifically, we propose an NTK-based kernel method for MLL, which aims to minimize Hamming and ranking loss simultaneously. Moreover, to efficiently train the model, we use the Nystr{ö}m method, which has rarely been used in MLL. Further, we give rigorous theoretical analyses on learning guarantees of the proposed algorithm w.r.t. these two measures. Finally, empirical results on small-scale datasets illustrate its superior performance along with efficiency over several related baselines.} }
Endnote
%0 Conference Paper %T Can Infinitely Wide Deep Nets Help Small-data Multi-label Learning? %A Guoqiang Wu %A Jun Zhu %B Proceedings of the 15th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Berrin Yanıkoğlu %E Wray Buntine %F pmlr-v222-wu24b %I PMLR %P 1494--1509 %U https://proceedings.mlr.press/v222/wu24b.html %V 222 %X In Multi-label Learning (MLL), kernel methods and deep neural networks (DNNs) are two typical families of approaches. Recent theory discovers an interesting connection between infinitely wide DNNs and neural tangent kernel (NTK) based methods. Further, recent work has shown the promising performance of NTK-based methods in \emph{small-data single-labeled tasks}. Then, a natural question arises: can infinitely wide DNNs help small-data multi-label learning? To answer this question, in this paper, we present to utilize infinitely wide DNNs for the MLL task. Specifically, we propose an NTK-based kernel method for MLL, which aims to minimize Hamming and ranking loss simultaneously. Moreover, to efficiently train the model, we use the Nystr{ö}m method, which has rarely been used in MLL. Further, we give rigorous theoretical analyses on learning guarantees of the proposed algorithm w.r.t. these two measures. Finally, empirical results on small-scale datasets illustrate its superior performance along with efficiency over several related baselines.
APA
Wu, G. & Zhu, J.. (2024). Can Infinitely Wide Deep Nets Help Small-data Multi-label Learning?. Proceedings of the 15th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 222:1494-1509 Available from https://proceedings.mlr.press/v222/wu24b.html.

Related Material