Q-Match: Self-Supervised Learning by Matching Distributions Induced by a Queue

Thomas Mulc, Debidatta Dwibedi
Proceedings of the 15th Asian Conference on Machine Learning, PMLR 222:911-926, 2024.

Abstract

In semi-supervised learning, student-teacher distribution matching has been successful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We introduce our algorithm, Q-Match, and show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We focus our study on tabular datasets and show that Q-Match outperforms previous self-supervised learning techniques when measuring downstream classification performance. Furthermore, we show that our method is sample efficient–in terms of both the labels required for downstream training and the amount of unlabeled data required for pre-training–and scales well to the sizes of both the labeled and unlabeled data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v222-mulc24a, title = {{Q-Match}: {S}elf-Supervised Learning by Matching Distributions Induced by a Queue}, author = {Mulc, Thomas and Dwibedi, Debidatta}, booktitle = {Proceedings of the 15th Asian Conference on Machine Learning}, pages = {911--926}, year = {2024}, editor = {Yanıkoğlu, Berrin and Buntine, Wray}, volume = {222}, series = {Proceedings of Machine Learning Research}, month = {11--14 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v222/mulc24a/mulc24a.pdf}, url = {https://proceedings.mlr.press/v222/mulc24a.html}, abstract = {In semi-supervised learning, student-teacher distribution matching has been successful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We introduce our algorithm, Q-Match, and show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We focus our study on tabular datasets and show that Q-Match outperforms previous self-supervised learning techniques when measuring downstream classification performance. Furthermore, we show that our method is sample efficient–in terms of both the labels required for downstream training and the amount of unlabeled data required for pre-training–and scales well to the sizes of both the labeled and unlabeled data.} }
Endnote
%0 Conference Paper %T Q-Match: Self-Supervised Learning by Matching Distributions Induced by a Queue %A Thomas Mulc %A Debidatta Dwibedi %B Proceedings of the 15th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Berrin Yanıkoğlu %E Wray Buntine %F pmlr-v222-mulc24a %I PMLR %P 911--926 %U https://proceedings.mlr.press/v222/mulc24a.html %V 222 %X In semi-supervised learning, student-teacher distribution matching has been successful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We introduce our algorithm, Q-Match, and show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We focus our study on tabular datasets and show that Q-Match outperforms previous self-supervised learning techniques when measuring downstream classification performance. Furthermore, we show that our method is sample efficient–in terms of both the labels required for downstream training and the amount of unlabeled data required for pre-training–and scales well to the sizes of both the labeled and unlabeled data.
APA
Mulc, T. & Dwibedi, D.. (2024). Q-Match: Self-Supervised Learning by Matching Distributions Induced by a Queue. Proceedings of the 15th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 222:911-926 Available from https://proceedings.mlr.press/v222/mulc24a.html.

Related Material