Negative Sampling in Semi-Supervised learning

John Chen, Vatsal Shah, Anastasios Kyrillidis
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:1704-1714, 2020.

Abstract

We introduce Negative Sampling in Semi-Supervised Learning (NS^3L), a simple, fast, easy to tune algorithm for semi-supervised learning (SSL). NS^3L is motivated by the success of negative sampling/contrastive estimation. We demonstrate that adding the NS^3L loss to state-of-the-art SSL algorithms, such as the Virtual Adversarial Training (VAT), significantly improves upon vanilla VAT and its variant, VAT with Entropy Minimization. By adding the NS^3L loss to MixMatch, the current state-of-the-art approach on semi-supervised tasks, we observe significant improvements over vanilla MixMatch. We conduct extensive experiments on the CIFAR10, CIFAR100, SVHN and STL10 benchmark datasets. Finally, we perform an ablation study for NS3L regarding its hyperparameter tuning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-chen20t, title = {Negative Sampling in Semi-Supervised learning}, author = {Chen, John and Shah, Vatsal and Kyrillidis, Anastasios}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {1704--1714}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/chen20t/chen20t.pdf}, url = {https://proceedings.mlr.press/v119/chen20t.html}, abstract = {We introduce Negative Sampling in Semi-Supervised Learning (NS^3L), a simple, fast, easy to tune algorithm for semi-supervised learning (SSL). NS^3L is motivated by the success of negative sampling/contrastive estimation. We demonstrate that adding the NS^3L loss to state-of-the-art SSL algorithms, such as the Virtual Adversarial Training (VAT), significantly improves upon vanilla VAT and its variant, VAT with Entropy Minimization. By adding the NS^3L loss to MixMatch, the current state-of-the-art approach on semi-supervised tasks, we observe significant improvements over vanilla MixMatch. We conduct extensive experiments on the CIFAR10, CIFAR100, SVHN and STL10 benchmark datasets. Finally, we perform an ablation study for NS3L regarding its hyperparameter tuning.} }
Endnote
%0 Conference Paper %T Negative Sampling in Semi-Supervised learning %A John Chen %A Vatsal Shah %A Anastasios Kyrillidis %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-chen20t %I PMLR %P 1704--1714 %U https://proceedings.mlr.press/v119/chen20t.html %V 119 %X We introduce Negative Sampling in Semi-Supervised Learning (NS^3L), a simple, fast, easy to tune algorithm for semi-supervised learning (SSL). NS^3L is motivated by the success of negative sampling/contrastive estimation. We demonstrate that adding the NS^3L loss to state-of-the-art SSL algorithms, such as the Virtual Adversarial Training (VAT), significantly improves upon vanilla VAT and its variant, VAT with Entropy Minimization. By adding the NS^3L loss to MixMatch, the current state-of-the-art approach on semi-supervised tasks, we observe significant improvements over vanilla MixMatch. We conduct extensive experiments on the CIFAR10, CIFAR100, SVHN and STL10 benchmark datasets. Finally, we perform an ablation study for NS3L regarding its hyperparameter tuning.
APA
Chen, J., Shah, V. & Kyrillidis, A.. (2020). Negative Sampling in Semi-Supervised learning. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:1704-1714 Available from https://proceedings.mlr.press/v119/chen20t.html.

Related Material