Strength from Weakness: Fast Learning Using Weak Supervision

Joshua Robinson, Stefanie Jegelka, Suvrit Sra
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:8127-8136, 2020.

Abstract

We study generalization properties of weakly supervised learning, that is, learning where only a few "strong" labels (the actual target for prediction) are present but many more "weak" labels are available. In particular, we show that pretraining using weak labels and finetuning using strong can accelerate the learning rate for the strong task to the fast rate of O(1/n), where n is the number of strongly labeled data points. This acceleration can happen even if, by itself, the strongly labeled data admits only the slower O(1/\sqrt{n}) rate. The acceleration depends continuously on the number of weak labels available, and on the relation between the two tasks. Our theoretical results are reflected empirically across a range of tasks and illustrate how weak labels speed up learning on the strong task.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-robinson20a, title = {Strength from Weakness: Fast Learning Using Weak Supervision}, author = {Robinson, Joshua and Jegelka, Stefanie and Sra, Suvrit}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {8127--8136}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/robinson20a/robinson20a.pdf}, url = {https://proceedings.mlr.press/v119/robinson20a.html}, abstract = {We study generalization properties of weakly supervised learning, that is, learning where only a few "strong" labels (the actual target for prediction) are present but many more "weak" labels are available. In particular, we show that pretraining using weak labels and finetuning using strong can accelerate the learning rate for the strong task to the fast rate of O(1/n), where n is the number of strongly labeled data points. This acceleration can happen even if, by itself, the strongly labeled data admits only the slower O(1/\sqrt{n}) rate. The acceleration depends continuously on the number of weak labels available, and on the relation between the two tasks. Our theoretical results are reflected empirically across a range of tasks and illustrate how weak labels speed up learning on the strong task.} }
Endnote
%0 Conference Paper %T Strength from Weakness: Fast Learning Using Weak Supervision %A Joshua Robinson %A Stefanie Jegelka %A Suvrit Sra %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-robinson20a %I PMLR %P 8127--8136 %U https://proceedings.mlr.press/v119/robinson20a.html %V 119 %X We study generalization properties of weakly supervised learning, that is, learning where only a few "strong" labels (the actual target for prediction) are present but many more "weak" labels are available. In particular, we show that pretraining using weak labels and finetuning using strong can accelerate the learning rate for the strong task to the fast rate of O(1/n), where n is the number of strongly labeled data points. This acceleration can happen even if, by itself, the strongly labeled data admits only the slower O(1/\sqrt{n}) rate. The acceleration depends continuously on the number of weak labels available, and on the relation between the two tasks. Our theoretical results are reflected empirically across a range of tasks and illustrate how weak labels speed up learning on the strong task.
APA
Robinson, J., Jegelka, S. & Sra, S.. (2020). Strength from Weakness: Fast Learning Using Weak Supervision. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:8127-8136 Available from https://proceedings.mlr.press/v119/robinson20a.html.

Related Material