Meta-learning for Mixed Linear Regression

Weihao Kong, Raghav Somani, Zhao Song, Sham Kakade, Sewoong Oh
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5394-5404, 2020.

Abstract

In modern supervised learning, there are a large number of tasks, but many of them are associated with only a small amount of labelled data. These include data from medical image processing and robotic interaction. Even though each individual task cannot be meaningfully trained in isolation, one seeks to meta-learn across the tasks from past experiences by exploiting some similarities. We study a fundamental question of interest: When can abundant tasks with small data compensate for lack of tasks with big data? We focus on a canonical scenario where each task is drawn from a mixture of $k$ linear regressions, and identify sufficient conditions for such a graceful exchange to hold; there is little loss in sample complexity even when we only have access to small data tasks. To this end, we introduce a novel spectral approach and show that we can efficiently utilize small data tasks with the help of $\tilde\Omega(k^{3/2})$ medium data tasks each with $\tilde\Omega(k^{1/2})$ examples.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-kong20a, title = {Meta-learning for Mixed Linear Regression}, author = {Kong, Weihao and Somani, Raghav and Song, Zhao and Kakade, Sham and Oh, Sewoong}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {5394--5404}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/kong20a/kong20a.pdf}, url = {https://proceedings.mlr.press/v119/kong20a.html}, abstract = {In modern supervised learning, there are a large number of tasks, but many of them are associated with only a small amount of labelled data. These include data from medical image processing and robotic interaction. Even though each individual task cannot be meaningfully trained in isolation, one seeks to meta-learn across the tasks from past experiences by exploiting some similarities. We study a fundamental question of interest: When can abundant tasks with small data compensate for lack of tasks with big data? We focus on a canonical scenario where each task is drawn from a mixture of $k$ linear regressions, and identify sufficient conditions for such a graceful exchange to hold; there is little loss in sample complexity even when we only have access to small data tasks. To this end, we introduce a novel spectral approach and show that we can efficiently utilize small data tasks with the help of $\tilde\Omega(k^{3/2})$ medium data tasks each with $\tilde\Omega(k^{1/2})$ examples.} }
Endnote
%0 Conference Paper %T Meta-learning for Mixed Linear Regression %A Weihao Kong %A Raghav Somani %A Zhao Song %A Sham Kakade %A Sewoong Oh %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-kong20a %I PMLR %P 5394--5404 %U https://proceedings.mlr.press/v119/kong20a.html %V 119 %X In modern supervised learning, there are a large number of tasks, but many of them are associated with only a small amount of labelled data. These include data from medical image processing and robotic interaction. Even though each individual task cannot be meaningfully trained in isolation, one seeks to meta-learn across the tasks from past experiences by exploiting some similarities. We study a fundamental question of interest: When can abundant tasks with small data compensate for lack of tasks with big data? We focus on a canonical scenario where each task is drawn from a mixture of $k$ linear regressions, and identify sufficient conditions for such a graceful exchange to hold; there is little loss in sample complexity even when we only have access to small data tasks. To this end, we introduce a novel spectral approach and show that we can efficiently utilize small data tasks with the help of $\tilde\Omega(k^{3/2})$ medium data tasks each with $\tilde\Omega(k^{1/2})$ examples.
APA
Kong, W., Somani, R., Song, Z., Kakade, S. & Oh, S.. (2020). Meta-learning for Mixed Linear Regression. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5394-5404 Available from https://proceedings.mlr.press/v119/kong20a.html.

Related Material