MAML and ANIL Provably Learn Representations

Liam Collins, Aryan Mokhtari, Sewoong Oh, Sanjay Shakkottai
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:4238-4310, 2022.

Abstract

Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks. However, the mechanics of GBML have remained largely mysterious from a theoretical perspective. In this paper, we prove that two well-known GBML methods, MAML and ANIL, as well as their first-order approximations, are capable of learning common representation among a set of given tasks. Specifically, in the well-known multi-task linear representation learning setting, they are able to recover the ground-truth representation at an exponentially fast rate. Moreover, our analysis illuminates that the driving force causing MAML and ANIL to recover the underlying representation is that they adapt the final layer of their model, which harnesses the underlying task diversity to improve the representation in all directions of interest. To the best of our knowledge, these are the first results to show that MAML and/or ANIL learn expressive representations and to rigorously explain why they do so.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-collins22a, title = {{MAML} and {ANIL} Provably Learn Representations}, author = {Collins, Liam and Mokhtari, Aryan and Oh, Sewoong and Shakkottai, Sanjay}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {4238--4310}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/collins22a/collins22a.pdf}, url = {https://proceedings.mlr.press/v162/collins22a.html}, abstract = {Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks. However, the mechanics of GBML have remained largely mysterious from a theoretical perspective. In this paper, we prove that two well-known GBML methods, MAML and ANIL, as well as their first-order approximations, are capable of learning common representation among a set of given tasks. Specifically, in the well-known multi-task linear representation learning setting, they are able to recover the ground-truth representation at an exponentially fast rate. Moreover, our analysis illuminates that the driving force causing MAML and ANIL to recover the underlying representation is that they adapt the final layer of their model, which harnesses the underlying task diversity to improve the representation in all directions of interest. To the best of our knowledge, these are the first results to show that MAML and/or ANIL learn expressive representations and to rigorously explain why they do so.} }
Endnote
%0 Conference Paper %T MAML and ANIL Provably Learn Representations %A Liam Collins %A Aryan Mokhtari %A Sewoong Oh %A Sanjay Shakkottai %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-collins22a %I PMLR %P 4238--4310 %U https://proceedings.mlr.press/v162/collins22a.html %V 162 %X Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks. However, the mechanics of GBML have remained largely mysterious from a theoretical perspective. In this paper, we prove that two well-known GBML methods, MAML and ANIL, as well as their first-order approximations, are capable of learning common representation among a set of given tasks. Specifically, in the well-known multi-task linear representation learning setting, they are able to recover the ground-truth representation at an exponentially fast rate. Moreover, our analysis illuminates that the driving force causing MAML and ANIL to recover the underlying representation is that they adapt the final layer of their model, which harnesses the underlying task diversity to improve the representation in all directions of interest. To the best of our knowledge, these are the first results to show that MAML and/or ANIL learn expressive representations and to rigorously explain why they do so.
APA
Collins, L., Mokhtari, A., Oh, S. & Shakkottai, S.. (2022). MAML and ANIL Provably Learn Representations. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:4238-4310 Available from https://proceedings.mlr.press/v162/collins22a.html.

Related Material