Stress Testing of Meta-learning Approaches for Few-shot Learning

Aroof Aimen, Sahil Sidheekh, Vineet Madan, Narayanan C. Krishnan
AAAI Workshop on Meta-Learning and MetaDL Challenge, PMLR 140:38-44, 2021.

Abstract

Meta-learning (ML) has emerged as a promising learning method under resource constraints such as few-shot learning. ML approaches typically propose a methodology to learn generalizable models. In this work-in-progress paper, we put the recent ML approaches to a stress test to discover their limitations. Precisely, we measure the performance of ML approaches for few-shot learning against increasing task complexity. Our results show a quick degradation in the performance of initialization strategies for ML (MAML, TAML, and MetaSGD), while surprisingly, approaches that use an optimization strategy (MetaLSTM) perform significantly better. We further demonstrate the effectiveness of an optimization strategy for ML (MetaLSTM++) trained in a MAML manner over a pure optimization strategy. Our experiments also show that the optimization strategies for ML achieve higher transferability from simple to complex tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v140-aimen21a, title = {Stress Testing of Meta-learning Approaches for Few-shot Learning}, author = {Aimen, Aroof and Sidheekh, Sahil and Madan, Vineet and Krishnan, Narayanan C.}, booktitle = {AAAI Workshop on Meta-Learning and MetaDL Challenge}, pages = {38--44}, year = {2021}, editor = {Guyon, Isabelle and van Rijn, Jan N. and Treguer, Sébastien and Vanschoren, Joaquin}, volume = {140}, series = {Proceedings of Machine Learning Research}, month = {09 Feb}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v140/aimen21a/aimen21a.pdf}, url = {https://proceedings.mlr.press/v140/aimen21a.html}, abstract = {Meta-learning (ML) has emerged as a promising learning method under resource constraints such as few-shot learning. ML approaches typically propose a methodology to learn generalizable models. In this work-in-progress paper, we put the recent ML approaches to a stress test to discover their limitations. Precisely, we measure the performance of ML approaches for few-shot learning against increasing task complexity. Our results show a quick degradation in the performance of initialization strategies for ML (MAML, TAML, and MetaSGD), while surprisingly, approaches that use an optimization strategy (MetaLSTM) perform significantly better. We further demonstrate the effectiveness of an optimization strategy for ML (MetaLSTM++) trained in a MAML manner over a pure optimization strategy. Our experiments also show that the optimization strategies for ML achieve higher transferability from simple to complex tasks.} }
Endnote
%0 Conference Paper %T Stress Testing of Meta-learning Approaches for Few-shot Learning %A Aroof Aimen %A Sahil Sidheekh %A Vineet Madan %A Narayanan C. Krishnan %B AAAI Workshop on Meta-Learning and MetaDL Challenge %C Proceedings of Machine Learning Research %D 2021 %E Isabelle Guyon %E Jan N. van Rijn %E Sébastien Treguer %E Joaquin Vanschoren %F pmlr-v140-aimen21a %I PMLR %P 38--44 %U https://proceedings.mlr.press/v140/aimen21a.html %V 140 %X Meta-learning (ML) has emerged as a promising learning method under resource constraints such as few-shot learning. ML approaches typically propose a methodology to learn generalizable models. In this work-in-progress paper, we put the recent ML approaches to a stress test to discover their limitations. Precisely, we measure the performance of ML approaches for few-shot learning against increasing task complexity. Our results show a quick degradation in the performance of initialization strategies for ML (MAML, TAML, and MetaSGD), while surprisingly, approaches that use an optimization strategy (MetaLSTM) perform significantly better. We further demonstrate the effectiveness of an optimization strategy for ML (MetaLSTM++) trained in a MAML manner over a pure optimization strategy. Our experiments also show that the optimization strategies for ML achieve higher transferability from simple to complex tasks.
APA
Aimen, A., Sidheekh, S., Madan, V. & Krishnan, N.C.. (2021). Stress Testing of Meta-learning Approaches for Few-shot Learning. AAAI Workshop on Meta-Learning and MetaDL Challenge, in Proceedings of Machine Learning Research 140:38-44 Available from https://proceedings.mlr.press/v140/aimen21a.html.

Related Material