Enabling Few-Shot Learning with PID Control: A Layer Adaptive Optimizer

Le Yu, Xinde Li, Pengfei Zhang, Zhentong Zhang, Fir Dunkin
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:57544-57558, 2024.

Abstract

Model-Agnostic Meta-Learning (MAML) and its variants have shown remarkable performance in scenarios characterized by a scarcity of labeled data during the training phase of machine learning models. Despite these successes, MAMLbased approaches encounter significant challenges when there is a substantial discrepancy in the distribution of training and testing tasks, resulting in inefficient learning and limited generalization across domains. Inspired by classical proportional-integral-derivative (PID) control theory, this study introduces a Layer-Adaptive PID (LA-PID) Optimizer, a MAML-based optimizer that employs efficient parameter optimization methods to dynamically adjust task-specific PID control gains at each layer of the network, conducting a first-principles analysis of optimal convergence conditions. A series of experiments conducted on four standard benchmark datasets demonstrate the efficacy of the LA-PID optimizer, indicating that LA-PID achieves state-oftheart performance in few-shot classification and cross-domain tasks, accomplishing these objectives with fewer training steps. Code is available on https://github.com/yuguopin/LA-PID.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-yu24h, title = {Enabling Few-Shot Learning with {PID} Control: A Layer Adaptive Optimizer}, author = {Yu, Le and Li, Xinde and Zhang, Pengfei and Zhang, Zhentong and Dunkin, Fir}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {57544--57558}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/yu24h/yu24h.pdf}, url = {https://proceedings.mlr.press/v235/yu24h.html}, abstract = {Model-Agnostic Meta-Learning (MAML) and its variants have shown remarkable performance in scenarios characterized by a scarcity of labeled data during the training phase of machine learning models. Despite these successes, MAMLbased approaches encounter significant challenges when there is a substantial discrepancy in the distribution of training and testing tasks, resulting in inefficient learning and limited generalization across domains. Inspired by classical proportional-integral-derivative (PID) control theory, this study introduces a Layer-Adaptive PID (LA-PID) Optimizer, a MAML-based optimizer that employs efficient parameter optimization methods to dynamically adjust task-specific PID control gains at each layer of the network, conducting a first-principles analysis of optimal convergence conditions. A series of experiments conducted on four standard benchmark datasets demonstrate the efficacy of the LA-PID optimizer, indicating that LA-PID achieves state-oftheart performance in few-shot classification and cross-domain tasks, accomplishing these objectives with fewer training steps. Code is available on https://github.com/yuguopin/LA-PID.} }
Endnote
%0 Conference Paper %T Enabling Few-Shot Learning with PID Control: A Layer Adaptive Optimizer %A Le Yu %A Xinde Li %A Pengfei Zhang %A Zhentong Zhang %A Fir Dunkin %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-yu24h %I PMLR %P 57544--57558 %U https://proceedings.mlr.press/v235/yu24h.html %V 235 %X Model-Agnostic Meta-Learning (MAML) and its variants have shown remarkable performance in scenarios characterized by a scarcity of labeled data during the training phase of machine learning models. Despite these successes, MAMLbased approaches encounter significant challenges when there is a substantial discrepancy in the distribution of training and testing tasks, resulting in inefficient learning and limited generalization across domains. Inspired by classical proportional-integral-derivative (PID) control theory, this study introduces a Layer-Adaptive PID (LA-PID) Optimizer, a MAML-based optimizer that employs efficient parameter optimization methods to dynamically adjust task-specific PID control gains at each layer of the network, conducting a first-principles analysis of optimal convergence conditions. A series of experiments conducted on four standard benchmark datasets demonstrate the efficacy of the LA-PID optimizer, indicating that LA-PID achieves state-oftheart performance in few-shot classification and cross-domain tasks, accomplishing these objectives with fewer training steps. Code is available on https://github.com/yuguopin/LA-PID.
APA
Yu, L., Li, X., Zhang, P., Zhang, Z. & Dunkin, F.. (2024). Enabling Few-Shot Learning with PID Control: A Layer Adaptive Optimizer. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:57544-57558 Available from https://proceedings.mlr.press/v235/yu24h.html.

Related Material