Flexible Phase Dynamics for Bio-Plausible Contrastive Learning

Ezekiel Williams, Colin Bredenberg, Guillaume Lajoie
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:37042-37065, 2023.

Abstract

Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with another. These Contrastive Learning (CL) algorithms are traditionally implemented with rigid, temporally non-local, and periodic learning dynamics, that could limit the range of physical systems capable of harnessing CL. In this study, we build on recent work exploring how CL might be implemented by biological or neurmorphic systems and show that this form of learning can be made temporally local, and can still function even if many of the dynamical requirements of standard training procedures are relaxed. Thanks to a set of general theorems corroborated by numerical experiments across several CL models, our results provide theoretical foundations for the study and development of CL methods for biological and neuromorphic neural networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-williams23a, title = {Flexible Phase Dynamics for Bio-Plausible Contrastive Learning}, author = {Williams, Ezekiel and Bredenberg, Colin and Lajoie, Guillaume}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {37042--37065}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/williams23a/williams23a.pdf}, url = {https://proceedings.mlr.press/v202/williams23a.html}, abstract = {Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with another. These Contrastive Learning (CL) algorithms are traditionally implemented with rigid, temporally non-local, and periodic learning dynamics, that could limit the range of physical systems capable of harnessing CL. In this study, we build on recent work exploring how CL might be implemented by biological or neurmorphic systems and show that this form of learning can be made temporally local, and can still function even if many of the dynamical requirements of standard training procedures are relaxed. Thanks to a set of general theorems corroborated by numerical experiments across several CL models, our results provide theoretical foundations for the study and development of CL methods for biological and neuromorphic neural networks.} }
Endnote
%0 Conference Paper %T Flexible Phase Dynamics for Bio-Plausible Contrastive Learning %A Ezekiel Williams %A Colin Bredenberg %A Guillaume Lajoie %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-williams23a %I PMLR %P 37042--37065 %U https://proceedings.mlr.press/v202/williams23a.html %V 202 %X Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with another. These Contrastive Learning (CL) algorithms are traditionally implemented with rigid, temporally non-local, and periodic learning dynamics, that could limit the range of physical systems capable of harnessing CL. In this study, we build on recent work exploring how CL might be implemented by biological or neurmorphic systems and show that this form of learning can be made temporally local, and can still function even if many of the dynamical requirements of standard training procedures are relaxed. Thanks to a set of general theorems corroborated by numerical experiments across several CL models, our results provide theoretical foundations for the study and development of CL methods for biological and neuromorphic neural networks.
APA
Williams, E., Bredenberg, C. & Lajoie, G.. (2023). Flexible Phase Dynamics for Bio-Plausible Contrastive Learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:37042-37065 Available from https://proceedings.mlr.press/v202/williams23a.html.

Related Material