AFRNN: Stable RNN with Top Down Feedback and Antisymmetry

Tim Schwabe, Tobias Glasmachers, Maribel Acosta
Proceedings of The 14th Asian Conference on Machine Learning, PMLR 189:880-894, 2023.

Abstract

Recurrent Neural Networks are an integral part of modern machine learning. They are good at performing tasks on sequential data. However, long sequences are still a problem for those models due to the well-known exploding/vanishing gradient problem. In this work, we build on recent approaches to interpreting the gradient problem as instability of the underlying dynamical system. We extend previous approaches to systems with top-down feedback, which is abundant in biological neural networks. We prove that the resulting system is stable for arbitrary depth and width and confirm this empirically. We further show that its performance is on par with LSTM and related approaches on standard benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v189-schwabe23a, title = {AFRNN: Stable RNN with Top Down Feedback and Antisymmetry}, author = {Schwabe, Tim and Glasmachers, Tobias and Acosta, Maribel}, booktitle = {Proceedings of The 14th Asian Conference on Machine Learning}, pages = {880--894}, year = {2023}, editor = {Khan, Emtiyaz and Gonen, Mehmet}, volume = {189}, series = {Proceedings of Machine Learning Research}, month = {12--14 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v189/schwabe23a/schwabe23a.pdf}, url = {https://proceedings.mlr.press/v189/schwabe23a.html}, abstract = {Recurrent Neural Networks are an integral part of modern machine learning. They are good at performing tasks on sequential data. However, long sequences are still a problem for those models due to the well-known exploding/vanishing gradient problem. In this work, we build on recent approaches to interpreting the gradient problem as instability of the underlying dynamical system. We extend previous approaches to systems with top-down feedback, which is abundant in biological neural networks. We prove that the resulting system is stable for arbitrary depth and width and confirm this empirically. We further show that its performance is on par with LSTM and related approaches on standard benchmarks.} }
Endnote
%0 Conference Paper %T AFRNN: Stable RNN with Top Down Feedback and Antisymmetry %A Tim Schwabe %A Tobias Glasmachers %A Maribel Acosta %B Proceedings of The 14th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Emtiyaz Khan %E Mehmet Gonen %F pmlr-v189-schwabe23a %I PMLR %P 880--894 %U https://proceedings.mlr.press/v189/schwabe23a.html %V 189 %X Recurrent Neural Networks are an integral part of modern machine learning. They are good at performing tasks on sequential data. However, long sequences are still a problem for those models due to the well-known exploding/vanishing gradient problem. In this work, we build on recent approaches to interpreting the gradient problem as instability of the underlying dynamical system. We extend previous approaches to systems with top-down feedback, which is abundant in biological neural networks. We prove that the resulting system is stable for arbitrary depth and width and confirm this empirically. We further show that its performance is on par with LSTM and related approaches on standard benchmarks.
APA
Schwabe, T., Glasmachers, T. & Acosta, M.. (2023). AFRNN: Stable RNN with Top Down Feedback and Antisymmetry. Proceedings of The 14th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 189:880-894 Available from https://proceedings.mlr.press/v189/schwabe23a.html.

Related Material