[edit]

# AFRNN: Stable RNN with Top Down Feedback and Antisymmetry

*Proceedings of The 14th Asian Conference on Machine Learning*, PMLR 189:880-894, 2023.

#### Abstract

Recurrent Neural Networks are an integral part of
modern machine learning. They are good at performing
tasks on sequential data. However, long sequences
are still a problem for those models due to the
well-known exploding/vanishing gradient problem. In
this work, we build on recent approaches to
interpreting the gradient problem as instability of
the underlying dynamical system. We extend previous
approaches to systems with top-down feedback, which
is abundant in biological neural networks. We prove
that the resulting system is stable for arbitrary
depth and width and confirm this empirically. We
further show that its performance is on par with
LSTM and related approaches on standard benchmarks.