On the Role of Depth in the Expressivity of RNNs

Maude Lizaire, Michael Rizvi-Martel, Éric Dupuis, Guillaume Rabusseau
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:2332-2340, 2026.

Abstract

The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth interacts with recurrence to shape expressive power. Here, we formally show that depth increases RNNs’ memory capacity efficiently with respect to the number of parameters, thus enhancing expressivity both by enabling more complex input transformations and improving the retention of past information. We extend our analysis to 2RNNs, a generalization of RNNs with multiplicative interactions between inputs and hidden states. Unlike RNNs, which remain linear without nonlinear activations, 2RNNs perform polynomial transformations whose maximal degree grows with depth. We further show that multiplicative interactions cannot, in general, be replaced by layerwise nonlinearities. Finally, we validate these insights empirically on synthetic and real-world tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-lizaire26a, title = { On the Role of Depth in the Expressivity of RNNs }, author = {Lizaire, Maude and Rizvi-Martel, Michael and Dupuis, \'{E}ric and Rabusseau, Guillaume}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {2332--2340}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/lizaire26a/lizaire26a.pdf}, url = {https://proceedings.mlr.press/v300/lizaire26a.html}, abstract = { The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth interacts with recurrence to shape expressive power. Here, we formally show that depth increases RNNs’ memory capacity efficiently with respect to the number of parameters, thus enhancing expressivity both by enabling more complex input transformations and improving the retention of past information. We extend our analysis to 2RNNs, a generalization of RNNs with multiplicative interactions between inputs and hidden states. Unlike RNNs, which remain linear without nonlinear activations, 2RNNs perform polynomial transformations whose maximal degree grows with depth. We further show that multiplicative interactions cannot, in general, be replaced by layerwise nonlinearities. Finally, we validate these insights empirically on synthetic and real-world tasks. } }
Endnote
%0 Conference Paper %T On the Role of Depth in the Expressivity of RNNs %A Maude Lizaire %A Michael Rizvi-Martel %A Éric Dupuis %A Guillaume Rabusseau %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-lizaire26a %I PMLR %P 2332--2340 %U https://proceedings.mlr.press/v300/lizaire26a.html %V 300 %X The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth interacts with recurrence to shape expressive power. Here, we formally show that depth increases RNNs’ memory capacity efficiently with respect to the number of parameters, thus enhancing expressivity both by enabling more complex input transformations and improving the retention of past information. We extend our analysis to 2RNNs, a generalization of RNNs with multiplicative interactions between inputs and hidden states. Unlike RNNs, which remain linear without nonlinear activations, 2RNNs perform polynomial transformations whose maximal degree grows with depth. We further show that multiplicative interactions cannot, in general, be replaced by layerwise nonlinearities. Finally, we validate these insights empirically on synthetic and real-world tasks.
APA
Lizaire, M., Rizvi-Martel, M., Dupuis, É. & Rabusseau, G.. (2026). On the Role of Depth in the Expressivity of RNNs . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:2332-2340 Available from https://proceedings.mlr.press/v300/lizaire26a.html.

Related Material