A unified theory of feature learning in RNNs and DNNs

Jan Philipp Bauer, Kirsten Fischer, Moritz Helias, Agostina Palmigiano
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:7092-7123, 2026.

Abstract

Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning ($\mu$P) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs’ weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs’ weight sharing furthermore induces an inductive bias that aids generalization by interpolating unobserved time steps. Overall, our theory offers a way to connect architectural structure to functional biases.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-bauer26a, title = {A unified theory of feature learning in {RNN}s and {DNN}s}, author = {Bauer, Jan Philipp and Fischer, Kirsten and Helias, Moritz and Palmigiano, Agostina}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {7092--7123}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/bauer26a/bauer26a.pdf}, url = {https://proceedings.mlr.press/v306/bauer26a.html}, abstract = {Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning ($\mu$P) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs’ weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs’ weight sharing furthermore induces an inductive bias that aids generalization by interpolating unobserved time steps. Overall, our theory offers a way to connect architectural structure to functional biases.} }
Endnote
%0 Conference Paper %T A unified theory of feature learning in RNNs and DNNs %A Jan Philipp Bauer %A Kirsten Fischer %A Moritz Helias %A Agostina Palmigiano %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-bauer26a %I PMLR %P 7092--7123 %U https://proceedings.mlr.press/v306/bauer26a.html %V 306 %X Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning ($\mu$P) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs’ weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs’ weight sharing furthermore induces an inductive bias that aids generalization by interpolating unobserved time steps. Overall, our theory offers a way to connect architectural structure to functional biases.
APA
Bauer, J.P., Fischer, K., Helias, M. & Palmigiano, A.. (2026). A unified theory of feature learning in RNNs and DNNs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:7092-7123 Available from https://proceedings.mlr.press/v306/bauer26a.html.

Related Material