Efficient Continuous-Depth Modeling with GRU Equivalents

Ayan Banerjee, Bin Xu, Sandeep Gupta
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:6194-6209, 2026.

Abstract

Continuous-Depth Neural Networks (CDNNs), including Neural Ordinary Differential Equations (ODEs) and Liquid-Time-Constant Neural Networks (LTC-NN), suffer from high computational costs due to solving numerous nonlinear ODEs during training and inference. We introduce Continuous Depth Acceleration (CoDA), a framework that leverages Mori–Zwanzig/Koopman operator theory to replace continuous-depth layers requiring multiple nonlinear ODEs with a compact GRU module, a single low-dimensional linear ODE, and a dense layer. We prove PAC learnability of CoDA, establishing that this transformation preserves accuracy and can be applied repeatedly across multiple layers with unified backpropagation. Experiments on the Liquid Foundation Model (LFM-1.2B) demonstrate $6.7\times$ training speedup and $1.8\times$ inference speedup without loss of accuracy. Across six real-world LTC-NN applications, CoDA consistently outperforms state-of-the-art acceleration techniques—including neural flows, model order reduction, and variational formulations—in both training and inference time while maintaining competitive or superior accuracy. The implementation and datasets are publicly available at https://github.com/ImpactLabASU/CoDA-ICML2026.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-banerjee26b, title = {Efficient Continuous-Depth Modeling with {GRU} Equivalents}, author = {Banerjee, Ayan and Xu, Bin and Gupta, Sandeep}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {6194--6209}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/banerjee26b/banerjee26b.pdf}, url = {https://proceedings.mlr.press/v306/banerjee26b.html}, abstract = {Continuous-Depth Neural Networks (CDNNs), including Neural Ordinary Differential Equations (ODEs) and Liquid-Time-Constant Neural Networks (LTC-NN), suffer from high computational costs due to solving numerous nonlinear ODEs during training and inference. We introduce Continuous Depth Acceleration (CoDA), a framework that leverages Mori–Zwanzig/Koopman operator theory to replace continuous-depth layers requiring multiple nonlinear ODEs with a compact GRU module, a single low-dimensional linear ODE, and a dense layer. We prove PAC learnability of CoDA, establishing that this transformation preserves accuracy and can be applied repeatedly across multiple layers with unified backpropagation. Experiments on the Liquid Foundation Model (LFM-1.2B) demonstrate $6.7\times$ training speedup and $1.8\times$ inference speedup without loss of accuracy. Across six real-world LTC-NN applications, CoDA consistently outperforms state-of-the-art acceleration techniques—including neural flows, model order reduction, and variational formulations—in both training and inference time while maintaining competitive or superior accuracy. The implementation and datasets are publicly available at https://github.com/ImpactLabASU/CoDA-ICML2026.} }
Endnote
%0 Conference Paper %T Efficient Continuous-Depth Modeling with GRU Equivalents %A Ayan Banerjee %A Bin Xu %A Sandeep Gupta %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-banerjee26b %I PMLR %P 6194--6209 %U https://proceedings.mlr.press/v306/banerjee26b.html %V 306 %X Continuous-Depth Neural Networks (CDNNs), including Neural Ordinary Differential Equations (ODEs) and Liquid-Time-Constant Neural Networks (LTC-NN), suffer from high computational costs due to solving numerous nonlinear ODEs during training and inference. We introduce Continuous Depth Acceleration (CoDA), a framework that leverages Mori–Zwanzig/Koopman operator theory to replace continuous-depth layers requiring multiple nonlinear ODEs with a compact GRU module, a single low-dimensional linear ODE, and a dense layer. We prove PAC learnability of CoDA, establishing that this transformation preserves accuracy and can be applied repeatedly across multiple layers with unified backpropagation. Experiments on the Liquid Foundation Model (LFM-1.2B) demonstrate $6.7\times$ training speedup and $1.8\times$ inference speedup without loss of accuracy. Across six real-world LTC-NN applications, CoDA consistently outperforms state-of-the-art acceleration techniques—including neural flows, model order reduction, and variational formulations—in both training and inference time while maintaining competitive or superior accuracy. The implementation and datasets are publicly available at https://github.com/ImpactLabASU/CoDA-ICML2026.
APA
Banerjee, A., Xu, B. & Gupta, S.. (2026). Efficient Continuous-Depth Modeling with GRU Equivalents. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:6194-6209 Available from https://proceedings.mlr.press/v306/banerjee26b.html.

Related Material