How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective

Keyan Miao, Konstantinos Gatsis
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:35528-35545, 2024.

Abstract

Neural Ordinary Differential Equations (ODEs) have shown promise in learning continuous dynamics. However, their slow training and inference speed hinder wider applications. In this paper, we propose to optimize Neural ODEs from a spatial and temporal perspective, drawing inspiration from control theory. We aim to find a reasonable depth of the network, accelerating both training and inference while maintaining network performance. Two approaches are proposed. One reformulates training as a minimum-time optimal control problem directly in a single stage to search for the terminal time and network weights. The second approach uses pre-training coupled with a Lyapunov method in an initial stage, and then at a secondary stage introduces a safe terminal time updating mechanism in the forward direction. Experimental results demonstrate the effectiveness of speeding up Neural ODEs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-miao24a, title = {How Deep Do We Need: Accelerating Training and Inference of Neural {ODE}s via Control Perspective}, author = {Miao, Keyan and Gatsis, Konstantinos}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {35528--35545}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/miao24a/miao24a.pdf}, url = {https://proceedings.mlr.press/v235/miao24a.html}, abstract = {Neural Ordinary Differential Equations (ODEs) have shown promise in learning continuous dynamics. However, their slow training and inference speed hinder wider applications. In this paper, we propose to optimize Neural ODEs from a spatial and temporal perspective, drawing inspiration from control theory. We aim to find a reasonable depth of the network, accelerating both training and inference while maintaining network performance. Two approaches are proposed. One reformulates training as a minimum-time optimal control problem directly in a single stage to search for the terminal time and network weights. The second approach uses pre-training coupled with a Lyapunov method in an initial stage, and then at a secondary stage introduces a safe terminal time updating mechanism in the forward direction. Experimental results demonstrate the effectiveness of speeding up Neural ODEs.} }
Endnote
%0 Conference Paper %T How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective %A Keyan Miao %A Konstantinos Gatsis %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-miao24a %I PMLR %P 35528--35545 %U https://proceedings.mlr.press/v235/miao24a.html %V 235 %X Neural Ordinary Differential Equations (ODEs) have shown promise in learning continuous dynamics. However, their slow training and inference speed hinder wider applications. In this paper, we propose to optimize Neural ODEs from a spatial and temporal perspective, drawing inspiration from control theory. We aim to find a reasonable depth of the network, accelerating both training and inference while maintaining network performance. Two approaches are proposed. One reformulates training as a minimum-time optimal control problem directly in a single stage to search for the terminal time and network weights. The second approach uses pre-training coupled with a Lyapunov method in an initial stage, and then at a secondary stage introduces a safe terminal time updating mechanism in the forward direction. Experimental results demonstrate the effectiveness of speeding up Neural ODEs.
APA
Miao, K. & Gatsis, K.. (2024). How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:35528-35545 Available from https://proceedings.mlr.press/v235/miao24a.html.

Related Material