On Numerical Integration in Neural Ordinary Differential Equations

Aiqing Zhu, Pengzhan Jin, Beibei Zhu, Yifa Tang
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:27527-27547, 2022.

Abstract

The combination of ordinary differential equations and neural networks, i.e., neural ordinary differential equations (Neural ODE), has been widely studied from various angles. However, deciphering the numerical integration in Neural ODE is still an open challenge, as many researches demonstrated that numerical integration significantly affects the performance of the model. In this paper, we propose the inverse modified differential equations (IMDE) to clarify the influence of numerical integration on training Neural ODE models. IMDE is determined by the learning task and the employed ODE solver. It is shown that training a Neural ODE model actually returns a close approximation of the IMDE, rather than the true ODE. With the help of IMDE, we deduce that (i) the discrepancy between the learned model and the true ODE is bounded by the sum of discretization error and learning loss; (ii) Neural ODE using non-symplectic numerical integration fail to learn conservation laws theoretically. Several experiments are performed to numerically verify our theoretical analysis.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-zhu22f, title = {On Numerical Integration in Neural Ordinary Differential Equations}, author = {Zhu, Aiqing and Jin, Pengzhan and Zhu, Beibei and Tang, Yifa}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {27527--27547}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/zhu22f/zhu22f.pdf}, url = {https://proceedings.mlr.press/v162/zhu22f.html}, abstract = {The combination of ordinary differential equations and neural networks, i.e., neural ordinary differential equations (Neural ODE), has been widely studied from various angles. However, deciphering the numerical integration in Neural ODE is still an open challenge, as many researches demonstrated that numerical integration significantly affects the performance of the model. In this paper, we propose the inverse modified differential equations (IMDE) to clarify the influence of numerical integration on training Neural ODE models. IMDE is determined by the learning task and the employed ODE solver. It is shown that training a Neural ODE model actually returns a close approximation of the IMDE, rather than the true ODE. With the help of IMDE, we deduce that (i) the discrepancy between the learned model and the true ODE is bounded by the sum of discretization error and learning loss; (ii) Neural ODE using non-symplectic numerical integration fail to learn conservation laws theoretically. Several experiments are performed to numerically verify our theoretical analysis.} }
Endnote
%0 Conference Paper %T On Numerical Integration in Neural Ordinary Differential Equations %A Aiqing Zhu %A Pengzhan Jin %A Beibei Zhu %A Yifa Tang %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-zhu22f %I PMLR %P 27527--27547 %U https://proceedings.mlr.press/v162/zhu22f.html %V 162 %X The combination of ordinary differential equations and neural networks, i.e., neural ordinary differential equations (Neural ODE), has been widely studied from various angles. However, deciphering the numerical integration in Neural ODE is still an open challenge, as many researches demonstrated that numerical integration significantly affects the performance of the model. In this paper, we propose the inverse modified differential equations (IMDE) to clarify the influence of numerical integration on training Neural ODE models. IMDE is determined by the learning task and the employed ODE solver. It is shown that training a Neural ODE model actually returns a close approximation of the IMDE, rather than the true ODE. With the help of IMDE, we deduce that (i) the discrepancy between the learned model and the true ODE is bounded by the sum of discretization error and learning loss; (ii) Neural ODE using non-symplectic numerical integration fail to learn conservation laws theoretically. Several experiments are performed to numerically verify our theoretical analysis.
APA
Zhu, A., Jin, P., Zhu, B. & Tang, Y.. (2022). On Numerical Integration in Neural Ordinary Differential Equations. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:27527-27547 Available from https://proceedings.mlr.press/v162/zhu22f.html.

Related Material