Linear Antisymmetric Recurrent Neural Networks

Signe Moe, Filippo Remonato, Esten Ingar Grøtli, Jan Tommy Gravdahl
Proceedings of the 2nd Conference on Learning for Dynamics and Control, PMLR 120:170-178, 2020.

Abstract

Recurrent Neural Networks (RNNs) have a form of memory where the output from a node at one timestep is fed back as input the next timestep in addition to data from the previous layer. This makes them highly suitable for timeseries analysis. However, standard RNNs have known weaknesses such as exploding/vanishing gradient and thereby struggle with a long-term memory. In this paper, we suggest a new recurrent network structure called Linear Antisymmetric RNN (LARNN). This structure is based on the numerical solution to an Ordinary Differential Equation (ODE) with stability properties resulting in a stable solution, which corresponds to long-term memory and trainability. Three different numerical methods are suggested to solve the ODE: Forward and Backward Euler and the midpoint method. The suggested structure has been implemented in Keras and several simulated datasets have been used to evaluate the performance. In the investigated cases, the LARNN performs better or similar to the Long Short Term Memory (LSTM) network which is the current state of the art for RNNs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v120-moe20a, title = {Linear Antisymmetric Recurrent Neural Networks}, author = {Moe, Signe and Remonato, Filippo and Gr{\o}tli, Esten Ingar and Gravdahl, Jan Tommy}, booktitle = {Proceedings of the 2nd Conference on Learning for Dynamics and Control}, pages = {170--178}, year = {2020}, editor = {Bayen, Alexandre M. and Jadbabaie, Ali and Pappas, George and Parrilo, Pablo A. and Recht, Benjamin and Tomlin, Claire and Zeilinger, Melanie}, volume = {120}, series = {Proceedings of Machine Learning Research}, month = {10--11 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v120/moe20a/moe20a.pdf}, url = {https://proceedings.mlr.press/v120/moe20a.html}, abstract = {Recurrent Neural Networks (RNNs) have a form of memory where the output from a node at one timestep is fed back as input the next timestep in addition to data from the previous layer. This makes them highly suitable for timeseries analysis. However, standard RNNs have known weaknesses such as exploding/vanishing gradient and thereby struggle with a long-term memory. In this paper, we suggest a new recurrent network structure called Linear Antisymmetric RNN (LARNN). This structure is based on the numerical solution to an Ordinary Differential Equation (ODE) with stability properties resulting in a stable solution, which corresponds to long-term memory and trainability. Three different numerical methods are suggested to solve the ODE: Forward and Backward Euler and the midpoint method. The suggested structure has been implemented in Keras and several simulated datasets have been used to evaluate the performance. In the investigated cases, the LARNN performs better or similar to the Long Short Term Memory (LSTM) network which is the current state of the art for RNNs.} }
Endnote
%0 Conference Paper %T Linear Antisymmetric Recurrent Neural Networks %A Signe Moe %A Filippo Remonato %A Esten Ingar Grøtli %A Jan Tommy Gravdahl %B Proceedings of the 2nd Conference on Learning for Dynamics and Control %C Proceedings of Machine Learning Research %D 2020 %E Alexandre M. Bayen %E Ali Jadbabaie %E George Pappas %E Pablo A. Parrilo %E Benjamin Recht %E Claire Tomlin %E Melanie Zeilinger %F pmlr-v120-moe20a %I PMLR %P 170--178 %U https://proceedings.mlr.press/v120/moe20a.html %V 120 %X Recurrent Neural Networks (RNNs) have a form of memory where the output from a node at one timestep is fed back as input the next timestep in addition to data from the previous layer. This makes them highly suitable for timeseries analysis. However, standard RNNs have known weaknesses such as exploding/vanishing gradient and thereby struggle with a long-term memory. In this paper, we suggest a new recurrent network structure called Linear Antisymmetric RNN (LARNN). This structure is based on the numerical solution to an Ordinary Differential Equation (ODE) with stability properties resulting in a stable solution, which corresponds to long-term memory and trainability. Three different numerical methods are suggested to solve the ODE: Forward and Backward Euler and the midpoint method. The suggested structure has been implemented in Keras and several simulated datasets have been used to evaluate the performance. In the investigated cases, the LARNN performs better or similar to the Long Short Term Memory (LSTM) network which is the current state of the art for RNNs.
APA
Moe, S., Remonato, F., Grøtli, E.I. & Gravdahl, J.T.. (2020). Linear Antisymmetric Recurrent Neural Networks. Proceedings of the 2nd Conference on Learning for Dynamics and Control, in Proceedings of Machine Learning Research 120:170-178 Available from https://proceedings.mlr.press/v120/moe20a.html.

Related Material