Exploring interpretable LSTM neural networks over multi-variable data

Tian Guo, Tao Lin, Nino Antulov-Fantulin
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:2494-2504, 2019.

Abstract

For recurrent neural networks trained on time series with target and exogenous variables, in addition to accurate prediction, it is also desired to provide interpretable insights into the data. In this paper, we explore the structure of LSTM recurrent neural networks to learn variable-wise hidden states, with the aim to capture different dynamics in multi-variable time series and distinguish the contribution of variables to the prediction. With these variable-wise hidden states, a mixture attention mechanism is proposed to model the generative process of the target. Then we develop associated training methods to jointly learn network parameters, variable and temporal importance w.r.t the prediction of the target variable. Extensive experiments on real datasets demonstrate enhanced prediction performance by capturing the dynamics of different variables. Meanwhile, we evaluate the interpretation results both qualitatively and quantitatively. It exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variable data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-guo19b, title = {Exploring interpretable {LSTM} neural networks over multi-variable data}, author = {Guo, Tian and Lin, Tao and Antulov-Fantulin, Nino}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {2494--2504}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/guo19b/guo19b.pdf}, url = {https://proceedings.mlr.press/v97/guo19b.html}, abstract = {For recurrent neural networks trained on time series with target and exogenous variables, in addition to accurate prediction, it is also desired to provide interpretable insights into the data. In this paper, we explore the structure of LSTM recurrent neural networks to learn variable-wise hidden states, with the aim to capture different dynamics in multi-variable time series and distinguish the contribution of variables to the prediction. With these variable-wise hidden states, a mixture attention mechanism is proposed to model the generative process of the target. Then we develop associated training methods to jointly learn network parameters, variable and temporal importance w.r.t the prediction of the target variable. Extensive experiments on real datasets demonstrate enhanced prediction performance by capturing the dynamics of different variables. Meanwhile, we evaluate the interpretation results both qualitatively and quantitatively. It exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variable data.} }
Endnote
%0 Conference Paper %T Exploring interpretable LSTM neural networks over multi-variable data %A Tian Guo %A Tao Lin %A Nino Antulov-Fantulin %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-guo19b %I PMLR %P 2494--2504 %U https://proceedings.mlr.press/v97/guo19b.html %V 97 %X For recurrent neural networks trained on time series with target and exogenous variables, in addition to accurate prediction, it is also desired to provide interpretable insights into the data. In this paper, we explore the structure of LSTM recurrent neural networks to learn variable-wise hidden states, with the aim to capture different dynamics in multi-variable time series and distinguish the contribution of variables to the prediction. With these variable-wise hidden states, a mixture attention mechanism is proposed to model the generative process of the target. Then we develop associated training methods to jointly learn network parameters, variable and temporal importance w.r.t the prediction of the target variable. Extensive experiments on real datasets demonstrate enhanced prediction performance by capturing the dynamics of different variables. Meanwhile, we evaluate the interpretation results both qualitatively and quantitatively. It exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variable data.
APA
Guo, T., Lin, T. & Antulov-Fantulin, N.. (2019). Exploring interpretable LSTM neural networks over multi-variable data. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:2494-2504 Available from https://proceedings.mlr.press/v97/guo19b.html.

Related Material