Predicting Sequential Data with LSTMs Augmented with Strictly 2-Piecewise Input Vectors

Chihiro Shibata, Jeffrey Heinz
; Proceedings of The 13th International Conference on Grammatical Inference, PMLR 57:137-142, 2017.

Abstract

Recurrent neural networks such as Long-Short Term Memory (LSTM) are often used to learn from various kinds of time-series data, especially those that involved long-distance dependencies. We introduce a vector representation for the Strictly 2-Piecewise (SP-2) formal languages, which encode certain kinds of long-distance dependencies using subsequences. These vectors are added to the LSTM architecture as an additional input. Through experiments with the problems in the SPiCe dataset, we demonstrate that for certain problems, these vectors slightly—but significantly—improve the top-5 score (normalized discounted cumulative gain) as well as the accuracy as compared to the LSTM architecture without the SP-2 input vector. These results are also compared to an LSTM architecture with an input vector based on bigrams.

Cite this Paper


BibTeX
@InProceedings{pmlr-v57-shibata16, title = {Predicting Sequential Data with {LSTM}s Augmented with Strictly 2-Piecewise Input Vectors}, author = {Chihiro Shibata and Jeffrey Heinz}, booktitle = {Proceedings of The 13th International Conference on Grammatical Inference}, pages = {137--142}, year = {2017}, editor = {Sicco Verwer and Menno van Zaanen and Rick Smetsers}, volume = {57}, series = {Proceedings of Machine Learning Research}, address = {Delft, The Netherlands}, month = {05--07 Oct}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v57/shibata16.pdf}, url = {http://proceedings.mlr.press/v57/shibata16.html}, abstract = {Recurrent neural networks such as Long-Short Term Memory (LSTM) are often used to learn from various kinds of time-series data, especially those that involved long-distance dependencies. We introduce a vector representation for the Strictly 2-Piecewise (SP-2) formal languages, which encode certain kinds of long-distance dependencies using subsequences. These vectors are added to the LSTM architecture as an additional input. Through experiments with the problems in the SPiCe dataset, we demonstrate that for certain problems, these vectors slightly—but significantly—improve the top-5 score (normalized discounted cumulative gain) as well as the accuracy as compared to the LSTM architecture without the SP-2 input vector. These results are also compared to an LSTM architecture with an input vector based on bigrams.} }
Endnote
%0 Conference Paper %T Predicting Sequential Data with LSTMs Augmented with Strictly 2-Piecewise Input Vectors %A Chihiro Shibata %A Jeffrey Heinz %B Proceedings of The 13th International Conference on Grammatical Inference %C Proceedings of Machine Learning Research %D 2017 %E Sicco Verwer %E Menno van Zaanen %E Rick Smetsers %F pmlr-v57-shibata16 %I PMLR %J Proceedings of Machine Learning Research %P 137--142 %U http://proceedings.mlr.press %V 57 %W PMLR %X Recurrent neural networks such as Long-Short Term Memory (LSTM) are often used to learn from various kinds of time-series data, especially those that involved long-distance dependencies. We introduce a vector representation for the Strictly 2-Piecewise (SP-2) formal languages, which encode certain kinds of long-distance dependencies using subsequences. These vectors are added to the LSTM architecture as an additional input. Through experiments with the problems in the SPiCe dataset, we demonstrate that for certain problems, these vectors slightly—but significantly—improve the top-5 score (normalized discounted cumulative gain) as well as the accuracy as compared to the LSTM architecture without the SP-2 input vector. These results are also compared to an LSTM architecture with an input vector based on bigrams.
RIS
TY - CPAPER TI - Predicting Sequential Data with LSTMs Augmented with Strictly 2-Piecewise Input Vectors AU - Chihiro Shibata AU - Jeffrey Heinz BT - Proceedings of The 13th International Conference on Grammatical Inference PY - 2017/01/16 DA - 2017/01/16 ED - Sicco Verwer ED - Menno van Zaanen ED - Rick Smetsers ID - pmlr-v57-shibata16 PB - PMLR SP - 137 DP - PMLR EP - 142 L1 - http://proceedings.mlr.press/v57/shibata16.pdf UR - http://proceedings.mlr.press/v57/shibata16.html AB - Recurrent neural networks such as Long-Short Term Memory (LSTM) are often used to learn from various kinds of time-series data, especially those that involved long-distance dependencies. We introduce a vector representation for the Strictly 2-Piecewise (SP-2) formal languages, which encode certain kinds of long-distance dependencies using subsequences. These vectors are added to the LSTM architecture as an additional input. Through experiments with the problems in the SPiCe dataset, we demonstrate that for certain problems, these vectors slightly—but significantly—improve the top-5 score (normalized discounted cumulative gain) as well as the accuracy as compared to the LSTM architecture without the SP-2 input vector. These results are also compared to an LSTM architecture with an input vector based on bigrams. ER -
APA
Shibata, C. & Heinz, J.. (2017). Predicting Sequential Data with LSTMs Augmented with Strictly 2-Piecewise Input Vectors. Proceedings of The 13th International Conference on Grammatical Inference, in PMLR 57:137-142

Related Material