Deriving Neural Architectures from Sequence and Graph Kernels

Tao Lei, Wengong Jin, Regina Barzilay, Tommi Jaakkola
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:2024-2033, 2017.

Abstract

The design of neural architectures for structured objects is typically guided by experimental insights rather than a formal process. In this work, we appeal to kernels over combinatorial structures, such as sequences and graphs, to derive appropriate neural operations. We introduce a class of deep recurrent neural operations and formally characterize their associated kernel spaces. Our recurrent modules compare the input to virtual reference objects (cf. filters in CNN) via the kernels. Similar to traditional neural operations, these reference objects are parameterized and directly optimized in end-to-end training. We empirically evaluate the proposed class of neural architectures on standard applications such as language modeling and molecular graph regression, achieving state-of-the-art results across these applications.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-lei17a, title = {Deriving Neural Architectures from Sequence and Graph Kernels}, author = {Tao Lei and Wengong Jin and Regina Barzilay and Tommi Jaakkola}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {2024--2033}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/lei17a/lei17a.pdf}, url = {https://proceedings.mlr.press/v70/lei17a.html}, abstract = {The design of neural architectures for structured objects is typically guided by experimental insights rather than a formal process. In this work, we appeal to kernels over combinatorial structures, such as sequences and graphs, to derive appropriate neural operations. We introduce a class of deep recurrent neural operations and formally characterize their associated kernel spaces. Our recurrent modules compare the input to virtual reference objects (cf. filters in CNN) via the kernels. Similar to traditional neural operations, these reference objects are parameterized and directly optimized in end-to-end training. We empirically evaluate the proposed class of neural architectures on standard applications such as language modeling and molecular graph regression, achieving state-of-the-art results across these applications.} }
Endnote
%0 Conference Paper %T Deriving Neural Architectures from Sequence and Graph Kernels %A Tao Lei %A Wengong Jin %A Regina Barzilay %A Tommi Jaakkola %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-lei17a %I PMLR %P 2024--2033 %U https://proceedings.mlr.press/v70/lei17a.html %V 70 %X The design of neural architectures for structured objects is typically guided by experimental insights rather than a formal process. In this work, we appeal to kernels over combinatorial structures, such as sequences and graphs, to derive appropriate neural operations. We introduce a class of deep recurrent neural operations and formally characterize their associated kernel spaces. Our recurrent modules compare the input to virtual reference objects (cf. filters in CNN) via the kernels. Similar to traditional neural operations, these reference objects are parameterized and directly optimized in end-to-end training. We empirically evaluate the proposed class of neural architectures on standard applications such as language modeling and molecular graph regression, achieving state-of-the-art results across these applications.
APA
Lei, T., Jin, W., Barzilay, R. & Jaakkola, T.. (2017). Deriving Neural Architectures from Sequence and Graph Kernels. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:2024-2033 Available from https://proceedings.mlr.press/v70/lei17a.html.

Related Material