Representing Unordered Data Using Complex-Weighted Multiset Automata

Justin DeBenedetto, David Chiang
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2412-2420, 2020.

Abstract

Unordered, variable-sized inputs arise in many settings across multiple fields. The ability for set- and multiset-oriented neural networks to handle this type of input has been the focus of much work in recent years. We propose to represent multisets using complex-weighted multiset automata and show how the multiset representations of certain existing neural architectures can be viewed as special cases of ours. Namely, (1) we provide a new theoretical and intuitive justification for the Transformer model’s representation of positions using sinusoidal functions, and (2) we extend the DeepSets model to use complex numbers, enabling it to outperform the existing model on an extension of one of their tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-debenedetto20a, title = {Representing Unordered Data Using Complex-Weighted Multiset Automata}, author = {{DeBenedetto}, Justin and Chiang, David}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {2412--2420}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/debenedetto20a/debenedetto20a.pdf}, url = {https://proceedings.mlr.press/v119/debenedetto20a.html}, abstract = {Unordered, variable-sized inputs arise in many settings across multiple fields. The ability for set- and multiset-oriented neural networks to handle this type of input has been the focus of much work in recent years. We propose to represent multisets using complex-weighted multiset automata and show how the multiset representations of certain existing neural architectures can be viewed as special cases of ours. Namely, (1) we provide a new theoretical and intuitive justification for the Transformer model’s representation of positions using sinusoidal functions, and (2) we extend the DeepSets model to use complex numbers, enabling it to outperform the existing model on an extension of one of their tasks. } }
Endnote
%0 Conference Paper %T Representing Unordered Data Using Complex-Weighted Multiset Automata %A Justin DeBenedetto %A David Chiang %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-debenedetto20a %I PMLR %P 2412--2420 %U https://proceedings.mlr.press/v119/debenedetto20a.html %V 119 %X Unordered, variable-sized inputs arise in many settings across multiple fields. The ability for set- and multiset-oriented neural networks to handle this type of input has been the focus of much work in recent years. We propose to represent multisets using complex-weighted multiset automata and show how the multiset representations of certain existing neural architectures can be viewed as special cases of ours. Namely, (1) we provide a new theoretical and intuitive justification for the Transformer model’s representation of positions using sinusoidal functions, and (2) we extend the DeepSets model to use complex numbers, enabling it to outperform the existing model on an extension of one of their tasks.
APA
DeBenedetto, J. & Chiang, D.. (2020). Representing Unordered Data Using Complex-Weighted Multiset Automata. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:2412-2420 Available from https://proceedings.mlr.press/v119/debenedetto20a.html.

Related Material