Rotary Informational Embeddings for Symbolic Music Generation

Felix Schön, Hans Tompits
Proceedings of the The 39th Canadian Conference on Artificial Intelligence, PMLR 318:1181-1185, 2026.

Abstract

In this paper, we present preliminary results on rotary informational embeddings (RotIE), an extension of rotary positional embeddings (RoPE) for Transformer-based symbolic music generation. With RotIE, we adapt the rotary mechanism to encode arbitrary integer-valued information such as pitch, absolute time, or intra-bar positions directly into the attention computation, allowing the model to depend on relative differences in musical attributes rather than on sequential position only. We focus on one representative per-head strategy and evaluate it on the Lakh MIDI and POP909 datasets. The presented results show improved perplexity over a regular Transformer, the Music Transformer, and a RoPE baseline, particularly on longer unseen sequences.

Cite this Paper

BibTeX
@InProceedings{pmlr-v318-schon26c, title = {Rotary Informational Embeddings for Symbolic Music Generation}, author = {Sch\"{o}n, Felix and Tompits, Hans}, booktitle = {Proceedings of the The 39th Canadian Conference on Artificial Intelligence}, pages = {1181--1185}, year = {2026}, editor = {Bouzar-Benlabiod, Lydia and Leung, Carson}, volume = {318}, series = {Proceedings of Machine Learning Research}, month = {25--29 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v318/main/assets/schon26c/schon26c.pdf}, url = {https://proceedings.mlr.press/v318/schon26c.html}, abstract = {In this paper, we present preliminary results on rotary informational embeddings (RotIE), an extension of rotary positional embeddings (RoPE) for Transformer-based symbolic music generation. With RotIE, we adapt the rotary mechanism to encode arbitrary integer-valued information such as pitch, absolute time, or intra-bar positions directly into the attention computation, allowing the model to depend on relative differences in musical attributes rather than on sequential position only. We focus on one representative per-head strategy and evaluate it on the Lakh MIDI and POP909 datasets. The presented results show improved perplexity over a regular Transformer, the Music Transformer, and a RoPE baseline, particularly on longer unseen sequences.} }
Endnote
%0 Conference Paper %T Rotary Informational Embeddings for Symbolic Music Generation %A Felix Schön %A Hans Tompits %B Proceedings of the The 39th Canadian Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Lydia Bouzar-Benlabiod %E Carson Leung %F pmlr-v318-schon26c %I PMLR %P 1181--1185 %U https://proceedings.mlr.press/v318/schon26c.html %V 318 %X In this paper, we present preliminary results on rotary informational embeddings (RotIE), an extension of rotary positional embeddings (RoPE) for Transformer-based symbolic music generation. With RotIE, we adapt the rotary mechanism to encode arbitrary integer-valued information such as pitch, absolute time, or intra-bar positions directly into the attention computation, allowing the model to depend on relative differences in musical attributes rather than on sequential position only. We focus on one representative per-head strategy and evaluate it on the Lakh MIDI and POP909 datasets. The presented results show improved perplexity over a regular Transformer, the Music Transformer, and a RoPE baseline, particularly on longer unseen sequences.
APA
Schön, F. & Tompits, H.. (2026). Rotary Informational Embeddings for Symbolic Music Generation. Proceedings of the The 39th Canadian Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 318:1181-1185 Available from https://proceedings.mlr.press/v318/schon26c.html.

Related Material