Efficient Additive Relative Information Attention for Transformer-based Symbolic Music Composition

Symbolic music generation deals with automatically composing music in which the latter is treated as a language whose words represent musical events. In recent years, approaches based on the Transformer architecture using relative positional attention showed particular promise. However, a drawback common between the existing approaches is their limitation to relative distances between the positions of tokens only, rather than properties of the elements represented by them. To overcome this limitation, we introduce an efficient novel method for additive relative information injection based on block-sparse matrix operations. We evaluate the effectiveness of our approach by comparing it to different network architectures and conducting an array of experiments which show improvements over previous approaches.