Musical Speech: A Transformer-based Composition Tool

Jason d’Eon, Sri Harsha Dumpla, Chandramouli Shama Sastry, Daniel Oore, Sageev Oore
Proceedings of the NeurIPS 2020 Competition and Demonstration Track, PMLR 133:253-274, 2021.

Abstract

In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline – which does not require a paired dataset for training – through examples of music created by musicians making use of our tool.

Cite this Paper

BibTeX
@InProceedings{pmlr-v133-d-eon21a, title = {Musical Speech: A Transformer-based Composition Tool}, author = {d'Eon, Jason and Dumpla, Sri Harsha and Sastry, Chandramouli Shama and Oore, Daniel and Oore, Sageev}, booktitle = {Proceedings of the NeurIPS 2020 Competition and Demonstration Track}, pages = {253--274}, year = {2021}, editor = {Escalante, Hugo Jair and Hofmann, Katja}, volume = {133}, series = {Proceedings of Machine Learning Research}, month = {06--12 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v133/d-eon21a/d-eon21a.pdf}, url = {https://proceedings.mlr.press/v133/d-eon21a.html}, abstract = {In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline – which does not require a paired dataset for training – through examples of music created by musicians making use of our tool.} }
Endnote
%0 Conference Paper %T Musical Speech: A Transformer-based Composition Tool %A Jason d’Eon %A Sri Harsha Dumpla %A Chandramouli Shama Sastry %A Daniel Oore %A Sageev Oore %B Proceedings of the NeurIPS 2020 Competition and Demonstration Track %C Proceedings of Machine Learning Research %D 2021 %E Hugo Jair Escalante %E Katja Hofmann %F pmlr-v133-d-eon21a %I PMLR %P 253--274 %U https://proceedings.mlr.press/v133/d-eon21a.html %V 133 %X In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline – which does not require a paired dataset for training – through examples of music created by musicians making use of our tool.
APA
d’Eon, J., Dumpla, S.H., Sastry, C.S., Oore, D. & Oore, S.. (2021). Musical Speech: A Transformer-based Composition Tool. Proceedings of the NeurIPS 2020 Competition and Demonstration Track, in Proceedings of Machine Learning Research 133:253-274 Available from https://proceedings.mlr.press/v133/d-eon21a.html.

Related Material