Non-Invasive Silent Speech Recognition in Multiple Sclerosis with Dysphonia

Arnav Kapur, Utkarsh Sarawgi, Eric Wadkins, Matthew Wu, Nora Hollenstein, Pattie Maes
Proceedings of the Machine Learning for Health NeurIPS Workshop, PMLR 116:25-38, 2020.

Abstract

We present the first non-invasive real-time silent speech system that helps patients with speech disorders to communicate in natural language voicelessly, merely by articulating words or sentences in the mouth without producing any sounds. We collected neuromus-cular recordings to build a dataset of 10 trials of 15 sentences from each of 3 multiple sclerosis (MS) patients with dysphonia, spanning a range of severity and subsequently affected speech attributes. We present a pipeline wherein we carefully preprocess the data, develop a convolutional neural architecture and employ personalized machine learning. In our experiments with multiple sclerosis patients, our system achieves a mean overall test accuracy of 0.81 at a mean information transfer rate of 203.73 bits per minute averaged over all patients. Our work demonstrates the potential of a reliable and promising human-computer interface that classifies intended sentences from silent speech and hence, paves the path for future work with further speech disorders in conditions such as amyotrophic lateral sclerosis (ALS), stroke, and oral cancer, among others.

Cite this Paper

BibTeX
@InProceedings{pmlr-v116-kapur20a, title = {{Non-Invasive Silent Speech Recognition in Multiple Sclerosis with Dysphonia}}, author = {Kapur, Arnav and Sarawgi, Utkarsh and Wadkins, Eric and Wu, Matthew and Hollenstein, Nora and Maes, Pattie}, booktitle = {Proceedings of the Machine Learning for Health NeurIPS Workshop}, pages = {25--38}, year = {2020}, editor = {Dalca, Adrian V. and McDermott, Matthew B.A. and Alsentzer, Emily and Finlayson, Samuel G. and Oberst, Michael and Falck, Fabian and Beaulieu-Jones, Brett}, volume = {116}, series = {Proceedings of Machine Learning Research}, month = {13 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v116/kapur20a/kapur20a.pdf}, url = {https://proceedings.mlr.press/v116/kapur20a.html}, abstract = {We present the first non-invasive real-time silent speech system that helps patients with speech disorders to communicate in natural language voicelessly, merely by articulating words or sentences in the mouth without producing any sounds. We collected neuromus-cular recordings to build a dataset of 10 trials of 15 sentences from each of 3 multiple sclerosis (MS) patients with dysphonia, spanning a range of severity and subsequently affected speech attributes. We present a pipeline wherein we carefully preprocess the data, develop a convolutional neural architecture and employ personalized machine learning. In our experiments with multiple sclerosis patients, our system achieves a mean overall test accuracy of 0.81 at a mean information transfer rate of 203.73 bits per minute averaged over all patients. Our work demonstrates the potential of a reliable and promising human-computer interface that classifies intended sentences from silent speech and hence, paves the path for future work with further speech disorders in conditions such as amyotrophic lateral sclerosis (ALS), stroke, and oral cancer, among others.} }
Endnote
%0 Conference Paper %T Non-Invasive Silent Speech Recognition in Multiple Sclerosis with Dysphonia %A Arnav Kapur %A Utkarsh Sarawgi %A Eric Wadkins %A Matthew Wu %A Nora Hollenstein %A Pattie Maes %B Proceedings of the Machine Learning for Health NeurIPS Workshop %C Proceedings of Machine Learning Research %D 2020 %E Adrian V. Dalca %E Matthew B.A. McDermott %E Emily Alsentzer %E Samuel G. Finlayson %E Michael Oberst %E Fabian Falck %E Brett Beaulieu-Jones %F pmlr-v116-kapur20a %I PMLR %P 25--38 %U https://proceedings.mlr.press/v116/kapur20a.html %V 116 %X We present the first non-invasive real-time silent speech system that helps patients with speech disorders to communicate in natural language voicelessly, merely by articulating words or sentences in the mouth without producing any sounds. We collected neuromus-cular recordings to build a dataset of 10 trials of 15 sentences from each of 3 multiple sclerosis (MS) patients with dysphonia, spanning a range of severity and subsequently affected speech attributes. We present a pipeline wherein we carefully preprocess the data, develop a convolutional neural architecture and employ personalized machine learning. In our experiments with multiple sclerosis patients, our system achieves a mean overall test accuracy of 0.81 at a mean information transfer rate of 203.73 bits per minute averaged over all patients. Our work demonstrates the potential of a reliable and promising human-computer interface that classifies intended sentences from silent speech and hence, paves the path for future work with further speech disorders in conditions such as amyotrophic lateral sclerosis (ALS), stroke, and oral cancer, among others.
APA
Kapur, A., Sarawgi, U., Wadkins, E., Wu, M., Hollenstein, N. & Maes, P.. (2020). Non-Invasive Silent Speech Recognition in Multiple Sclerosis with Dysphonia. Proceedings of the Machine Learning for Health NeurIPS Workshop, in Proceedings of Machine Learning Research 116:25-38 Available from https://proceedings.mlr.press/v116/kapur20a.html.

Related Material