EEGtoText: Learning to Write Medical Reports from EEG Recordings

Siddharth Biswal, Cao Xiao, M. Brandon Westover, Jimeng Sun
Proceedings of the 4th Machine Learning for Healthcare Conference, PMLR 106:513-531, 2019.

Abstract

Electroencephalography (EEG) is widely used in hospitals and clinics for the diagnosis of many neurological conditions. Such diagnoses require accurate and timely clinical reports to summarize the findings from raw EEG data. In this paper, we investigate whether it is possible to automatically generate text reports directly from EEG data. To address the challenges, we proposed EEGtoText , which first extracted shift invariant and temporal patterns using stacked convolutional neural networks and recurrent neural networks (RCNN). These temporal patterns are used to classify key phenotypes including EEG normality, sleep, generalized and focal slowing, epileptiform discharges, spindles, vertex waves and seizures. Based on these phenotypes, the impression section of the EEG report is generated. Next, we adopted a hierarchical long short-term memory network(LSTM) that comprises of paragraph-level and sentence-level LSTMs to generate the detail explanation of the impression. Within the hierarchical LSTM, we used an attention module to localize the abnormal areas in the EEG which provide another explanation and justification of the extracted phenotypes. We conducted large-scale evaluations on two different EEG datasets Dataset1 (n=12,980) and TUH (n=16,950). We achieved an area under the ROC curve (AUC) between .658 to .915 on phenotype classification, which is significantly higher than CRNN and RCNN with attention. We also conducted a quantitative evaluation of the detailed explanation, which achieved METEOR score .371 and BLEU score 4.583. Finally, our initial clinical reviews confirmed the effectiveness of the generated reports.

Cite this Paper

BibTeX
@InProceedings{pmlr-v106-biswal19a, title = {EEGtoText: Learning to Write Medical Reports from EEG Recordings}, author = {Biswal, Siddharth and Xiao, Cao and Westover, M. Brandon and Sun, Jimeng}, booktitle = {Proceedings of the 4th Machine Learning for Healthcare Conference}, pages = {513--531}, year = {2019}, editor = {Doshi-Velez, Finale and Fackler, Jim and Jung, Ken and Kale, David and Ranganath, Rajesh and Wallace, Byron and Wiens, Jenna}, volume = {106}, series = {Proceedings of Machine Learning Research}, month = {09--10 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v106/biswal19a/biswal19a.pdf}, url = {https://proceedings.mlr.press/v106/biswal19a.html}, abstract = {Electroencephalography (EEG) is widely used in hospitals and clinics for the diagnosis of many neurological conditions. Such diagnoses require accurate and timely clinical reports to summarize the findings from raw EEG data. In this paper, we investigate whether it is possible to automatically generate text reports directly from EEG data. To address the challenges, we proposed EEGtoText , which first extracted shift invariant and temporal patterns using stacked convolutional neural networks and recurrent neural networks (RCNN). These temporal patterns are used to classify key phenotypes including EEG normality, sleep, generalized and focal slowing, epileptiform discharges, spindles, vertex waves and seizures. Based on these phenotypes, the impression section of the EEG report is generated. Next, we adopted a hierarchical long short-term memory network(LSTM) that comprises of paragraph-level and sentence-level LSTMs to generate the detail explanation of the impression. Within the hierarchical LSTM, we used an attention module to localize the abnormal areas in the EEG which provide another explanation and justification of the extracted phenotypes. We conducted large-scale evaluations on two different EEG datasets Dataset1 (n=12,980) and TUH (n=16,950). We achieved an area under the ROC curve (AUC) between .658 to .915 on phenotype classification, which is significantly higher than CRNN and RCNN with attention. We also conducted a quantitative evaluation of the detailed explanation, which achieved METEOR score .371 and BLEU score 4.583. Finally, our initial clinical reviews confirmed the effectiveness of the generated reports.} }
Endnote
%0 Conference Paper %T EEGtoText: Learning to Write Medical Reports from EEG Recordings %A Siddharth Biswal %A Cao Xiao %A M. Brandon Westover %A Jimeng Sun %B Proceedings of the 4th Machine Learning for Healthcare Conference %C Proceedings of Machine Learning Research %D 2019 %E Finale Doshi-Velez %E Jim Fackler %E Ken Jung %E David Kale %E Rajesh Ranganath %E Byron Wallace %E Jenna Wiens %F pmlr-v106-biswal19a %I PMLR %P 513--531 %U https://proceedings.mlr.press/v106/biswal19a.html %V 106 %X Electroencephalography (EEG) is widely used in hospitals and clinics for the diagnosis of many neurological conditions. Such diagnoses require accurate and timely clinical reports to summarize the findings from raw EEG data. In this paper, we investigate whether it is possible to automatically generate text reports directly from EEG data. To address the challenges, we proposed EEGtoText , which first extracted shift invariant and temporal patterns using stacked convolutional neural networks and recurrent neural networks (RCNN). These temporal patterns are used to classify key phenotypes including EEG normality, sleep, generalized and focal slowing, epileptiform discharges, spindles, vertex waves and seizures. Based on these phenotypes, the impression section of the EEG report is generated. Next, we adopted a hierarchical long short-term memory network(LSTM) that comprises of paragraph-level and sentence-level LSTMs to generate the detail explanation of the impression. Within the hierarchical LSTM, we used an attention module to localize the abnormal areas in the EEG which provide another explanation and justification of the extracted phenotypes. We conducted large-scale evaluations on two different EEG datasets Dataset1 (n=12,980) and TUH (n=16,950). We achieved an area under the ROC curve (AUC) between .658 to .915 on phenotype classification, which is significantly higher than CRNN and RCNN with attention. We also conducted a quantitative evaluation of the detailed explanation, which achieved METEOR score .371 and BLEU score 4.583. Finally, our initial clinical reviews confirmed the effectiveness of the generated reports.
APA
Biswal, S., Xiao, C., Westover, M.B. & Sun, J.. (2019). EEGtoText: Learning to Write Medical Reports from EEG Recordings. Proceedings of the 4th Machine Learning for Healthcare Conference, in Proceedings of Machine Learning Research 106:513-531 Available from https://proceedings.mlr.press/v106/biswal19a.html.

Related Material