Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization

Bharath Chintagunta, Namit Katariya, Xavier Amatriain, Anitha Kannan
Proceedings of the 6th Machine Learning for Healthcare Conference, PMLR 149:354-372, 2021.

Abstract

In medical dialogue summarization, summaries must be coherent and must capture all the medically relevant information in the dialogue. However, learning effective models for summarization require large amounts of labeled data which is especially hard to obtain. We present an algorithm to create synthetic training data with an explicit focus on capturing medically relevant information. We utilize GPT-3 as the backbone of our algorithm and scale 210 human labeled examples to yield results comparable to using 6400 human labeled examples (∼30x) leveraging low-shot learning and an ensemble method. In detailed experiments, we show that this approach produces high quality training data that can further be combined with human labeled data to get summaries that are strongly preferable to those produced by models trained on human data alone both in terms of medical accuracy and coherency.

Cite this Paper

BibTeX
@InProceedings{pmlr-v149-chintagunta21a, title = {Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization}, author = {Chintagunta, Bharath and Katariya, Namit and Amatriain, Xavier and Kannan, Anitha}, booktitle = {Proceedings of the 6th Machine Learning for Healthcare Conference}, pages = {354--372}, year = {2021}, editor = {Jung, Ken and Yeung, Serena and Sendak, Mark and Sjoding, Michael and Ranganath, Rajesh}, volume = {149}, series = {Proceedings of Machine Learning Research}, month = {06--07 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v149/chintagunta21a/chintagunta21a.pdf}, url = {https://proceedings.mlr.press/v149/chintagunta21a.html}, abstract = {In medical dialogue summarization, summaries must be coherent and must capture all the medically relevant information in the dialogue. However, learning effective models for summarization require large amounts of labeled data which is especially hard to obtain. We present an algorithm to create synthetic training data with an explicit focus on capturing medically relevant information. We utilize GPT-3 as the backbone of our algorithm and scale 210 human labeled examples to yield results comparable to using 6400 human labeled examples (∼30x) leveraging low-shot learning and an ensemble method. In detailed experiments, we show that this approach produces high quality training data that can further be combined with human labeled data to get summaries that are strongly preferable to those produced by models trained on human data alone both in terms of medical accuracy and coherency.} }
Endnote
%0 Conference Paper %T Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization %A Bharath Chintagunta %A Namit Katariya %A Xavier Amatriain %A Anitha Kannan %B Proceedings of the 6th Machine Learning for Healthcare Conference %C Proceedings of Machine Learning Research %D 2021 %E Ken Jung %E Serena Yeung %E Mark Sendak %E Michael Sjoding %E Rajesh Ranganath %F pmlr-v149-chintagunta21a %I PMLR %P 354--372 %U https://proceedings.mlr.press/v149/chintagunta21a.html %V 149 %X In medical dialogue summarization, summaries must be coherent and must capture all the medically relevant information in the dialogue. However, learning effective models for summarization require large amounts of labeled data which is especially hard to obtain. We present an algorithm to create synthetic training data with an explicit focus on capturing medically relevant information. We utilize GPT-3 as the backbone of our algorithm and scale 210 human labeled examples to yield results comparable to using 6400 human labeled examples (∼30x) leveraging low-shot learning and an ensemble method. In detailed experiments, we show that this approach produces high quality training data that can further be combined with human labeled data to get summaries that are strongly preferable to those produced by models trained on human data alone both in terms of medical accuracy and coherency.
APA
Chintagunta, B., Katariya, N., Amatriain, X. & Kannan, A.. (2021). Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization. Proceedings of the 6th Machine Learning for Healthcare Conference, in Proceedings of Machine Learning Research 149:354-372 Available from https://proceedings.mlr.press/v149/chintagunta21a.html.

Related Material