Directing Human Attention in Event Localization for Clinical Timeline Creation

Jason Zhao, Monica Agrawal, Pedram Razavi, David Sontag
Proceedings of the 6th Machine Learning for Healthcare Conference, PMLR 149:80-102, 2021.

Abstract

Many variables useful for clinical research (e.g. patient disease state, treatment regimens) are trapped in free-text clinical notes. Structuring such variables for downstream use typically involves a tedious process in which domain experts manually search through long clinical timelines. Natural language processing systems present an opportunity for automating this workflow, but algorithms still have trouble accurately parsing the most complex patient cases, which may be best deferred to experts. In this work, we present a framework that automatically structures simple patient cases, but when required, iteratively requests human input, specifically a label for a single note in the patient’s timeline that would decrease uncertainty in model output. Our method provides a lightweight way to leverage domain experts. We test our system on two tasks from a cohort of oncology patients: identification of the date of (i) metastasis onset and (ii) oral therapy start. Compared to standard search heuristics, we show we can reduce 80% of model errors with less than 15% of the manual annotation effort that may otherwise be required.

Cite this Paper

BibTeX
@InProceedings{pmlr-v149-zhao21a, title = {Directing Human Attention in Event Localization for Clinical Timeline Creation}, author = {Zhao, Jason and Agrawal, Monica and Razavi, Pedram and Sontag, David}, booktitle = {Proceedings of the 6th Machine Learning for Healthcare Conference}, pages = {80--102}, year = {2021}, editor = {Jung, Ken and Yeung, Serena and Sendak, Mark and Sjoding, Michael and Ranganath, Rajesh}, volume = {149}, series = {Proceedings of Machine Learning Research}, month = {06--07 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v149/zhao21a/zhao21a.pdf}, url = {https://proceedings.mlr.press/v149/zhao21a.html}, abstract = {Many variables useful for clinical research (e.g. patient disease state, treatment regimens) are trapped in free-text clinical notes. Structuring such variables for downstream use typically involves a tedious process in which domain experts manually search through long clinical timelines. Natural language processing systems present an opportunity for automating this workflow, but algorithms still have trouble accurately parsing the most complex patient cases, which may be best deferred to experts. In this work, we present a framework that automatically structures simple patient cases, but when required, iteratively requests human input, specifically a label for a single note in the patient’s timeline that would decrease uncertainty in model output. Our method provides a lightweight way to leverage domain experts. We test our system on two tasks from a cohort of oncology patients: identification of the date of (i) metastasis onset and (ii) oral therapy start. Compared to standard search heuristics, we show we can reduce 80% of model errors with less than 15% of the manual annotation effort that may otherwise be required.} }
Endnote
%0 Conference Paper %T Directing Human Attention in Event Localization for Clinical Timeline Creation %A Jason Zhao %A Monica Agrawal %A Pedram Razavi %A David Sontag %B Proceedings of the 6th Machine Learning for Healthcare Conference %C Proceedings of Machine Learning Research %D 2021 %E Ken Jung %E Serena Yeung %E Mark Sendak %E Michael Sjoding %E Rajesh Ranganath %F pmlr-v149-zhao21a %I PMLR %P 80--102 %U https://proceedings.mlr.press/v149/zhao21a.html %V 149 %X Many variables useful for clinical research (e.g. patient disease state, treatment regimens) are trapped in free-text clinical notes. Structuring such variables for downstream use typically involves a tedious process in which domain experts manually search through long clinical timelines. Natural language processing systems present an opportunity for automating this workflow, but algorithms still have trouble accurately parsing the most complex patient cases, which may be best deferred to experts. In this work, we present a framework that automatically structures simple patient cases, but when required, iteratively requests human input, specifically a label for a single note in the patient’s timeline that would decrease uncertainty in model output. Our method provides a lightweight way to leverage domain experts. We test our system on two tasks from a cohort of oncology patients: identification of the date of (i) metastasis onset and (ii) oral therapy start. Compared to standard search heuristics, we show we can reduce 80% of model errors with less than 15% of the manual annotation effort that may otherwise be required.
APA
Zhao, J., Agrawal, M., Razavi, P. & Sontag, D.. (2021). Directing Human Attention in Event Localization for Clinical Timeline Creation. Proceedings of the 6th Machine Learning for Healthcare Conference, in Proceedings of Machine Learning Research 149:80-102 Available from https://proceedings.mlr.press/v149/zhao21a.html.

Related Material